2.5 mmap, virtio-fs, and DAX

中文

English

Contents

Chapter 2: FUSE Implementation Analysis · Article 5 of 6

This article analyzes ordinary mmap and virtio-fs DAX, with three specific questions: what the current code does when the DAX window is full; whether a timed-out wait can fall back to non-DAX direct I/O; and why dax_iomap_rw() partial-completion semantics define the safety boundary.

1. Ordinary FUSE mmap

A non-DAX file normally maps the page cache. A missing folio causes FUSE READ; a shared writable mapping dirties folios that FUSE writeback sends later.

mmap()
  -> fuse_file_mmap()
  -> page-cache vm_operations
  -> fault -> read_folio/readahead -> FUSE_READ
  -> shared write fault -> dirty folio
  -> writepages -> FUSE_WRITE

Most I/O occurs on later faults, not during the mmap() syscall itself.

2. virtio-fs DAX model

virtio-fs can expose a shared-memory DAX window to the guest. Files are not permanently mapped in full. Instead, a fixed-size file extent is temporarily associated with a window slot.

The current fs/fuse/dax.c uses:

#define FUSE_DAX_SHIFT 21
#define FUSE_DAX_SZ    (1 << FUSE_DAX_SHIFT)  /* 2 MiB */

Each struct fuse_dax_mapping represents one 2 MiB range and records its window offset, inode/file offset, writable state, references, and free/busy-list membership.

3. Mapping setup and removal

When fuse_iomap_begin() finds no mapping, it reaches fuse_setup_new_dax_mapping():

1. allocate a range from free_ranges;
2. if empty, try inline reclaim;
3. serialize duplicate setup with the inode DAX semaphore;
4. send FUSE_SETUPMAPPING with file offset, window offset, length, and access flags;
5. insert the mapping into the inode interval tree;
6. fill an iomap that points DAX core at shared memory.

Reclaim sends FUSE_REMOVEMAPPING; the slot cannot safely return to the free pool until the host/backend association is gone.

4. Why slots become exhausted

The range count is DAX-window size divided by 2 MiB. Exhaustion can result from:

• many simultaneously mapped files;
• a working set larger than the window;
• mappings pinned by page tables or active I/O;
• writable layouts that must be broken before reclaim;
• reclaim blocked by inode or invalidate locks;
• slow host-side REMOVEMAPPING processing.

A full window is initially a resource-pressure condition, not necessarily a permanent access failure.

5. Current read/write behavior when full

The non-fault path calls alloc_dax_mapping_reclaim(). Its loop:

1. tries the free list;
2. tries to reclaim a mapping from the current inode;
3. retries temporary reclaim failures;
4. when this inode has no reclaimable mapping and the connection has no free range, waits with wait_event_killable_exclusive() on range_waitq;
5. wakes when a returned range increments nr_free_ranges.

The current implementation has no fixed wait timeout. A signal can produce -EINTR; otherwise it may wait until a slot is available. Setup/remove protocol failures propagate their own errors.

6. Why the fault path differs

A page fault already holds mapping->invalidate_lock shared. Inline reclaim can need lock operations that are illegal in that context. Therefore a fault-side allocation miss returns -EAGAIN from fuse_setup_new_dax_mapping().

__fuse_dax_fault() recognizes that result, leaves the invalidate-lock domain, waits for range_waitq, and retries the fault. This wait also has no built-in deadline.

A fault returns vm_fault_t, not a byte count. Its fallback semantics therefore cannot be copied from read_iter.

7. DAX read/write call chain

file.c checks FUSE_IS_DAX(inode) before per-open direct I/O and passthrough:

fuse_file_read_iter
  -> fuse_dax_read_iter
  -> shared inode lock
  -> dax_iomap_rw(..., fuse_iomap_ops)
  -> fuse_iomap_begin
  -> find, allocate, or reclaim a mapping
  -> copy between DAX window and iov_iter

fuse_file_write_iter
  -> fuse_dax_write_iter
  -> non-extending write: dax_iomap_rw
  -> extending write: fuse_direct_io

Extending writes avoid DAX because window data update and persistent i_size growth are not atomic in the existing protocol.

8. How dax_iomap_rw() hides a later error

The decisive DAX-core code is:

while ((ret = iomap_iter(&iomi, ops)) > 0)
        iomi.status = dax_iomap_iter(&iomi, iter);

done = iomi.pos - iocb->ki_pos;
iocb->ki_pos = iomi.pos;
return done ? done : ret;

Suppose a large I/O copies 2 MiB through its first mapping, then times out waiting for a second slot. done is positive, so the function returns 2 MiB instead of -ETIMEDOUT.

That is standard partial-I/O behavior: completed bytes take precedence. Userspace observes a short I/O and can issue another syscall for the remainder.

9. Adding a range-wait timeout

The wait can technically be changed to a killable timeout and range allocation can return -ETIMEDOUT. A real design must still specify:

• whether the deadline covers only a free-slot wait or also SETUP/REMOVEMAPPING replies;
• whether timing begins at the first allocation miss or syscall entry;
• signal-versus-timeout errno precedence;
• conversion into vm_fault_t for faults;
• reconciliation of late mapping/setup completion;
• metrics distinguishing window pressure from a stalled daemon.

Replacing one wait macro creates a mechanism, not complete semantics.

10. Timeout fallback to non-DAX direct I/O

A conservative design is:

1. Do not apply this policy to small I/O if its selection overhead is unjustified.
2. Fall back only for the specific error “timed out waiting for a DAX slot.”
3. Replay the original range through direct I/O only when DAX completed zero bytes.
4. If DAX completed a prefix, return the short I/O rather than replaying it.
5. Preserve the open handle, current offset, and iterator state.
6. Exclude DAX layouts and page-cache aliases before a write fallback.

The top-level shape is:

snapshot = iov_iter_count(iter);
ret = dax_iomap_rw(iocb, iter, &fuse_iomap_ops);
if (ret == -ETIMEDOUT && iov_iter_count(iter) == snapshot)
        ret = fuse_direct_io(...);   /* zero progress only */
return ret;

Iterator count alone is not a full proof: file position must not have advanced, and mapping setup must not have left published state. A dedicated internal range-wait-timeout result is preferable to conflating it with an unrelated backend timeout.

11. The remaining blind spot

“Use direct I/O only when DAX returns timeout” works for zero progress. It does not trigger after a completed prefix, because dax_iomap_rw() returns a positive count.

Continuing the suffix inside one syscall would require exposing both done and terminal error across the DAX/FUSE boundary, then:

• submit direct I/O from the advanced ki_pos and remaining iterator;
• never rewind or replay the prefix;
• avoid overlap at the DAX mapping boundary;
• merge return values under short-I/O conventions;
• prove DAX/direct write locking and invalidation are compatible.

A first implementation should support zero-progress fallback only and let mid-operation exhaustion surface as short I/O.

12. mmap fault is a separate design

A direct read copies into a transient buffer; it cannot install a page-table mapping for mmap. To degrade a timed-out DAX fault, the implementation would have to switch the VMA or inode to page-cache-backed faults, remove existing DAX PTEs, break layouts, and serialize concurrent faults.

Explicit read/write fallback and mmap fallback should therefore be designed separately. An initial change can intentionally cover only read/write.

13. Required observability

Useful metrics include:

• free, busy, and total DAX ranges;
• allocation fast hits, inline reclaim, waits, and timeouts;
• SETUPMAPPING/REMOVEMAPPING latency and errno;
• zero-progress direct fallbacks and bytes;
• partial DAX short-I/O count;
• fault wait duration;
• unreclaimable causes: reference, busy layout, lock contention, or daemon failure.

14. Summary

The DAX window is a finite pool of 2 MiB mappings. Current read/write code reclaims or waits interruptibly when full; the fault path leaves its lock domain and retries after -EAGAIN. Neither has a fixed timeout. A slot-wait timeout can safely trigger direct-I/O fallback at zero progress. Once a prefix was copied, dax_iomap_rw() returns the positive count and hides the later error; replaying the full operation would duplicate I/O.

Previous: Request Lifecycle and /dev/fuse Transport

Next: Teardown, Debugging, and Performance