1.3 Caching and Coherency

中文

English

Contents

Chapter 1: FUSE Subsystem Design · Article 3 of 4

Caching is the main reason FUSE can perform well, and the main reason it can return stale results. A complete design must specify namespace, attribute, and data-cache policy separately, then explain how external changes invalidate each layer.

1. Three independent cache layers

1.1 Dentry cache

A dentry caches whether a name under a parent resolves to an inode. Positive and negative results may both be cached. The entry_valid duration in a lookup reply controls how long the namespace result may be trusted.

1.2 Attribute cache

An inode can remain known while its mode, size, timestamps, ownership, or block count becomes stale. attr_valid controls this lease. Expiration may trigger GETATTR; it does not necessarily discard the dentry.

1.3 Page cache

File contents can live in the kernel page cache. Buffered reads reuse clean pages; buffered writes dirty pages and may reach the daemon later through writeback. Data validity therefore cannot be inferred from entry or attribute validity alone.

2. Validity intervals are leases

The daemon tells the kernel, in effect: “this fact may be reused until this deadline.” Long leases reduce round trips but enlarge the stale-data window. Zero or tiny leases improve freshness but can turn every pathname walk and stat() into protocol traffic.

A useful policy derives lease length from the source of truth:

• immutable archive: long namespace and data lifetimes are safe;
• daemon-exclusive mutable store: leases can be long if all changes pass through the daemon;
• shared directory modified externally: leases must be short or actively invalidated;
• distributed backend: lease duration must reflect the backend’s own consistency guarantees.

3. Active invalidation

When the backend changes independently, the daemon can send notifications to invalidate:

• an inode’s attributes or data range;
• a named entry under a parent;
• a directory entry and associated cache state;
• other negotiated cache state.

Notifications close the gap between long leases and freshness, but they require a reliable change feed and careful ordering. Invalidating after exposing a backend change may leave a short stale window; invalidating before a change can produce refetches that still see old data.

4. Four main data paths

4.1 Buffered cached I/O

The ordinary path uses the page cache. It supports readahead, write aggregation, and mmap naturally. It is usually best for repeated reads and small writes, but it requires explicit coherency policy.

4.2 Direct I/O

FOPEN_DIRECT_IO makes reads and writes use the direct FUSE path rather than ordinary page-cache I/O. This avoids persistent cache aliases, but still uses FUSE protocol requests and may split a large I/O into multiple operations.

Direct I/O is a semantic mode as well as a performance choice. Alignment, short I/O, partial completion, and asynchronous behavior must be handled consistently.

4.3 Passthrough

When supported and negotiated, a daemon can associate a FUSE open file with a backing kernel file so eligible I/O bypasses repeated daemon round trips. The daemon remains responsible for establishing a safe relationship between the visible FUSE object and the backing file.

4.4 DAX

In virtio-fs, DAX maps file ranges through a finite shared-memory window. Page faults request mappings; mappings may later be reclaimed. DAX avoids the ordinary page cache for mapped ranges, so its allocation, invalidation, writeback, and fallback rules must be designed explicitly.

5. Open flags that affect caching

The daemon’s OPEN reply can influence behavior, including concepts such as:

• direct I/O;
• keeping existing cached data across open;
• nonseekable or stream behavior;
• cache invalidation requirements;
• parallel direct-write support;
• passthrough or DAX eligibility through negotiated mechanisms.

These flags form part of the per-open contract. They cannot safely be toggled midway through an operation without reconciling existing cached or mapped state.

6. Writeback cache

With writeback cache enabled, writes can complete into dirty page-cache state before the daemon receives them. The kernel may merge, reorder, and issue larger writeback requests. Reads may be satisfied from dirty cached pages.

The daemon must therefore tolerate writes whose shape differs from application syscalls and must implement FLUSH, FSYNC, RELEASE, truncation, and error reporting coherently. “The application wrote once, therefore the daemon receives one identical WRITE” is not a valid assumption.

7. Coherency scenarios

7.1 All changes pass through one mount

This is the easiest case. The kernel and daemon observe every mutation, so relatively long leases and writeback cache can work well.

7.2 Backend changes outside the mount

Now the daemon needs change notifications, shorter leases, or a documented weak-consistency model. Updating only attributes is insufficient if file data or directory names also changed.

7.3 Multiple clients

A distributed daemon must coordinate cache leases across clients. Local kernel invalidation solves only one client’s cache; the backend protocol must distribute ordering and revocation.

7.4 Mixed cached I/O, direct I/O, mmap, and DAX

Multiple aliases to the same bytes can observe different values unless transitions flush dirty data, invalidate stale cache, and wait for outstanding faults or I/O. A fallback from DAX to direct I/O is particularly delicate: it must not duplicate already-completed bytes or race an existing DAX mapping.

8. Why fallback is not a simple retry

Assume a large DAX operation maps and transfers its first portion, then times out while waiting for another mapping slot. Linux I/O conventions commonly return the positive byte count when partial progress exists, hiding the later error from that syscall. Retrying the original full request through direct I/O would duplicate the prefix.

A safe fallback design must define:

1. the exact point before which no visible progress has occurred;
2. whether the I/O iterator and file position have advanced;
3. how dirty or writable DAX mappings are excluded;
4. whether only the untransferred suffix may use another path;
5. how short I/O differs from resource-timeout failure;
6. whether read and write need different guarantees.

The cleanest policy is normally to choose the path before transfer starts, or to fall back only when DAX allocation fails with zero progress.

9. Cache-policy checklist

For each object type, document:

• source of truth;
• cached representation;
• validity duration;
• passive expiration behavior;
• active invalidation source and ordering;
• behavior during daemon disconnect;
• interaction with mmap, direct I/O, passthrough, and DAX;
• observability needed to detect stale-cache bugs.

10. Summary

FUSE has separate namespace, attribute, and data caches. Validity intervals are time-bounded promises, while notifications provide active revocation. Buffered I/O, direct I/O, passthrough, and DAX change both performance and semantics; switching between them safely requires zero-progress detection and cache reconciliation.

Previous: Object Model and Protocol

Next: Concurrency, Security, and Failure Semantics