1.4 Concurrency, Security, and Failure Semantics

中文

English

Contents

Chapter 1: FUSE Subsystem Design · Article 4 of 4

FUSE adds another scheduler, queue, and failure domain to filesystem I/O. A design is incomplete until it specifies ordering, backpressure, cancellation, trust boundaries, daemon failure, and teardown.

1. Sources of concurrency

Requests may originate from:

• unrelated application threads;
• concurrent operations on the same inode or directory;
• page faults and readahead;
• background writeback;
• invalidation and forget work;
• daemon workers and backend completions;
• mount shutdown or connection abort.

The daemon must not assume requests arrive in syscall order, nor that completing them in read order preserves filesystem semantics.

2. Synchronous and background requests

A foreground request usually makes the caller wait for a reply. Background requests allow the initiating kernel path to proceed while completion happens later. The connection limits the number of active background operations and marks congestion before exhaustion.

Background execution improves throughput, but it also means errors may be reported later through writeback, fsync, close, or mapping faults rather than through the original write call.

3. Backpressure

Without bounds, a slow daemon could cause unbounded kernel memory use. FUSE therefore needs controls such as:

• maximum background request count;
• congestion threshold;
• daemon-side worker and buffer limits;
• bounded backend concurrency;
• fair scheduling across mounts, inodes, and tenants.

Queue depth is part of latency. Increasing it can improve throughput while making tail latency and cancellation worse.

4. Locking and ordering

Several lock domains can interact:

VFS namespace/inode locks
        -> FUSE connection and request-queue locks
        -> daemon locks
        -> backend filesystem, database, or RPC locks

A daemon that recursively accesses its own mount can deadlock: the kernel waits for the daemon, while the daemon blocks on a request that only itself can service. Backing storage and runtime dependencies should be kept outside the served mount unless recursion is deliberately designed and bounded.

Namespace operations need especially careful ordering. Rename, lookup, forget, readdir, and invalidation can overlap even when the VFS serializes only part of the operation.

5. Interruption is a protocol race

When a waiting task receives a signal, the kernel may send FUSE_INTERRUPT for the original request. This does not prove the daemon stopped it. Possible outcomes include:

1. the interrupt arrives before the daemon starts the operation;
2. it arrives while the backend operation is cancellable;
3. it races with normal completion;
4. normal completion has already won;
5. the backend operation cannot be cancelled.

The implementation needs a single completion owner and must tolerate a late reply without use-after-free or double completion. For mutating operations, returning EINTR does not automatically prove that no side effect occurred.

6. Object lifetime under concurrency

Consider an open file that is concurrently unlinked:

• the namespace entry disappears;
• lookup references may later be forgotten;
• the VFS inode can remain alive;
• daemon open handles remain valid until release;
• buffered or mapped writes may still need completion.

This is why lookup count, link count, VFS references, mapping references, and open-handle count must not be collapsed into one lifetime counter.

7. Security boundary

The daemon receives kernel-supplied caller context, but policy is split between the kernel and daemon. Security-relevant choices include:

• whether the kernel performs permission checks or delegates them;
• allow_other and who may access the mount;
• user namespace, idmapped mount, UID/GID translation, and supplementary context;
• treatment of setuid, device nodes, capabilities, ACLs, and xattrs;
• validation of untrusted daemon replies;
• privilege separation between the protocol front end and backend worker;
• protection against malformed lengths, node IDs, and notification floods.

The kernel cannot trust a daemon simply because it communicates through a kernel interface. Conversely, the daemon must treat request payloads and backend data as untrusted inputs.

8. Daemon stalls and crashes

A stalled daemon can block application tasks, reclaim, writeback, and unmount. A dead connection must transition outstanding and future requests to deterministic errors and wake all waiters.

Operational controls exposed through the FUSE control filesystem help inspect waiting requests and abort a connection. They are emergency and diagnostic mechanisms, not a replacement for daemon health management.

A production design should define:

• health checks and liveness ownership;
• whether requests have deadlines;
• abort and unmount policy;
• restart and remount strategy;
• what happens to dirty data;
• which errors applications observe.

9. Timeout semantics

A timeout is not merely a timer value. It must identify:

• the resource being waited for: daemon reply, queue space, DAX slot, or backend completion;
• the point at which the timer starts;
• whether the operation has made visible progress;
• whether cancellation prevents later side effects;
• how a late completion is discarded or reconciled;
• whether the timeout is reported directly or hidden by a positive partial byte count.

For I/O, Linux normally prefers returning bytes already transferred over a later error. Therefore, “fallback only on timeout” works safely only if that timeout is observable with zero progress, or if the fallback continues precisely from the untransferred suffix.

10. Teardown ordering

Unmount and abort must coordinate:

1. preventing new requests;
2. waking or failing queued waiters;
3. detaching device or transport endpoints;
4. finishing or invalidating outstanding I/O;
5. dropping lookup and open-handle state;
6. freeing connection and mount objects only after all references drain.

Errors in teardown often appear as rare hangs or use-after-free bugs because the ordinary request path is already gone.

11. Reliability checklist

Before deployment, test at least:

• daemon kill during lookup, read, write, fsync, and mmap fault;
• signal interruption before and after daemon dequeue;
• connection abort under heavy background writeback;
• lookup/forget/open/unlink races;
• malformed and late replies;
• queue saturation and daemon worker exhaustion;
• backend recursion and dependency failure;
• DAX-window exhaustion and mapping recall;
• unmount with open files and dirty mappings.

12. Summary

FUSE concurrency spans the VFS, kernel queues, daemon workers, and backend. Backpressure, cancellation, security, timeout, and teardown are protocol properties, not afterthoughts. A reliable filesystem defines one owner for every completion and one unambiguous outcome for every failure race.

Previous: Caching and Coherency

Next chapter: Source Map, Mount, and Initialization