2.6 Teardown, Debugging, and Performance

中文

English

Contents

Chapter 2: FUSE Implementation Analysis · Article 6 of 6

The final article covers engineering outside the happy path: connection teardown, forced abort, fusectl, tracepoints, latency decomposition, tuning, and fault injection. A filesystem is complete only when daemon disappearance and resource pressure have deterministic outcomes.

1. Exit is more than umount

Relevant events include:

• normal or lazy unmount;
• daemon transport close;
• administrator-triggered fusectl abort;
• mount-namespace destruction;
• virtio device removal or host backend disconnect;
• failure before INIT completes;
• system shutdown.

All should converge on consistent connection teardown rather than independently freeing partial state.

2. Teardown ordering constraints

A conceptual safe sequence is:

reject new requests
  -> publish disconnected/aborted state
  -> stop transport delivery
  -> finish pending and processing requests with errors
  -> wake application, daemon, and background waiters
  -> release DAX, passthrough, open, and lookup state
  -> drain workqueue, async, RCU, and ordinary references
  -> free mount, channel, and connection

The implementation can overlap some phases, but no object may be freed before its last possible accessor is gone.

3. Dirty data during disconnect

When the daemon disappears, dirty page-cache data may remain. The kernel cannot invent a path to persist it and must not report persistence success. Later writeback and fsync need deterministic connection errors, recorded through mapping error state where appropriate.

Forced abort exists to end hangs and release resources, not to rescue dirty data. Production control planes should distinguish graceful drain, bounded shutdown, and final forced abort.

4. fusectl

After mounting fusectl, each connection exposes:

File	Purpose
waiting	number of waiting or in-flight FUSE requests
abort	write to abort the connection and wake requests
max_background	read/write background-request limit
congestion_threshold	point at which the connection is considered congested

A persistently nonzero waiting value proves unfinished work, but not whether the bottleneck is the daemon, backend, transport, or DAX range allocator.

5. Kernel tracepoints

fs/fuse/fuse_trace.h defines request-lifecycle events:

• fuse_request_send;
• fuse_request_sent;
• fuse_request_end.

Opcode, node ID, unique, and error identify transactions. Daemon-side dequeue and backend-completion timestamps make finer decomposition possible.

Complementary tools include syscall tracing, VFS/filemap/writeback/iomap tracepoints, perf, temporary eBPF probes, scheduler and PSI data, and block/network backend metrics.

6. Four-part latency model

T_total = T_kernel_queue
        + T_transport_and_daemon_queue
        + T_backend
        + T_reply_and_wakeup

A cache hit may produce no request, and one syscall may produce several. Measure requests by unique first, then correlate them into syscall-level work rather than subtracting unrelated log timestamps.

7. Tuning order

A useful optimization sequence is:

1. reduce request count with suitable leases, readdirplus, and batched FORGET;
2. reduce unnecessary copies with suitable large requests, passthrough, splice, or DAX;
3. increase useful parallelism in workers, background slots, and backend pools;
4. control tail latency with limits, fairness, deadlines, and bounded jobs;
5. improve locality through readahead, writeback aggregation, and DAX working-set policy;
6. only then micro-optimize locks and allocation.

If every pathname component pays a remote RTT, shaving a few percent from context-switch cost will not fix the scale of the problem.

8. Parameter tradeoffs

• Larger max_background can raise throughput and also memory, backend queueing, and abort time.
• Larger requests amortize headers but occupy workers longer and enlarge short-I/O effects.
• Longer cache leases reduce metadata traffic and enlarge stale windows.
• Writeback cache aggregates writes but delays errors and complicates coherency.
• A larger DAX window reduces reclaim pressure while consuming more shared memory and address space.
• Aggressive DAX reclaim can produce layout-break and remapping thrash.

Evaluate p50/p99 latency, request rate, queue depth, throughput, and error rate together.

9. Symptom-to-entry table

Symptom	First checks
mount stuck at INIT	daemon read/reply, version, and response length
slow stat	dentry/attribute lease, LOOKUP/GETATTR rate, backend RTT
low read throughput	cache hit, request size, readahead, copies, backend
fast writes but slow fsync	writeback queue, daemon persistence, mapping errors
steadily growing waiting	daemon workers, backend saturation, deadlock
random mmap stalls	fault, invalidate lock, writeback, DAX-range wait
short large DAX I/O	window pressure, reclaim, error after partial progress
stuck unmount	open handles, background work, recursion, dirty data

10. Fault-injection matrix

Test at least:

• daemon death before dequeue, during processing, and before reply for every operation class;
• truncated, oversized, wrong-unique, duplicate, and late replies;
• signal versus normal-completion races;
• lookup/forget/open/unlink/rename interleavings;
• ENOSPC, EIO, and disconnect during writeback;
• background-limit saturation;
• daemon memory reclaim and backing-filesystem I/O;
• zero DAX free ranges, all mappings referenced, and delayed REMOVEMAPPING;
• truncate racing DAX read/write/fault;
• zero progress, partial progress, timeout, and late completion;
• unmount with mmap, open files, and dirty folios.

11. Verification layers for kernel changes

1. Build relevant configurations: FUSE, CUSE, virtio-fs, DAX, and io_uring.
2. Run existing xfstests and FUSE/virtio-fs tests.
3. Use a minimal daemon for protocol-boundary fault injection.
4. Run KASAN, KCSAN, and lockdep for lifetime and race defects.
5. Stress reclaim, writeback, mmap, and unmount.
6. Compare request count, CPU, throughput, and tail latency before and after.

Timeout/fallback work must verify iterator state, ki_pos, and exact return values—not merely that the application no longer hangs.

12. Tutorial conclusion

The FUSE implementation is a collection of state machines crossing VFS, protocol queues, daemon workers, and a backend. Caching and concurrency supply performance; identity, leases, ownership, and explicit failure semantics supply correctness. Following VFS entry → fuse_args → channel/request → daemon reply → cache/object update reconstructs the complete path from the distributed source files.

Previous: mmap, virtio-fs, and DAX

English contents

Tutorial home