Skip to content

Add NATS-distributed OpenFiles server mode with cache invalidation and docs#1

Closed
dywongcloud wants to merge 2 commits into
mainfrom
openfiles-pr-clean
Closed

Add NATS-distributed OpenFiles server mode with cache invalidation and docs#1
dywongcloud wants to merge 2 commits into
mainfrom
openfiles-pr-clean

Conversation

@dywongcloud
Copy link
Copy Markdown
Owner

This PR adds optional NATS-based distribution to openfiles-server, allowing multiple OpenFiles server instances to share work across a NATS queue group while using the object backend as the durable source of truth.

What changed

  • Added optional [nats] configuration support.
  • Added a NATS transport layer for openfiles-server.
  • Added queue-group based distributed execution for mutating operations:
    • write
    • delete
    • rename
    • flush
    • expire cache
  • Added mutation event publishing for peer cache invalidation.
  • Added clean cache invalidation helpers to preserve dirty local writes.
  • Kept reads local to avoid cross-process cache-file mismatch.
  • Updated README with:
    • NATS setup
    • MinIO setup
    • distributed test commands
    • HTTP API examples
    • troubleshooting
    • CRDT design notes

Why

OpenFiles can now run as multiple HTTP gateways/workers against the same object-backed filesystem. This improves horizontal scalability and supports distributed processing without changing the core correctness model.

The object store remains the source of truth. NATS is used for coordination, work distribution, and cache invalidation — not as durable file storage.

Validation

Tested locally with:

  • MinIO running at 127.0.0.1:9000
  • NATS running at 127.0.0.1:4222
  • Two OpenFiles server instances:
    • 127.0.0.1:8787
    • 127.0.0.1:8788

Validated write/read across instances:

name="dist-$(date +%s).txt"

curl -i -X PUT --data-binary "hello $name" \
  "http://127.0.0.1:8787/v1/write/$name"

for i in $(seq 1 20); do
  curl -s "http://127.0.0.1:8788/v1/read/$name"
  echo
done

Reverse direction was also validated:

name="dist-reverse-$(date +%s).txt"

curl -i -X PUT --data-binary "hello $name" \
  "http://127.0.0.1:8788/v1/write/$name"

for i in $(seq 1 20); do
  curl -s "http://127.0.0.1:8787/v1/read/$name"
  echo
done

Both tests returned 20/20 successful reads with the expected content.

Notes

Reads intentionally remain local even when NATS is enabled. The object backend is shared, but cache files are per-process. Routing reads through a NATS queue group can hit a worker with stale cache metadata or missing local cache files. Local reads avoid that issue while still reading from the shared object backend.

CRDTs are not added for arbitrary file contents. This PR preserves the existing object-store conflict model and lost+found behavior. CRDT support can be layered later for specific structured file types such as JSON, collaborative text documents, or append-only logs.

@dywongcloud dywongcloud closed this May 5, 2026
@dywongcloud dywongcloud deleted the openfiles-pr-clean branch May 5, 2026 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant