Skip to content

feat: add experimental on-demand pinning#11252

Open
ihlec wants to merge 1 commit intoipfs:masterfrom
ihlec:master
Open

feat: add experimental on-demand pinning#11252
ihlec wants to merge 1 commit intoipfs:masterfrom
ihlec:master

Conversation

@ihlec
Copy link
Copy Markdown

@ihlec ihlec commented Mar 26, 2026

Summary

Automatically pin content when DHT provider counts fall below a configurable replication target, and unpin once
replication has been above target for a grace period.

Helps keeping critical data around, without wasting storage on overly replicated CIDs.

The feature is described in this draft: ipfs/specs#532
The feature is gated behind Experimental.OnDemandPinningEnabled.

CLI commands for ipfs pin ondemand:

  • add -- register CIDs for on-demand pinning
  • rm -- deregister and unpin
  • ls -- list registered CIDs

Design

  • Pin partitioning: the checker needs to distinguish its pins from user
    pins to avoid accidental deletion. This implementation uses boxo's pin name
    field ("on-demand") for that.
  • Storage budget: skips pinning when repo usage exceeds
    StorageMax * StorageGCWatermark.
  • Idle timeout: DAG fetches timeout after 2 minutes
    without receiving new blocks (allows large downloads while skipping dead records).
  • Provide after pin: the checker publishes a DHT provider record after pinning.
  • Sybil limitation: provider counts come from DHT queries, which are
    susceptible to Sybil manipulation. Documented as a known limitation.

Configuration Parameters (TBD)

Option Default Description
OnDemandPinning.ReplicationTarget 5 Minimum providers in DHT (excluding self)
OnDemandPinning.CheckInterval "10m" How often the checker runs
OnDemandPinning.UnpinGracePeriod "24h" How long above target before unpinning

Tests

Unit tests for the checker and store in ondemandpin/.

Visualization of Feature

(out of scope for this CLI PR, but helpful to get the idea)
UI-flow

Add a background checker that automatically pins content when DHT provider counts fall below a configurable replication target and unpins once enough providers exist again after a grace period.

Gated behind Experimental.OnDemandPinningEnabled.
New CLI commands: ipfs pin ondemand {add,rm,ls}

Safety measures:
- storage budget check (respects StorageMax/GCWatermark)
- idle timeout on recursive DAG fetches (2 min without progress)
- pin partitioning via pin name to distinguish on-demand pins from persitent pins.
@ihlec
Copy link
Copy Markdown
Author

ihlec commented Mar 31, 2026

Please focus the review on:

1. Core logic — ondemandpin/

  • ondemandpin/store.go (143 lines) — Datastore with CRUD to manage on-demand pinning CIDs.
  • ondemandpin/checker.go (282 lines) — Frequently queries the DHT for provider counts and pins/unpins accordingly.

2. Kubo integration — wiring and CLI

  • core/node/ondemandpin.go (168 lines) — translates Kubo's pinning system into a simple interface for checker.go.
  • core/commands/pin/ondemandpin.go (280 lines) — Standard cmds boilerplate for on-demand pinning.

*The rest of the files are configuration, glue code, and documentation.

@guillaumemichel guillaumemichel mentioned this pull request Mar 31, 2026
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant