From 1cf1a9e620583aadc314cef47d6962d2ff778ccd Mon Sep 17 00:00:00 2001 From: ihlec Date: Wed, 25 Mar 2026 11:50:57 +0100 Subject: [PATCH 1/2] IPIP: on-demand pinning based on DHT provider counts --- src/ipips/ipip-0000.md | 153 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 src/ipips/ipip-0000.md diff --git a/src/ipips/ipip-0000.md b/src/ipips/ipip-0000.md new file mode 100644 index 000000000..f0cdf1daf --- /dev/null +++ b/src/ipips/ipip-0000.md @@ -0,0 +1,153 @@ +--- +title: "IPIP-0000: On-Demand Pinning Based on DHT Provider Counts" +date: 2026-03-25 +ipip: proposal +editors: + - name: Cornelius Ihle + github: ihlec +relatedIssues: [] +order: 0000 +tags: ['ipips'] +--- + +## Summary + +Defines a mechanism for IPFS nodes to automatically pin content when DHT +provider counts fall below a configurable replication target, and unpin when +replication has recovered above target for a grace period. + +## Motivation + +IPFS content availability depends on at least one provider remaining online. +In practice, content hosted by a small number of volunteer nodes is fragile -- +if those nodes go offline, the content becomes unretrievable. There is +currently no standard mechanism for nodes to react to this situation. + +Node operators who want to help keep content alive must manually monitor +provider counts and re-pin content when replication drops. This is tedious, +error-prone, and doesn't scale. Conversely, nodes may continue pinning content +that already has abundant providers elsewhere, wasting local storage that could +serve under-replicated content instead. + +A standardized on-demand pinning mechanism would let community nodes act as an +automatic safety net: pinning content that is at risk of disappearing, and +releasing it once enough other providers exist. Without a common specification, +each implementation would invent its own incompatible approach, fragmenting the +ecosystem. + +## Detailed design + +This IPIP defines an on-demand pinning mechanism with three components: a +registry of monitored CIDs, a background checker, and pin/unpin behavior based +on DHT provider counts. + +### Registry + +Implementations maintain a persistent registry of CIDs to monitor. Each entry +tracks: + +- The CID being monitored +- Whether the implementation currently holds a pin for it +- A timestamp for when providers were last observed above target (used for + grace period tracking) +- A creation timestamp + +Users add and remove CIDs from the registry explicitly. Adding a CID does not +immediately pin it -- it registers it for monitoring. + +### Background checker + +A periodic loop evaluates every registered CID: + +1. Query the DHT for providers of the CID (excluding self) +2. If providers < replication target and not currently pinned: recursively pin + the content +3. If providers >= replication target and currently pinned: start the grace + period timer (if not already running) +4. If grace period has elapsed: unpin the content +5. If providers drop below target again while grace period is running: reset + the timer + +The checker skips CIDs that have a user-created pin (to avoid interfering with +manual pin management). + +### Configuration parameters + +Implementations MUST support at minimum: + +- **Replication target**: the desired number of providers (excluding self). + Recommended default: 5 +- **Check interval**: time between full evaluation cycles. Recommended + default: 10 minutes +- **Unpin grace period**: how long providers must stay above target before + unpinning. Recommended default: 24 hours + +### Pin naming + +When the checker creates a pin, it SHOULD use a well-known name (e.g., +`"on-demand"`) to distinguish it from user-created pins. The checker MUST NOT +unpin content whose pin name does not match, to avoid removing user pins. + +### Pinning scope + +The checker MUST use recursive pins. Direct pins are not sufficient to preserve +content availability since they do not protect linked blocks. + +## Design rationale + +The design favors simplicity: it relies entirely on existing DHT infrastructure +for provider discovery and existing pin semantics for storage. No new wire +protocols or peer coordination are introduced. + +The grace period mechanism prevents thrashing. Without it, a CID hovering near +the replication target would be pinned and unpinned on every check cycle. A +24-hour default gives enough time to confirm that new providers are stable. + +The pin naming convention lets the checker coexist with user pins. If a user +manually pins a CID that is also registered for on-demand pinning, the checker +will not interfere. + +### User benefit + +Nodes can contribute storage where it matters most. Instead of pinning content +indefinitely regardless of how many other providers exist, on-demand pinning +frees storage automatically when content is well-replicated, making room for +content that actually needs help. + +### Compatibility + +This feature is purely additive. Nodes that do not implement on-demand pinning +are unaffected. On-demand pinning nodes interact with the network using only +existing DHT queries and standard pin operations -- no protocol changes are +required. + +### Security + +DHT provider counts can be gamed. A Sybil attack could inflate provider counts +by announcing many fake provider records, tricking nodes into unpinning content +that is not actually well-replicated. Implementations SHOULD document this +limitation. The grace period provides partial mitigation: an attacker would +need to sustain fake provider records for the full grace duration. + +Implementations SHOULD also consider storage exhaustion: if the registry is +exposed to untrusted input, an attacker could register many large CIDs to fill +a node's disk. Access to the registry SHOULD be restricted to the node +operator. + +### Alternatives + +A dedicated replication protocol between cooperating nodes was considered. This +would allow nodes to explicitly coordinate who pins what, avoiding redundant +work. However, it would require a new network protocol, peer discovery for +replication partners, and consensus mechanisms -- dramatically increasing +complexity. The DHT-based approach reuses existing infrastructure and requires +no coordination between nodes. + +## Test fixtures + +Not applicable. This IPIP defines node behavior, not content-addressed data +formats. + +### Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From c6839fce75d2bd9961544446922d61c1e9106a0a Mon Sep 17 00:00:00 2001 From: Cornelius Ihle Date: Wed, 25 Mar 2026 12:29:58 +0100 Subject: [PATCH 2/2] Improve clarity and details in IPIP-0000 Refine motivation, registry structure, and configuration parameters. --- src/ipips/ipip-0000.md | 43 +++++++++++++++++++----------------------- 1 file changed, 19 insertions(+), 24 deletions(-) diff --git a/src/ipips/ipip-0000.md b/src/ipips/ipip-0000.md index f0cdf1daf..20031ca43 100644 --- a/src/ipips/ipip-0000.md +++ b/src/ipips/ipip-0000.md @@ -19,9 +19,7 @@ replication has recovered above target for a grace period. ## Motivation IPFS content availability depends on at least one provider remaining online. -In practice, content hosted by a small number of volunteer nodes is fragile -- -if those nodes go offline, the content becomes unretrievable. There is -currently no standard mechanism for nodes to react to this situation. +Content hosted by a small number of nodes is fragile. Node operators who want to help keep content alive must manually monitor provider counts and re-pin content when replication drops. This is tedious, @@ -31,15 +29,15 @@ serve under-replicated content instead. A standardized on-demand pinning mechanism would let community nodes act as an automatic safety net: pinning content that is at risk of disappearing, and -releasing it once enough other providers exist. Without a common specification, -each implementation would invent its own incompatible approach, fragmenting the -ecosystem. +releasing it once enough other providers exist. ## Detailed design -This IPIP defines an on-demand pinning mechanism with three components: a -registry of monitored CIDs, a background checker, and pin/unpin behavior based -on DHT provider counts. +This IPIP defines an on-demand pinning mechanism with three components: + +- registry of monitored CID +- background checker +- pin/unpin behavior based on DHT provider counts ### Registry @@ -48,9 +46,8 @@ tracks: - The CID being monitored - Whether the implementation currently holds a pin for it -- A timestamp for when providers were last observed above target (used for - grace period tracking) -- A creation timestamp +- A timestamp for when providers were last observed above target (used for grace period tracking) +- A creation timestamp (purely informational, no logic depends on the timestamp) Users add and remove CIDs from the registry explicitly. Adding a CID does not immediately pin it -- it registers it for monitoring. @@ -69,11 +66,12 @@ A periodic loop evaluates every registered CID: the timer The checker skips CIDs that have a user-created pin (to avoid interfering with -manual pin management). +manual pin management). +The periodic query replaces the periodic re-provide and only adds overhead for actively pinnned CIDs (below replication target). ### Configuration parameters -Implementations MUST support at minimum: +Implementations MUST support the following parameters (values are TBD): - **Replication target**: the desired number of providers (excluding self). Recommended default: 5 @@ -90,7 +88,7 @@ unpin content whose pin name does not match, to avoid removing user pins. ### Pinning scope -The checker MUST use recursive pins. Direct pins are not sufficient to preserve +The checker MUST use recursive pins. Direct pins do not preserve content availability since they do not protect linked blocks. ## Design rationale @@ -114,6 +112,9 @@ indefinitely regardless of how many other providers exist, on-demand pinning frees storage automatically when content is well-replicated, making room for content that actually needs help. +On-demand pinning can be easily integrated into existing UI-flow. +![UI-flow](https://github.com/user-attachments/assets/2cc1109a-1005-4a5c-b843-7821499c5f0e) + ### Compatibility This feature is purely additive. Nodes that do not implement on-demand pinning @@ -129,19 +130,13 @@ that is not actually well-replicated. Implementations SHOULD document this limitation. The grace period provides partial mitigation: an attacker would need to sustain fake provider records for the full grace duration. -Implementations SHOULD also consider storage exhaustion: if the registry is -exposed to untrusted input, an attacker could register many large CIDs to fill -a node's disk. Access to the registry SHOULD be restricted to the node -operator. - ### Alternatives A dedicated replication protocol between cooperating nodes was considered. This would allow nodes to explicitly coordinate who pins what, avoiding redundant -work. However, it would require a new network protocol, peer discovery for -replication partners, and consensus mechanisms -- dramatically increasing -complexity. The DHT-based approach reuses existing infrastructure and requires -no coordination between nodes. +work. However, it would require a overlay network protocol, and peer discovery for +replication partners -- dramatically increasing [complexity](https://github.com/gipplab/D-LOCKSS) or introducing [centralized components](https://github.com/gipplab/ipfs-archive-tracker). +The DHT-based approach reuses existing infrastructure and requires no coordination between nodes. ## Test fixtures