IPIP: On-Demand Pinning Based on DHT Provider Counts#532
IPIP: On-Demand Pinning Based on DHT Provider Counts#532
Conversation
Refine motivation, registry structure, and configuration parameters.
|
This looks like useful functionality, but I'm not convinced it belongs in the IPFS specs repo. The proposal composes existing DHT and pinning primitives without introducing any new interoperability surface: no new wire format, cross-node protocol, nor behavior that other peers depend on. It reads more like a Kubo feature or a pinning service policy than a network-level spec. Would it make sense to move the discussion to a kubo issue? |
|
@guillaumemichel I asked myself the same question. I went with IPFS specs repo because I believe it would make sense for other IPFS implementations to support this feature as well (Helia, etc). If this reason does not justify having it here, we can move it. No problem. |
| 2. If providers < replication target and not currently pinned: recursively pin | ||
| the content |
There was a problem hiding this comment.
Presumably the pinning node needs to execute a provide for the CID too? Otherwise peers trying to fetch the content will not know that the pinning node has it.
There was a problem hiding this comment.
True. It would wait for the next re-provide interval (worst case 22h).
Immediate explicit provide call after pinning is better.
There was a problem hiding this comment.
Ah, right - I think I misread things - does this spec use the term "pinning" when it means "providing"?
It's certainly possible to pin something without providing it, and vice versa.
There was a problem hiding this comment.
Let's check.
My perception is:
Pinning = store content permanently on the node. Do not garbage collect.
Providing = write a provider record to the DHT (advertise content), pointing to my node.
It is possible to pin something without providing it, until re-provide picks it up.
It is possible to provide something without pinning, until garbage collector hits it.
The Spec assumes providing to be implicit after at most 22h, due to the re-provide system providing all pinned data by default.
I agree the spec should explicitly state the provide operation and not assume implicit providing. A explicit provide right after pinning makes the content discover-able early and prevents others from unnecessary pinning.
Please check if my perception is accurate. Thank you for your detailed review so far!
There was a problem hiding this comment.
It is possible to pin something without providing it, until re-provide picks it up.
It is possible to provide something without pinning, until garbage collector hits it.
This may be implementation-specific - js-libp2p's kad-dht implementation only re-provides previously provided CIDs, it doesn't look at what's pinned or not, or even if the blocks are present.
One day this might let a delegated endpoint publish provider records on behalf of less powerful/well connected nodes, for example.
| ### Pin naming | ||
|
|
||
| When the checker creates a pin, it SHOULD use a well-known name (e.g., | ||
| `"on-demand"`) to distinguish it from user-created pins. The checker MUST NOT | ||
| unpin content whose pin name does not match, to avoid removing user pins. |
There was a problem hiding this comment.
This is too specific - implementations like Helia allow attaching arbitrary metadata to pins, so it's easy to mark a pin as "on-demand" without dictating what the pin name should be.
Better to just say that implementations should partition their pin store in some way to prevent accidental deletion of user pins, or words to that effect.
🚀 Build Preview on IPFS ready
|
lidel
left a comment
lidel
left a comment
There was a problem hiding this comment.
Thanks for writing this up @ihlec, and for the PoC implementation.
I agree with @guillaumemichel -- this is a very nice feature, enables loosly-coordinated collaboration.
But it is not a spec change. It composes existing DHT queries and pin operations without introducing any new interoperability surface. No new wire protocol, no cross-implementation behavior or wire format that peers depend on, so it doesn't need to be a spec/IPIP.
Happy to keep this open as a draft PR so there's a linkable description of the idea for any implementation that want to add this to their pin code.
Suggested next step
This feels like a perfect feature for Kubo / WebUI / IPFS Desktop users.
@ihlec since you already did the work and have working PoC, do you mind opening a PR against ipfs/kubo with the implementation? That's where we can review and discuss the implementation and UX details (CLI ergonomics, pin naming, provide-after-pin,
concurrency, etc.) in the context of actual code.
Once the Kubo RPC adds new commands and ships, UI can be added to ipfs-webui, and ipfs-desktop gets it for free.
Happy to review a Kubo PR if someone is willing to invest the time. Other implementations may look into this independently too.
|
opened PR against Kubo: ipfs/kubo#11252 |
4 participants
Automatically pin content when DHT provider counts fall below a replication target, and unpin when replication recovers.
A working implementation exists in Kubo: https://github.com/ihlec/kubo-on-demand-pinning