Stratum v2 Template Provider via IPC interface (multiprocess) by Sjors · Pull Request #48 · Sjors/bitcoin

Sjors · 2024-07-17T15:16:20Z

This is an alternative approach to #68. Rather than integrating the Template Provider directly into Bitcoin Core, we make it an entirely separate application that communicates via IPC, see design/multiprocess.md.

Usage

Compile as usual, multiprocess is enabled by default.

Alternatively get a recent binary, see (unofficial, by me) releases ending in "ICP".

Start node:

build/src/bitcoin-node -chain=testnet4 -ipcbind=unix

Start the Template Provider:

build/bin/bitcoin-mine -chain=testnet4 -debug=sv2 -loglevel=sv2:trace

See general Stratum v2 usage instructions in bitcoin#29432.

Rationale, PRs this builds on

Upstream PRs used:

There are multiple ways to create such an application. It could be built from scratch in Rust, using bits and pieces of the Stratum Reference Implementation SRI. It could be a c++ application built from scratch, perhaps using some libraries from Bitcoin Core.

But this PR goes about in a slightly different way. Similar to Elements or Knots it's a set of patches on top of the full Bitcoin Core codebase. This allows me to leverage the build system, subclass Transport and use various other useful bits.

This lets me reuse almost all code from the integrated Template Provider bitcoin#29432. Both build direct on:

I modified the guix script to only produce bitcoin-node (for now) and bitcoin-mine binaries.

Goal

The goal is to drop bitcoin-node from the release here. Ideally users would install Bitcoin Core in the manner they're familiar with. They would then install bitcoin-mine separately and it should just work(tm), at most having to add a line to bitcoin.conf to turn IPC on.

This won't work until we ship a release with multiprocess and the mining interface. See bitcoin#30437 for progress towards that.

Rebase hint

Rebase hint:

git reset --hard 2024/07/sv2-tp-common
git fetch ryanofsky
git merge ryanofsky/pr/mine

git cherry-pick sjors/sv2-ipc~12^..sjors/sv2-ipc

Sjors · 2024-07-17T15:30:49Z

@ryanofsky I slightly mangled e3736422b87e9f5b67d058fc278fe0838e073e07 because of bitcoin@4e1a434#r1681193702. I'll clean that up later.

Sjors · 2024-07-17T16:56:53Z

Incorporated the changes from bitcoin#30437 (comment) and cleaned up commit history.

Sjors · 2024-07-18T08:58:20Z

@ryanofsky do you have a commit somewhere that adds libmultiprocess to the guix build?

Sjors · 2024-07-18T09:34:54Z

Getting a cryptic error: '__NVCC___WORKAROUND_GUARD' is not defined error for the multiprocess job: https://cirrus-ci.com/task/4940413969104896?logs=ci#L2225

Somehow related to the Template Provider depending on boost/multi_index via its use of the mempool.

I can't make sense of the personality(old_personality | ADDR_NO_RANDOMIZE) error thrown by MSAN and TSAN, but it's possible that's related to my self-hosted CI setup and/or the modification in 33196682f15caf6fb54d05ed4c53a0d711a91241.

ryanofsky · 2024-07-18T20:10:16Z

re: #48 (comment)

@ryanofsky do you have a commit somewhere that adds libmultiprocess to the guix build?

I don't, since I haven't really experimented with the guix build. Maybe simplest approach to start with would enable the multiprocess build option and build and package all the multiprocess executables including bitcoin-tp and bitcoin-node. I know in the longer run you want to have separate releases as described in bitcoin#30437 (comment).

re: #48 (comment)

Getting a cryptic error: '__NVCC___WORKAROUND_GUARD' is not defined error for the multiprocess job: https://cirrus-ci.com/task/4940413969104896?logs=ci#L2225

Somehow related to the Template Provider depending on boost/multi_index via its use of the mempool.

This is a -Wundef error caused by bitcoin#29876 that happens when a boost header is included from a source file that is compiled with -I /path/to/boost/headers rather than -isystem /path/to/boost/headers. The fix for this is probably:

-bitcoin_mine_CPPFLAGS = $(bitcoin_bin_cppflags)
+bitcoin_mine_CPPFLAGS = $(bitcoin_bin_cppflags) $(BOOST_CPPFLAGS)

to add the right -isystem option.

I can't make sense of the personality(old_personality | ADDR_NO_RANDOMIZE) error thrown by MSAN and TSAN, but it's possible that's related to my self-hosted CI setup and/or the modification in 3319668.

Not sure about this error. Link seems to be https://cirrus-ci.com/task/4759864851824640?logs=ci#L2486

Sjors · 2024-07-19T15:11:10Z

I'm getting the MSAN / TSAN error inconsistently on my self-hosted CI. Does not seem related to this PR. Trying to reproduce in isolation in #51.

Add a MakeBasicInit() function so simpler standalone IPC clients like bitcoin-mine in bitcoin#30437 and bitcoin-cli in bitcoin#32297 that only initiate IPC connections without exposing any IPC interfaces themselves can to avoid needing to implement their own specialized interfaces::Init subclasses.

Expose ENABLE_IPC build option to function tests so new tests for bitcoin-mine in bitcoin#30437 and bitcoin-cli in bitcoin#32297 can test IPC-only features.

Set new `BitcoinTestFramework.binary_paths.bitcoin_bin` property with path to the `bitcoin` wrapper binary. This allows new tests for `bitcoin-mine` in bitcoin#30437 and `bitcoin-cli` in bitcoin#32297 to find the `bitcoin` binary and call `bitcoin -m` to start nodes with IPC support. This way the new tests can run whenever the ENABLE_IPC build option is enabled, instead of only running when the `BITCOIN_CMD` environment variable is set to `bitcoin -m`

With this change, tests can specify `self.extra_init = [{ipcbind: True}]` to start a node listening on an IPC socket, instead of needing to choose which node binary to invoke and what `self.extra_args=[["-ipcbind=..."]]` value to pass to it. The eliminates boilerplate code bitcoin#30437 (interface_ipc_mining.py), bitcoin#32297 (interface_ipc_cli.py), and bitcoin#33201 (interface_ipc.py) previously needed in their test setup.

Allows InitError and InitWarning to be used by other executables beside bitcoind and bitcoin-node.

See src/bitcoin-mine.cpp for usage information. Co-authored-by: Sjors Provoost <sjors@sprovoost.nl>

Co-authored-by: Sjors Provoost <sjors@sprovoost.nl>

Co-authored-by: Ryan Ofsky <ryan@ofsky.org>

Suggested by Sjors Provoost <sjors@sprovoost.nl> in bitcoin#30437 (comment)

For now also build bitcoin-node.

The Template Provider binary: * won't have a wallet * probably won't have a gui * doesn't use the functional test framework Disable some CI stuff to speed it up. This is not a complete list, because: 1. That would be too hard to rebase 2. Some things might be added later, e.g. tracepoints

1b8d4a6f1e Merge bitcoin-core/libmultiprocess#194: mpgen: Work around c++20 / capnproto 0.8 incompatibility f1fad396bf Merge bitcoin-core/libmultiprocess#195: ci: Add openbsd eed42f210d ci: Bump all tasks to actions/checkout@v5 486a510bbe ci: Remove ancient and problematic -lstdc++fs in mpexample dd40897efe Add missing thread include 98414e7d28 ci: Add openbsd dc3ba22046 cmake, doc: Add check for CVE-2022-46149 cb170d4913 Merge bitcoin-core/libmultiprocess#193: build: require CapnProto 0.7.0 or better 8ceeaa6ae4 ci: Add olddeps job to test old dependencies versions c4cb758ecc mpgen: Work around c++20 / capnproto 0.8 incompatibility 30930dff7b build: require CapnProto 0.7.0 or better git-subtree-dir: src/ipc/libmultiprocess git-subtree-split: 1b8d4a6f1e54b92708bd2ad627ec6d440a1daf3d

Sjors · 2025-08-25T16:18:17Z

bitcoin#31802 landed!

Last rebase before I'm going to split off the sv2 functionality into a fresh codebase.

Sjors · 2025-08-27T11:46:54Z

This code is now part of https://github.com/Sjors/sv2-tp; feedback and pull requests are welcome there!

I'll be tagging a release shortly. Usage will be the same as in this PR, except:

the bitcoin-mine command was renamed to sv2-tp
you need to bring your own Bitcoin Core node (I'll make a preview release of v30)

This was referenced Jul 17, 2024

Stratum v2 Template Provider (take 3) bitcoin/bitcoin#29432

Closed

Stratum v2 Template Provider common functionality bitcoin/bitcoin#30475

Closed

ipc: add bitcoin-mine test program bitcoin/bitcoin#30437

Open

Sjors force-pushed the sv2-ipc branch from 301d15a to d70406e Compare July 17, 2024 16:56

Sjors force-pushed the sv2-ipc branch from d70406e to 78b1607 Compare July 17, 2024 17:11

Sjors force-pushed the sv2-ipc branch 2 times, most recently from c75792b to 91720b7 Compare July 18, 2024 09:26

Sjors force-pushed the sv2-ipc branch from 91720b7 to 25b4286 Compare July 18, 2024 09:39

Sjors force-pushed the 2024/07/sv2-tp-common branch 2 times, most recently from db58076 to 1ff4660 Compare July 18, 2024 12:24

Sjors force-pushed the sv2-ipc branch 3 times, most recently from 4354e96 to 3aa6f0a Compare July 18, 2024 13:11

Sjors force-pushed the 2024/07/sv2-tp-common branch from 1ff4660 to ee010f5 Compare July 18, 2024 13:16

Sjors force-pushed the sv2-ipc branch from 3aa6f0a to f2dd3bd Compare July 18, 2024 13:17

Sjors mentioned this pull request Jul 19, 2024

guix: build multiprocess bitcoin/bitcoin#30483

Closed

Sjors force-pushed the 2024/07/sv2-tp-common branch 2 times, most recently from 19cc9f1 to dabc32c Compare July 19, 2024 08:59

Sjors force-pushed the sv2-ipc branch 3 times, most recently from 6c24b45 to a8bb152 Compare July 19, 2024 13:55

Sjors mentioned this pull request Jul 19, 2024

ci: skip Github CI on branch pushes for forks bitcoin/bitcoin#30487

Merged

Sjors force-pushed the sv2-ipc branch from a8bb152 to 7e0de5a Compare July 19, 2024 16:23

Sjors mentioned this pull request Jul 19, 2024

Add sjors attestations for sv2-tp-0.1.5-icp Sjors/guix.sigs#1

Merged

Sjors mentioned this pull request Jun 24, 2025

IPC via TCP Sockets bitcoin/bitcoin#32802

Open

ryanofsky and others added 9 commits August 18, 2025 06:54

test: add is_ipc_enabled() and skip_if_no_ipc() functions

c3d82ef

Expose ENABLE_IPC build option to function tests so new tests for bitcoin-mine in bitcoin#30437 and bitcoin-cli in bitcoin#32297 can test IPC-only features.

interface_ui: move from node to common library

894297c

Allows InitError and InitWarning to be used by other executables beside bitcoind and bitcoin-node.

ipc: Add bitcoin-mine test program

33217f3

See src/bitcoin-mine.cpp for usage information. Co-authored-by: Sjors Provoost <sjors@sprovoost.nl>

test: add interface_ipc_mining.py test calling bitcoin-mine

bb033ef

Co-authored-by: Sjors Provoost <sjors@sprovoost.nl>

bitcoin-mine: Extend example to mine a block

295f9c2

Co-authored-by: Ryan Ofsky <ryan@ofsky.org>

ipc: Add bitcoin-mine to bitcoin wrapper interface

1e8a832

Suggested by Sjors Provoost <sjors@sprovoost.nl> in bitcoin#30437 (comment)

Sjors mentioned this pull request Aug 25, 2025

Stratum v2 Template Provider inside Bitcoin Core (deprecated) #68

Closed

Sjors added 16 commits August 25, 2025 16:59

Merge remote-tracking branch 'ryanofsky/pr/mine' into sv2-ipc

ad6d9c3

build: disable sv2 on Windows

a05b46d

Drop tip hash print from demo

8af16b2

Add event loop with interrupt

4d73dd5

build: add bitcoin_sv2 as bitcoin-mine dep

12864c9

Run Template Provider from bitcoin-mine

aab62cc

Document -sv2 arguments

2bd7e5b

build: add bitcoin-mine to executables

ce817da

depends: no sv2 without multiprocess

ef2ef5e

build: reduce Windows deploy stuff

1574763

guix: only build Template Provider

5028c31

For now also build bitcoin-node.

build: disable bitcoin-mine in fuzzer build

7c1fa9f

test: drop interface_ipc_mining.py

00f709a

Merge commit '8ce3ac4bb8117f086199eae673a215ddca118394' into sv2-ipc

021e099

Sjors mentioned this pull request Aug 25, 2025

Standalone Stratum v2 Template Provider #102

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stratum v2 Template Provider via IPC interface (multiprocess)#48

Stratum v2 Template Provider via IPC interface (multiprocess)#48
Sjors wants to merge 25 commits into2024/07/sv2-tp-commonfrom
sv2-ipc

Sjors commented Jul 17, 2024 •

edited

Loading

Uh oh!

Sjors commented Jul 17, 2024

Uh oh!

Sjors commented Jul 17, 2024

Uh oh!

Sjors commented Jul 18, 2024

Uh oh!

Sjors commented Jul 18, 2024 •

edited

Loading

Uh oh!

ryanofsky commented Jul 18, 2024

Uh oh!

Sjors commented Jul 19, 2024 •

edited

Loading

Uh oh!

Sjors commented Aug 25, 2025

Uh oh!

Sjors commented Aug 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Sjors commented Jul 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Usage

Rationale, PRs this builds on

Goal

Rebase hint

Uh oh!

Sjors commented Jul 17, 2024

Uh oh!

Sjors commented Jul 17, 2024

Uh oh!

Sjors commented Jul 18, 2024

Uh oh!

Sjors commented Jul 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ryanofsky commented Jul 18, 2024

Uh oh!

Sjors commented Jul 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sjors commented Aug 25, 2025

Uh oh!

Sjors commented Aug 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Sjors commented Jul 17, 2024 •

edited

Loading

Sjors commented Jul 18, 2024 •

edited

Loading

Sjors commented Jul 19, 2024 •

edited

Loading