Skip to content
5 changes: 5 additions & 0 deletions .changeset/por-indexer-streams-btc-indexer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@chainlink/por-indexer-adapter': minor
---

Add opt-in streams Bitcoin indexer support for Bitcoin mainnet balances in `por-indexer`. Default behaviour remains the existing `bitcoin-por-indexer` HTTP path via `BITCOIN_*_POR_INDEXER_URL`. Enable BCY/streams UTXO queries by setting `BITCOIN_MAINNET_USE_STREAMS_INDEXER=true` and `BITCOIN_MAINNET_RPC_URL`. Preserves `minConfirmations` on the streams path.
2 changes: 0 additions & 2 deletions .pnp.cjs

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

26 changes: 15 additions & 11 deletions packages/sources/por-indexer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,25 +8,29 @@ This document was generated automatically. Please see [README Generator](../../s

### Dependencies

The `por-indexer` external adapter is strongly dependent on a synced `bitcoin-por-indexer` service and will not be functional without it.
By default, Bitcoin balances are fetched from a NOP-run **`bitcoin-por-indexer`** HTTP service via `BITCOIN_*_POR_INDEXER_URL`.

Follow [this deployment documentation](https://chainlink.notion.site/Bitcoin-Proof-of-Reserves-v2-e670b124e429466bbb31988c6836a9da) to set up a `bitcoin-por-indexer` service. Note that sync time may take weeks in certain cases.
Optionally, a NOP can opt into the **streams Bitcoin indexer** (Electrs-compatible REST API, same as `dlc-cbtc-por`) for **Bitcoin mainnet** by setting `BITCOIN_MAINNET_USE_STREAMS_INDEXER=true` and `BITCOIN_MAINNET_RPC_URL`. Dogecoin continues to use `DOGECOIN_*_POR_INDEXER_URL`.

See [custom.md](./docs/custom.md) for more details.

### MAX_PAYLOAD_SIZE_LIMIT configuration

The `MAX_PAYLOAD_SIZE_LIMIT` environment variable is used for controlling the maximum size of the incoming request body that the EA can handle. If you decided to customize this value it's essential to ensure that any reverse proxy or web server in front of the EA, such as Nginx, is also configured with a corresponding limit. This alignment prevents scenarios where Nginx rejects a request for exceeding its payload size limit before it reaches the EA.

## Environment Variables

| Required? | Name | Description | Type | Options | Default |
| :-------: | :------------------------------: | :---------------------------------------------------------------------------------------: | :----: | :-----: | :-----------------------------------------------------------------: |
| | BITCOIN_MAINNET_POR_INDEXER_URL | Indexer URL for Bitcoin mainnet | string | | `` |
| | BITCOIN_TESTNET_POR_INDEXER_URL | Indexer URL for Bitcoin testnet | string | | `` |
| | DOGECOIN_MAINNET_POR_INDEXER_URL | Indexer URL for Dogecoin mainnet | string | | `` |
| | DOGECOIN_TESTNET_POR_INDEXER_URL | Indexer URL for Dogecoin testnet | string | | `` |
| | ZEUS_ZBTC_API_URL | API url for zeus zBTC | string | | `https://hermes.zeusnetwork.xyz/api/v2/chainlink/proof-of-reserves` |
| | BATCH_SIZE | Maximum number of addresses to send in a single request to the balance indexer | number | | `5000` |
| | BACKGROUND_EXECUTE_MS | The amount of time the background execute should sleep before performing the next request | number | | `10000` |
| Required? | Name | Description | Type | Options | Default |
| :-------: | :---------------------------------: | :---------------------------------------------------------------------------------------: | :-----: | :-----: | :-----------------------------------------------------------------: |
| | BITCOIN_MAINNET_POR_INDEXER_URL | bitcoin-por-indexer HTTP service URL for Bitcoin mainnet (default) | string | | `` |
| | BITCOIN_TESTNET_POR_INDEXER_URL | bitcoin-por-indexer HTTP service URL for Bitcoin testnet (default) | string | | `` |
| | BITCOIN_MAINNET_RPC_URL | Streams Bitcoin indexer endpoint for Bitcoin mainnet UTXO queries | string | | `` |
| | BITCOIN_MAINNET_USE_STREAMS_INDEXER | Use streams Bitcoin indexer for mainnet when `true` (requires `BITCOIN_MAINNET_RPC_URL`) | boolean | | `false` |
| | DOGECOIN_MAINNET_POR_INDEXER_URL | Indexer URL for Dogecoin mainnet | string | | `` |
| | DOGECOIN_TESTNET_POR_INDEXER_URL | Indexer URL for Dogecoin testnet | string | | `` |
| | ZEUS_ZBTC_API_URL | API url for zeus zBTC | string | | `https://hermes.zeusnetwork.xyz/api/v2/chainlink/proof-of-reserves` |
| | BATCH_SIZE | Maximum number of addresses to send in a single request to the balance indexer | number | | `5000` |
| | BACKGROUND_EXECUTE_MS | The amount of time the background execute should sleep before performing the next request | number | | `10000` |

---

Expand Down
11 changes: 9 additions & 2 deletions packages/sources/por-indexer/docs/custom.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,16 @@

### Dependencies

The `por-indexer` external adapter is strongly dependent on a synced `bitcoin-por-indexer` service and will not be functional without it.
**Default (unchanged):** Bitcoin balances use a NOP-run **`bitcoin-por-indexer`** HTTP service. Set `BITCOIN_MAINNET_POR_INDEXER_URL` / `BITCOIN_TESTNET_POR_INDEXER_URL`.

Follow [this deployment documentation](https://chainlink.notion.site/Bitcoin-Proof-of-Reserves-v2-e670b124e429466bbb31988c6836a9da) to set up a `bitcoin-por-indexer` service. Note that sync time may take weeks in certain cases.
**Opt-in streams indexer (mainnet only):** To use the shared BCY/streams Bitcoin indexer instead, set:

- `BITCOIN_MAINNET_USE_STREAMS_INDEXER=true`
- `BITCOIN_MAINNET_RPC_URL` to the Electrs-compatible base URL (same as `BITCOIN_RPC_ENDPOINT` in `dlc-cbtc-por`)

PoR jobspecs can stay on `indexer: por_indexer`; only the `por-indexer` EA deployment env vars change.

Dogecoin continues to use `DOGECOIN_*_POR_INDEXER_URL`.

### MAX_PAYLOAD_SIZE_LIMIT configuration

Expand Down
23 changes: 23 additions & 0 deletions packages/sources/por-indexer/src/config/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,17 @@ export const configDefinition = {
type: 'string',
default: '',
},
BITCOIN_MAINNET_RPC_URL: {
description: 'Streams Bitcoin indexer URL for mainnet UTXO queries (opt-in path)',
type: 'string',
default: '',
},
BITCOIN_MAINNET_USE_STREAMS_INDEXER: {
description:
'When true, use BITCOIN_MAINNET_RPC_URL for mainnet Bitcoin instead of BITCOIN_MAINNET_POR_INDEXER_URL',
type: 'boolean',
default: false,
},
DOGECOIN_MAINNET_POR_INDEXER_URL: {
description: 'Indexer URL for Dogecoin mainnet',
type: 'string',
Expand Down Expand Up @@ -50,3 +61,15 @@ export const config = new AdapterConfig(
},
},
)

export const balanceEnvVarForAddress = (
network: string,
chainId: string,
useStreamsMainnet: boolean,
): string => {
if (network === 'bitcoin' && chainId === 'mainnet' && useStreamsMainnet) {
return 'BITCOIN_MAINNET_RPC_URL'
}

return `${network}_${chainId}`.toUpperCase() + '_POR_INDEXER_URL'
}
29 changes: 15 additions & 14 deletions packages/sources/por-indexer/src/endpoint/balance.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
import { InputParameters } from '@chainlink/external-adapter-framework/validation'
import { PoRTotalBalanceEndpoint } from '@chainlink/external-adapter-framework/adapter/por'
import { config, configDefinition } from '../config'
import { balanceTransport } from '../transport/balance'
import { AdapterRequest } from '@chainlink/external-adapter-framework/util'
import { InputParameters } from '@chainlink/external-adapter-framework/validation'
import { AdapterInputError } from '@chainlink/external-adapter-framework/validation/error'
import { balanceEnvVarForAddress, config } from '../config'
import { balanceTransport } from '../transport/balance'

export const inputParameters = new InputParameters(
{
Expand Down Expand Up @@ -82,25 +82,26 @@ export const endpoint = new PoRTotalBalanceEndpoint({
})
}

// Check if based on input parameters corresponding env vars are set
const foundEnv = new Map()
const envVarsToCheck = Object.keys(configDefinition)
const checkedNetworkIds = new Set<string>()

for (const address of addresses) {
const id = `${address.network}_${address.chainId}`
const env = `${id}_POR_INDEXER_URL`.toUpperCase()
const networkId = `${address.network}_${address.chainId}`.toUpperCase()
if (checkedNetworkIds.has(networkId)) {
continue
}
checkedNetworkIds.add(networkId)

const env = balanceEnvVarForAddress(
address.network,
address.chainId,
settings.BITCOIN_MAINNET_USE_STREAMS_INDEXER,
)
if (!settings[env as keyof typeof settings]) {
throw new AdapterInputError({
statusCode: 400,
message: `'${env}' environment variable is required.`,
})
}

foundEnv.set(env, true)
// Stop the loop if all `network_chainId` env vars are found
if (foundEnv.size === envVarsToCheck.length) {
break
}
}

return
Expand Down
153 changes: 153 additions & 0 deletions packages/sources/por-indexer/src/lib/btc/por.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
import { Requester } from '@chainlink/external-adapter-framework/util/requester'

type UTXO = {
txid: string
vout: number
value: number
status: { confirmed: boolean; block_height?: number }
}

type MempoolTransaction = {
txid: string
vin: Array<{
txid: string
vout: number
prevout: { scriptpubkey_address: string; value: number }
}>
}

type TxStatus = {
confirmed: boolean
block_height?: number
}

const MAX_STREAMS_ADDRESS_BATCH_SIZE = 10

const joinUrl = (base: string, path: string): string => {
const url = new URL(base)
url.pathname = url.pathname.replace(/\/$/, '') + path
return url.toString()
}

const getConfirmations = (status: TxStatus, blockHeight: number): number => {
if (!status.confirmed || !status.block_height) return 0
return blockHeight - status.block_height + 1
}

const sumConfirmedUtxos = (utxos: UTXO[], blockHeight: number, minConfirmations: number): bigint =>
utxos
.filter((utxo) => getConfirmations(utxo.status, blockHeight) >= minConfirmations)
.reduce((sum, utxo) => sum + BigInt(utxo.value), 0n)

const fetchTxStatus = async (
requester: Requester,
endpoint: string,
txid: string,
): Promise<TxStatus> => {
const txStatusResponse = await requester.request<TxStatus>(
joinUrl(endpoint, `/tx/${txid}/status`),
{
url: joinUrl(endpoint, `/tx/${txid}/status`),
},
)
return txStatusResponse.response.data
}

const sumPendingSpendInputs = async (
requester: Requester,
endpoint: string,
mempoolTxs: MempoolTransaction[],
address: string,
blockHeight: number,
minConfirmations: number,
): Promise<bigint> => {
const matchingInputs = mempoolTxs.flatMap((tx) =>
tx.vin.filter((input) => input.prevout.scriptpubkey_address === address),
)

if (matchingInputs.length === 0) {
return 0n
}

if (minConfirmations <= 0) {
return matchingInputs.reduce((sum, input) => sum + BigInt(input.prevout.value), 0n)
}

const txStatusById = new Map<string, Promise<TxStatus>>()
const eligiblePendingSpends = await Promise.all(
matchingInputs.map(async (input) => {
let txStatusPromise = txStatusById.get(input.txid)
if (!txStatusPromise) {
txStatusPromise = fetchTxStatus(requester, endpoint, input.txid)
txStatusById.set(input.txid, txStatusPromise)
}

const txStatus = await txStatusPromise
return getConfirmations(txStatus, blockHeight) >= minConfirmations
? BigInt(input.prevout.value)
: 0n
}),
)

let total = 0n
for (const value of eligiblePendingSpends) {
total += value
}
return total
}

export async function calculateReserves(
requester: Requester,
endpoint: string,
addresses: string[],
minConfirmations: number,
batchSize: number,
): Promise<bigint> {
const blockHeightResponse = await requester.request<number>(
joinUrl(endpoint, '/blocks/tip/height'),
{
url: joinUrl(endpoint, '/blocks/tip/height'),
},
)
const blockHeight = blockHeightResponse.response.data as number

let totalReserves = 0n
const addressBatchSize = Math.max(1, Math.min(batchSize, MAX_STREAMS_ADDRESS_BATCH_SIZE))

for (let i = 0; i < addresses.length; i += addressBatchSize) {
const batch = addresses.slice(i, i + addressBatchSize)
const batchTotals = await Promise.all(
batch.map(async (address) => {
const utxoResponse = await requester.request<UTXO[]>(
joinUrl(endpoint, `/address/${address}/utxo`),
{ url: joinUrl(endpoint, `/address/${address}/utxo`) },
)
const mempoolResponse = await requester.request<MempoolTransaction[]>(
joinUrl(endpoint, `/address/${address}/txs/mempool`),
{ url: joinUrl(endpoint, `/address/${address}/txs/mempool`) },
)

const confirmed = sumConfirmedUtxos(
utxoResponse.response.data,
blockHeight,
minConfirmations,
)
const pending = await sumPendingSpendInputs(
requester,
endpoint,
mempoolResponse.response.data,
address,
blockHeight,
minConfirmations,
)
return confirmed + pending
}),
)

for (const reserves of batchTotals) {
totalReserves += reserves
}
}

return totalReserves
}
Loading
Loading