Skip to content

feat(evmrpc): bound eth_getLogs peak memory with a matched-log cap #3759

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
amir-deris wants to merge 4 commits into
base: main
Choose a base branch
from
Open

feat(evmrpc): bound eth_getLogs peak memory with a matched-log cap #3759

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
76f591d
feat(evmrpc): bound eth_getLogs peak memory with a matched-log cap
amir-deris Jul 15, 2026
5669275
Updated config comment
amir-deris Jul 15, 2026
4977fab
Added more test coverage
amir-deris Jul 15, 2026
000b292
Addressing feedbacks
amir-deris Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ Ref: https://keepachangelog.com/en/1.0.0/
# Changelog
## v6.6
sei-chain
* [#3759](https://github.com/sei-protocol/sei-chain/pull/3759) feat(evmrpc): bound `eth_getLogs` peak memory with a matched-log cap. `max_log_no_block` now caps both bounded and open-ended block ranges; exceeding it returns an error instead of silently truncating.
* [#3743](https://github.com/sei-protocol/sei-chain/pull/3743) Backport `release/v6.6`: feat(cosmos): range-check Dec conversions and DecCoin validation (CON-369)
* [#3732](https://github.com/sei-protocol/sei-chain/pull/3732) Bump version in prep to release v6.6-rc2
* [#3731](https://github.com/sei-protocol/sei-chain/pull/3731) Backport `release/v6.6`: Update v6.6 changelog in prep to cut RC2
Expand Down
7 changes: 5 additions & 2 deletions evmrpc/config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,9 @@ type Config struct {
// Deny list defines list of methods that EVM RPC should fail fast
DenyList []string `mapstructure:"deny_list"`

// max number of logs returned if block range is open-ended
// max number of logs a single eth_getLogs query may match before it errors,
// for both bounded and open-ended block ranges (a non-positive value falls
// back to DefaultMaxLogLimit)
MaxLogNoBlock int64 `mapstructure:"max_log_no_block"`

// max number of blocks to query logs for
Expand Down Expand Up @@ -695,7 +697,8 @@ enabled_legacy_sei_apis = [
# "sei2_getBlockTransactionCountByNumber",
]

# max number of logs returned if block range is open-ended
# max number of logs a single eth_getLogs query may match before it errors,
# for both bounded and open-ended block ranges
max_log_no_block = {{ .EVM.MaxLogNoBlock }}

# max number of blocks to query logs for
Expand Down
102 changes: 74 additions & 28 deletions evmrpc/filter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"fmt"
"sort"
"sync"
"sync/atomic"
"time"

"github.com/ethereum/go-ethereum/common"
Expand Down Expand Up @@ -802,23 +803,31 @@ func (f *LogFetcher) GetLogsByFilters(ctx context.Context, crit filters.FilterCr
return nil, 0, fmt.Errorf("block range too large (%d), maximum allowed is %d blocks", blockRange, f.filterConfig.maxBlock)
}

// maxLog caps the number of matching logs a single query may return, for
// both bounded and open-ended queries. Exceeding it is an error (not
// silent truncation), so peak memory stays bounded to the cap. NewFilterAPI
// coerces a non-positive value to DefaultMaxLogLimit, so limit is always > 0
// here; downstream helpers still treat limit <= 0 as uncapped.
limit := f.filterConfig.maxLog

@seidroid seidroid Bot Jul 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] limit is sourced from MaxLogNoBlock, whose config doc reads "max number of logs returned if block range is open-ended." On main, bounded queries were uncapped on both the range path (tryFilterLogsRange passed no limit) and the block-by-block path (limit only applied when applyOpenEndedLogLimit). Applying limit unconditionally here silently repurposes an open-ended-only setting into a hard error threshold for bounded queries: a bounded eth_getLogs matching more than MaxLogNoBlock (default 10000) logs now returns ErrTooManyLogs where it previously succeeded. If this broader cap is intended, update the max_log_no_block doc comment (config.go:103 and the config template) and the changelog; otherwise consider preserving the open-ended-only semantics or introducing a distinct general-cap setting.


// blockHash queries must use the hash-aware block fetch path below.
// Range-query receipt stores only constrain by numeric block range and
// do not enforce crit.BlockHash.
if crit.BlockHash == nil {
// Try efficient range query first
// #nosec G115 -- begin and end are validated to be positive block heights above
if logs, rangeErr := f.tryFilterLogsRange(ctx, uint64(begin), uint64(end), crit); rangeErr == nil {
if logs, rangeErr := f.tryFilterLogsRange(ctx, uint64(begin), uint64(end), crit, limit); rangeErr == nil {
return logs, end, nil
} else if !errors.Is(rangeErr, receipt.ErrRangeQueryNotSupported) {
// If it's a real error (not just unsupported), return it
// A real error (including receipt.ErrTooManyLogs) short-circuits;
// only "unsupported" falls through to the block-by-block path.
return nil, 0, rangeErr
}
}
// Fall back to block-by-block querying for backends that don't support range queries

bloomIndexes := EncodeFilters(crit.Addresses, crit.Topics)
blocks, end, applyOpenEndedLogLimit, err := f.fetchBlocksByCrit(ctx, crit, lastToHeight, bloomIndexes)
blocks, end, err := f.fetchBlocksByCrit(ctx, crit, lastToHeight, bloomIndexes)
if err != nil {
return nil, 0, err
}
Expand All @@ -828,14 +837,27 @@ func (f *LogFetcher) GetLogsByFilters(ctx context.Context, crit filters.FilterCr
sortedBatches := make([][]*ethtypes.Log, 0)
var wg sync.WaitGroup
var submitError error
// collected tracks matched logs across workers so the fan-out can stop
// queueing batches once the cap is exceeded.
var collected int64
capExceeded := func() bool {
return limit > 0 && atomic.LoadInt64(&collected) > limit
}

processBatch := func(batch []*coretypes.ResultBlock) {
defer wg.Done()
// Each worker gets a clean slice from the pool
localLogs := f.globalLogSlicePool.Get()

for _, block := range batch {
// Cooperative early-abort: once any worker has pushed the matched
// count past the cap, stop materializing further blocks.
if capExceeded() {
break
}
before := len(localLogs)
f.GetLogsForBlockPooled(block, crit, &localLogs)

@seidroid seidroid Bot Jul 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Nit: the cap is checked only between blocks, so GetLogsForBlockPooled materializes a full block's matching logs before collected is incremented. Peak matched-log memory is therefore limit + (workers × single-block logs), not strictly limit. This is inherent (at least one block must be materialized) and matches the PR's documented "cap plus at most one in-flight block per worker" — memory stays bounded — so no change needed, just flagging that the bound is cap + overshoot rather than the cap alone.

atomic.AddInt64(&collected, int64(len(localLogs)-before))
}

// Sort the local batch
Expand All @@ -855,6 +877,11 @@ func (f *LogFetcher) GetLogsByFilters(ctx context.Context, crit filters.FilterCr
// Batch process with fail-fast
blockBatch := make([]*coretypes.ResultBlock, 0, evmrpcconfig.WorkerBatchSize)
for block := range blocks {
// Stop queueing once the cap is exceeded; in-flight batches finish and
// the overflow is reported after wg.Wait.
if capExceeded() {
break
}
blockBatch = append(blockBatch, block)

if len(blockBatch) >= evmrpcconfig.WorkerBatchSize {
Expand All @@ -874,8 +901,8 @@ func (f *LogFetcher) GetLogsByFilters(ctx context.Context, crit filters.FilterCr
return nil, 0, submitError
}

// Process remaining blocks
if len(blockBatch) > 0 {
// Process remaining blocks, unless the cap is already exceeded
if len(blockBatch) > 0 && !capExceeded() {
wg.Add(1)
if err := runner.SubmitWithMetrics(func() { processBatch(blockBatch) }); err != nil {
wg.Done()
Expand All @@ -893,13 +920,15 @@ func (f *LogFetcher) GetLogsByFilters(ctx context.Context, crit filters.FilterCr
}
}()

// Push the cap into the merge so it stops popping once the limit is reached,
// bounding the merged result allocation to O(maxLog) instead of O(total
// matching logs). A limit of 0 means no cap.
var limit int64
if applyOpenEndedLogLimit {
limit = f.filterConfig.maxLog
// Drain the blocks channel so its producer goroutines are not left blocked

@seidroid seidroid Bot Jul 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] The rationale in this comment is slightly inaccurate. fetchBlocksByCrit sizes the channel buffer to the full range (end-begin+1) and blocks on errChan until every producer goroutine has finished and res is closed, so by the time blocks is returned here the producers have already completed — they are never "left blocked on a full buffer." The drain loop is still fine (it just releases buffered ResultBlocks for GC and terminates on the closed channel), but the comment's justification doesn't hold. Consider rewording to reflect that it's freeing buffered blocks rather than unblocking producers.

// on a full buffer when we broke out of the loop above on overflow.
for range blocks {
}

if limit > 0 && collected > limit {
return nil, 0, receipt.NewTooManyLogsError(limit)
}

res = f.mergeSortedLogs(sortedBatches, limit)

// Ensure we never return nil, always return an array (even if empty)
Expand Down Expand Up @@ -974,7 +1003,9 @@ func (f *LogFetcher) earliestHeight(ctx context.Context) (int64, error) {

// tryFilterLogsRange attempts to use the efficient range query if supported by the backend.
// Returns ErrRangeQueryNotSupported if the backend doesn't support range queries.
func (f *LogFetcher) tryFilterLogsRange(ctx context.Context, fromBlock, toBlock uint64, crit filters.FilterCriteria) ([]*ethtypes.Log, error) {
// When limit > 0 the backend aborts with receipt.ErrTooManyLogs once more than
// limit logs match, bounding peak memory; limit <= 0 means no cap.
func (f *LogFetcher) tryFilterLogsRange(ctx context.Context, fromBlock, toBlock uint64, crit filters.FilterCriteria, limit int64) ([]*ethtypes.Log, error) {
store := f.k.ReceiptStore()
if store == nil {
return nil, receipt.ErrRangeQueryNotSupported
Expand All @@ -984,7 +1015,7 @@ func (f *LogFetcher) tryFilterLogsRange(ctx context.Context, fromBlock, toBlock
// #nosec G115 -- toBlock is a block height which fits in int64
sdkCtx := f.ctxProvider(int64(toBlock))

logs, err := store.FilterLogs(sdkCtx, fromBlock, toBlock, crit)
logs, err := store.FilterLogs(sdkCtx, fromBlock, toBlock, crit, limit)
Comment thread
cursor[bot] marked this conversation as resolved.

@seidroid seidroid Bot Jul 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] The range path applies the cap against the store's pre-normalization count. filterLogsByTags counts every indexed matching log (including synthetic/Cosmos receipts), but normalizeRangeQueryLogs (called just below) rebuilds only from EVM-visible txs — for the eth_* namespace includeSyntheticReceipts=false, so synthetic logs are dropped afterward. Result: eth_getLogs can return ErrTooManyLogs even when the actual RPC result is below maxLog. The block-by-block fallback counts correctly (its collectLogs already excludes synthetic), so the two paths are inconsistent. Consider capping against the normalized count, or documenting that the range-path cap is a conservative over-count. (Codex P1.)

@seidroid seidroid Bot Jul 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] The cap is enforced by the store on the raw tag-index match count, which includes non-EVM-visible / synthetic logs, before normalizeRangeQueryLogs filters the result down to the EVM-visible set. Since normalized <= raw, a query whose EVM-visible result is within limit can still trip ErrTooManyLogs when the raw match count exceeds it — a false positive that the block-by-block path (which counts already-normalized logs via GetLogsForBlockPooled) does not have. Consider whether the range path should cap on the EVM-visible count for consistency, or document that the cap is intentionally a conservative pre-normalization bound. (Matches Codex's finding.)

if err != nil {
return nil, err
}
Expand All @@ -993,7 +1024,18 @@ func (f *LogFetcher) tryFilterLogsRange(ctx context.Context, fromBlock, toBlock
return []*ethtypes.Log{}, nil
}

return f.normalizeRangeQueryLogs(ctx, logs, crit)
normalized, err := f.normalizeRangeQueryLogs(ctx, logs, crit)
if err != nil {
return nil, err
}

// Re-check the cap against the normalized (EVM-visible) count, which the
// store's tag-index count does not match.
if limit > 0 && int64(len(normalized)) > limit {

@seidroid seidroid Bot Jul 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This post-normalization re-check appears to be dead code: it only fires when len(normalized) > limit, but the store already aborts with ErrTooManyLogs whenever the raw count exceeds limit, and normalized <= raw. So whenever the store returns successfully (raw <= limit), normalized <= limit too and this branch can never trigger. It's harmless as defense-in-depth, but it cannot recover the false-positive case described above — it can only tighten the cap further, never loosen it. A brief comment clarifying it's defensive would help, or it could be removed.

return nil, receipt.NewTooManyLogsError(limit)
}

return normalized, nil
}

// normalizeRangeQueryLogs corrects BlockHash, TxIndex, and LogIndex on logs
Expand Down Expand Up @@ -1170,57 +1212,61 @@ func MatchesCriteria(log *ethtypes.Log, crit filters.FilterCriteria) bool {
}

// Optimized fetchBlocksByCrit with batch processing
func (f *LogFetcher) fetchBlocksByCrit(ctx context.Context, crit filters.FilterCriteria, lastToHeight int64, bloomIndexes [][]BloomIndexes) (chan *coretypes.ResultBlock, int64, bool, error) {
func (f *LogFetcher) fetchBlocksByCrit(ctx context.Context, crit filters.FilterCriteria, lastToHeight int64, bloomIndexes [][]BloomIndexes) (chan *coretypes.ResultBlock, int64, error) {
if crit.BlockHash != nil {
// Check for invalid zero hash
zeroHash := common.Hash{}
if *crit.BlockHash == zeroHash {
// For invalid hash, return empty channel instead of error
res := make(chan *coretypes.ResultBlock)
close(res)
return res, 0, false, nil
return res, 0, nil
}

block, err := blockByHashRespectingWatermarks(ctx, f.tmClient, f.watermarks, crit.BlockHash[:], 1)
if err != nil {
// For non-existent blocks, return empty channel instead of error
res := make(chan *coretypes.ResultBlock)
close(res)
return res, 0, false, nil
return res, 0, nil
}
res := make(chan *coretypes.ResultBlock, 1)
res <- block
close(res)
return res, 0, false, nil
return res, 0, nil
}

applyOpenEndedLogLimit := f.filterConfig.maxLog > 0 && (crit.FromBlock == nil || crit.ToBlock == nil)
// Open-ended queries (missing fromBlock or toBlock) have their block range
// windowed down to the most recent maxBlock blocks rather than erroring; a
// bounded query whose range is too large still errors. The matched-log cap
// (maxLog) is enforced separately by the caller.
applyOpenEndedBlockWindow := f.filterConfig.maxLog > 0 && (crit.FromBlock == nil || crit.ToBlock == nil)
latest, err := f.watermarks.LatestHeight(ctx)
if err != nil {
return nil, 0, false, err
return nil, 0, err
}
earliest, err := f.watermarks.EarliestHeight(ctx)
if err != nil {
earliest = 0
}
begin, end, err := ComputeBlockBounds(latest, earliest, lastToHeight, crit)
if err != nil {
return nil, 0, false, err
return nil, 0, err
}

blockRange := end - begin + 1
if applyOpenEndedLogLimit && blockRange > f.filterConfig.maxBlock {
if applyOpenEndedBlockWindow && blockRange > f.filterConfig.maxBlock {
begin = end - f.filterConfig.maxBlock + 1
if begin < earliest {
begin = earliest
}
} else if !applyOpenEndedLogLimit && f.filterConfig.maxBlock > 0 && blockRange > f.filterConfig.maxBlock {
} else if !applyOpenEndedBlockWindow && f.filterConfig.maxBlock > 0 && blockRange > f.filterConfig.maxBlock {
// Use consistent error message format
return nil, 0, false, fmt.Errorf("block range too large (%d), maximum allowed is %d blocks", blockRange, f.filterConfig.maxBlock)
return nil, 0, fmt.Errorf("block range too large (%d), maximum allowed is %d blocks", blockRange, f.filterConfig.maxBlock)
}

if begin > end {
return nil, 0, false, fmt.Errorf("fromBlock %d is after toBlock %d", begin, end)
return nil, 0, fmt.Errorf("fromBlock %d is after toBlock %d", begin, end)
}

res := make(chan *coretypes.ResultBlock, end-begin+1)
Expand All @@ -1243,7 +1289,7 @@ func (f *LogFetcher) fetchBlocksByCrit(ctx context.Context, crit filters.FilterC
}
}(batchStart, batchEnd)); err != nil {
wg.Done()
return nil, 0, false, fmt.Errorf("system overloaded, please reduce request frequency: %w", err)
return nil, 0, fmt.Errorf("system overloaded, please reduce request frequency: %w", err)
}
}

Expand All @@ -1262,10 +1308,10 @@ func (f *LogFetcher) fetchBlocksByCrit(ctx context.Context, crit filters.FilterC
}

if firstErr != nil {
return nil, 0, false, firstErr
return nil, 0, firstErr
}

return res, end, applyOpenEndedLogLimit, nil
return res, end, nil
}

// Batch processing function for blocks
Expand Down
29 changes: 29 additions & 0 deletions evmrpc/filter_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,35 @@ func testFilterGetLogs(t *testing.T, namespace string, tests []GetFilterLogTests
}
}

// TestFilterGetLogsMatchedLogCap exercises the eth_getLogs matched-log cap
// end-to-end over the live JSON-RPC handler.
// This test is intentionally not parallel: it runs during the serial phase, so
// its cache warm-up below completes before the parallel filter tests resume.
func TestFilterGetLogsMatchedLogCap(t *testing.T) {
// Warm the shared block cache with MockHeight8's canonical block via an
// address-less number-range query (no address ⇒ no bloom pre-filter, so the
// block is fetched and cached).
warmCriteria := map[string]interface{}{
"fromBlock": "0x8",
"toBlock": "0x8",
}
warmRes := sendRequestGood(t, "getLogs", warmCriteria)
_, warmErr := warmRes["error"]
require.False(t, warmErr, "warm-up number query should not error, got: %v", warmRes)

filterCriteria := map[string]interface{}{
"blockHash": LogCapBlockHash,
"address": []common.Address{common.HexToAddress(LogCapAddr)},
}
resObj := sendRequestGood(t, "getLogs", filterCriteria)

_, hasResult := resObj["result"]
require.False(t, hasResult, "expected no result once the log cap is exceeded, got: %v", resObj["result"])
errObj, ok := resObj["error"].(map[string]interface{})
require.True(t, ok, "expected an error object, got: %v", resObj)
require.Contains(t, errObj["message"].(string), "too many logs")
}

func TestFilterGetFilterLogs(t *testing.T) {
t.Skip()
filterCriteria := map[string]interface{}{
Expand Down
Loading
Loading