snapshots: fix pipeline state machine edge cases #7570
Merged
+130
−152
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
amass-jump
commented
Dec 9, 2025
Comment on lines
-135
to
-140
| if( FD_UNLIKELY( ctx->is_zstd && ctx->dirty ) ) { | ||
| FD_LOG_WARNING(( "encountered end-of-file in the middle of a compressed frame" )); | ||
| ctx->state = FD_SNAPSHOT_STATE_ERROR; | ||
| fd_stem_publish( stem, 0UL, FD_SNAPSHOT_MSG_CTRL_ERROR, 0UL, 0UL, 0UL, 0UL, 0UL ); | ||
| return; | ||
| } |
Contributor
Author
There was a problem hiding this comment.
This check could never trigger before (we always reset dirty to 0 above), but also returning here deadlocks the pipeline
Comment on lines
-127
to
-128
| FD_TEST( ctx->state==FD_SNAPSHOT_STATE_PROCESSING || | ||
| ctx->state==FD_SNAPSHOT_STATE_ERROR ); |
Contributor
Author
There was a problem hiding this comment.
It's also possible for tiles to receive the FAIL control message in the IDLE state. This is the cause of several of the FD_TEST assertions we've seen.
This happens when snapct sends out a DONE, and an early tile immediately handles it and goes to IDLE. But a later tile may fail and generate ERROR, which causes snapct to send out a FAIL control message which is looped back through the pipeline
Comment on lines
162
to
167
| transition_malformed( fd_snapin_tile_t * ctx, | ||
| fd_stem_context_t * stem ) { | ||
| if( FD_UNLIKELY( ctx->state==FD_SNAPSHOT_STATE_ERROR ) ) return; | ||
| ctx->state = FD_SNAPSHOT_STATE_ERROR; | ||
| fd_stem_publish( stem, ctx->out_ct_idx, FD_SNAPSHOT_MSG_CTRL_ERROR, 0UL, 0UL, 0UL, 0UL, 0UL ); | ||
| } |
Contributor
Author
There was a problem hiding this comment.
Not a bug, but no need to generate extra ERROR messages when we are already in that state
Comment on lines
-316
to
-323
| static void | ||
| after_credit( fd_snapls_tile_t * ctx, | ||
| fd_stem_context_t * stem, | ||
| int * opt_poll_in FD_PARAM_UNUSED, | ||
| int * charge_busy FD_PARAM_UNUSED ) { | ||
| if( FD_UNLIKELY( ctx->hash_accum.received_lthashes==ctx->num_hash_tiles && ctx->hash_accum.awaiting_ack ) ) { | ||
| fd_lthash_sub( &ctx->hash_accum.calculated_lthash, &ctx->running_lthash ); | ||
| if( FD_UNLIKELY( memcmp( &ctx->hash_accum.expected_lthash, &ctx->hash_accum.calculated_lthash, sizeof(fd_lthash_value_t) ) ) ) { |
Contributor
Author
There was a problem hiding this comment.
This doesn't need to be asynchronous, because receiving all the NEXT/DONE "acks" from the snapla's implies that we must have already seen the HASH_RESULT messages from each of them. So as soon as we get the last ack we can check the lthash for correctness.
amass-jump
force-pushed
the
amass/snap-error
branch
from
d435810 to
9ac6b51
Compare
amass-jump
changed the title
Performance Measurements ⏳
|
amass-jump
force-pushed
the
amass/snap-error
branch
from
d9a07ab to
61e85bc
Compare
Performance Measurements ⏳
|
amass-jump
changed the title
amass-jump
changed the title
amass-jump
force-pushed
the
amass/snap-error
branch
from
61e85bc to
4536805
Compare
Performance Measurements ⏳
|
| if( FD_UNLIKELY( ctx->state!=FD_SNAPSHOT_STATE_FINISHING ) ) { | ||
| transition_malformed( ctx, stem ); | ||
| return; | ||
| break; |
Contributor
There was a problem hiding this comment.
why the break here instead of return? Do we still need to forward the control message if we are generating an error?
Contributor
Author
There was a problem hiding this comment.
Yes, every control message needs to be forwarded down the pipeline immediately, or deferred for later (such as with snapls). But we can't outright drop a control message or the pipeline will be locked on waiting for that message to be flushed forever (each control message is only generated a single time and we do not generate new control messages until it's been flushed).
amass-jump
force-pushed
the
amass/snap-error
branch
from
4536805 to
c5191c3
Compare
Performance Measurements ⏳
|
amass-jump
force-pushed
the
amass/snap-error
branch
from
c5191c3 to
b304768
Compare
amass-jump
changed the title
Performance Measurements ⏳
|
amass-jump
force-pushed
the
amass/snap-error
branch
from
b304768 to
95e973c
Compare
Performance Measurements ⏳
|
cali-jumptrading
approved these changes
Jan 6, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.