Skip to content

fix: saturate transport in-flight counter to prevent underflow wedge#155

Merged
eli-r-ph merged 1 commit into
mainfrom
fix/in-flight-counter-underflow
Jun 24, 2026
Merged

fix: saturate transport in-flight counter to prevent underflow wedge#155
eli-r-ph merged 1 commit into
mainfrom
fix/in-flight-counter-underflow

Conversation

@eli-r-ph

@eli-r-ph eli-r-ph commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

The in-flight event counter is a shared AtomicUsize decremented from many paths, and that accounting is tricky. This hardens it against a double-decrement bug underflowing the counter.

💡 Why

  • An underflow wraps to a huge value.
    • try_reserve then sees the queue as permanently full.
    • Every later event is silently dropped for the life of the client.
  • The counter is decremented from 5+ paths, so a future bug is plausible.
    • before_send drops, partial v1 results, terminal sends, shutdown drops, channel drain.
  • Flagged in review on feat: runtime-independent background event transport #145.

🔨 What

  • Decrements now saturate at 0 instead of fetch_sub.
    • fetch_update + saturating_sub in dec_len.
  • debug_assert catches a real underflow bug in tests.
    • Release builds clamp instead of wedging.
  • Both "worker gone" enqueue paths now route through dec_len too.

💚 How tested

  • fmt + clippy -D warnings clean (default + capture-v1).
  • 14 transport:: tests pass across async-v1, blocking-v1, v0.
    • Includes the two that drive dec_len to 0 with the assert live.
  • No saturation-only test: it needs --release (assert panics in debug), so it would not run in CI.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file

The in-flight `len` counter is decremented from several paths (before_send
drops, partial v1 batch results, terminal outcomes, shutdown drops, channel
drain). A double-decrement on any of them would underflow the AtomicUsize via
plain fetch_sub and wrap to a huge value, making the bounded queue look
permanently full and silently dropping every later event.

Route all production decrements through dec_len and have it saturate at 0
(fetch_update + saturating_sub), with a debug_assert to surface such a bug in
tests. Release builds now degrade gracefully instead of wedging the queue.
@greptile-apps

greptile-apps Bot commented Jun 24, 2026

Copy link
Copy Markdown

Reviews (1): Last reviewed commit: "fix: saturate transport in-flight counte..." | Re-trigger Greptile

@eli-r-ph eli-r-ph self-assigned this Jun 24, 2026
@eli-r-ph eli-r-ph requested a review from cat-ph June 24, 2026 18:36
@github-actions

Copy link
Copy Markdown
Contributor

posthog-rs-v1 Compliance Report

Date: 2026-06-24 18:37:34 UTC
Duration: 23006ms

⚠️ Some Tests Failed

96/110 tests passed, 14 failed


Capture_V1 Tests

94/94 tests passed

View Details
Test Status Duration
Endpoint And Method.Targets V1 Endpoint 147ms
Endpoint And Method.Does Not Use Legacy Endpoints 147ms
Required Headers.Has Authorization Bearer Header 152ms
Required Headers.Has Content Type Json 146ms
Required Headers.Has Posthog Sdk Info Format 146ms
Required Headers.Has Posthog Attempt Header 147ms
Required Headers.Has Posthog Request Id 146ms
Required Headers.Has Posthog Request Timestamp 145ms
Required Headers.Has User Agent 145ms
Body Format.Body Has Created At And Batch 144ms
Body Format.No Api Key In Body 139ms
Body Format.No Sent At In Body 140ms
Event Format.Event Has Required Root Fields 140ms
Event Format.Event Uuid Is Valid 138ms
Event Format.Event Timestamp Is Rfc3339 138ms
Event Format.Distinct Id Is String 142ms
Event Format.Distinct Id At Root Not Properties 140ms
Event Format.Custom Properties Preserved 138ms
Event Format.Set Properties Preserved 139ms
Event Format.Set Once Properties Preserved 134ms
Event Format.Groups Properties Preserved 134ms
Event Format.Sdk Generates Uuid If Not Provided 134ms
Event Format.Event Has Required Root Fields Batch 159ms
Event Format.Event Uuid Is Valid Batch 167ms
Event Format.Event Timestamp Is Rfc3339 Batch 178ms
Event Format.Distinct Id Is String Batch 167ms
Event Format.Distinct Id At Root Not Properties Batch 158ms
Event Format.Custom Properties Preserved Batch 178ms
Event Format.Set Properties Preserved Batch 177ms
Event Format.Set Once Properties Preserved Batch 176ms
Event Format.Groups Properties Preserved Batch 139ms
Event Format.Sdk Generates Uuid If Not Provided Batch 146ms
Batch Behavior.Multiple Events In Single Batch 189ms
Batch Behavior.Batch Envelope Smoke 190ms
Batch Behavior.Flush With No Events Sends Nothing 80ms
Batch Behavior.Flush At Triggers Batch 1128ms
Batch Behavior.Created At Reflects Batch Creation Time 118ms
Deduplication.Generates Unique Uuids 167ms
Deduplication.Different Events Same Content Different Uuids 145ms
Deduplication.Preserves Uuid On Retry 5120ms
Deduplication.Preserves Timestamp On Retry 5076ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry 5093ms
Deduplication.No Duplicate Events In Batch 134ms
Header Behavior On Retry.Attempt Header Starts At One 64ms
Header Behavior On Retry.Attempt Header Increments On Retry 10089ms
Header Behavior On Retry.Request Id Preserved On Retry 5071ms
Header Behavior On Retry.Different Requests Have Different Request Ids 2074ms
Header Behavior On Retry.Request Timestamp Changes On Retry 5068ms
Response Format Validation.Success Response Has Uuid Keyed Results 59ms
Response Format Validation.Success Response Has Ok For Each Event 34ms
Response Format Validation.Success No Retry After When All Ok 37ms
Response Format Validation.Success Retry After Present When Retry Events 29ms
Response Format Validation.Success No Retry After When Drop Only 32ms
Response Format Validation.Response Echoes Request Id 29ms
Retry Behavior.Retries On 408 5028ms
Retry Behavior.Retries On 500 5023ms
Retry Behavior.Retries On 503 5025ms
Retry Behavior.Retries On 504 5022ms
Retry Behavior.Retryable Errors Have Retry After 2027ms
Retry Behavior.Respects Retry After On Retryable Error 8023ms
Retry Behavior.Does Not Retry On 400 2037ms
Retry Behavior.Does Not Retry On 401 2027ms
Retry Behavior.Does Not Retry On 402 2035ms
Retry Behavior.Does Not Retry On 413 2026ms
Retry Behavior.Does Not Retry On 415 2024ms
Retry Behavior.Non Retryable Errors Have No Retry After 2024ms
Retry Behavior.Implements Backoff 15032ms
Retry Behavior.Max Retries Respected 15024ms
Partial Batch Handling.Handles 200 Full Success 2028ms
Partial Batch Handling.Handles 200 With All Ok 3043ms
Partial Batch Handling.Does Not Retry Dropped Events 3042ms
Partial Batch Handling.Does Not Retry Limited Events 3027ms
Partial Batch Handling.Prunes Ok Events On Partial Retry 5027ms
Partial Batch Handling.Prunes Dropped Events On Partial Retry 5027ms
Partial Batch Handling.Retries Only Retry Events From Partial 5026ms
Partial Batch Handling.Partial Retry Preserves Uuids 5025ms
Partial Batch Handling.Partial Retry Attempt Header Increments 5029ms
Partial Batch Handling.Partial Retry Request Id Preserved 5030ms
Partial Batch Handling.Respects Retry After On Partial 5024ms
Partial Batch Handling.Unknown Result Treated As Terminal 3027ms
Partial Batch Handling.Mixed Ok Drop Limited No Retry 3026ms
Compression.Sends Gzip Content Encoding 22ms
Compression.No Content Encoding When Disabled 21ms
Compression.Compressed Body Is Decompressible 21ms
Error Handling.Does Not Retry On Unknown 4Xx 2024ms
Event Options.Cookieless Mode Override 23ms
Event Options.Disable Skew Correction Override 22ms
Event Options.Process Person Profile Override 22ms
Event Options.Product Tour Id Override 22ms
Event Options.Unset Options Omitted 21ms
Event Options.Options Override In Batch 23ms
Geoip And Historical Migration.Geoip Disable Injected Into Properties 22ms
Geoip And Historical Migration.Historical Migration Set In Body 22ms
Geoip And Historical Migration.Historical Migration Absent By Default 22ms

Feature_Flags Tests

⚠️ 2/16 tests passed, 14 failed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id 12ms
Request Payload.Flags Request Uses V2 Query Param 16ms
Request Payload.Flags Request Hits Flags Path Not Decide 15ms
Request Payload.Flags Request Omits Authorization Header 15ms
Request Payload.Token In Flags Body Matches Init 15ms
Request Payload.Groups Round Trip 14ms
Request Payload.Groups Default To Empty Object 15ms
Request Payload.Person Properties Distinct Id Auto Populated When Caller Omits It 15ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False 15ms
Request Payload.Disable Geoip Omitted Defaults To False 15ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key 14ms
Request Lifecycle.No Flags Request On Init Alone 12ms
Request Lifecycle.No Flags Request On Normal Capture 23ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests 12ms
Request Lifecycle.Mock Response Value Is Returned To Caller 14ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event 15ms

Failures

request_payload.request_with_person_properties_device_id

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-18792331-c448-4e87-8021-0f9dc24b914a'

request_payload.flags_request_uses_v2_query_param

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-1ca4d46f-16d5-4893-bae4-b62e2dc46a49'

request_payload.flags_request_hits_flags_path_not_decide

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-2c93b787-5468-4715-b11b-04fd21bbdf02'

request_payload.flags_request_omits_authorization_header

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-15a9cd3b-bd31-475f-9a45-b09adfddb7c9'

request_payload.token_in_flags_body_matches_init

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-c6ce700f-1305-4fad-993f-226f0fee61c3'

request_payload.groups_round_trip

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-67b56ac8-b926-4b57-8fc1-c5aca0534a99'

request_payload.groups_default_to_empty_object

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-a5a192c0-b813-4fef-888c-656e2c059f9e'

request_payload.person_properties_distinct_id_auto_populated_when_caller_omits_it

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-8c74f023-025f-46b6-aad9-db35ed20dc49'

request_payload.disable_geoip_false_propagates_as_geoip_disable_false

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-bdbdafb3-dd7e-4b61-bbf5-40054887efb7'

request_payload.disable_geoip_omitted_defaults_to_false

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-032ece5c-c76b-4352-a87e-ea31cd02f395'

request_payload.flag_keys_to_evaluate_contains_only_requested_key

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-3e4e4963-cec6-465f-a54d-1e2ecb3db97d'

request_lifecycle.two_flag_calls_produce_two_remote_requests

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-41a675fa-4bc9-4c17-a1d2-fe5ea073a8d5'

request_lifecycle.mock_response_value_is_returned_to_caller

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-d0ae8559-35c6-43c9-85dd-93f0f1d7d2a3'

side_effect_events.get_feature_flag_captures_feature_flag_called_event

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-9f56e0a1-6504-4099-aa6a-6d03d881c84f'

@github-actions

Copy link
Copy Markdown
Contributor

posthog-rs-v0 Compliance Report

Date: 2026-06-24 18:40:03 UTC
Duration: 15547ms

⚠️ Some Tests Failed

31/45 tests passed, 14 failed


Capture Tests

29/29 tests passed

View Details
Test Status Duration
Format Validation.Event Has Required Fields 156ms
Format Validation.Event Has Uuid 162ms
Format Validation.Event Has Lib Properties 163ms
Format Validation.Distinct Id Is String 157ms
Format Validation.Token Is Present 161ms
Format Validation.Custom Properties Preserved 160ms
Format Validation.Event Has Timestamp 162ms
Retry Behavior.Retries On 503 5165ms
Retry Behavior.Does Not Retry On 400 2165ms
Retry Behavior.Does Not Retry On 401 2161ms
Retry Behavior.Respects Retry After Header 5103ms
Retry Behavior.Implements Backoff 15102ms
Retry Behavior.Retries On 500 5094ms
Retry Behavior.Retries On 502 5098ms
Retry Behavior.Retries On 504 5093ms
Retry Behavior.Max Retries Respected 15099ms
Deduplication.Generates Unique Uuids 102ms
Deduplication.Preserves Uuid On Retry 5024ms
Deduplication.Preserves Uuid And Timestamp On Retry 10029ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry 5035ms
Deduplication.No Duplicate Events In Batch 25ms
Deduplication.Different Events Have Different Uuids 23ms
Compression.Sends Gzip When Enabled 22ms
Batch Format.Uses Proper Batch Structure 22ms
Batch Format.Flush With No Events Sends Nothing 25ms
Batch Format.Multiple Events Batched Together 103ms
Error Handling.Does Not Retry On 403 2069ms
Error Handling.Does Not Retry On 413 2069ms
Error Handling.Retries On 408 5086ms

Feature_Flags Tests

⚠️ 2/16 tests passed, 14 failed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id 47ms
Request Payload.Flags Request Uses V2 Query Param 51ms
Request Payload.Flags Request Hits Flags Path Not Decide 28ms
Request Payload.Flags Request Omits Authorization Header 32ms
Request Payload.Token In Flags Body Matches Init 27ms
Request Payload.Groups Round Trip 37ms
Request Payload.Groups Default To Empty Object 44ms
Request Payload.Person Properties Distinct Id Auto Populated When Caller Omits It 22ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False 31ms
Request Payload.Disable Geoip Omitted Defaults To False 31ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key 40ms
Request Lifecycle.No Flags Request On Init Alone 19ms
Request Lifecycle.No Flags Request On Normal Capture 45ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests 36ms
Request Lifecycle.Mock Response Value Is Returned To Caller 34ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event 19ms

Failures

request_payload.request_with_person_properties_device_id

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-962b6223-9615-4f88-9a76-bc5349d05843'

request_payload.flags_request_uses_v2_query_param

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-e598ee15-9252-48ab-8959-ceeb3e57e225'

request_payload.flags_request_hits_flags_path_not_decide

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-dc5c0287-42a6-4472-a0d0-1b702c3fc616'

request_payload.flags_request_omits_authorization_header

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-d83f95de-d435-45f2-a67d-d3ee7d75a2e5'

request_payload.token_in_flags_body_matches_init

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-4c832ae1-1608-405a-a269-1e285e419045'

request_payload.groups_round_trip

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-af7b9ccb-6a37-479e-ac87-d5b44518c50d'

request_payload.groups_default_to_empty_object

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-b9c451c1-9d54-4890-a332-efcc7076c092'

request_payload.person_properties_distinct_id_auto_populated_when_caller_omits_it

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-cc338808-5f59-4113-b895-dbb54b0b6528'

request_payload.disable_geoip_false_propagates_as_geoip_disable_false

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-755ac7e9-b91c-4207-9397-a2c26870d676'

request_payload.disable_geoip_omitted_defaults_to_false

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-9b7b1f84-ff37-4eea-b0e5-8b4ff321adc5'

request_payload.flag_keys_to_evaluate_contains_only_requested_key

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-7e8ac85d-21d8-4b86-9147-c589b96d176a'

request_lifecycle.two_flag_calls_produce_two_remote_requests

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-d01b09f7-389e-42b7-8611-9bcbf86b3a77'

request_lifecycle.mock_response_value_is_returned_to_caller

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-da902d0a-47d3-4fa8-80d3-f9be7ac4e500'

side_effect_events.get_feature_flag_captures_feature_flag_called_event

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-bbb25e6b-16e2-4e12-9f60-91e5e6b1d1f9'

@cat-ph cat-ph left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ooh, good point, thanks! 💙

@eli-r-ph eli-r-ph marked this pull request as ready for review June 24, 2026 18:40
@eli-r-ph eli-r-ph requested a review from a team as a code owner June 24, 2026 18:40
@eli-r-ph eli-r-ph merged commit e447b36 into main Jun 24, 2026
25 checks passed
@eli-r-ph eli-r-ph deleted the fix/in-flight-counter-underflow branch June 24, 2026 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants