fix: saturate transport in-flight counter to prevent underflow wedge#155
Conversation
The in-flight `len` counter is decremented from several paths (before_send drops, partial v1 batch results, terminal outcomes, shutdown drops, channel drain). A double-decrement on any of them would underflow the AtomicUsize via plain fetch_sub and wrap to a huge value, making the bounded queue look permanently full and silently dropping every later event. Route all production decrements through dec_len and have it saturate at 0 (fetch_update + saturating_sub), with a debug_assert to surface such a bug in tests. Release builds now degrade gracefully instead of wedging the queue.
|
Reviews (1): Last reviewed commit: "fix: saturate transport in-flight counte..." | Re-trigger Greptile |
posthog-rs-v1 Compliance ReportDate: 2026-06-24 18:37:34 UTC
|
| Test | Status | Duration |
|---|---|---|
| Endpoint And Method.Targets V1 Endpoint | ✅ | 147ms |
| Endpoint And Method.Does Not Use Legacy Endpoints | ✅ | 147ms |
| Required Headers.Has Authorization Bearer Header | ✅ | 152ms |
| Required Headers.Has Content Type Json | ✅ | 146ms |
| Required Headers.Has Posthog Sdk Info Format | ✅ | 146ms |
| Required Headers.Has Posthog Attempt Header | ✅ | 147ms |
| Required Headers.Has Posthog Request Id | ✅ | 146ms |
| Required Headers.Has Posthog Request Timestamp | ✅ | 145ms |
| Required Headers.Has User Agent | ✅ | 145ms |
| Body Format.Body Has Created At And Batch | ✅ | 144ms |
| Body Format.No Api Key In Body | ✅ | 139ms |
| Body Format.No Sent At In Body | ✅ | 140ms |
| Event Format.Event Has Required Root Fields | ✅ | 140ms |
| Event Format.Event Uuid Is Valid | ✅ | 138ms |
| Event Format.Event Timestamp Is Rfc3339 | ✅ | 138ms |
| Event Format.Distinct Id Is String | ✅ | 142ms |
| Event Format.Distinct Id At Root Not Properties | ✅ | 140ms |
| Event Format.Custom Properties Preserved | ✅ | 138ms |
| Event Format.Set Properties Preserved | ✅ | 139ms |
| Event Format.Set Once Properties Preserved | ✅ | 134ms |
| Event Format.Groups Properties Preserved | ✅ | 134ms |
| Event Format.Sdk Generates Uuid If Not Provided | ✅ | 134ms |
| Event Format.Event Has Required Root Fields Batch | ✅ | 159ms |
| Event Format.Event Uuid Is Valid Batch | ✅ | 167ms |
| Event Format.Event Timestamp Is Rfc3339 Batch | ✅ | 178ms |
| Event Format.Distinct Id Is String Batch | ✅ | 167ms |
| Event Format.Distinct Id At Root Not Properties Batch | ✅ | 158ms |
| Event Format.Custom Properties Preserved Batch | ✅ | 178ms |
| Event Format.Set Properties Preserved Batch | ✅ | 177ms |
| Event Format.Set Once Properties Preserved Batch | ✅ | 176ms |
| Event Format.Groups Properties Preserved Batch | ✅ | 139ms |
| Event Format.Sdk Generates Uuid If Not Provided Batch | ✅ | 146ms |
| Batch Behavior.Multiple Events In Single Batch | ✅ | 189ms |
| Batch Behavior.Batch Envelope Smoke | ✅ | 190ms |
| Batch Behavior.Flush With No Events Sends Nothing | ✅ | 80ms |
| Batch Behavior.Flush At Triggers Batch | ✅ | 1128ms |
| Batch Behavior.Created At Reflects Batch Creation Time | ✅ | 118ms |
| Deduplication.Generates Unique Uuids | ✅ | 167ms |
| Deduplication.Different Events Same Content Different Uuids | ✅ | 145ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 5120ms |
| Deduplication.Preserves Timestamp On Retry | ✅ | 5076ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 5093ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 134ms |
| Header Behavior On Retry.Attempt Header Starts At One | ✅ | 64ms |
| Header Behavior On Retry.Attempt Header Increments On Retry | ✅ | 10089ms |
| Header Behavior On Retry.Request Id Preserved On Retry | ✅ | 5071ms |
| Header Behavior On Retry.Different Requests Have Different Request Ids | ✅ | 2074ms |
| Header Behavior On Retry.Request Timestamp Changes On Retry | ✅ | 5068ms |
| Response Format Validation.Success Response Has Uuid Keyed Results | ✅ | 59ms |
| Response Format Validation.Success Response Has Ok For Each Event | ✅ | 34ms |
| Response Format Validation.Success No Retry After When All Ok | ✅ | 37ms |
| Response Format Validation.Success Retry After Present When Retry Events | ✅ | 29ms |
| Response Format Validation.Success No Retry After When Drop Only | ✅ | 32ms |
| Response Format Validation.Response Echoes Request Id | ✅ | 29ms |
| Retry Behavior.Retries On 408 | ✅ | 5028ms |
| Retry Behavior.Retries On 500 | ✅ | 5023ms |
| Retry Behavior.Retries On 503 | ✅ | 5025ms |
| Retry Behavior.Retries On 504 | ✅ | 5022ms |
| Retry Behavior.Retryable Errors Have Retry After | ✅ | 2027ms |
| Retry Behavior.Respects Retry After On Retryable Error | ✅ | 8023ms |
| Retry Behavior.Does Not Retry On 400 | ✅ | 2037ms |
| Retry Behavior.Does Not Retry On 401 | ✅ | 2027ms |
| Retry Behavior.Does Not Retry On 402 | ✅ | 2035ms |
| Retry Behavior.Does Not Retry On 413 | ✅ | 2026ms |
| Retry Behavior.Does Not Retry On 415 | ✅ | 2024ms |
| Retry Behavior.Non Retryable Errors Have No Retry After | ✅ | 2024ms |
| Retry Behavior.Implements Backoff | ✅ | 15032ms |
| Retry Behavior.Max Retries Respected | ✅ | 15024ms |
| Partial Batch Handling.Handles 200 Full Success | ✅ | 2028ms |
| Partial Batch Handling.Handles 200 With All Ok | ✅ | 3043ms |
| Partial Batch Handling.Does Not Retry Dropped Events | ✅ | 3042ms |
| Partial Batch Handling.Does Not Retry Limited Events | ✅ | 3027ms |
| Partial Batch Handling.Prunes Ok Events On Partial Retry | ✅ | 5027ms |
| Partial Batch Handling.Prunes Dropped Events On Partial Retry | ✅ | 5027ms |
| Partial Batch Handling.Retries Only Retry Events From Partial | ✅ | 5026ms |
| Partial Batch Handling.Partial Retry Preserves Uuids | ✅ | 5025ms |
| Partial Batch Handling.Partial Retry Attempt Header Increments | ✅ | 5029ms |
| Partial Batch Handling.Partial Retry Request Id Preserved | ✅ | 5030ms |
| Partial Batch Handling.Respects Retry After On Partial | ✅ | 5024ms |
| Partial Batch Handling.Unknown Result Treated As Terminal | ✅ | 3027ms |
| Partial Batch Handling.Mixed Ok Drop Limited No Retry | ✅ | 3026ms |
| Compression.Sends Gzip Content Encoding | ✅ | 22ms |
| Compression.No Content Encoding When Disabled | ✅ | 21ms |
| Compression.Compressed Body Is Decompressible | ✅ | 21ms |
| Error Handling.Does Not Retry On Unknown 4Xx | ✅ | 2024ms |
| Event Options.Cookieless Mode Override | ✅ | 23ms |
| Event Options.Disable Skew Correction Override | ✅ | 22ms |
| Event Options.Process Person Profile Override | ✅ | 22ms |
| Event Options.Product Tour Id Override | ✅ | 22ms |
| Event Options.Unset Options Omitted | ✅ | 21ms |
| Event Options.Options Override In Batch | ✅ | 23ms |
| Geoip And Historical Migration.Geoip Disable Injected Into Properties | ✅ | 22ms |
| Geoip And Historical Migration.Historical Migration Set In Body | ✅ | 22ms |
| Geoip And Historical Migration.Historical Migration Absent By Default | ✅ | 22ms |
Feature_Flags Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ❌ | 12ms |
| Request Payload.Flags Request Uses V2 Query Param | ❌ | 16ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ❌ | 15ms |
| Request Payload.Flags Request Omits Authorization Header | ❌ | 15ms |
| Request Payload.Token In Flags Body Matches Init | ❌ | 15ms |
| Request Payload.Groups Round Trip | ❌ | 14ms |
| Request Payload.Groups Default To Empty Object | ❌ | 15ms |
| Request Payload.Person Properties Distinct Id Auto Populated When Caller Omits It | ❌ | 15ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ❌ | 15ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ❌ | 15ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ❌ | 14ms |
| Request Lifecycle.No Flags Request On Init Alone | ✅ | 12ms |
| Request Lifecycle.No Flags Request On Normal Capture | ✅ | 23ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ❌ | 12ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ❌ | 14ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ❌ | 15ms |
Failures
request_payload.request_with_person_properties_device_id
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-18792331-c448-4e87-8021-0f9dc24b914a'
request_payload.flags_request_uses_v2_query_param
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-1ca4d46f-16d5-4893-bae4-b62e2dc46a49'
request_payload.flags_request_hits_flags_path_not_decide
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-2c93b787-5468-4715-b11b-04fd21bbdf02'
request_payload.flags_request_omits_authorization_header
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-15a9cd3b-bd31-475f-9a45-b09adfddb7c9'
request_payload.token_in_flags_body_matches_init
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-c6ce700f-1305-4fad-993f-226f0fee61c3'
request_payload.groups_round_trip
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-67b56ac8-b926-4b57-8fc1-c5aca0534a99'
request_payload.groups_default_to_empty_object
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-a5a192c0-b813-4fef-888c-656e2c059f9e'
request_payload.person_properties_distinct_id_auto_populated_when_caller_omits_it
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-8c74f023-025f-46b6-aad9-db35ed20dc49'
request_payload.disable_geoip_false_propagates_as_geoip_disable_false
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-bdbdafb3-dd7e-4b61-bbf5-40054887efb7'
request_payload.disable_geoip_omitted_defaults_to_false
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-032ece5c-c76b-4352-a87e-ea31cd02f395'
request_payload.flag_keys_to_evaluate_contains_only_requested_key
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-3e4e4963-cec6-465f-a54d-1e2ecb3db97d'
request_lifecycle.two_flag_calls_produce_two_remote_requests
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-41a675fa-4bc9-4c17-a1d2-fe5ea073a8d5'
request_lifecycle.mock_response_value_is_returned_to_caller
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-d0ae8559-35c6-43c9-85dd-93f0f1d7d2a3'
side_effect_events.get_feature_flag_captures_feature_flag_called_event
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-9f56e0a1-6504-4099-aa6a-6d03d881c84f'
posthog-rs-v0 Compliance ReportDate: 2026-06-24 18:40:03 UTC
|
| Test | Status | Duration |
|---|---|---|
| Format Validation.Event Has Required Fields | ✅ | 156ms |
| Format Validation.Event Has Uuid | ✅ | 162ms |
| Format Validation.Event Has Lib Properties | ✅ | 163ms |
| Format Validation.Distinct Id Is String | ✅ | 157ms |
| Format Validation.Token Is Present | ✅ | 161ms |
| Format Validation.Custom Properties Preserved | ✅ | 160ms |
| Format Validation.Event Has Timestamp | ✅ | 162ms |
| Retry Behavior.Retries On 503 | ✅ | 5165ms |
| Retry Behavior.Does Not Retry On 400 | ✅ | 2165ms |
| Retry Behavior.Does Not Retry On 401 | ✅ | 2161ms |
| Retry Behavior.Respects Retry After Header | ✅ | 5103ms |
| Retry Behavior.Implements Backoff | ✅ | 15102ms |
| Retry Behavior.Retries On 500 | ✅ | 5094ms |
| Retry Behavior.Retries On 502 | ✅ | 5098ms |
| Retry Behavior.Retries On 504 | ✅ | 5093ms |
| Retry Behavior.Max Retries Respected | ✅ | 15099ms |
| Deduplication.Generates Unique Uuids | ✅ | 102ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 5024ms |
| Deduplication.Preserves Uuid And Timestamp On Retry | ✅ | 10029ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 5035ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 25ms |
| Deduplication.Different Events Have Different Uuids | ✅ | 23ms |
| Compression.Sends Gzip When Enabled | ✅ | 22ms |
| Batch Format.Uses Proper Batch Structure | ✅ | 22ms |
| Batch Format.Flush With No Events Sends Nothing | ✅ | 25ms |
| Batch Format.Multiple Events Batched Together | ✅ | 103ms |
| Error Handling.Does Not Retry On 403 | ✅ | 2069ms |
| Error Handling.Does Not Retry On 413 | ✅ | 2069ms |
| Error Handling.Retries On 408 | ✅ | 5086ms |
Feature_Flags Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ❌ | 47ms |
| Request Payload.Flags Request Uses V2 Query Param | ❌ | 51ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ❌ | 28ms |
| Request Payload.Flags Request Omits Authorization Header | ❌ | 32ms |
| Request Payload.Token In Flags Body Matches Init | ❌ | 27ms |
| Request Payload.Groups Round Trip | ❌ | 37ms |
| Request Payload.Groups Default To Empty Object | ❌ | 44ms |
| Request Payload.Person Properties Distinct Id Auto Populated When Caller Omits It | ❌ | 22ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ❌ | 31ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ❌ | 31ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ❌ | 40ms |
| Request Lifecycle.No Flags Request On Init Alone | ✅ | 19ms |
| Request Lifecycle.No Flags Request On Normal Capture | ✅ | 45ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ❌ | 36ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ❌ | 34ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ❌ | 19ms |
Failures
request_payload.request_with_person_properties_device_id
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-962b6223-9615-4f88-9a76-bc5349d05843'
request_payload.flags_request_uses_v2_query_param
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-e598ee15-9252-48ab-8959-ceeb3e57e225'
request_payload.flags_request_hits_flags_path_not_decide
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-dc5c0287-42a6-4472-a0d0-1b702c3fc616'
request_payload.flags_request_omits_authorization_header
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-d83f95de-d435-45f2-a67d-d3ee7d75a2e5'
request_payload.token_in_flags_body_matches_init
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-4c832ae1-1608-405a-a269-1e285e419045'
request_payload.groups_round_trip
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-af7b9ccb-6a37-479e-ac87-d5b44518c50d'
request_payload.groups_default_to_empty_object
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-b9c451c1-9d54-4890-a332-efcc7076c092'
request_payload.person_properties_distinct_id_auto_populated_when_caller_omits_it
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-cc338808-5f59-4113-b895-dbb54b0b6528'
request_payload.disable_geoip_false_propagates_as_geoip_disable_false
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-755ac7e9-b91c-4207-9397-a2c26870d676'
request_payload.disable_geoip_omitted_defaults_to_false
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-9b7b1f84-ff37-4eea-b0e5-8b4ff321adc5'
request_payload.flag_keys_to_evaluate_contains_only_requested_key
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-7e8ac85d-21d8-4b86-9147-c589b96d176a'
request_lifecycle.two_flag_calls_produce_two_remote_requests
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-d01b09f7-389e-42b7-8611-9bcbf86b3a77'
request_lifecycle.mock_response_value_is_returned_to_caller
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-da902d0a-47d3-4fa8-80d3-f9be7ac4e500'
side_effect_events.get_feature_flag_captures_feature_flag_called_event
404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag?test_id=t-bbb25e6b-16e2-4e12-9f60-91e5e6b1d1f9'
cat-ph
left a comment
There was a problem hiding this comment.
ooh, good point, thanks! 💙
The in-flight event counter is a shared
AtomicUsizedecremented from many paths, and that accounting is tricky. This hardens it against a double-decrement bug underflowing the counter.💡 Why
try_reservethen sees the queue as permanently full.before_senddrops, partial v1 results, terminal sends, shutdown drops, channel drain.🔨 What
fetch_sub.fetch_update+saturating_subindec_len.debug_assertcatches a real underflow bug in tests.dec_lentoo.💚 How tested
fmt+clippy -D warningsclean (default +capture-v1).transport::tests pass across async-v1, blocking-v1, v0.dec_lento 0 with the assert live.--release(assert panics in debug), so it would not run in CI.📝 Checklist
If releasing new changes
sampo addto generate a changeset file