A note for the community
We are seeing data loss in production when using Vector's Elasticsearch sink. The root cause is that es_rejected_execution_exception — a transient, recoverable error indicating the ES write thread pool is temporarily full — is treated as non-retriable and causes entire batches to be dropped immediately.
We have configured retry_attempts = 80 and expect retries for temporary failures, but Vector's retry logic only looks at HTTP status codes. Since ES returns HTTP 200 for _bulk requests even when individual documents fail, Vector never retries.
Other established tools like Logstash and Flink properly retry these errors. This makes Vector's behavior a critical issue for production use cases where data durability is non-negotiable. We are seeking confirmation on whether this is intentional, and if there are any workarounds or plans to improve it.
Problem
Vector Version: v0.56.0
Configuration (minimal example):
toml
[sinks.my_elasticsearch_sink]
type = "elasticsearch"
inputs = ["my_source"]
endpoint = "http://my-es:9200"
index = "my-index-%Y-%m-%d"
[sinks.my_elasticsearch_sink.request]
retry_attempts = 5
retry_backoff_secs = 5
Problem Summary
We are using Vector as the log shipping component to write data into Elasticsearch. When the Elasticsearch cluster experiences occasional load spikes, we observe two types of rejection errors:
circuit_breaking_exception — JVM memory pressure exceeds the breaker limit
es_rejected_execution_exception — the Elasticsearch write thread pool is full (queue capacity reached)
Our understanding is that both of these are transient, recoverable conditions. A brief backoff and retry should allow the Elasticsearch node to recover and accept the request.
However, Vector's Elasticsearch sink does not retry es_rejected_execution_exception at all. Instead, it logs:
text
ERROR sink{...}: vector::sinks::util::retries: Not retriable; dropping the request.
reason="error type: es_rejected_execution_exception, reason: rejected execution of ..."
And then:
text
ERROR sink{...}: vector_common::internal_event::component_events_dropped: Events dropped
intentional=false count=1000
1000 events are dropped immediately, with zero retry attempts.
Evidence from Vector Code/Documentation
According to Vector's documented retry policy:
Vector will retry failed requests (status == 429, >= 500, and != 501). Other responses will not be retried.
The es_rejected_execution_exception error is returned by Elasticsearch's _bulk API with HTTP status 200 (because the bulk request itself succeeded, but some individual documents failed). Since the HTTP status is 200, Vector's retry logic does not trigger, and the error is treated as "non-retriable".
Additional context:
Our Elasticsearch cluster is generally stable. The issue occurs only when individual nodes experience occasional write rejections during brief load spikes — after a short period, the nodes recover and become healthy again. In these scenarios, we need the data to be retried and successfully delivered, rather than being dropped permanently.
Our current Vector Elasticsearch sink configuration:
toml
batch.max_events = 1000
batch.max_bytes = 5242880
batch.timeout_secs = 3
request.retry_attempts = 80
request.retry_initial_backoff_secs = 5
request.retry_max_duration_secs = 300
request.timeout_secs = 60
request.retry_partial = false
request.concurrency = "adaptive"
request.adaptive_concurrency.max_concurrency_limit = 10
request.adaptive_concurrency.decrease_ratio = 0.7
request.rate_limit_duration_secs = 1
request.rate_limit_num = 50
healthcheck.enabled = false
[sinks.my_elasticsearch_sink.buffer]
type = "memory"
max_events = 100000
when_full = "block"
We would like to ask:
In Vector v0.56.1, when writing to Elasticsearch, are es_rejected_execution_exception and circuit_breaking_exception retryable? Or is there any alternative approach (e.g., disk buffer, dead-letter queue, or custom retry logic) to guarantee no data loss for these transient errors?
If both types of errors are intentionally designed to be dropped without retry, does that mean Vector's Elasticsearch sink is essentially unsuitable for production use cases where data durability is critical? Are there any plans to improve this behavior in future releases?
Comparison with Other Components
Logstash (Elastic's own log shipping tool) does retry es_rejected_execution_exception. The Logstash Elasticsearch output plugin will retry events that are rejected due to transient queue full conditions. It only treats certain specific errors (like 409 conflict) as non-retryable.
Apache Flink also provides a RetryRejectedExecutionFailureHandler that specifically re-adds requests that failed due to EsRejectedExecutionException.
Many other data pipeline components treat es_rejected_execution_exception as a temporary, retryable error.
Please help to reply, thanks for your answer!
Configuration
Version
0.56.0
Debug Output
Example Data
No response
Additional Context
No response
References
No response
A note for the community
We are seeing data loss in production when using Vector's Elasticsearch sink. The root cause is that es_rejected_execution_exception — a transient, recoverable error indicating the ES write thread pool is temporarily full — is treated as non-retriable and causes entire batches to be dropped immediately.
We have configured retry_attempts = 80 and expect retries for temporary failures, but Vector's retry logic only looks at HTTP status codes. Since ES returns HTTP 200 for _bulk requests even when individual documents fail, Vector never retries.
Other established tools like Logstash and Flink properly retry these errors. This makes Vector's behavior a critical issue for production use cases where data durability is non-negotiable. We are seeking confirmation on whether this is intentional, and if there are any workarounds or plans to improve it.
Problem
Vector Version: v0.56.0
Configuration (minimal example):
toml
[sinks.my_elasticsearch_sink]
type = "elasticsearch"
inputs = ["my_source"]
endpoint = "http://my-es:9200"
index = "my-index-%Y-%m-%d"
[sinks.my_elasticsearch_sink.request]
retry_attempts = 5
retry_backoff_secs = 5
Problem Summary
We are using Vector as the log shipping component to write data into Elasticsearch. When the Elasticsearch cluster experiences occasional load spikes, we observe two types of rejection errors:
circuit_breaking_exception — JVM memory pressure exceeds the breaker limit
es_rejected_execution_exception — the Elasticsearch write thread pool is full (queue capacity reached)
Our understanding is that both of these are transient, recoverable conditions. A brief backoff and retry should allow the Elasticsearch node to recover and accept the request.
However, Vector's Elasticsearch sink does not retry es_rejected_execution_exception at all. Instead, it logs:
text
ERROR sink{...}: vector::sinks::util::retries: Not retriable; dropping the request.
reason="error type: es_rejected_execution_exception, reason: rejected execution of ..."
And then:
text
ERROR sink{...}: vector_common::internal_event::component_events_dropped: Events dropped
intentional=false count=1000
1000 events are dropped immediately, with zero retry attempts.
Evidence from Vector Code/Documentation
According to Vector's documented retry policy:
Vector will retry failed requests (status == 429, >= 500, and != 501). Other responses will not be retried.
The es_rejected_execution_exception error is returned by Elasticsearch's _bulk API with HTTP status 200 (because the bulk request itself succeeded, but some individual documents failed). Since the HTTP status is 200, Vector's retry logic does not trigger, and the error is treated as "non-retriable".
Additional context:
Our Elasticsearch cluster is generally stable. The issue occurs only when individual nodes experience occasional write rejections during brief load spikes — after a short period, the nodes recover and become healthy again. In these scenarios, we need the data to be retried and successfully delivered, rather than being dropped permanently.
Our current Vector Elasticsearch sink configuration:
toml
batch.max_events = 1000
batch.max_bytes = 5242880
batch.timeout_secs = 3
request.retry_attempts = 80
request.retry_initial_backoff_secs = 5
request.retry_max_duration_secs = 300
request.timeout_secs = 60
request.retry_partial = false
request.concurrency = "adaptive"
request.adaptive_concurrency.max_concurrency_limit = 10
request.adaptive_concurrency.decrease_ratio = 0.7
request.rate_limit_duration_secs = 1
request.rate_limit_num = 50
healthcheck.enabled = false
[sinks.my_elasticsearch_sink.buffer]
type = "memory"
max_events = 100000
when_full = "block"
We would like to ask:
In Vector v0.56.1, when writing to Elasticsearch, are es_rejected_execution_exception and circuit_breaking_exception retryable? Or is there any alternative approach (e.g., disk buffer, dead-letter queue, or custom retry logic) to guarantee no data loss for these transient errors?
If both types of errors are intentionally designed to be dropped without retry, does that mean Vector's Elasticsearch sink is essentially unsuitable for production use cases where data durability is critical? Are there any plans to improve this behavior in future releases?
Comparison with Other Components
Logstash (Elastic's own log shipping tool) does retry es_rejected_execution_exception. The Logstash Elasticsearch output plugin will retry events that are rejected due to transient queue full conditions. It only treats certain specific errors (like 409 conflict) as non-retryable.
Apache Flink also provides a RetryRejectedExecutionFailureHandler that specifically re-adds requests that failed due to EsRejectedExecutionException.
Many other data pipeline components treat es_rejected_execution_exception as a temporary, retryable error.
Please help to reply, thanks for your answer!
Configuration
Version
0.56.0
Debug Output
Example Data
No response
Additional Context
No response
References
No response