Vector internal_logs incorrect behavior or flooding

[Zava2012]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

We are running Vector in Kubernetes.
Starting with version 0.51.1 where internal logs rate limiting was disabled, I've started getting the following errors with huge amount of suppressed events (and the same thousands of logs from internal_logs are going to sinks):

2026-07-17T20:38:31.997351Z ERROR transform{component_kind="transform" component_id=routes component_type=route}: vector::internal_events::conditions: Internal log [VRL condition execution failed.] has been suppressed 177029 times.
2026-07-17T20:38:31.997375Z ERROR transform{component_kind="transform" component_id=routes component_type=route}: vector::internal_events::conditions: VRL condition execution failed. error=function call error for "starts_with" at (0:57): function call error for "string" at (12:41): expected string, got null error_type="script_failed" stage="processing"
2026-07-17T20:38:31.997396Z ERROR transform{component_kind="transform" component_id=routes component_type=route}: vector::internal_events::conditions: Internal log [VRL condition execution failed.] is being suppressed to avoid flooding.

The Descheduler component generates only 5 log lines per 15 seconds that are writing to sinks and I can see the same logs in pod logs and in Loki. I cannot imagine where these 177029 times are coming from.
Redis had less than 1 thousand logs during this testing.

The issue is not related to the particular Descheduler component. The same behavior happens for any application with the same or similar route condition with functions.

The debug logs do not have any useful information for this case.

Now I'm using version 0.56.0 and the behavior is the same. The last normally working version is 0.51.0.

Can someone explain what is going on?

Configuration

sources:
  redis:
    type: redis
    url: redis://:${REDIS_PASSWORD}@redis:6379/0
    key: vector
    decoding:
      codec: json
  vector_logs:
    type: internal_logs
transforms:
  routes:
    type: route
    inputs:
      - redis
      - vector_logs
    route:
      descheduler: starts_with(string!(.kubernetes.pod_name), "descheduler")
      vector: .source_type == "internal_logs"
  parse_klog:
    type: remap
    inputs:
      - routes.descheduler
    source: |
      .message_full = .message
      structured = parse_klog(.message) ?? parse_regex!(.message, r'(?P<message>(?s).*)')
      . = merge(., structured)
  parse_vector:
    type: remap
    inputs:
      - routes.vector
    source: |
      ...
sinks:
  loki:
    type: loki
    inputs:
      - parse_*
    ...

Version

0.56.0

Debug Output

Example Data

I0717 20:48:11.659722       1 topologyspreadconstraint.go:135] "Processing namespaces for topology spread constraints" plugin="RemovePodsViolatingTopologySpreadConstraint" ExtensionPoint="Balance"
I0717 20:48:11.659999       1 profile.go:378] "Total number of evictions/requests" extension point="Balance" evictedPods=0 evictionRequests=0
I0717 20:48:11.660247       1 topologyspreadconstraint.go:135] "Processing namespaces for topology spread constraints" plugin="RemovePodsViolatingTopologySpreadConstraint" ExtensionPoint="Balance"
I0717 20:48:11.660605       1 profile.go:378] "Total number of evictions/requests" extension point="Balance" evictedPods=0 evictionRequests=0
I0717 20:48:11.660626       1 descheduler.go:404] "Number of evictions/requests" totalEvicted=0 evictionRequests=0

Additional Context

No response

References

No response