Fluentbit Loki output Tenant_Id_Key routing not working properly.

[shivamvishwakarma27]

Bug Report

Describe the bug

Fluentbit version: 5.0.3
EKS version: 1.30
Fluentbit deployment mode: Daemonset

I am performing POC to replace the promtail by fluentbit for Loki stack. In my current setup, I am using multi-cluster with multi tenant - using dynamic tenancy in Promtail. While performing POC with promtail, the resource utilisation is very less at scale and very impressive but get blocked at tenancy behaviour. I see there is option to set Tenant_Id_Key to support multi tenant dynamic mode but observed It's intermittently adding different tenant log in different tenant causes inconsistent loglines for any tenant.

To Reproduce

• Rubular link if applicable:
• I am using below config for Log pipeline from ref https://docs.fluentbit.io/manual/data-pipeline/outputs/loki .

  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Tag kube.*
        Mem_Buf_Limit 25MB
        Skip_Long_Lines Off
        multiline.parser  docker, cri
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [FILTER]
        Name parser
        Match kube.*
        Key_Name log
        Parser json
        Reserve_Data On

    [FILTER]
        Name lua
        Match kube.*
        script /fluent-bit/etc/conf/log_pipeline.lua
        call enrich_log

    # Group records by namespace before Loki output flush.
    [FILTER]
        Name rewrite_tag
        Match kube.*
        Rule $tenant_id ^(.+)$ loki.$1 false
        Emitter_Name re_emitted

  ## https://docs.fluentbit.io/manual/pipeline/outputs
  outputs: |
    [OUTPUT]
        Name Loki
        Match loki.*
        Host loki-distributor.observability.svc.cluster.local
        Uri /loki/api/v1/push
        Port 3100
        Line_Format key_value
        Drop_Single_Key raw
        Tenant_Id_Key tenant_id
        workers 0
        Retry_Limit False
        Labels agent=fluentbit,cluster_name=qa,deployed_on=k8s
        Label_Map_Path /fluent-bit/etc/conf/loki_label_map.json
        Remove_keys kubernetes,app,level,tenant_id

      Lua function for tenant_id label
    
     function enrich_log(tag, timestamp, record)
        local k8s = record["kubernetes"] or {}
        local out = {}
        out["kubernetes"] = k8s
        out["tenant_id"] = k8s["namespace_name"] or "default"

• Example log message if applicable:

2026-05-19 17:07:38.828 | I0519 11:37:38.828040       1 topology.go:175] "Skipping topology aware endpoint filtering since one or more endpoints is missing a zone hint" endpoint="172.16.82.162:3000" 

Fields
  | I | __tenant_id__ | revenue
  | I | agent | fluentbit
  | I | app | unknown
  | I | cluster_name | qa
  | I | container_name | kube-proxy
  | I | deployed_on | k8s
  | S | detected_level | info
  | I | level | info
  | I | namespace | kube-system
  | I | pod | kube-proxy-sqdfb
  | I | service_name | unknown

• Steps to reproduce the problem:
  1: Deploy the Fluentbit as Daemonset mode on multiple EKS nodes with tail input.
  2: Use namespace for Tenant_Id_Key for Loki output plugin.
  3: Crearte datasource for a single tennat datasource on grafana and list the namespaces, even we have specified a single namespace tenant still the intermittent data is getting pushed in wrong tenant.

Expected behavior

• Each tenantkey should contain logs of the specific namespace services only.
  Screenshots

Your Environment

• Version used: 5.0.3
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?): Env: QA, EKS: 1.30
• Server type and version: Loki 3.4.2
• Operating System and version: Linux 1.30 compatible with EKS
• Filters and plugins: Tail input with Loki output pluging

Additional context

• We are trying to replace promtail with fluentbit because of it's lightweight low resource usage and high performance. But blocked because of Tenant_Id_Key tenant_id behaviour is not working.

NA

[patrick-stephens]

May be related to #8809 (comment)

[shivamvishwakarma27]

Yes @patrick-stephens It's similar issue, but that bug is marked as Not planned. Can you please confirm if the bug is fixed or any alternative workaround you know. Please let us know, we are having blocker due to this. It will be great if can get some help on it. Thanks