Skip to content

feat(fluentbit): Support network relevant settings for s3 output#1955

Open
smallc2009 wants to merge 1 commit into
fluent:masterfrom
smallc2009:feature/support-options-for-s3-output
Open

feat(fluentbit): Support network relevant settings for s3 output#1955
smallc2009 wants to merge 1 commit into
fluent:masterfrom
smallc2009:feature/support-options-for-s3-output

Conversation

@smallc2009
Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

This pull request adds a number of advanced network and connection management options to the S3 output plugin for Fluent Bit, enhancing its configurability and robustness.

Enhancements to S3 Plugin Networking and Connection Handling:

New S3 Plugin Fields and Parameter Mapping:

  • Added fields to the S3 struct in s3_types.go for fine-grained network and connection control (e.g., ConnectTimeout, ConnectTimeoutLogError, DNSMode, DNSPreferIPv4, DNSPreferIPv6, IoTimeout, KeepaliveMaxRecycle, MaxWorkerConnections, ProxyEnvIgnore, SourceAddress, Keepalive, TCPKeepaliveInterval, TCPKeepaliveProbes, TCPKeepaliveTime, KeepaliveIdleTimeout). These fields are now mapped to their corresponding Fluent Bit configuration parameters in the Params method.

CRD and Helm Chart Updates:

  • Updated the CRD YAMLs and Helm chart templates to expose the new S3 networking and connection fields, including validation and documentation for each option. This allows users to configure these parameters via Kubernetes manifests.

Test Coverage:

  • Extended the S3 output plugin test (s3_types_test.go) to cover all new fields, ensuring correct parameter insertion and behavior.

Which issue(s) this PR fixes:

Fixes #1930

Does this PR introduced a user-facing change?


Additional documentation, usage docs, etc.:


Copilot AI review requested due to automatic review settings May 18, 2026 07:30
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Fluent Bit networking/connection-management options (timeouts, DNS preferences, keepalive, TCP keepalive, source address, proxy env handling, worker connections) to the S3 output plugin, exposing them via the CRDs and Helm charts and adding test coverage.

Changes:

  • Add ~15 new networking fields to the S3 struct and emit them as net.* params in S3.Params.
  • Regenerate the affected CRD YAMLs, Helm chart templates, deepcopy, and API docs.
  • Extend TestOutput_S3_Params to cover the new fields.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
apis/fluentbit/v1alpha2/plugins/output/s3_types.go Adds new networking fields and maps them to net.* params
apis/fluentbit/v1alpha2/plugins/output/s3_types_test.go Tests new fields produce the expected net.* KVs
apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go Generated deepcopy for new S3 pointer fields
config/crd/bases/fluentbit.fluent.io_outputs.yaml Regenerated CRD with new S3 fields
config/crd/bases/fluentbit.fluent.io_clusteroutputs.yaml Regenerated CRD with new S3 fields
charts/fluent-operator/crds/fluentbit.fluent.io_outputs.yaml Helm CRD chart updated
charts/fluent-operator/crds/fluentbit.fluent.io_clusteroutputs.yaml Helm CRD chart updated
charts/fluent-operator-fluent-bit-crds/templates/fluentbit.fluent.io_outputs.yaml Helm CRD chart updated
charts/fluent-operator-fluent-bit-crds/templates/fluentbit.fluent.io_clusteroutputs.yaml Helm CRD chart updated
manifests/setup/setup.yaml Generated manifest updated
manifests/setup/fluent-operator-crd.yaml Generated manifest updated
docs/plugins/fluentbit/output/s3.md Doc lists new fields
Files not reviewed (1)
  • apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Language not supported
Comments suppressed due to low confidence (1)

apis/fluentbit/v1alpha2/plugins/output/s3_types.go:150

  • Keepalive is documented as a boolean on/off switch for connection keepalive support, but it is being emitted as net.tcp_keepalive. Per Fluent Bit's networking documentation (and the existing plugins.Networking.Params implementation at apis/fluentbit/v1alpha2/plugins/net_types.go:55-56), the on/off keepalive toggle is the net.keepalive setting. net.tcp_keepalive is a different (TCP-level) option. As written, enabling keepalive: "on" on S3 will set the wrong Fluent Bit parameter.
	plugins.InsertKVField(kvs, "net.tcp_keepalive", o.Keepalive)

Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go
@joshuabaird joshuabaird changed the title Support network relevant settings for s3 output feat(fluentbit): Support network relevant settings for s3 output May 21, 2026
@cw-Guo
Copy link
Copy Markdown
Collaborator

cw-Guo commented May 29, 2026

Hi @smallc2009 can you please help resolve the conflicts? Thank you.

Signed-off-by: Anson <anson.liu@live.com>
Copilot AI review requested due to automatic review settings May 29, 2026 04:05
@smallc2009 smallc2009 force-pushed the feature/support-options-for-s3-output branch from cce8ea9 to ac49398 Compare May 29, 2026 04:05
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 11 changed files in this pull request and generated 9 comments.

Files not reviewed (1)
  • apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Language not supported
Comments suppressed due to low confidence (2)

manifests/setup/setup.yaml:1

  • Fix grammar/casing in descriptions: change "this include" to "this includes", and "IPV6" to "IPv6" for consistency and correctness in user-facing CRD docs.
    manifests/setup/setup.yaml:1
  • Fix grammar/casing in descriptions: change "this include" to "this includes", and "IPV6" to "IPv6" for consistency and correctness in user-facing CRD docs.

Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go
Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go
Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go
Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go
Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go
Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go
Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go
Comment thread docs/plugins/fluentbit/output/s3.md
Comment thread docs/plugins/fluentbit/output/s3.md
*plugins.TLS `json:"tls,omitempty"`
Workers *int32 `json:"Workers,omitempty"`
// Set maximum time expressed in seconds to wait for a TCP connection to be established, this include the TLS handshake time.
ConnectTimeout *int32 `json:"ConnectTimeout,omitempty"`
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible we can make such configurations shareable among different plugins, just like *plugins.TLS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S3 networking options

3 participants