file source: `encoding.charset = "auto"` - safe per-file encoding detection (BOM + UTF-8/UTF-16 only)

[klondikedragon]

A note for the community

I plan on submitting a PR implementing this enhancement, and I'd love feedback on the design before I submit the PR.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Collecting Windows log files. Windows log directories routinely mix encodings:

• Windows servicing logs (C:\Windows\Logs\CBS\CBS.log,
  C:\Windows\Logs\DISM\dism.log) are UTF-8 on some Windows versions and
  UTF-16 on others: the same file path needs a different fixed charset on
  different systems in the same fleet, so no single static config is correct.
• SQL Server ERRORLOG is UTF-16LE (version-dependent, sometimes without BOM).
• MSI/installer logs are UTF-16 or ANSI depending on the producer, and can
  change encoding across rotations of the same logical log.
• PowerShell redirection produces UTF-16LE with BOM; many app logs next to
  these files are UTF-8/ASCII.

encoding.charset is per-source today, and the line delimiter is encoded into
that charset for framing, so a single file source with a directory glob cannot
handle a mix of encodings no matter how it is configured. Splitting into one
source per encoding requires knowing each file's encoding in advance, which is
exactly what operators of fleet-deployed agents do not know.

The current failure mode is also mostly silent. UTF-16LE ASCII content read
with charset: utf-8 (or with no encoding at all) contains no invalid UTF-8
byte sequences (0x00 is valid UTF-8), so DecoderMalformedReplacement never
fires and NULL-riddled lines ship to sinks. When replacement warnings do fire,
they carry only the encoding name, not the file, so a single bad file in a
large glob is hard to locate.

Attempted Solutions

• Explicit encoding.charset per source: works only when every file matched by the source shares one encoding, and breaks when a log changes encoding across rotation.
• Pre-transcoding files outside Vector: not viable for logs being written in place by third-party software.

Proposal

Add an auto mode to the file source's encoding option:

encoding:
  charset: auto            # existing charset labels keep working unchanged
  fallback_charset: utf-8  # optional; used when detection is inconclusive
                           # (default utf-8 = today's behavior)

Detection is per file, at file open, over the first read chunk (a few bytes
suffice for BOM; a bounded window, e.g. up to 1 KiB, for the heuristics), and
deliberately limited to cases that are (nearly) impossible to get wrong:

1. BOM (authoritative): EF BB BF -> UTF-8, FF FE -> UTF-16LE,
   FE FF -> UTF-16BE (exactly the set encoding_rs::Encoding::for_bom
   recognizes).
2. BOM-less UTF-16 heuristic: NULL bytes heavily concentrated at one byte
   parity and near-absent at the other, AND the window decodes without errors
   as UTF-16LE/BE (decode_without_bom_handling_and_without_replacement).
3. Strict UTF-8 validation of the window -> UTF-8.
4. Otherwise -> fallback_charset, with a once-per-file warning that
   detection was inconclusive.

Ordering note: step 2 must precede step 3, because BOM-less UTF-16LE ASCII is
byte-valid UTF-8; a UTF-8-first ladder can never detect the file class that
motivates this feature.

Non-goals (deliberate):

• No detection of legacy single-byte codepages (windows-125x etc.): they are
  statistically indistinguishable from each other, and a wrong guess silently
  corrupts data. Operators who know the encoding set an explicit charset (or
  fallback_charset). This also keeps chardetng-style content guessing out.
• No UTF-32 (not supported by encoding_rs / the Encoding Standard, and
  effectively absent from real log files).

Implementation sketch:

• Detection lives with per-file state in lib/file-source (FileWatcher),
  because the line delimiter must become per-file: today the delimiter is
  encoded once per source into the configured charset and used for byte-level
  framing. With auto, the watcher sniffs at open, picks the per-file
  delimiter bytes, and annotates emitted Lines with the detected encoding.
• The source-side transcoding keeps today's decoder-sharing compromise, one
  decoder per detected charset instead of one per source.
• Rotation/truncation already produces fresh per-file state via
  fingerprinting, so re-detection across rotation (where real logs do change
  encoding) falls out naturally.
• Observability: a debug-level internal event per file on detection (file,
  detected encoding, via bom|utf16-heuristic|utf8-valid), a warning on
  inconclusive fallback, and (small companion improvement) file attribution on
  DecoderMalformedReplacement, which currently logs only the encoding name.
• No new dependencies: encoding_rs (already used for transcoding) provides
  BOM sniffing and strict validity checks; UTF-8 validation can use simdutf8
  (already in the tree).

Existing behavior is unchanged unless charset: auto is configured.

I want to open a PR and would like feedback before we implement.

References

• Add encoding.charset option to sources and sinks #2166 (original encoding.charset option)
• fix(windows platform): report SERVICE_STOPPED to SCM when startup fails #25810 (windows service startup-failure fix; same motivation context: Vector as a fleet-managed Windows collector as part of https://sparklogs.com/ )

Version

vector 0.56.0 (x86_64-pc-windows-msvc 7923556 2026-05-02 05:50:46.918905925)

[klondikedragon]

Design refinement (feedback welcome before the PR)

After grounding this in the file source + lib/file-source path, here is a tighter design than the original sketch. Would love feedback!

Config (file source encoding only)

encoding:
  charset: auto                    # all new behavior is gated under this value
  fallback_charset: utf-8          # optional; default utf-8; only valid with charset: auto
  auto_detect_min_bytes: 128       # optional; default 128; clamp >= 32
  auto_detect_max_bytes: 2048      # optional; default 2048; clamp <= 65536
  max_replacement_ratio: 0.33      # optional; default 0.33; 0 disables the reject gate

Byte fields follow existing file-source conventions (max_line_bytes, max_read_bytes, …): _bytes suffix, raw integer byte counts, docs::type_unit = "bytes". Not human-size strings.

Per-file lifecycle (files can be empty then grow)

Vector is a tailer; there is no reliable “file is complete” signal, so we do not try to detect that.

Per watched file:

1. Pending until we can decide safely.
2. Decided once charset is chosen; per-file line delimiter + decode from then on.
3. Rejected (optional gate) if the sniff window looks like mostly undecodable garbage under the chosen charset.

While Pending or Rejected: emit no lines; do not advance checkpoints through unread content.

Peek safety: always sniff from offset 0 (up to auto_detect_max_bytes), then seek back to the checkpoint offset before framing. Resume must stay byte-exact.

Detection ladder (order is load-bearing)

On the sniff window:

1. BOM (encoding_rs::Encoding::for_bom) → definitive at any size (UTF-8 / UTF-16LE / UTF-16BE).
2. If size < auto_detect_min_bytes and no BOM → stay Pending (do not accept “strict UTF-8” yet).
3. If size ≥ min (peek up to max):
   • BOM-less UTF-16 NUL-parity heuristic + strict UTF-16 decode, then
   • strict UTF-8, then
   • else fallback_charset + once-per-file warning.

Why min exists: UTF-16LE ASCII is byte-valid UTF-8, and a short partial flush of UTF-16 can look like plain ASCII. Declaring UTF-8 too early is exactly the silent NULL-mangling failure mode. ~128 bytes (64 UTF-16 code units) is the default confidence floor before non-BOM decisions; operators can raise it.

Non-goals unchanged: no windows-125x / chardetng guessing; no UTF-32.

Reject gate (binary / garbage globs)

After charset is chosen, decode the sniff window under that charset. If max_replacement_ratio > 0 and U+FFFD replacements / codepoints ≥ threshold → Rejected: warn once with filename + reason + ratio, never ingest that fingerprint.

• Default 0.33 (1-in-3 replacements is already badly wrong). 0 turns the gate off.
• Auto-only (explicit charset: utf-16le / utf-8 behavior unchanged).
• This catches typical png/exe/zip under fallback UTF-8. It does not replace the UTF-16-before-UTF-8 ladder (UTF-16-as-UTF-8 has ~0% U+FFFD).
• Residual: ASCII-heavy “binary” may still look like UTF-8.

Also planning a small companion change: add file path to DecoderMalformedReplacement (today it logs only the encoding name).

Intentionally orthogonal to fingerprint.ignored_header_bytes

That option only skips bytes when computing the checksum fingerprint. It does not skip bytes on the read/decode path. Auto-detect therefore peeks from the same start the decoder will see (offset 0), and does not require ignored_header_bytes + min before leaving Pending. Coupling those would miss a BOM at 0 and still decode the header with the body charset.

Checkpoint encoding field?

Considered and dropped for this PR: with Pending + re-peek, sticky checkpoint encoding adds schema/review surface for little gain and can poison if content is replaced under the same fingerprint. Detection is fast/cheap and only happens if configured to auto anyway. Revisit later only with evidence.

Scope

Still file-source only (sources::util::EncodingConfig / encoding_transcode are file-source consumers today). stdin/exec use codecs; out of scope unless there is demand.

Questions for maintainers

1. Defaults (min=128, max=2048, ratio=0.33) feel right for upstream, with knobs for stricter fleets?
2. Reject gate default-on for auto, disable with max_replacement_ratio: 0, acceptable?
3. Any objection to Pending state for sub-min files without BOM?
4. Prefer the reject / malformed events’ naming to follow a particular existing pattern?

[klondikedragon]

One more small thing in case it wasn't clear: if the file-source encoding mode is set to the new auto mode and a BOM is present, the BOM will NOT be included in data that is forwarded by the source. It will be treated as a file-charset-encoding signal and since data sent upstream has a standardized text encoding already, the BOM is not data we should include (IMO) if charset mode is auto. does that sound reasonable? (could also add a config knob if desired)