BOR-518: add raw line ingestion fallback

[bdclaw2026]

Summary

• add a dedicated raw log line ingestion entry point that always produces an event/store record
• preserve original raw text verbatim while surfacing parser errors without dropping storage
• cover fallback and parsed ingestion behavior with unit tests

Validation

• gofmt -l -w .
  go vet ./...
  go build -o output/lapp ./cmd/lapp/
  golangci-lint run
  0 issues.
  go test ./pkg/...
  ok github.com/strrl/lapp/pkg/analyzer (cached)
  ? github.com/strrl/lapp/pkg/config [no test files]
  ok github.com/strrl/lapp/pkg/event (cached)
  ok github.com/strrl/lapp/pkg/ingest (cached)
  ok github.com/strrl/lapp/pkg/logsource (cached)
  ok github.com/strrl/lapp/pkg/multiline (cached)
  ok github.com/strrl/lapp/pkg/pattern (cached)
  ok github.com/strrl/lapp/pkg/semantic (cached)
  ok github.com/strrl/lapp/pkg/store (cached)
  ? github.com/strrl/lapp/pkg/tracing [no test files]
  ? github.com/strrl/lapp/pkg/workspace [no test files]
• ? github.com/strrl/lapp/cmd/lapp [no test files]
  ok github.com/strrl/lapp/integration_test (cached)
  ok github.com/strrl/lapp/integration_test/loghub (cached)
  ok github.com/strrl/lapp/pkg/analyzer (cached)
  ? github.com/strrl/lapp/pkg/config [no test files]
  ok github.com/strrl/lapp/pkg/event (cached)
  ok github.com/strrl/lapp/pkg/ingest (cached)
  ok github.com/strrl/lapp/pkg/logsource (cached)
  ok github.com/strrl/lapp/pkg/multiline (cached)
  ok github.com/strrl/lapp/pkg/pattern (cached)
  ok github.com/strrl/lapp/pkg/semantic (cached)
  ok github.com/strrl/lapp/pkg/store (cached)
  ? github.com/strrl/lapp/pkg/tracing [no test files]
  ? github.com/strrl/lapp/pkg/workspace [no test files]

Linear: https://linear.app/boringdesign/issue/BOR-518/phase-2ingestion-foundation-build-raw-log-line-ingestion-with-text

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a more resilient log ingestion mechanism. It ensures that all raw log lines are captured and stored, regardless of whether they can be successfully parsed into structured events. This enhancement improves data retention and diagnostic capabilities by providing a complete record of all incoming log data, while still leveraging parsing for richer event data when possible.

Highlights

• Raw Log Line Ingestion: A dedicated entry point for raw log line ingestion has been added, ensuring that an event/store record is always produced.
• Data Preservation: The original raw text is now preserved verbatim, even when parser errors occur, preventing data loss during storage.
• Test Coverage: Unit tests have been added to cover both fallback and successful parsed ingestion behaviors, ensuring robustness.

Changelog

• pkg/ingest/raw.go
  • Added new types ParseResult, Parser, and Outcome to define the structure for parsing results and ingestion outcomes.
  • Implemented FromRawLine function to normalize a raw log line, providing a fallback event/log entry even if parsing fails.
  • Implemented StoreRawLine function to persist the raw line outcome, ensuring storage even with parser failures.
  • Added a cloneMap helper function for safely copying map data.
• pkg/ingest/raw_test.go
  • Added comprehensive unit tests for FromRawLine and StoreRawLine functions.
  • Included tests for scenarios where no parser is provided, ensuring fallback behavior.
  • Tested the persistence of raw lines even when the parser encounters errors.
  • Verified that parsed fields are correctly used while the original raw text is preserved in the stored log entry.

Activity

• No human activity (comments, reviews) was detected on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a robust raw log line ingestion mechanism with a well-designed fallback strategy for parser failures. The implementation ensures that log lines are always stored, even if parsing fails, which is crucial for data integrity. The FromRawLine and StoreRawLine functions are clear and handle various scenarios, including the absence of a parser or errors during parsing. The cloneMap utility is correctly used to prevent unintended side effects with map references. The accompanying unit tests are comprehensive, covering both successful parsing and fallback behaviors, demonstrating a thorough approach to validation. Overall, the changes are well-implemented, maintainable, and correctly address the stated objective.