Add LLM wiki + binwalk + firmware corpus, improve loader/dbt models, and unify EyeOnData.toml discovery

[grantjllnl]

Summary

This PR substantially expands pEyeON-Analytics into a more complete appliance-ready analytics stack. It introduces an LLM-maintained wiki and feature-work workflow, adds first-class Binwalk metadata support end-to-end, begins curating a firmware corpus with tooling and tests, adds new Streamlit exploration pages, and hardens the DLT/dbt pipeline for multi-batch loads and first-run bootstrap behavior. It also aligns configuration discovery and wrapper behavior with the companion pEyeON appliance/container work so both repos can share a single EyeOnData.toml, including VM layouts.

Major Additions

LLM Wiki System And Workflow

• Add wiki/ with index/log, component docs, pipeline docs, schema docs, file format pages, decisions, tensions, and feature-work artifacts.
• Add AGENTS.md describing wiki-maintainer vs code-development modes and repo-specific operating rules.
• Add LLM-assisted feature workflow docs and templates under wiki/work/ and wiki/concepts/.
• Add feature-work records for Binwalk support, firmware corpus work, metadata drift, VM appliance work, parse multiprocessing hang diagnostics, and future parse terminal-output improvements.

Binwalk Support End-To-End

• Add utils/binwalk_cli.py wrapper to run Binwalk v3 and parse JSON output.
• Add tests for the Binwalk wrapper in tests/test_binwalk_cli.py.
• Add dbt support for Binwalk metadata via dbt_eyeon_gold/models/staging/stg_metadata_binwalk_file.sql.
• Add supporting docs under raw/binwalk/ and wiki/work/binwalk-support/.

Firmware Corpus Tooling

• Add curated firmware corpus manifest at data/firmware_corpus/manifest.json.
• Add utils/firmware_corpus.py to list, fetch, and verify corpus entries.
• Add tests/test_firmware_corpus.py.
• Add firmware corpus design, verification, references, and handoff docs under wiki/work/firmware-corpus/.

Streamlit UX Improvements

• Add pages/ObservationHierarchy.py for browsing parent/child observation relationships from container and firmware extraction.
• Improve Schema Blame behavior in pages/Schema_Blame.py.
• Update page wiring in pages/pages.py.
• Simplify hierarchy metadata-type discovery and improve multi-batch usability.

DLT Loader And Schema Evolution

• Fix first-run DLT pipeline bootstrap behavior in load_eyeon.py.
• Improve multi-batch load behavior.
• Add metadata drift visibility so silver-discovered metadata tables that are not modeled in gold are surfaced.
• Add/recover schema blame and base schema derivation notes, including extras/Schema_Blame.md and wiki/pipeline/base_schema_derivation.md.

dbt Model Improvements

• Add gold.metadata_type_drift.
• Add macros/safe_silver_source.sql.
• Expand and standardize staging models for additional metadata types, including Binwalk/container/generic/opkg/symlink/text/device-tree metadata.
• Update gold.all_metadata, source definitions, and schema docs for metadata curation.

Config And Wrapper Alignment

• Unify EyeOnData.toml discovery between shell wrappers and Python code.
• Add EYEON_EYEONDATA_TOML override support in utils/config.py and wrappers.
• Update eyeon-parse.sh and eyeon-batch-summary.sh for consistent discovery and VM behavior.
• Keep the Analytics eyeon-parse.sh copy synchronized with the pEyeON wrapper copy.
• Improve wrapper interactive behavior by allocating a TTY for normal terminal runs and passing unbuffered Python/terminal settings.
• Default wrapper parse log level to WARNING, with EYEON_LOG_LEVEL / --log-level overrides and WARN alias support.

Parse Hang Diagnostics And Future Output Planning

• Document the pEyeON parse multiprocessing hang investigation and mitigation under wiki/work/parse-multiprocessing-hang/.
• Track fragile mitigations to revisit:
  • exact-message suppression for Surfactant's Possible nested set at position 81 warning,
  • threshold-based large-file serialization,
  • LOGURU_LEVEL coordination rather than a fully centralized worker logging design.
• Add wiki/work/parse-terminal-output/brief.md as future work for parent-owned worker events and potential Rich-based progress/log rendering.

VM / Container Documentation Alignment

• Add/update container and deployment documentation pages to reflect Debian 12 container/VM build paths, shared provision scripts, DuckDB CLI, Surfactant DB warmup, and external VM deployment guidance.
• Add builder glossary and VM size-reduction future-work notes.
• Record UTM/qcow2 and external deployment details under the VM work packet.

Validation

Validation performed across the branch includes:

• uv run python -m unittest tests.test_firmware_corpus
• uv run python -m unittest tests.test_firmware_corpus tests.test_binwalk_cli
• uv run ruff check utils/firmware_corpus.py tests/test_firmware_corpus.py utils/binwalk_cli.py tests/test_binwalk_cli.py
• Firmware corpus list/fetch/verify smoke checks.
• Expanded corpus parsing with Binwalk-enabled and generic-metadata-enabled pEyeON images.
• Schema validation over generated EyeON JSON batches.
• uv run python -m py_compile pages/ObservationHierarchy.py pages/pages.py
• uv run ruff check pages/ObservationHierarchy.py pages/pages.py
• Wrapper syntax checks for eyeon-parse.sh.
• Companion pEyeON targeted parse/logging tests are documented in wiki/work/parse-multiprocessing-hang/verification.md.

Notes / Follow-Ups

• This PR intentionally introduces the wiki as an LLM-maintained project memory and feature-work system; future code-development tasks should continue updating relevant wiki/work/*/verification.md and wiki/log.md entries.
• Parse terminal output is improved enough for current deployment testing, but a future parent-owned logging/progress design is documented in wiki/work/parse-terminal-output/brief.md.
• Some workflow artifacts document companion changes in pEyeON; durable implementation lives in the core repo, while this repo records analytics/wrapper/wiki context and verification.