Skip to content

feat: port ateam STRATIX instrumentation engine into layerlens.instrument#64

Closed
garrettallen14 wants to merge 1 commit into
mainfrom
feat/instrument
Closed

feat: port ateam STRATIX instrumentation engine into layerlens.instrument#64
garrettallen14 wants to merge 1 commit into
mainfrom
feat/instrument

Conversation

@garrettallen14

Copy link
Copy Markdown
Contributor
Screenshot from 2026-03-18 20-03-50 Example of a trace uploaded to our enterprise app via the new SDK adapters.

Summary

Ports Marc's ateam STRATIX instrumentation engine into the SDK as the layerlens.instrument
module. This is not a rewrite — 206 of 208 source files are the ateam originals with only
mechanical import path changes (stratix.*layerlens.instrument.*).

What's included

  • Schema layer (L1–L6): All event types from the ateam spec — agent I/O, code, model,
    environment, tools, protocols, plus cross-cutting events (state, cost, policy, handoff) and
    feedback/evaluation/replay
  • Core SDK: STRATIX class, context propagation, decorators, emit helpers, cost tracking,
    policy enforcement
  • 13 framework adapters: LangChain, LangGraph, CrewAI, AutoGen, OpenAI Agents, Google
    ADK, Bedrock, LlamaIndex, PydanticAI, Semantic Kernel, SmolAgents, Langfuse, Agentforce
  • 3 protocol adapters: A2A, AG-UI, MCP
  • 7 LLM provider adapters: OpenAI, Anthropic, Azure, Bedrock, Vertex, Ollama, LiteLLM
  • OTel exporters: span + metrics
  • Trace simulator: scenarios, sources, outputs, error injection, content templates
  • 97-file sample library: frameworks, protocols, providers, 13 industry verticals,
    OpenClaw, CopilotKit, CLI

What changed vs ateam (only 2 files)

  1. _core.py: replay() stubbed with NotImplementedError (server-side only)
  2. _sinks.py: Replaced TraceStoreSink/IngestionPipelineSink with APIUploadSink
    (bridges capture → layerlens.Stratix platform API) and LoggingSink

Everything else is 1:1 with ateam, plus underscore-prefix convention on internal modules.

Import rewriting

Automated via scripts/rewrite_imports.py — 6 prefix substitutions, zero manual edits to
business logic.

Stats

  • 208 source files under src/layerlens/instrument/
  • 131 test files under tests/instrument/
  • 97 sample files under samples/
  • 2,360 tests pass (1,868 new + 492 existing), 0 failures
  • 0 stale from stratix.* imports in production code
  • Base pip install layerlens unchanged — framework adapters are opt-in via extras

Test plan

  • python -c "from layerlens.instrument import STRATIX" — imports clean
  • python -c "from layerlens import Stratix" — existing SDK unaffected
  • grep -r "from stratix\." src/ — 0 results
  • pytest tests/instrument/ -x — 1,868 passed
  • pytest tests/ --ignore=tests/instrument/ -x — 492 passed
  • Full suite: pytest tests/ -x — 2,360 passed, 0 failures

@m-peko

m-peko commented Mar 19, 2026

Copy link
Copy Markdown
Collaborator

can you make sure CI is passing?
should samples directory content be actually in examples directory?

" pip install crewai crewai-tools"
)

from layerlens.instrument import STRATIX, emit_handoff, emit_tool_call

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is STRATIX here and why it's all uppercase?

@m-peko

m-peko commented Mar 19, 2026

Copy link
Copy Markdown
Collaborator

i don't see the docs for this. can we add some docs to docs folder?

@Lzok

Lzok commented Mar 19, 2026

Copy link
Copy Markdown
Collaborator

Leaving here the first Claude's analysis. Number one is funny

1. Likely AI-Generated, Not "Ported"

Code has strong AI-generation hallmarks:

  • 97 sample files covering 13 industry verticals (education, energy, financial, healthcare, insurance, legal, manufacturing, media, real_estate, retail, telecom, travel) — this breadth is classic LLM padding
  • Extremely uniform structure across all adapters and samples
  • CopilotKit + OpenClaw samples look like vaporware concepts
  • Formulaic docstrings everywhere

2. Entire Module Excluded from Static Analysis

The pyproject.toml changes are the most suspicious part:

"src/layerlens/instrument/**.py" = ["T201", "T203", "F401"]  # blanket unused-import suppression
exclude = ["**/__pycache__", "src/layerlens/instrument/**"]  # entirely excluded from pyright

208 new source files with zero type checking. If this were genuinely ported from a well-tested internal repo, you wouldn't need to suppress the type checker — you'd fix the errors. This reads as "the generated code doesn't pass pyright, so turn it off." claude went a bit 'violent' on this one lol

3. Stale stratix Import in Code

The PR description boasts "0 stale from stratix.* imports" — but lifecycle.py still contains:

Usage (new):
    from stratix import STRATIX

The grep check grep -r "from stratix\." src/ misses this because it's from stratix import (no dot). Sloppy.

4. Silent Data Loss in _sinks.py

APIUploadSink.flush() catches all exceptions at logger.debug() level and continues. Trace data can be silently lost in production with zero visibility:

except Exception:
    logger.debug("APIUploadSink.flush() failed for %d events ...")

This should at minimum be logger.warning() or logger.error(), and probably should surface failures to the caller.

5. Breaking Python Version Bump

Silently changes requires-python from >= 3.8 to >= 3.9. This is a breaking change for existing users on Python 3.8 — buried in a 70K-line PR.

6. Temp File IO on Every Flush

APIUploadSink writes a temp JSONL file on every flush then uploads it. This will break in read-only container filesystems and is unnecessarily slow for what could be an in-memory buffer upload.

8. No CI Verification

The PR claims "2,360 tests pass, 0 failures" but there's no CI status check visible. The tests are all self-contained with simple mocks — they test the generated code against its own assumptions, not against real framework behavior.

@m-peko m-peko deleted the feat/instrument branch July 2, 2026 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants