fix(result): normalize result artifact timestamps

[bingran-you]

Summary

Closes #528.

This keeps agent_result as the canonical result metrics object while preserving the existing top-level n_tool_calls, n_skill_invocations, and n_prompts fields as compatibility mirrors. Both native rollout and hosted-env writers now source those mirrors from agent_result, so the old flat fields cannot drift independently.

It also normalizes artifact timestamps at the result/config serialization boundary:

• result.json.started_at and result.json.finished_at now serialize as timezone-aware UTC ISO 8601 strings with a Z suffix.
• native config.json.started_at and hosted-env config.json.started_at use the same formatter.
• native rollout started_at is now captured with datetime.now(UTC) at source, so non-UTC hosts do not stamp local wall time as a UTC instant.
• aware non-UTC started_at inputs are handled without naive/aware datetime arithmetic failures.

CI security follow-up

Fresh CI hit two transitive pip-audit advisories after the first push:

• click 8.3.1 -> 8.4.2
• cryptography 46.0.7 -> 48.0.1
• current-head CI later found mcp 1.27.2 -> 1.28.1 for CVE-2026-59950

cryptography>=48.0.1 requires moving the exact litellm[proxy] pin from 1.89.0 to 1.91.0, the first stable release in this range whose proxy extra permits the fixed cryptography line. LiteLLM-focused tests and the full suite pass on the updated lock.

Why

Issue #528 called out two related artifact-shape problems:

• consumers had to choose between duplicated top-level metrics and agent_result metrics;
• result timestamps were emitted via str(datetime), producing space-separated values with no timezone while other artifact fields already used ISO 8601.

Removing the top-level metric mirrors in one shot would break existing contract tests and likely downstream consumers, so this PR makes agent_result the source of truth first and leaves removal of compatibility mirrors as a deliberate later schema break.

Validation

• uv sync --extra dev --extra sandbox-daytona --locked
• uv run --extra dev python -m pytest tests/test_verify.py::TestResultJson tests/test_hosted_env_rollout_contract.py::test_hosted_env_writes_contract_result_json -q -> 7 passed
• uv run --extra dev python -m pytest tests/test_verify.py tests/test_hosted_env_rollout_contract.py tests/test_sdk_internals.py::TestBuildResult tests/test_config_redaction.py tests/test_usage_litellm.py tests/test_native_acp_usage.py tests/test_metrics.py -q -> 133 passed
• uv run --extra dev python -m pytest tests/test_litellm_config.py tests/test_litellm_hardening.py tests/test_litellm_logging.py tests/test_litellm_smoke.py tests/test_usage_litellm.py tests/test_agent_registry.py -q -> 134 passed, 2 skipped
• uv run --extra dev python -m pytest tests/test_trace_import_cli.py::test_tasks_generate_rejects_removed_short_options -q -> 4 passed
• uv run --extra dev python -m pytest tests/ -q -> 4855 passed, 16 skipped, 7 deselected
• uv export --locked --extra dev --extra sandbox-daytona --extra sandbox-modal --format requirements-txt --no-hashes --output-file /tmp/req-audit && uv run --with pip-audit pip-audit -r /tmp/req-audit --no-deps --disable-pip -> no known vulnerabilities found
• uv run ruff format --check pyproject.toml src/benchflow/_utils/timestamps.py src/benchflow/rollout/_results.py src/benchflow/hosted_env.py tests/test_verify.py tests/test_hosted_env_rollout_contract.py tests/test_trace_import_cli.py
• uv run ruff check .
• uv run ty check src/
• uv run --extra dev --locked python -m pytest tests/test_artifact_timestamps.py tests/test_hosted_env_rollout_contract.py::test_hosted_env_writes_contract_result_json -q -> 4 passed
• uv run --extra dev --locked python -m pytest tests/test_artifact_timestamps.py tests/test_verify.py tests/test_hosted_env_rollout_contract.py tests/test_sdk_internals.py::TestBuildResult tests/test_config_redaction.py tests/test_usage_litellm.py tests/test_native_acp_usage.py tests/test_metrics.py tests/test_trace_import_cli.py::test_tasks_generate_rejects_removed_short_options -q -> 138 passed
• uv export --locked --extra dev --extra sandbox-daytona --extra sandbox-modal --format requirements-txt --no-hashes --output-file /tmp/pr928-req-audit && uv run --with pip-audit pip-audit -r /tmp/pr928-req-audit --no-deps --disable-pip -> no known vulnerabilities found
• uv run ruff format --check src tests
• uv lock --check

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4a8baaead0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bingran-you]

Daily scan follow-up (2026-07-13): pushed d285ba940 to clear the fresh pip-audit failure on this PR.

What changed:

• click lock entry updated from 8.3.1 to 8.4.2 for PYSEC-2026-2132.
• cryptography lock entry updated from 46.0.7 to 48.0.1 for GHSA-537c-gmf6-5ccf.
• litellm[proxy] exact pin moved from 1.89.0 to 1.91.0 because 1.89.0 requires cryptography<47, while 1.91.0 permits cryptography>=48.0.1.
• The Click 8.4 error-message formatting regression test now asserts the removed option is rejected without pinning Click's old punctuation.

Local validation after the dependency update:

• exact CI pip-audit command -> no known vulnerabilities found
• LiteLLM/proxy-focused tests: 134 passed, 2 skipped
• removed-short-option CLI regression: 4 passed
• full unit suite: 4855 passed, 16 skipped, 7 deselected
• repo-wide ruff check . and ty check src/ passed

Waiting on refreshed GitHub checks for head d285ba940.

[bingran-you]

CI follow-up (2026-07-13): refreshed GitHub checks are now green on head d285ba940.

Passed checks:

• test
• pip-audit
• manifest-parity / parity
• integration-light / rollout-smoke

Labels moved back to status:ready + review:pending; remaining gate is non-author human review before merge.

[bingran-you]

Users Simulation automation review (2026-07-13): ready by simulation on head d285ba9401aa06918201d55c1463463672b1bfb1.

Scope checked: result/config timestamp normalization, agent_result as the source of truth for compatibility metric mirrors, hosted-env result shape, dependency audit follow-up, and broad CLI/SDK regression risk.

Evidence:

• Shared base smoke passed: uv sync --extra dev --extra sandbox-daytona --locked, ruff check ., ty check src, bench --help, bench agent list, bench tasks check --help, bench eval run --help, SDK Scene.single(agent="oracle") + RolloutConfig.
• PR worktree checks passed: uv sync --extra dev --extra sandbox-daytona --locked; pytest tests/test_verify.py tests/test_hosted_env_rollout_contract.py tests/test_sdk_internals.py::TestBuildResult tests/test_config_redaction.py tests/test_usage_litellm.py tests/test_native_acp_usage.py tests/test_metrics.py tests/test_trace_import_cli.py::test_tasks_generate_rejects_removed_short_options -q -> 137 passed; focused ruff; ty check src; CLI + SDK smoke.
• GitHub current-head checks are green: test, pip-audit, manifest-parity / parity, and integration-light / rollout-smoke passed.

Artifact health:

• Downloaded run 29236955437 and validated it with benchflow-experiment-review: 1/1 healthy, result.json, nested and rollout-level results.jsonl, trajectory/acp_trajectory.jsonl, and trajectory/llm_trajectory.jsonl present.
• The model rollout has 27 ACP events, 18/18 LLM exchanges with provider usage, one training-ready row, reward 1.0, 309,996 total tokens, 16 tool calls, and timing total 296.1s.
• result.json.started_at, result.json.finished_at, and config.json.started_at are UTC ISO timestamps with Z; top-level n_tool_calls / n_skill_invocations / n_prompts mirror agent_result.
• Caveat: GLM user-endpoint cost accounting is still unpriced (agent_result.cost_usd=null, price_source=null) even though token/timing telemetry is complete.

Thermo-nuclear review: no blocker. The timestamp helper is small and canonical, the compatibility mirrors now read from agent_result, and no changed production file crosses the 1k-line threshold. Keeping status:ready + review:pending; remaining gate is non-author human review.

[bingran-you]

Users Simulation automation review (2026-07-23): blocked on current head d285ba9401aa06918201d55c1463463672b1bfb1.

What still passes:

• PR-scoped sidecar ran uv sync --extra dev --locked, 137 focused tests, ruff check ., format check, ty check src/, SDK artifact smoke, and CLI bench eval list / bench eval metrics --json.
• Current exact-head integration-light artifact from run 29236955437 validates healthy with benchflow-experiment-review: ACP trajectory, LLM trajectory, one training-ready results.jsonl row, reward 1.0, 309,996 tokens, 16 tool calls, and 296.1s timing.
• Thermo-nuclear review did not find a file-size or abstraction-growth blocker; the timestamp helper itself is small and in a reasonable shared layer.

Blocking finding:

• src/benchflow/_utils/timestamps.py treats naive datetimes as UTC, but src/benchflow/rollout/_setup.py still creates the rollout started_at with local datetime.now(). On a non-UTC host this produces the wrong instant while adding Z. Repro in TZ=America/Los_Angeles: a local naive 2026-07-23T05:10:58 formats as 2026-07-23T05:10:58Z, while the actual UTC instant is 2026-07-23T12:10:58Z.

Repo-convention blocker:

• New regression docstrings cite issue result.json schema bloat: top-level/agent_result duplication and started_at/finished_at not ISO 8601 #528 / PR Hosted-env evaluations bypass the rollout reward and trajectory engine #419 but do not name PR fix(result): normalize result artifact timestamps #928 as required by AGENTS.md: tests/test_verify.py timestamp tests and tests/test_hosted_env_rollout_contract.py::test_hosted_env_writes_contract_result_json.

Recommended fix: make runtime artifact timestamps UTC-aware at source, add a non-UTC timezone regression, and update the regression-test docstrings to name PR #928. Keeping this blocked until that is fixed.

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[bingran-you]

Users Simulation follow-up (2026-07-23): ready by simulation on current head 089022bc9339f6a4be91947fd94afd52f212d560.

What changed:

• ba7686f09 fixes the timestamp blocker by sourcing native rollout started_at with datetime.now(UTC) and moving the PR fix(result): normalize result artifact timestamps #928 timestamp regressions into focused tests/test_artifact_timestamps.py instead of growing tests/test_verify.py.
• 089022bc9 fixes the current-head pip-audit failure by moving mcp from 1.27.2 to 1.28.1 for CVE-2026-59950.
• Replied to and resolved the two current blocker threads: discussion_r3569233988 and discussion_r3569233993.

Local validation:

• uv run --extra dev --locked python -m pytest tests/test_artifact_timestamps.py tests/test_hosted_env_rollout_contract.py::test_hosted_env_writes_contract_result_json -q -> 4 passed
• uv run --extra dev --locked python -m pytest tests/test_artifact_timestamps.py tests/test_verify.py tests/test_hosted_env_rollout_contract.py tests/test_sdk_internals.py::TestBuildResult tests/test_config_redaction.py tests/test_usage_litellm.py tests/test_native_acp_usage.py tests/test_metrics.py tests/test_trace_import_cli.py::test_tasks_generate_rejects_removed_short_options -q -> 138 passed
• uv export --locked --extra dev --extra sandbox-daytona --extra sandbox-modal --format requirements-txt --no-hashes --output-file /tmp/pr928-req-audit && uv run --with pip-audit pip-audit -r /tmp/pr928-req-audit --no-deps --disable-pip -> no known vulnerabilities found
• uv run ruff check ., uv run ruff format --check src tests, uv run ty check src/, uv lock --check, and git diff --check passed.

Current-head GitHub checks are green: test, pip-audit, manifest-parity / parity, integration-light / rollout-smoke, and both detect-scope jobs passed; matrix/review-pack jobs were skipped by scope.

Artifact health:

• Downloaded run 30010318870 artifact integration-light-jobs and validated it with .agents/skills/benchflow-experiment-review/scripts/validate_run_artifacts.py /tmp/pr928-artifacts.0meKUG --json: 1/1 healthy.
• Required artifacts are present: result.json, rollout-level and nested results.jsonl, trajectory/acp_trajectory.jsonl, and trajectory/llm_trajectory.jsonl.
• The model rollout has reward 1.0, 25 ACP events, 16/16 LLM exchanges with usage, one training-ready row, 261,995 tokens, 15 tool calls, and 263.3s timing.
• result.json.started_at, result.json.finished_at, and config.json.started_at are UTC ISO timestamps with Z; the top-level metric mirrors match agent_result.

Thermo-nuclear review: no remaining blocker. Production change is a one-line UTC source fix plus a small timestamp helper doc clarification; no changed production file crosses the 1k-line threshold, and timestamp regression coverage now lives in a dedicated 99-line test module. Labels moved to status:ready + review:pending; remaining gate is non-author human review.