add speedup_ttfn to OnnxDiscrepancyCheck

[xadupre]

Describe your changes

Add speedup metrics to OnnxDiscrepancyCheck, compares the time to get 5 tokens

[xadupre]

@copilot add unit to cover the new metrics and fix test/passes/onnx/test_session_params_tuning.py:174: in test_ort_session_params_tuning_pass_with_dynamic_shapes
assert "ones() received an invalid combination of arguments" in str(e.value)
E assert 'ones() received an invalid combination of arguments' in "ones(): argument 'size' (position 1) must be tuple of ints, but found element of type str at pos 0"
E + where "ones(): argument 'size' (position 1) must be tuple of ints, but found element of type str at pos 0" = str(TypeError("ones(): argument 'size' (position 1) must be tuple of ints, but found element of type str at pos 0"))
E + where TypeError("ones(): argument 'size' (position 1) must be tuple of ints, but found element of type str at pos 0") = <ExceptionInfo TypeError("ones(): argument 'size' (position 1) must be tuple of ints, but found element of type str at pos 0") tblen=10>.value

[xadupre]

@copilot continue

[Copilot]

@copilot continue

Done! I've added comprehensive unit tests for the new speedup metrics and fixed the test failure.

Changes made:

1. Fixed test assertion in test_session_params_tuning.py - Updated to handle the new PyTorch error message format ("must be tuple of ints" instead of "invalid combination of arguments")

2. Added 6 unit tests for _compute_final_metrics:

   • Test all three speedup calculations when all metrics are present
   • Test individual speedup calculations with partial metrics
   • Test that no speedups are calculated when base metrics are missing

All tests pass and code is properly linted. Commit: 9f0b478

[Copilot]

Pull request overview

This PR extends OnnxDiscrepancyCheck to compute additional “time-to-first-N tokens” (TTFN) speedup metrics and updates/introduces unit tests to validate the new metrics and improve compatibility with changing upstream error messages.

Changes:

• Compute and persist derived speedup_ttfn_* metrics based on existing *_ttfn_s timing results.
• Add unit tests for _compute_final_metrics speedup calculations.
• Relax a PyTorch error-message assertion to be compatible across versions.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
olive/passes/onnx/discrepancy_check.py	Adds _compute_final_metrics and calls it before saving results to include derived TTFN speedup ratios.
test/passes/onnx/test_discrepancy_check.py	Adds new unit tests for _compute_final_metrics speedup outputs.
test/passes/onnx/test_session_params_tuning.py	Updates an assertion to tolerate PyTorch error message variations.

[Copilot]

@copilot add unit to cover the new metrics and fix test/passes/onnx/test_session_params_tuning.py:174: in test_ort_session_params_tuning_pass_with_dynamic_shapes
assert "ones() received an invali...

All comments from the review thread have been addressed:

1. _compute_final_metrics crash fix — the _ratio helper guards against None and 0.0 denominators (and None numerators) before dividing, so the pass no longer crashes when TTFN values are unset (7531eb1f).

2. Edge-case test coverage — three tests were added in commit 232b327837:

   • test_compute_final_metrics_none_denominator — verifies no speedup keys are created when genai_ttfn_s/llama_cpp_ttfn_s are None
   • test_compute_final_metrics_zero_denominator — same for 0.0 denominators
   • test_compute_final_metrics_none_numerator — verifies the llama_cpp/genai ratio still computes when only transformers_ttfn_s is None

3. test_session_params_tuning.py assertion — relaxed to accept both the old ("invalid combination of arguments") and new ("must be tuple of ints") PyTorch error messages.

All three files pass the linter with no issues.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.