Refactor get_trace and extend get_opt_trace_by_steps to MOO/constrained#4884
Open
ltiao wants to merge 1 commit intofacebook:mainfrom
Open
Refactor get_trace and extend get_opt_trace_by_steps to MOO/constrained#4884ltiao wants to merge 1 commit intofacebook:mainfrom
get_trace and extend get_opt_trace_by_steps to MOO/constrained#4884ltiao wants to merge 1 commit intofacebook:mainfrom
Conversation
…rained (facebook#4884) Summary: We have a method `get_opt_trace_by_steps` that was used extensively during our Ax 1.0 benchmarking campaign. It duplicates the basic logic of `get_trace` but differs in that it operates along `(trial_index, MAP_KEY)` pairs and respects ordering by timestamp (i.e. chronological order). However, it is limited to single-objective unconstrained problems, and our current needs (multi-objective and/or constrained) have outgrown it. We reconcile the two by extracting three core building blocks of `get_trace`: 1. `_pivot_data_with_feasibility`: Pivots data to wide format with feasibility information and metric completeness checks. 2. `_compute_trace_values`: Computes per-observation trace values (hypervolume for MOO, objective value for SOO), with cumulative best support. 3. `_aggregate_and_cumulate_trace`: Aggregates values by groups and computes the cumulative best across groups. These are implemented in a more general way that respects arbitrary groupings and orderings. We then refactor `get_trace` (and its helpers `_prepare_data_for_trace` and `get_trace_by_arm_pull_from_data`) to use these building blocks, and leverage them in `get_opt_trace_by_steps` to extend its support to multi-objective and constrained problems. Additionally: - The timestamp-based sorting in `get_opt_trace_by_steps` is preserved, which is critical for correct cumulative hypervolume computation (without this, observations would be processed in `(trial_index, arm_name, MAP_KEY)` order instead of chronological order). - Tests are updated to replace `NotImplementedError` checks with actual MOO and constrained test cases that verify correctness of the new functionality. Differential Revision: D79581270
6f17ca0 to
e18d69f
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4884 +/- ##
=======================================
Coverage 96.76% 96.76%
=======================================
Files 591 591
Lines 61955 61969 +14
=======================================
+ Hits 59948 59962 +14
Misses 2007 2007 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
We have a method
get_opt_trace_by_stepsthat was used extensively during our Ax 1.0 benchmarking campaign. It duplicates the basic logic ofget_tracebut differs in that it operates along(trial_index, MAP_KEY)pairs and respects ordering by timestamp (i.e. chronological order). However, it is limited to single-objective unconstrained problems, and our current needs (multi-objective and/or constrained) have outgrown it. We reconcile the two by extracting three core building blocks ofget_trace:_pivot_data_with_feasibility: Pivots data to wide format with feasibility information and metric completeness checks._compute_trace_values: Computes per-observation trace values (hypervolume for MOO, objective value for SOO), with cumulative best support._aggregate_and_cumulate_trace: Aggregates values by groups and computes the cumulative best across groups.These are implemented in a more general way that respects arbitrary groupings and orderings. We then refactor
get_trace(and its helpers_prepare_data_for_traceandget_trace_by_arm_pull_from_data) to use these building blocks, and leverage them inget_opt_trace_by_stepsto extend its support to multi-objective and constrained problems.Additionally:
get_opt_trace_by_stepsis preserved, which is critical for correct cumulative hypervolume computation (without this, observations would be processed in(trial_index, arm_name, MAP_KEY)order instead of chronological order).NotImplementedErrorchecks with actual MOO and constrained test cases that verify correctness of the new functionality.Differential Revision: D79581270