Roadmap
Welcome to the CyclOps roadmap!
This page shows you what we're working on, what we're about to work on, and what we're thinking about working on.
Want to contribute? We've compiled some good first issues.
Got a feature idea? Let us know by opening a GitHub issue.
Thanks for popping by!
Summary: Evaluation reports will provide machine learning engineers, data scientists and other decision makers with a visual way to evaluate models on clinical prediction tasks.
Intended outcome: The reports will provide metrics and plots that allow practitioners to assess model performance. Reports provide an easy-to-understand, visual overview of model performance across various axes (e.g. time, sub-population, different metrics). Practitioners can use the information to decide if a model in production is operating within desired margins.
How it will work: The reports will be built using the model card framework. Practitioners will be able to interact with graphs, plots, and tables, as well as compare two or more models on various factors including versions and time.
Summary: Expand functionality for fairness metrics with equity measurement.
Summary: Monitoring reports will explain the source of detected dataset shifts and provide estimates of model degradation to a clinical team.
Intended outcome: Currently, a shortcoming of the monitor API is that while it can detect data distribution shifts, it cannot explain the source of the shifts. Along with detection of dataset shift, it is important for a clinical team to know when the shift can degrade model performance.
How it will work: We are considering an approach described in this paper that could provide estimates of model degradation and detected shift.
Summary: We'd like to expand our support beyond mortality decompensation prediction.
Intended outcome: Build a library of use cases on clinical datasets.
How it will work: We are considering adding length of stay prediction, patient readmission prediction and prediction of specific pathologies.
Summary: The idea is to use workflow orchestration tools to build pipelines with Cyclops packages.
Intended outcome: With a workflow orchestrator, one can define each of those processes as tasks and link them in a workflow in a way that the output of one flows into the other. A workflow can be repeated, replicated, and automated.
How it will work: The orchestrator could be built using workflow orchestration tools like Flyte. An example pipeline could look something like this: query an EHR database using cyclops query API, process the data using cyclops process API, train/validate an ML model on the data using cyclops tasks API, evaluate the model using cyclops evaluate API, monitor the model using cyclops monitor API.
