Figure out optimal project packaging strategy

[mosmar99]

Currently its unclear exactly what the most optimal packaging strategy is. Neptune does not allow model registratry which is not optimal for scalability, however, its very good at model tracking and storage (for as a hedge against single points of failures). Practically, it also is pretty easy to sort amongst runs to find the model runs you are interested in, given that they are named in a logical manner!

We also have a docker dev. file, it works.

What needs to be done is to explore different tools and what the optimal manner in which to use them for our usecase.

• Look at MLFlow, Kubernetes, Azure, KServe, GentoML, etc.
• Reads:
  • https://neptune.ai/blog/ml-model-registry
  • https://neptune.ai/blog/best-ml-experiment-tracking-tools

[mosmar99]

Notes: First Blog Post

An ML model registry serves as a centralized repository, enabling effective model management and documentation. It allows for clear naming conventions, comprehensive metadata, and improved collaboration between data scientists and operations teams, ensuring smooth deployment and utilization of trained models.

Model store is a superset of a Model registry. Meaning within a model store, you can find the model registry component. The store is a service other services can interface with to retrieve models from the registry.

ML vs. Ops: Who has what reponsibility?

Model registries provide interfaces that enable downstream services to consume the model through API integration. The integration can also track offline and online evaluation metrics for the models.

Model registry enables model versioning by tracking the different versions of a model as they are developed and improved. This allows you to compare different versions of the model, track its performance, and select the best version for deployment.

One thing that the model registry does really well is centralizing ML models and organizing their relevant details. With the registry, you have a central source of truth for your models throughout different stages of their lifecycle, including:

1. Development,
2. Validation,
3. Deployment,
4. And monitoring.

I do like to note that all of them have their CONS. MLFlow is only local (although can be paired with online cloud-like solutions), Verta is not free and Neptune has a lacking Model Registry. I hence chose to search further for alternative tools and found the second blog article. That article is more comprehensive, I presented my insights below.

Notes: Second Blog Post

The second blog post compares a lot more of the available tools, in fact, there's a total of 11 of them. After reading the article, it seemed to me that weights and biases looked what we may be interested in. I found out that many large AI/ML related companies are using it: OpenAI, Salesforce, and Microsoft Source. It has both an experiment tracker, model repository, CI/CD automation, 200GB of free storage, and PRO features for academics — enabling us students to use to its full extent. Beyond that, it looked user friendly (clear categorizations and plenty of documentation), allows parameter sweeps, can create automatic reports and I found it personally aesthetic. Check out the interactive comparison between model runs visualize in the figure below:

Hence, just in virtue of the specifications, weights and biases seemed like a tool of high interest to us. Also, it is American owned, and a part of NATO (which Sweden also is). Some of the other companies were owned by non-nato countries and non-eu countries, rising concerns about data security.

I made a toy project which I made in order to try weights and biases out. The first picture is from the experiment tracking tab, called "runs". In all honesty, I could not say that I find Neptunes run tracker much better, maybe slightly (how easy it is to structure your output, such as creating folders and attaching metadata and artifacts).

As seen in the last picture, in each runs artifact page, it links to the actual location where the artifact is saved. Note that it also specifies which dataset (also an artifiact) that was used during the training of the model (which is the artifact that is outputed in this case).

In the artifact tab one can get an overview of all the artifacts, both data and model versions, including model weights and data files.

I do like to note the smart versioning feature, that is, if one tries to push the same dataset with the same name in the same folder, it will not generate a new version for it but recognizes that such a version already exists and instead refers to it.

Subsequently one can simply link a model to a collection inside specified model registry simply by clicking "Link to registry".

The model is now in the production category and ready for real usage and accessible via API. See image below:

[mosmar99]

Cost Analysis

Source 1:

https://pantheon.io/learning-center/hosting/managed-vs-self-hosting

Source 2:

https://gcore.com/learning/kubernetes-tco-comparison

Source 3:

https://mlflow.org/docs/latest/self-hosting

Allows investment into model developement instead of maintance costs.

[mosmar99]

Added MongoDB for storage of structural data. It is self-hosted as a service. In addition, also self-hosting express for UI. MongoDB was chosen because over the existing minIO s3 bucket since it has O(1) retrieval (upon prompting it with a query) when using a created, which I did.