Dispatch with model ID to a machine learning online service with a CLI.
- The model will first be downloaded or already downloaded by model hub manager
- The model will be dispatched to as a service by its specific serving engine
- Current support engine dispatch with CLI: TorchScript, ONNX, TFS and TensorRT(optional)
- The dispatch device (CPU, CUDA:0) should be specified with CLI options
- The CLI also support for distributed dispatch later, leave the IP as localhost for extension.
- Use typer to implement
- add unit test
Dispatch with model ID to a machine learning online service with a CLI.