example Yolo deployment server (#104)

* example Yolo deployment server

* add pre-nms postprocessing

* pre-processing note

Author: bfineran

integrations/ultralytics/README.md

@@ -100,3 +100,8 @@ python train.py \
   --sparseml-recipe /PATH/TO/RECIPE/recipe.yaml \
   <regular yolov5/train.py parameters>
 ```  
+
+
+## Server
+The `server/` directory contians an self-documented example of deploying a sparsified
+Yolo model with the DeepSparse engine.

integrations/ultralytics/server/README.md

@@ -0,0 +1,86 @@
+<!--
+Copyright (c) 2021 - present / Neuralmagic, Inc. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Example Yolo Model Server and Client Using DeepSparse Flask
+
+To illustrate how the DeepSparse engine can be used for Yolo model deployments, this directory
+contains a sample model server and client. 
+
+The server uses Flask to create an app with the DeepSparse Engine hosting a
+compiled Yolo model.
+The client can make requests into the server returning object detection results for given images.
+
+
+## Installation
+
+Similarly to the SparseML integration, dependencies can be installed via `pip` and the files in
+this self-contained example can be copied directly into the `ultralytics/yolov5` for execution.
+
+If both repositories are already cloned, you may skip that step.
+
+```bash
+# clone
+git clone https://github.com/ultralytics/yolov5.git
+git clone https://github.com/neuralmagic/sparseml.git
+
+# copy script
+cp sparseml/integrations/ultralytics/server/*.py yolov5
+cd yolov5
+
+# install dependencies
+pip install -r requirements.txt
+pip install deepsparse flask flask-cors
+```
+
+## Execution
+
+### Server
+
+First, start up the host `server.py` with your model of choice.
+
+Example command:
+```bash
+python server.py ~/models/yolov3-pruned_quant.onnx
+```
+
+You can leave that running as a detached process or in a spare terminal.
+
+This starts a Flask app with the DeepSparse Engine as the inference backend, accessible at `http://0.0.0.0:5543` by default.
+
+The app exposes HTTP endpoints at:
+- `/info` to get information about the compiled model
+- `/predict` to send images to the model and receive detected in response.
+    The number of images should match the compiled model's batch size.
+
+For a full list of options, run `python server.py -h`.
+
+Currently, the server is set to do pre-processing for the yolov3-spp
+model, if other models are used, the image shape, output shapes, and
+anchor grids should be updated. 
+
+### Client
+
+`client.py` provides a `YoloDetectionClient` object to make requests to the server easy.
+The file is self documented.  See example usage below:
+
+```python
+from client import YoloDetectionClient
+
+remote_model = YoloDetectionClient()
+image_path = "/PATH/TO/EXAMPLE/IMAGE.jpg"
+
+model_outputs = remote_model.detect(image_path)
+```

integrations/ultralytics/server/client.py

@@ -0,0 +1,68 @@
+# Copyright (c) 2021 - present / Neuralmagic, Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Client object for making requests to the example DeepSparse Yolo inference server
+"""
+
+
+import time
+from typing import List, Union
+
+import numpy
+import requests
+
+import cv2
+from deepsparse.utils import arrays_to_bytes, bytes_to_arrays
+
+
+class YoloDetectionClient:
+    """
+    Client object for making requests to the example DeepSparse Yolo inference server
+
+    :param address: IP address of the server, default is 0.0.0.0
+    :param port: Port the server is hosted on, default is 5543
+    """
+
+    def __init__(self, address: str = "0.0.0.0", port: str = 5543):
+        self._url = f"http://{address}:{port}/predict"
+
+    def detect(
+        self, images: Union[str, numpy.ndarray, List[str], List[numpy.ndarray]]
+    ) -> List[numpy.ndarray]:
+        """
+        :param images: list of or singular file paths or numpy arrays of images to
+            run the detection model on
+        :return: list of post-processed object detection results for each image including
+            class label, likelihood, and bounding box coordinates
+        """
+        if not isinstance(images, List):
+            images = [images]
+        images = [
+            cv2.imread(image) if isinstance(image, str) else image for image in images
+        ]
+
+        print(f"Sending batch of {len(images)} images for detection to {self._url}")
+
+        start = time.time()
+        # Encode inputs
+        data = arrays_to_bytes(images)
+        # Send data to server for inference
+        response = requests.post(self._url, data=data)
+        # Decode outputs
+        outputs = bytes_to_arrays(response.content)
+        total_time = time.time() - start
+        print(f"Round-trip time took {total_time * 1000.0:.4f}ms")
+
+        return outputs