Yolo benchmarking (#108)

* Benchmarking for YOLOv3

* edits from Jeannie

* support for other inference backends, updates to preprocecssing flow, quantized inputs, etc

* load data from sparsezoo, ORT batch override, ORT num cores override, documentation update

* qat bug fix

Author: bfineran

integrations/ultralytics/README.md

@@ -20,7 +20,7 @@ This directory provides a SparseML integrated training script for the popular
 repository.
 
 Using this integration, you will be able to apply SparseML optimizations
-to the powerful training flows provided in the yolov5 repository.
+to the powerful training flows provided in the `yolov5` repository.
 
 Some of the tasks you can perform using this integration include, but are not limited to:
 * model pruning
@@ -30,27 +30,29 @@ Some of the tasks you can perform using this integration include, but are not li
 
 ## Installation
 To use the script, clone both repositories, install their dependencies,
-and copy the integrated training script into the yolov5 directory to run from.
+and copy the integrated training script into the `yolov5` directory to run from.
 
 ```bash
 # clone
 git clone https://github.com/ultralytics/yolov5.git
 git clone https://github.com/neuralmagic/sparseml.git
 
 # copy script
-cp sparseml/integrations/ultralytics/train.py sparseml/integrations/ultralytics/test.py yolov5
 cd yolov5
+git checkout c9bda11  # latest tested integration commit hash
+cp ../sparseml/integrations/ultralytics/*.py yolov5
+cp ../sparseml/integrations/ultralytics/deepsparse/*.py
 
 # install dependencies
 pip install -r requirements.txt
-pip install sparseml
+pip install sparseml deepsparse
 ```
 
 
 ## Script
 `integrations/ultralytics/train.py` modifies
 [`train.py`](https://github.com/ultralytics/yolov5/blob/master/train.py)
-from yolov5 to include a `sparseml-recipe` argument
+from `yolov5` to include a `sparseml-recipe` argument
 to run SparseML optimizations with.  This can be a file path to a local
 SparseML recipe or a SparseZoo model stub prefixed by `zoo:` such as
 `zoo:cv/detection/yolo_v3-spp/pytorch/ultralytics/coco/pruned-aggressive`.
@@ -61,13 +63,13 @@ initial checkpoint, pass that model's SparseZoo stub prefixed by `zoo:` to the
 `--initial-checkpoint` argument.
 
 Running the script will
-follow the normal yolov5 training flow with the given SparseML optimizations enabled.
+follow the normal `yolov5` training flow with the given SparseML optimizations enabled.
 
 Some considerations:
 
 * `--sparseml-recipe` is a required parameter
 * `--epochs` will now be overridden by the epochs set in the SparseML recipe
-* if using learning rate schedulers both with the yolov5 script and your recipe, they
+* if using learning rate schedulers both with the `yolov5` script and your recipe, they
 may conflict with each other causing unintended side effects, so choose
 hyperparameters accordingly
 * Modifiers will log their outputs to the console as well as to the TensorBoard file
@@ -102,6 +104,39 @@ python train.py \
 ```  
 
 
-## Server
-The `server/` directory contians an self-documented example of deploying a sparsified
-Yolo model with the DeepSparse engine.
+## DeepSparse Examples
+The [DeepSparse](https://github.com/neuralmagic/deepsparse) engine can be used for effcient
+inference on sparsified YOLOv3 models.  The `deepsparse` directory contains examples of using
+DeepSparse for benchmarking performance and in an inference server environment.
+
+### Installation
+To use these examples after performing the previous installations, copy over the python files
+to the `yolov5` directory and install DeepSparse.
+
+```bash
+# copy DeepSparse python files
+cp sparseml/integrations/ultralytics/deepsparse/*.py yolov5
+cd yolov5
+
+# install deepsparse and server dependencies
+pip install deepsparse flask flask-cors
+```
+
+
+### Benchmarking
+`benchmarking.py` is a script for benchmarking sparsified and quantized YOLOv3
+performance with DeepSparse.  For a full list of options run `python benchmarking.py -h`.
+
+To run a benchmark run:
+```bash
+python benchmark.py
+    zoo:cv/detection/yolo_v3-spp/pytorch/ultralytics/coco/pruned_quant-aggressive_90 \
+    --batch-size 32 \
+    --quantized-inputs
+```
+
+Note for quantized performance, your CPU must support VNNI instructions.
+
+### Server
+`sparseml/integrations/ultralytics/deepsparse/SERVER.md` contains relevant
+documentation for running the server and client examples.

integrations/ultralytics/server/README.md …rations/ultralytics/deepsparse/SERVER.mdintegrations/ultralytics/server/README.md renamed to integrations/ultralytics/deepsparse/SERVER.md integrations/ultralytics/server/README.md renamed to integrations/ultralytics/deepsparse/SERVER.md

@@ -14,13 +14,13 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 
-# Example Yolo Model Server and Client Using DeepSparse Flask
+# Example YOLO Model Server and Client Using DeepSparse Flask
 
-To illustrate how the DeepSparse engine can be used for Yolo model deployments, this directory
+To illustrate how the DeepSparse engine can be used for YOLO model deployments, this directory
 contains a sample model server and client. 
 
 The server uses Flask to create an app with the DeepSparse Engine hosting a
-compiled Yolo model.
+compiled YOLOv3 model.
 The client can make requests into the server returning object detection results for given images.
 
 
@@ -36,13 +36,13 @@ If both repositories are already cloned, you may skip that step.
 git clone https://github.com/ultralytics/yolov5.git
 git clone https://github.com/neuralmagic/sparseml.git
 
-# copy script
-cp sparseml/integrations/ultralytics/server/*.py yolov5
+# copy DeepSparse python files
+cp sparseml/integrations/ultralytics/deepsparse/*.py yolov5
 cd yolov5
 
 # install dependencies
 pip install -r requirements.txt
-pip install deepsparse flask flask-cors
+pip install deepsparse sparseml flask flask-cors
 ```
 
 ## Execution