Skip to content
This repository was archived by the owner on Jun 3, 2025. It is now read-only.

Commit 1169027

Browse files
bfineranjeanniefinksmarkurtz
authored
Using pytorch recipes tutorial (#330)
* Using pytorch recipes tutorial * style * jfinks review edits Co-authored-by: Jeannie Finks <74554921+jeanniefinks@users.noreply.github.com> * fix formatting post edits * Correct title Co-authored-by: Jeannie Finks <74554921+jeanniefinks@users.noreply.github.com> Co-authored-by: Mark Kurtz <mark@neuralmagic.com>
1 parent 985079c commit 1169027

File tree

2 files changed

+282
-0
lines changed

2 files changed

+282
-0
lines changed

‎integrations/pytorch/recipes/resnet50-imagenette-pruned.md‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
<!--
2+
Copyright (c) 2021 - present / Neuralmagic, Inc. All Rights Reserved.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing,
11+
software distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
-->
16+
17+
---
18+
# General Epoch/LR variables
19+
num_epochs: &num_epochs 10
20+
lr: &lr 0.008
21+
22+
# Pruning variables
23+
pruning_start_epoch: &pruning_start_epoch 1.0
24+
pruning_end_epoch: &pruning_end_epoch 8.0
25+
pruning_update_frequency: &pruning_update_frequency 0.5
26+
init_sparsity: &init_sparsity 0.05
27+
28+
prune_low_target_sparsity: &prune_low_target_sparsity 0.8
29+
prune_mid_target_sparsity: &prune_mid_target_sparsity 0.85
30+
prune_high_target_sparsity: &prune_high_target_sparsity 0.9
31+
32+
training_modifiers:
33+
- !EpochRangeModifier
34+
start_epoch: 0.0
35+
end_epoch: *num_epochs
36+
37+
- !SetLearningRateModifier
38+
start_epoch: 0.0
39+
learning_rate: *lr
40+
41+
pruning_modifiers:
42+
- !GMPruningModifier
43+
params:
44+
- sections.0.0.conv1.weight
45+
- sections.0.0.conv2.weight
46+
- sections.0.0.conv3.weight
47+
- sections.0.0.identity.conv.weight
48+
- sections.0.1.conv1.weight
49+
- sections.0.1.conv3.weight
50+
- sections.0.2.conv1.weight
51+
- sections.0.2.conv3.weight
52+
- sections.1.0.conv1.weight
53+
- sections.1.0.conv3.weight
54+
- sections.1.2.conv3.weight
55+
- sections.1.3.conv1.weight
56+
- sections.2.0.conv1.weight
57+
- sections.3.0.conv1.weight
58+
- classifier.fc.weight
59+
init_sparsity: *init_sparsity
60+
final_sparsity: *prune_low_target_sparsity
61+
start_epoch: *pruning_start_epoch
62+
end_epoch: *pruning_end_epoch
63+
update_frequency: *pruning_update_frequency
64+
65+
- !GMPruningModifier
66+
params:
67+
- sections.0.1.conv2.weight
68+
- sections.0.2.conv2.weight
69+
- sections.1.0.identity.conv.weight
70+
- sections.1.1.conv1.weight
71+
- sections.1.1.conv2.weight
72+
- sections.1.1.conv3.weight
73+
- sections.1.2.conv1.weight
74+
- sections.1.2.conv2.weight
75+
- sections.1.3.conv2.weight
76+
- sections.1.3.conv3.weight
77+
- sections.2.0.conv3.weight
78+
- sections.2.0.identity.conv.weight
79+
- sections.2.1.conv1.weight
80+
- sections.2.1.conv3.weight
81+
- sections.2.2.conv1.weight
82+
- sections.2.2.conv3.weight
83+
- sections.2.3.conv1.weight
84+
- sections.2.3.conv3.weight
85+
- sections.2.4.conv1.weight
86+
- sections.2.4.conv3.weight
87+
- sections.2.5.conv1.weight
88+
- sections.2.5.conv3.weight
89+
- sections.3.1.conv1.weight
90+
- sections.3.2.conv1.weight
91+
init_sparsity: *init_sparsity
92+
final_sparsity: *prune_mid_target_sparsity
93+
start_epoch: *pruning_start_epoch
94+
end_epoch: *pruning_end_epoch
95+
update_frequency: *pruning_update_frequency
96+
97+
- !GMPruningModifier
98+
params:
99+
- sections.1.0.conv2.weight
100+
- sections.2.0.conv2.weight
101+
- sections.2.1.conv2.weight
102+
- sections.2.2.conv2.weight
103+
- sections.2.3.conv2.weight
104+
- sections.2.4.conv2.weight
105+
- sections.2.5.conv2.weight
106+
- sections.3.0.conv2.weight
107+
- sections.3.0.conv3.weight
108+
- sections.3.0.identity.conv.weight
109+
- sections.3.1.conv2.weight
110+
- sections.3.1.conv3.weight
111+
- sections.3.2.conv2.weight
112+
- sections.3.2.conv3.weight
113+
init_sparsity: *init_sparsity
114+
final_sparsity: *prune_high_target_sparsity
115+
start_epoch: *pruning_start_epoch
116+
end_epoch: *pruning_end_epoch
117+
update_frequency: *pruning_update_frequency
118+
---
119+
120+
# ResNet-50 Imagenette Conservative Sparse
121+
122+
This recipe creates a sparse [ResNet-50](https://arxiv.org/abs/1512.03385) model that
123+
achieves full recovery of its baseline accuracy on the Imagenette dataset.
124+
Training was done using 1 GPU with a total training batch size of 128
125+
using a SGD optimizer.
126+
127+
When running, adjust hyperparameters based on training environment and dataset.
128+
129+
## Training
130+
The training script can be found at `sparseml/integrations/pytorch/scripts/vision.py`.
131+
Alternatively, a full walk-through notebook is located at `sparseml/notebooks/pytorch_classification.ipynb`.
132+
133+
*script command:*
134+
135+
```
136+
python integrations/pytorch/scripts/vision.py train \
137+
--recipe-path zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenette/pruned-conservative?recipe_type=original \
138+
--pretrained True \
139+
--arch-key resnet50 \
140+
--dataset imagenette \
141+
--dataset-path /PATH/TO/IMAGENETTE \
142+
--train-batch-size 128 --test-batch-size 256 \
143+
--loader-num-workers 8 \
144+
--model-tag resnet50-imagenette-pruned-conservative
145+
```

‎integrations/pytorch/tutorials/sparsifying_pytorch_models_using_recipes.md‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
<!--
2+
Copyright (c) 2021 - present / Neuralmagic, Inc. All Rights Reserved.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing,
11+
software distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
-->
16+
17+
# Sparsifying PyTorch Models Using Recipes
18+
19+
This tutorial shows how Neural Magic recipes simplify the sparsification process by encoding the hyperparameters and instructions needed to create highly accurate pruned and pruned-quantized PyTorch models.
20+
21+
## Overview
22+
23+
Neural Magic’s ML team creates recipes that allow anyone to plug in their data and leverage SparseML’s recipe-driven approach on top of their existing training pipelines.
24+
Sparsifying involves removing redundant information from neural networks using algorithms such as pruning and quantization, among others.
25+
This sparsification process results in many benefits for deployment environments, including faster inference and smaller file sizes.
26+
Unfortunately, many have not realized the benefits due to the complicated process and number of hyperparameters involved.
27+
28+
Working through this tutorial, you will experience how Neural Magic recipes simplify the sparsification process by pruning a PyTorch [ResNet-50](https://arxiv.org/abs/1512.03385) model.
29+
30+
1. Setting Up
31+
2. Inspecting a Recipe
32+
3. Applying a Recipe
33+
3. Exporting for Inference
34+
35+
## Need Help?
36+
37+
For Neural Magic Support, sign up or log in to get help with your questions in our **Tutorials channel**: [Discourse Forum](https://discuss.neuralmagic.com/) and/or [Slack](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ).
38+
39+
## Setting Up
40+
41+
This tutorial can be run by cloning and installing the `sparseml` repository which contains scripts and recipes for
42+
this pruning example:
43+
44+
```bash
45+
git clone https://github.com/neuralmagic/sparseml.git
46+
pip install sparseml[torchvision]
47+
```
48+
49+
## Inspecting a Recipe
50+
51+
Recipes are YAML or Markdown files that SparseML uses to easily define and control the sparsification of a model.
52+
Recipes consist of a series of `Modifiers` that can influence the training process in different ways. A list of
53+
common modifiers and their uses is provided
54+
[here](https://github.com/neuralmagic/sparseml/blob/main/docs/source/recipes.md#modifiers-intro).
55+
56+
SparseML provides a recipe for sparsifying a ResNet-50 model trained on the tiny Imagenette dataset. The recipe can
57+
be viewed in the browser
58+
[here](https://github.com/neuralmagic/sparseml/blob/main/integrations/pytorch/recipes/resnet50-imagenette-pruned.md)
59+
or in the command line from the file `sparseml/integrations/pytorch/recipes/resnet50-imagenette-pruned.md`.
60+
61+
The recipe contains three kinds of modifiers:
62+
63+
- an `EpochRangeModifier` that sets the number of epochs used for sparsification
64+
- a `SetLearningRateModifier` that sets the sparsification learning rate
65+
- three `GMPruningModifier`s that set various layers of the model to be pruned to different levels.
66+
67+
Notice that each `GMPruningModifier` sets specific parameters of the ResNet-50 model to be pruned. To
68+
prune a different model, the parameter names should be adjusted to match the new model. To avoid naming
69+
individual parameter names, all parameters can be set for pruning by passing `__ALL_PRUNABLE__` instead
70+
of the parameters list for a single `GMPruningModifier`.
71+
72+
### Applying a Recipe
73+
74+
Recipes can integrated into training flows with a couple of lines of code by using a `ScheduledModifierManager`
75+
that wraps the PyTorch `Optimizer` step. An example of how this is done can be found
76+
[here](https://github.com/neuralmagic/sparseml/blob/main/docs/source/code.md#pytorch-sparsification).
77+
78+
For this example, we can use the `sparseml/integrations/pytorch/vision.py` script. This script runs a
79+
PyTorch training flow that is modified by a `ScheduledModifierManager` and takes a recipe as an input.
80+
It natively supports ResNet-50 and other popular image classification models.
81+
82+
To prune a ResNet-50 model trained on the Imagenette dataset with the recipe above, run the following command.
83+
If the Imagenette dataset has not been downloaded yet, the script will automatically download it to the given
84+
`--dataset-path`. If you have a local copy of the dataset, `--dataset-path` may be updated.
85+
86+
*Note:* If an out of memory exception occurs, the train batch size should be lowered to fit on your device
87+
88+
```bash
89+
cd sparseml
90+
91+
python integrations/pytorch/vision.py train \
92+
--recipe-path integrations/pytorch/recipes/resnet50-imagenette-pruned.md \
93+
--dataset-path ./data \
94+
--pretrained True \
95+
--arch-key resnet50 \
96+
--dataset imagenette \
97+
--train-batch-size 128 \
98+
--test-batch-size 256 \
99+
--loader-num-workers 8 \
100+
--save-dir sparsification_example \
101+
--logs-dir sparsification_example \
102+
--model-tag resnet50-imagenette-pruned \
103+
--save-best-after 8
104+
```
105+
106+
A full list of arguments and their descriptions can be found by running
107+
108+
```bash
109+
python integrations/pytorch/vision.py train --help
110+
```
111+
112+
113+
## Exporting for Inference
114+
115+
After, the training pruned model and training log files will be found under
116+
`sparsification_example/resnet50-imagenette-pruned`. To run the sparsified model for accelerated inference with
117+
an inference engine such as [DeepSparse](https://github.com/neuralmagic/deepsparse), the pruned model must be
118+
exported to the ONNX format.
119+
120+
This step has already been done for you by the `train.py` script which uses the
121+
`sparseml.pytorch.utils.ModuleExporter` class. The exported sparse ResNet-50 Imagenette ONNX model will be
122+
saved to `sparsification_example/resnet50-imagenette-pruned/model.onnx`
123+
124+
The DeepSparse Engine is explicitly coded to support running sparsified models for significant improvements in
125+
inference performance. An example for benchmarking and deploying image classification models with DeepSparse can
126+
be found [here](https://github.com/neuralmagic/deepsparse/tree/main/examples/classification).
127+
128+
## Wrap-Up
129+
130+
Neural Magic recipes simplify the sparsification process by encoding the hyperparameters and instructions needed to create highly accurate sparsified models.
131+
In this tutorial, you installed SparseML, downloaded and inspected a pruning recipe, applied it to a ResNet-50 model,
132+
and exported the sparse model for inference.
133+
134+
Next steps will include creating new recipes and applying them to different models. Check out other
135+
[SparseML tutorials](https://github.com/neuralmagic/sparseml/tree/main/integrations) to learn more.
136+
137+
For Neural Magic Support, sign up or log in to get help with your questions in our **Tutorials channel**: [Discourse Forum](https://discuss.neuralmagic.com/) and/or [Slack](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ).

0 commit comments

Comments
 (0)