Lr scheduler notebook

[SamSandwich07]

Adds an notebook demonstrating the use of an LR scheduler

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.17%. Comparing base (464960b) to head (3b4ddce).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #702   +/-   ##
=======================================
  Coverage   64.17%   64.17%           
=======================================
  Files          61       61           
  Lines        5892     5892           
=======================================
  Hits         3781     3781           
  Misses       2111     2111           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Before [464960b]	After [ee57f85]	Ratio	Benchmark (Parameter)
1.92±0.02s	1.95±0.01s	1.02	benchmarks.time_help
1.43G	1.45G	1.02	vector_db_benchmarks.VectorDBInsertBenchmarks.peakmem_load_vector_db(2048, 'qdrant')
10.8±0.01s	11.0±0.03s	1.02	vector_db_benchmarks.VectorDBInsertBenchmarks.time_load_vector_db(2048, 'qdrant')
9.31±0.03ms	9.49±0.05ms	1.02	vector_db_benchmarks.VectorDBSearchBenchmarks.time_search_by_vector_many_shards(128, 'chromadb')
203±2ms	206±3ms	1.01	benchmarks.time_import
15.317196486583306	15.396910452865157	1.01	data_cache_benchmarks.DataCacheBenchmarks.track_cache_hsc1k_hyrax_size_undercount
54.1±0.2s	54.4±0.4s	1.01	vector_db_benchmarks.VectorDBInsertBenchmarks.time_load_vector_db(16384, 'qdrant')
4.61±0.01s	4.64±0.04s	1.01	vector_db_benchmarks.VectorDBInsertBenchmarks.time_load_vector_db(64, 'qdrant')
9.53±0.03ms	9.58±0.04ms	1.01	vector_db_benchmarks.VectorDBSearchBenchmarks.time_search_by_vector_many_shards(64, 'chromadb')
1.92±0.01s	1.93±0.01s	1	benchmarks.time_infer_help

Click here to view all benchmarks.

[drewoldag]

I would recommend reverting the changes to this notebook.

[drewoldag]

"Using Learning Rate Schedulers" as the title - avoid the "LR".

In the first sentence, let's go with "... configure simple learning rate (LR) schedulers with ..."

I would recommend putting the last sentence in a "Note" comment toward the end of the notebook to avoid making the user wonder about it before they even get to the notebook.

Reply via ReviewNB

[drewoldag]

Second line: spelling error, "CIFAR10 dataset". Cut the "For more information" part.

Reply via ReviewNB

[drewoldag]

Instead of "Assuming everything is working properly", let's go with "By using the default learning rate scheduler, the resulting ..."

Reply via ReviewNB

[drewoldag]

Would it be worth doing this with a different scheduler? CosineAnnealing is popular https://docs.pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CosineAnnealingLR.html

Alternatively, perhaps we could do this in two sections.

1) Modifying parameters for the default scheduler

2) Defining a different scheduler

Reply via ReviewNB

[drewoldag]

Also please add the new notebook to the common_workflows.rst page. :)

[drewoldag]

Still worth changing the heading for this section I think. Perhaps "Specify Learning Rate Parameters"?

Reply via ReviewNB

[drewoldag]

Maybe a markdown section after this like:

"Now that we've updated the parameters, we'll retrain and infer with the model."

Reply via ReviewNB

[drewoldag]

I don't want to be overly picky, but can we add the regex "train" so that we only see the training results in the list? Also, I think that "Ignore outliers" might be selected, since you can't see the last two epochs for the gamma=0.9 run. Might be worth changing the colors to be more distinct, and calling out that orange is the default and red is the gamma=0.9 run.

I really like the way that you have the two training runs separated here. Definitely keep that. In the second Tensorboard screen shot, let's layer on the new CosineAnnealing as well. Similar to above, call out in a comment what run each color corresponds to.

Reply via ReviewNB

[drewoldag]

Hyrax offers support for more than just the ExponentialLR, here we demonstrate this with CosineAnnealingLR. Any other PyTorch learning rate scheduler can be used in the same way. The list of other schedulers can be seen here: https://docs.pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.LRScheduler.html.

Reply via ReviewNB

[drewoldag]

In the same way that you called out that the model performance was worse with gamma=0.9, you could do so here. Noting that the goal is to demonstrate the use of different learning rate schedulers more than to tune the model for optimal performance.

Reply via ReviewNB