This repository was archived by the owner on Jul 1, 2024. It is now read-only.
support lr_scale to support scaled LR in head modules#569
Open
stephenyan1231 wants to merge 4 commits intofacebookresearch:mainfrom
Open
support lr_scale to support scaled LR in head modules#569stephenyan1231 wants to merge 4 commits intofacebookresearch:mainfrom
stephenyan1231 wants to merge 4 commits intofacebookresearch:mainfrom
Conversation
Differential Revision: D21949491 fbshipit-source-id: 1fb62f0280553cbbdf7194cbe857086fb9bb765e
Differential Revision: D22372256 fbshipit-source-id: a986a1f150374ef535b40e850f2c49a2fe780ac4
Differential Revision: D22618966 fbshipit-source-id: 294a9a23f4777601bdf577cdce61a086b167f61e
Summary: For fine-tuning xrayvideo 2019a model, we want to use small LR for trunk which is already pre-trained, and larger LR for heads which are randomly initialized. Thus, we store parameters from heads (and losses) in a separate parameter group, which use a `lr_scale` larger than default 1.0. Differential Revision: D22618972 fbshipit-source-id: 1011cff648761f3c9fc3e6b370b773799ca78296
facebook-github-bot
added
CLA Signed
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D22618972 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
For fine-tuning xrayvideo 2019a model, we want to use small LR for trunk which is already pre-trained, and larger LR for heads which are randomly initialized.
Thus, we store parameters from heads (and losses) in a separate parameter group, which use a
lr_scalelarger than default 1.0.Differential Revision: D22618972