Commit 3e1f0e53 authored by Yuxin Wu's avatar Yuxin Wu

update FAQ about freezing variables

parent cc2f9c12
...@@ -44,5 +44,14 @@ Therefore, if you want to load some model, just use the same variable name. ...@@ -44,5 +44,14 @@ Therefore, if you want to load some model, just use the same variable name.
If you want to re-train some layer, just rename it. If you want to re-train some layer, just rename it.
Unmatched variables on both sides will be printed as a warning. Unmatched variables on both sides will be printed as a warning.
To freeze some variables, there are [different ways](https://github.com/ppwwyyxx/tensorpack/issues/87#issuecomment-270545291) ## How to freeze some variables in training
with pros and cons.
1. You can simply use `tf.stop_gradient` in your model code in some situations (e.g. to freeze first several layers).
2. [varreplace.freeze_variables](http://tensorpack.readthedocs.io/en/latest/modules/tfutils.html#tensorpack.tfutils.varreplace.freeze_variables) can wrap some variables with `tf.stop_gradient`.
3. [ScaleGradient](http://tensorpack.readthedocs.io/en/latest/modules/tfutils.html#tensorpack.tfutils.gradproc.ScaleGradient) can be used to set the gradients of some variables to 0.
Note that the above methods only prevent variables being updated by SGD.
Some variables may be updated by other means,
e.g., BatchNorm statistics are updated through the `UPDATE_OPS` collection and the [RunUpdateOps](http://tensorpack.readthedocs.io/en/latest/modules/callbacks.html#tensorpack.callbacks.RunUpdateOps) callback.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment