Decoupled weight decay (#164)

* Add decoupled weight decay optimizers and helper class for optimizer
tests.

* Adapt README.md. Fix broken link to keras_utils.

* Remove TF private API calls.

* Add imports to __init__.py

* Fix indentation of comments, remove call to tf.test.main from optimizer_test_base.

* Move optimizer_test_base into weight_decay_test for now. In the optimizer tests, optimizer params are now keywords instead of a dict. Fix code in comments to support tf-2.0, naming errors, line length.

* Delete optimizer_test_base.py Remove keras object registration in the
factory function.

* Fix code formatting via patch file.

Author: PhilJd

tensorflow_addons/optimizers/BUILD

@@ -8,6 +8,7 @@ py_library(
         "__init__.py",
         "lazy_adam.py",
         "moving_average.py",
+        "weight_decay_optimizers.py",
     ],
     srcs_version = "PY2AND3",
     deps = [
@@ -40,3 +41,16 @@ py_test(
         ":optimizers",
     ],
 )
+
+py_test(
+    name = "weight_decay_optimizers_test",
+    size = "small",
+    srcs = [
+        "weight_decay_optimizers_test.py",
+    ],
+    main = "weight_decay_optimizers_test.py",
+    srcs_version = "PY2AND3",
+    deps = [
+        ":optimizers",
+    ],
+)

tensorflow_addons/optimizers/README.md

@@ -5,20 +5,23 @@
 |:---------- |:------------- |:--------------|
 | lazy_adam |  SIG-Addons | addons@tensorflow.org   |
 | moving_average | Dheeraj R. Reddy | dheeraj98reddy@gmail.com |
+| weight_decay_optimizers |  Phil Jund | ijund.phil@googlemail.com   |
+
 
 ## Components
 | Submodule | Optimizer  | Reference                                   |
-|:----------------------- |:---------------------- |:---------|
+|:--------- |:---------- |:---------|
 | lazy_adam | LazyAdam | https://arxiv.org/abs/1412.6980      |
 | moving_average | MovingAverage | |
+| weight_decay_optimizers | SGDW, AdamW, extend_with_decoupled_weight_decay | https://arxiv.org/pdf/1711.05101.pdf |
 
 
 ## Contribution Guidelines
 #### Standard API
 In order to conform with the current API standard, all optimizers
 must:
  * Inherit from either `keras.optimizer_v2.OptimizerV2` or its subclasses.
- * [Register as a keras global object](https://github.com/tensorflow/addons/blob/master/tensorflow_addons/utils/python/keras_utils.py)
+ * [Register as a keras global object](https://github.com/tensorflow/addons/blob/master/tensorflow_addons/utils/keras_utils.py)
   so it can be serialized properly.
  * Add the addon to the `py_library` in this sub-package's BUILD file.
 

tensorflow_addons/optimizers/__init__.py

@@ -19,4 +19,8 @@
 from __future__ import print_function
 
 from tensorflow_addons.optimizers.lazy_adam import LazyAdam
+from tensorflow_addons.optimizers.weight_decay_optimizers import AdamW
+from tensorflow_addons.optimizers.weight_decay_optimizers import SGDW
+from tensorflow_addons.optimizers.weight_decay_optimizers import (
+    extend_with_decoupled_weight_decay)
 from tensorflow_addons.optimizers.moving_average import MovingAverage