Home Arrow Icon Knowledge base Arrow Icon Global Arrow Icon How do I handle lazy variable creation in Sonnet modules


How do I handle lazy variable creation in Sonnet modules


To handle lazy variable creation in Sonnet modules, you can follow these steps:

1. Define variables as attributes of the module class, but do not initialize them in the `__init__` method. For example:

python
class MyModule(snt.Module):
    def __init__(self, output_size, name=None):
        super().__init__(name=name)
        self.output_size = output_size
        self.w = None
        self.b = None

2. Initialize the variables lazily in the `__call__` method, the first time the module is used. Check if the variables are `None` and create them if needed:

python
def __call__(self, x):
    if self.w is None:
        self.w = tf.Variable(tf.random.normal([x.shape[-1], self.output_size]), name="w")
    if self.b is None:
        self.b = tf.Variable(tf.zeros([self.output_size]), name="b")
    return tf.matmul(x, self.w) + self.b

3. Access the variables using the `trainable_variables` property to get all trainable variables owned by the module and its submodules:

python
variables = self.trainable_variables

4. If you want to access variables before they are created, you can use the `allow_empty_variables` context manager to suppress the exception raised when there are no variables:

python
with snt.allow_empty_variables():
    variables = self.trainable_variables

By following these steps, you can create variables lazily in Sonnet modules, initializing them only when they are first used. This can help reduce memory usage and speed up module initialization if you have many variables that are not needed in every forward pass.

Remember to use the `trainable_variables` property to access the variables, as it recursively includes variables from submodules as well. The `allow_empty_variables` context manager is useful for accessing variables before they are created, such as during module construction.

Citations:
[1] https://sonnet.readthedocs.io/en/latest/api.html
[2] https://github.com/google-deepmind/sonnet/issues/69
[3] https://en.wikipedia.org/wiki/Lazy_initialization
[4] https://www.sonnetsoftware.com/support/help-17/Sonnet_Suites/..%5Cusers_guide/HowtoCreateaVariable.html
[5] https://jsonnet.org/learning/tutorial.html