What is the role of memory_sequence_length in Bahdanau attention

When `memory_sequence_length` is provided, the attention mechanism will mask the memory tensor rows with zeros for values past the respective sequence lengths. This ensures that the attention mechanism only considers the relevant parts of the input sequence, which can improve performance and prevent the model from attending to irrelevant parts of the input sequence.

Here is an example of how to use `memory_sequence_length` in Bahdanau attention:

python
import tensorflow as tf
from tensorflow_addons.seq2seq import BahdanauAttention

# Define the Bahdanau attention object
attnUnits = 128
attention = BahdanauAttention(
    units=attnUnits,
    memory=encoderOutput,
    memory_sequence_length=encoderSequenceLength,
    normalize=True,
    probability_fn='softmax',
    kernel_initializer='glorot_uniform',
    dtype=tf.float32,
    name='BahdanauAttention'
)

In this example, `encoderSequenceLength` is a tensor that contains the sequence lengths for each batch entry in the `encoderOutput` tensor. The `BahdanauAttention` object will use these sequence lengths to mask the `encoderOutput` tensor before computing the attention scores.

Benefits of Using `memory_sequence_length`

2. Reduced Computational Cost: By avoiding unnecessary computations on the memory tensor, the attention mechanism can reduce its computational cost, making it more efficient.

3. Better Interpretability: The use of `memory_sequence_length` can provide more interpretable results by highlighting the relevant parts of the input sequence that the model is attending to.

Example Usage

python
# Example usage
encoderOutput = tf.constant([
    [[1, 2, 3], [4, 5, 6]],
    [[7, 8, 9], [10, 11, 12]]
])
encoderSequenceLength = tf.constant([2, 2])

# Define the Bahdanau attention object
attnUnits = 128
attention = BahdanauAttention(
    units=attnUnits,
    memory=encoderOutput,
    memory_sequence_length=encoderSequenceLength,
    normalize=True,
    probability_fn='softmax',
    kernel_initializer='glorot_uniform',
    dtype=tf.float32,
    name='BahdanauAttention'
)

# Compute the attention scores
attentionScores = attention(encoderOutput)

print(attentionScores)

In this example, the `BahdanauAttention` object is used to compute the attention scores for the `encoderOutput` tensor, which is masked based on the `encoderSequenceLength` tensor. The output will be a tensor where the attention scores are computed only for the relevant parts of the input sequence, based on the sequence lengths provided.