Question 8: Sequence Length : t Number of Heads : h Embedding dimensi…

• Sequence Length : $t$
• Number of Heads : $h$
• Embedding dimension : $d_{\text{model}}$
• Input $X \in \mathbb{R}^{d_{\text{model}} \times t}$
• $d_k = d_q = \frac{d_{\text{model}}}{h}$
• $W_Q \in \mathbb{R}^{d_q \times d_{\text{model}}}$
• $W_K \in \mathbb{R}^{d_k \times d_{\text{model}}}$
• $W_V \in \mathbb{R}^{d_v \times d_{\text{model}}}$
• $W_o \in \mathbb{R}^{d_{\text{model}} \times (h \times d_v)}$

Based on the above data, answer the given subquestions.

Suppose $t = 32$, $d_{\text{model}} = 64$, $h = 2$ and $d_v = 16$. What will be the shape of the output of the scaled dot-product attention operation for a single head, given by:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T$

Compute the resulting output dimension and report the total number of elements in the resulting attention output.