Notes

1-1 - classification of images
1 to many - image capturing
many to one - actio predictio
many to many - language trans
many to many - frame classification

LSTM has 2 output
GRU doesn't have state

encder - decoder - lang trans
last state for encoder first state for decoder
context vector - weighted sum of all hidden layer
i/p seq is bottlenck but get problem if we have big seq
ans - use context vec at each step of decoder
HOW ?
1) finad initial score
We compute alignment score from initial decoder to final hidden
attention weights - proability
instead of weighted sum; we put more weighyt / attentio to more related part of sentence
apply softmax to get attention score

in decoder attentio is taken as a query --> the hidden state is starting to find it or decode it
attention - permutation invariant , doesn't care abt ordering

in softmax pro goes down
i/p + hidden calculate alignment -> use softmax to get attention --> output
from i/p vec we can cal : 1) Key vec & 2) Value vec
if we put query key and value as i/p vec then its called self-attention layer

Self att do not care abt order --> So, do positiojnal encoding
position enc - encode position of vec at positins

Masked self-attention : we do not have the data for the future ( e.g. stock market prediction, text summarization )
It manually set alignment scores to -infinity
Multi-head self attention : to learn diff features diff layer or itteration

CNN with Self attention
``````````````````````````````````````
residual conn : help to address the vanishing gradient problem and improve the performance of the network


Batch normalization normalizes each feature independently across the mini-batch. Layer normalization normalizes each of the inputs in the batch independently across all features.