SS23_IR: Question about exercise sheet 3

Hello all,

I have a question about exercise sheet 3 | Semantic Retrieval - Representation Learning | Task 3:

Calculate the output of the last layer (softmax layer) for the current sentence in which Sam is the center word.

How do I calculate "logits" and "y"?

Thanks a lot for your answers :)

von Daniel Weller - Freitag, 21. Juli 2023, 14:44

Hi Alexander,

logits:
u₁= 0,0625*0,64 + -0,5875*1,4 + -0,5225*-0,83 + 0,2225*-0,23 = -0.4

softmax:
y₁ = e^u1/ e^u1+ e^u2+ e^u3+ e^u4+ e^u5+ e^u6+e^u7+ e^u8y₁= e^-0,4/ e^-0,4+ e^1,6070+ ... + e^0,8874= 0,04

I found this page very helpful for understanding (point 3.2 Forward Propagation):
https://thinkinfi.com/continuous-bag-of-words-cbow-single-word-model-how-it-works/?utm_content=cmp-true

von gelöscht - Freitag, 21. Juli 2023, 15:26

Thank you very much!