Large-Scale Pretraining with Transformers

English Version mxnet
#1

https://d2l.ai/chapter_attention-mechanisms-and-transformers/large-pretraining-transformers.html

#2

Fig. 11.9.7.

Je suis malade = I am sick

#3

Yes.
And “He’s calm” = “il est calme” | “Elle court” = “She’s running”

#4

I’m pretty sure there will be some additions to this section about GPT-4 and I’m looking forward to reading that! While that happens, may I ask the admins to check whether the Figure with the description “Zero-shot, one-shot, few-shot in-context learning with language models…” is correctly depicting what’s happening in the Decoder when it says “(no parameter update).”

If I’m not misunderstanding the concepts, I think when we’re giving the pre-trained Decoder - a task, an example, and a prompt - and ask it to learn, thus fine-tune the model, we are indeed updating the Decoder parameters so that it “learns” how to translate sequences. Is that not true? What am I missing here?

#5

I think the decoder in Fig. 11.9.3 should start with a beginning of sentence tag (e.g. <bos>). At inference time, we won’t know what the first decoded token will be (currently it shows <X>), hence why we would start decoding with <bos>.

#6

I think the key is that, this doesn’t involving back propagating process.

#7

My solutions to the exs: 11.9

1 Like
#8

I might be misunderstanding, but shouldn’t the sentence:

Following the autoregressive language model training as described in Section 9.3.3, Fig. 11.9.6 illustrates GPT pretraining with a Transformer encoder, …

…say decoder?