The Dataset for Pretraining BERT

English Version mxnet
1

https://d2l.ai/chapter_natural-language-processing-pretraining/bert-dataset.html

2

In bert, replace the word with a random word instead of random number.

red line should be made:
vocab.to_tokens(random.randint(0, len(vocab) - 1))

image
image1151×408 38.5 KB

1 Like
3

image
image1077×429 45.1 KB

1 Like
4

I agree. Contribute if you like.
https://d2l.ai/chapter_appendix-tools-for-deep-learning/contributing.html
@goldpiggy

5

Hi @HeartSea15, feel free to post a PR as @StevenJokess suggested! Thanks!

6

in masked LM task, should input token whether to consider ## ? in our code, there’s no part of it.

image
image772×224 23.1 KB

7

hi, the value of max_mum_mlm_preds according to paper followed or what?
thanks reply.

image
image1279×454 37.6 KB

image

8

In BERT paper, https://arxiv.org/abs/1810.04805 Page7
image
But the deep reason of 15%?

9

Is the two 15% correspondence to the paper ???

image
image1337×753 217 KB

1 Like
10

I saw your dataset in the attachment. I also need to build the customized dataset to pretrain for our model. But I don’t understand your way a lot. Could you tell me more detail or provide some sample explanation about your dataset? I’m fresh in this area.