Continual pre-training vs. Fine-tuning a language model with MLM

Question

I have some custom data I want to use to further pre-train the BERT model. I’ve tried the two following approaches so far:

1. Starting with a pre-trained BERT checkpoint and continuing the pre-training with Masked Language Modeling (MLM) + Next Sentence Prediction (NSP) heads (e.g. using BertForPreTraining model)
2. Starting with a pre-trained BERT model with the MLM objective (e.g. using the BertForMaskedLM model assuming we don’t need NSP for the pretraining part.)

But I’m still confused that if using either BertForPreTraining or BertForMaskedLM actually does the continual pre-training on BERT or these are just two models for fine-tuning that use MLM+NSP and MLM for fine-tuning BERT, respectively. Is there even any difference between fine-tuning BERT with MLM+NSP or continually pre-train it using these two heads or this is something we need to test?

I've reviewed similar questions such as this one but still, I want to make sure that whether technically there's a difference between continual pre-training a model from an initial checkpoint and fine-tuning it using the same objective/head.

Answer 1

The answer is a mere difference in the terminology used. When the model is trained on a large generic corpus, it is called 'pre-training'. When it is adapted to a particular task or dataset it is called as 'fine-tuning'.

Technically speaking, in either cases ('pre-training' or 'fine-tuning'), there are updates to the model weights.

For example, usually, you can just take the pre-trained model and then fine-tune it for a specific task (such as classification, question-answering, etc.). However, if you find that the target dataset is from a specific domain, and you have a few unlabled data that might help the model to adapt to the particular domain, then you can do a MLM or MLM+NSP 'fine-tuning' (unsupervised learning) (some researchers do call this as 'pre-training' especially when a huge corpus is used to train the model), followed by using the target corpus with target task fine-tuning.

Answer 2

Yes, there is a difference between pre-training and "further pre-training".

Pre-training usually would mean take the original model, initialize the weights randomly, and train the model from absolute scratch on some large corpora.

Further pre-training means take some already pre-trained model, and basically apply transfer learning - use the already saved weights from the trained model and train it on some new domain. This is usually beneficial if you don't have a very large corpora.

Regarding BertForPreTraining and BertForMaskedLM, you can use either one of them for both of the above purposes. It has been shown that further pre-training on MLM is very beneficial, and often NSP is not needed at all. So you will be good to go with BertForMaskedLM.

NB! You can initialize a model checkpoint by: BertForMaskedLM.from_pretrained({model_name}) and then apply the training procedure, otherwise just pass it a config (from the huggingface API)