I have a 1GB raw text dataset in a niche domain. I want to train an ELECTRA model but I couldn’t find any tutorial/examples to do so. Can anyone help me? I tried using the simpletransformers package, but it has memory issues at the moment and after a few epochs my colab session crashes.

Do you want to fine-tune or pre-train an ELECTRA model? If you want to fine-tune it, you can leverage the examples/run_language_modeling.py script. If you want to pre-train it, your best bet is to use the original implementation (in TF1) and then convert it to our library using our conversion scrip…

@lysandre I am planning to pretrain the model, but I would like to also give a shot to examples/run_language_modeling.py script. How can I finetune the base Electra model on MLM task on my data? The script is a bit ambigious.

I’m not sure if fine-tuning ELECTRA with MLM is good idea since the main idea behind ELECTRA was to train it as a discriminator rather than generator and overcome the problems of MLM’s Any thoughts @lysandre

[image] ELECTRA training reimplementation and discussion Research After months of development and debugging, I finally successfully train a model from scratch and replicate the official results. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators …

How pretrain ELECTRA on custom dataset?

Beginners

valhalla September 6, 2020, 10:53am 6

Topic		Replies	Views
ELECTRA training reimplementation and discussion Research	14	6950	September 17, 2023
PreTrain ELECTRA from scratch in Portuguese Flax/JAX Projects	2	1389	November 8, 2021
How hard is it to finetune an ELECTRA model for multihead regression & classification? 🤗Transformers	0	332	October 15, 2021
ELECTRA: Accounting for mask tokens that are correctly predicted by MLM 🤗Transformers	9	1373	May 15, 2021
Retrain Electra model with different embedings from scratch Beginners	0	386	April 1, 2022

How pretrain ELECTRA on custom dataset?

Related topics