预训练任务

分类

TLM : Translation Language Modeling

DAE: Denoising Autoencoder

CTL: Contrastive Learning

RTD： Replaced Token Detection

SOP：Sentence Order Prediction

DIM：Deep InfoMAx

参考

https://arxiv.org/pdf/2003.08271v4.pdf

https://zhuanlan.zhihu.com/p/360892229