大模型基础
1、Attention Is All You Needhttps://arxiv.org/abs/1706.03762
2、Sequence to Sequence Learning with Neural Networks https://arxiv.org/abs/1409.3215
3、Neural Machine Translation by Jointly Learning to Align and Translate https://arxiv.org/abs/1409.0473
4、BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding https://arxiv.org/abs/1810.04805
5、Scaling Laws for Neural Language Models https://arxiv.org/pdf/2001.08361.pdf
6、Emergent Abilities of Large Language Models https://openreview.net/pdf?id=yzkSU5zdwD
Emergent Abilities of Large Language Models
7、Training Compute-Optimal Large Language Models (ChinChilla scaling law) https://arxiv.org/abs/2203.15556
8、Scaling Instruction-Finetuned Language Models https://arxiv.org/pdf/2210.11416.pdf
Direct Preference Optimization:
9、Your Language Model is Secretly a Reward Model https://arxiv.org/pdf/2305.18290.pdf
10、Progress measures for grokking via mechanistic interpretability https://arxiv.org/abs/2301.05217
11、Language Models Represent Space and Time https://arxiv.org/abs/2310.02207
12、GLaM: Efficient Scaling of Language Models with Mixture-of-Experts https://arxiv.org/abs/2112.06905
13、Adam: A Method for Stochastic Optimization https://arxiv.org/abs/1412.6980
14、Efficient Estimation of Word Representations in Vector Space (Word2Vec) https://arxiv.org/abs/1301.3781
15、Distributed Representations of Words and Phrases and their Compositionality https://arxiv.org/abs/1310.4546