目录
- Keywords
- Introduction
- [Backgroud for LLMs](#Backgroud for LLMs)
-
- [Scaling Laws for LLMs](#Scaling Laws for LLMs)
-
- [KM scaling law](#KM scaling law)
- [Chinchilla scaling law](#Chinchilla scaling law)
- [Emergent Abilities of LLMs](#Emergent Abilities of LLMs)
-
- [In-context learning](#In-context learning)
- [Instruction following](#Instruction following)
- [Step-by-step reasoning](#Step-by-step reasoning)
- [Key Techniques for LLMs](#Key Techniques for LLMs)
Keywords
PLMs:pre-trained language models
NLP:natural language processing
LLM:large language models
LM:language modeling
AI:artificial intelligence
SLM:statistical language models
NLM:Neural language models
RNNs:recurrent neural networks
ELMo:Embedding from Language Models
AGI:artificial general intelligence
ICL:In-context learning
Introduction
https://github.com/RUCAIBox/LLMSurvey
SLM
SLM 's basic idea is based on Markov assumption.The SLMs with a fixed context length n are also called n-gram language models.
瓶颈：维度问题，由于指数增长的转换概率需要计算，SLM无法准确估计高位语言模型
衍生：backoff estimation and Good-Tuning estimation, 用于解决数据稀疏的问题
NLM
通过神经网络来表征单词序列的概率问题。开启了用语言模型来做表征建模(representation learning, the beyond is word sequence modeling词序建模)
distributed representation of words
word prediction function conditioned on distributed word vectors
word2vec
PLM
ELMo通过bidirectional LSTM (biLSTM)网络捕获了上下文信息，并可以通过特定的下游任务进行fine-tuning.ELMo简介
BERT可以使用大规模的未标注数据进行特定的预训练任务
LLM
scaling PLMs(scaling model size or data size)
Three differences between PLMs and LLMs:
1.LLMs表现出在更小的PLMs中可能无法观察到的更惊人的能力
2.通过prompting interface来访问LLMs（eg：gpt-4 API）
3.LLMs的发展不需要明确区分以研究或是工程化为目的，LLMs的训练需要大数据处理和并行训练这些更实际的经验。
Backgroud for LLMs
LLMs refers to Transformer language models that contain hundreds of billions (or more) of params, which are trained on massive text data.
Scaling Laws for LLMs
LLMs 可以适配相同结构的transformer 并可以作为小模型的与训练模型
KM scaling law
通过 model size (N), dataset size (D), and the amount of training compute © 三个因素来衡量神经网络模型的表现
The three laws were derived by fitting the model performance with
varied data sizes (22M to 23B tokens), model sizes (768M to 1.5B
non-embedding parameters) and training compute,under some assumptions
(e.g., the analysis of one factor should be not bottlenecked by the
other two factors).
Chinchilla scaling law
.They conducted rigorous experiments by varying a larger range of
model sizes (70M to 16B) and data sizes (5B to 500B tokens) and fitted a similar
scaling law yet with different coefficients
the KM scaling law favors a larger budget allocation in model sizethan the data size, while the Chinchilla scaling law argues that the
two sizes should be increased in equal scales
问题：
However, some abilities (e.g., in-context learning) are
unpredictable according to the scaling law, which can be observed only
when the model size exceeds a certain level (as discussed below).
Emergent Abilities of LLMs
emergent abilities of LLMs are formally defined as "the abilities that
are not present in small models but arise in large models"
three typical emergent abilities for LLMs：
In-context learning
assuming that the language model has been provided with a natural
language instruction and/or several task demonstrations, it can
generate the expected output for the test instances by completing the
word sequence of input text, without requiring additional training or
gradient update
Instruction following
By fine-tuning with a mixture of multi-task datasets formatted via
natural language descriptions (called instruction tuning), LLMs are
shown to perform well on unseen tasks that are also described in the
form of instructions
Step-by-step reasoning
with the chain-of-thought (CoT) prompting strategy, LLMs can solve
such tasks by utilizing the prompting mechanism that involves
intermediate reasoning steps for deriving the final answer
Key Techniques for LLMs
Scaling
larger model/data sizes and more training compute typically lead to an improved model capacity
Training
To support distributed training, several optimization frameworks have been released to facilitate the implementation and deployment of parallel algorithms, such as DeepSpeed and Megatron-LM
Ability eliciting
These abilities might not be explicitly exhibited when LLMs perform some specific tasks.As the technical approach, it is useful to design suitable task instructions or specific in-context learning strategies to elicit such abilities
Alignment tuning
they are likely to generate toxic, biased, or even harmful content for humans. It is necessary to align LLMs with human values
InstructGPT designs an effective tuning approach that enables LLMs to follow the expected instructions, which utilizes the technique of reinforcement learning with human feedback
Tools manipulation
For example, LLMs can utilize the calculator for accurate computation and employ search engines to retrieve unknown information