大模型综述论文笔记6-15

这里写自定义目录标题

  • Keywords
  • [Backgroud for LLMs](#Backgroud for LLMs)
    • [Technical Evolution of GPT-series Models](#Technical Evolution of GPT-series Models)
      • [Research of OpenAI on LLMs can be roughly divided into the following stages](#Research of OpenAI on LLMs can be roughly divided into the following stages)
        • [Early Explorations](#Early Explorations)
        • [Capacity Leap](#Capacity Leap)
        • [Capacity Enhancement](#Capacity Enhancement)
        • [The Milestones of Language Models](#The Milestones of Language Models)
  • Resources
  • Pre-training
    • [Data Collection](#Data Collection)
    • [Data Preprocessing](#Data Preprocessing)

Keywords

GPT:Generative Pre-Training

Backgroud for LLMs

Technical Evolution of GPT-series Models

Two key points to GPT's success are (I) training decoder-onlly Transformer language models that can accurately predict the next word and (II) scaling up the size of language models

Research of OpenAI on LLMs can be roughly divided into the following stages

Early Explorations

Capacity Leap

ICT

Capacity Enhancement

1.training on code data

Codex: a GPT model fine-tuned on a large corpus of GitHub

code
2.alignment with human preference

reinforcement learning from human feedback (RLHF) algorithm

Note that it seems that the wording of "instruction tuning" has seldom

been used in OpenAI's paper and documentation, which is substituted by

supervised fine-tuning on human demonstrations (i.e., the first step

of the RLHF algorithm).

The Milestones of Language Models

chatGPT(based on gpt3.5 and gpt4) and GPT-4(multimodal)

Resources

Stanford Alpaca is the first open instruct-following model fine-tuned based on LLaMA (7B).

Alpaca LoRA (a reproduction of Stanford Alpaca using LoRA)

model 、data、library

Pre-training

Data Collection

General Text Data:webpages, books, and conversational text

Specialized Text Data:Multilingual text, Scientific text, Code

Data Preprocessing

Quality Filtering

  1. The former approach trains a selection classifier based on highquality texts and leverages it to identify and filter out low quality data.
  2. heuristic based approaches to eliminate low-quality texts through a set of well-designed rules: Language based filtering, Metric based filtering, Statistic based filtering, Keyword based filtering

De-duplication

Existing work has found that duplicate data in a corpus would reduce the diversity of language models, which may cause the training process to become unstable and thus affect the model performance.

  1. Privacy Redaction: (PII:personally identifiable information )
  2. Tokenization:(It aims to segment raw text into sequences of individual tokens, which are subsequently used as the inputs of LLMs.) Byte-Pair Encoding (BPE) tokenization; WordPiece tokenization; WordPiece tokenization
相关推荐
张较瘦_16 小时前
[论文阅读] 软件工程 | GPS算法:用“路径摘要”当向导,软件模型检测从此告别“瞎找bug”
论文阅读·算法·bug
Caaacy_YU19 小时前
多模态大模型研究每日简报【2025-08-25】
论文阅读·人工智能·深度学习·计算机视觉
berling002 天前
【论文阅读 | arXiv 2025 | WaveMamba:面向RGB-红外目标检测的小波驱动Mamba融合方法】
论文阅读·人工智能·目标检测
爱补鱼的猫猫2 天前
11、Informer论文笔记
论文阅读
勤劳的进取家2 天前
论文阅读:Inner Monologue: Embodied Reasoning through Planning with Language Models
论文阅读·人工智能·机器学习·语言模型·自然语言处理
有Li3 天前
CXR-LT 2024:一场关于基于胸部X线的长尾、多标签和零样本疾病分类的MICCAI挑战赛|文献速递-深度学习人工智能医疗图像
论文阅读·人工智能·算法·医学生
B612 little star king4 天前
UNIKGQA论文笔记
论文阅读·人工智能·笔记·自然语言处理·知识图谱
勤劳的进取家4 天前
论文阅读:Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
论文阅读·人工智能·机器学习·语言模型·自然语言处理
0x2115 天前
[论文阅读]Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Tools
论文阅读
张较瘦_5 天前
[论文阅读] 人工智能 + 软件工程 | 当AI成为文学研究员:Agentic DraCor如何用MCP解锁戏剧数据分析
论文阅读·人工智能·软件工程