解决ValueError: too many values to unpack (expected 2)

报错: ValueError: too many values to unpack (expected 2)

复现RGCL 模型,遇到的问题,该代码里面报错原因是: collate_fn 里的 tokens 是多层嵌套列表 (每个样本是分词后的 token 列表,batch 打包后变成 [[token1,token2...], [xxx]]),但旧版 transformers(3.1.0)在 is_pretokenized=True 时内部解包逻辑不兼容,或者数据里混入了非法空样本、维度错乱。

解决方案

原 collate_fn:

复制代码
def collate_fn(data):
    u, i, r, tokens = zip(*data)
    encoding = bert_tokenizer(tokens, return_tensors='pt', padding=True,
                              truncation=True, is_pretokenized=True)
    return torch.Tensor(u), torch.Tensor(i), torch.Tensor(r), \
        encoding['input_ids'], encoding['attention_mask']

改成下面版本,去掉 is_pretokenized=True,改用字符串拼接 / 直接传原文本,彻底规避解包 bug:

复制代码
def collate_fn(data):
    u, i, r, token_lists = zip(*data)
    # 把分词列表还原成字符串,不用is_pretokenized
    text_list = [" ".join(toks) for toks in token_lists]
    encoding = bert_tokenizer(text_list, 
                             return_tensors='pt', 
                             padding=True,
                             truncation=True,
                             max_length=args.review_max_length)
    return torch.Tensor(u), torch.Tensor(i), torch.Tensor(r), \
        encoding['input_ids'], encoding['attention_mask']

infact,在修改代码之前,我还多次尝试改变transformers的版本但是还是报一样的错误,可能是没有改到正好可以用的那个版本。(ps.之前跑自己的模型也遇到过版本问题,修改transformers的版本就重新运行了)

完整报错:

复制代码
(base) root@autodl-container-6d25459816-01b42b10:~/Work/ReviewGraph-main/BERT# python bert_whitening.py

libgomp: Invalid value for environment variable OMP_NUM_THREADS
2026-07-06 00:23:43 - Load_Data - Start reading data to pandas.
Clean string: 100%|███████████████████████████████████████████████████████████████████████| 963441/963441 [01:36<00:00, 9971.91it/s]
Delete unused words: 100%|██████████████████████████████████████████████████████████████| 963441/963441 [00:08<00:00, 107299.97it/s]
2026-07-06 00:25:55 - Load_Data - Truncate review length to 56 words
Delete unused words: 100%|██████████████████████████████████████████████████████████████| 963441/963441 [00:03<00:00, 255554.45it/s]
check data split: 770753it [00:33, 23069.12it/s]
/root/Work/ReviewGraph-main/BERT/../load_data.py:190: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  train_data = train_data.append([valid_data.loc[valid_drop_user_data_index],
pre tokenize: 100%|███████████████████████████████████████████████████████████████████████| 973763/973763 [09:04<00:00, 1789.32it/s]
  0%|                                                                                                      | 0/7608 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "bert_whitening.py", line 175, in <module>
    save_sentence_feat(args)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "bert_whitening.py", line 149, in save_sentence_feat
    for u, i, r, input_ids, mask in tqdm(data_loader):
  File "/root/miniconda3/lib/python3.8/site-packages/tqdm/std.py", line 1185, in __iter__
    for obj in iterable:
  File "/root/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 530, in __next__
    data = self._next_data()
  File "/root/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 570, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/root/miniconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "bert_whitening.py", line 100, in collate_fn
    encoding = bert_tokenizer(tokens, return_tensors='pt', padding=True,
  File "/root/miniconda3/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 3021, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/root/miniconda3/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 3109, in _call_one
    return self.batch_encode_plus(
  File "/root/miniconda3/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 3311, in batch_encode_plus
    return self._batch_encode_plus(
  File "/root/miniconda3/lib/python3.8/site-packages/transformers/tokenization_utils.py", line 886, in _batch_encode_plus
    ids, pair_ids = ids_or_pair_ids
ValueError: too many values to unpack (expected 2)
相关推荐
吴佳浩4 小时前
我把 HAMi 摆回一台双卡机器上跑了一遍:GPU 编排的性能账,到底记在谁头上
人工智能·gpu
少司府8 小时前
嵌入式外设篇(二)GPIO·二:GPIO输出与LED控制
linux·stm32·单片机·嵌入式硬件·mcu·51单片机·系统
forest_098 小时前
《AI 编程 Harness 实战手册》04|审批这道门:点了一百次“允许”,还剩多少作用?
人工智能·算法·prompt·agi
Rabitebla9 小时前
【Linux操作系统】 文件篇(一):文件描述符与 Linux 文件管理机制
linux·网络·c++·笔记·学习
火山引擎开发者社区9 小时前
当 RPA 遇上 Agent:MUA 混合执行 Workflow 深度拆解
人工智能
严肃小白兔10 小时前
第 10/24 期:深度神经网络的自动求导机制与梯度验证
人工智能·神经网络·dnn
火山引擎开发者社区10 小时前
定档12.16-12.18!FORCE原动力大会上海见!
人工智能
努力的MM12310 小时前
CAIE 二级和阿里云大模型 ACP 怎么选?企业 AI 工程与云平台开发能力对比
人工智能
老金带你玩AI10 小时前
Kling Image 4.0来了!8K和组图直出,电商人的提效神器?
人工智能