MODEL: #MODEL field
framework: "RecognizerGCN" #Mandatory, indicate the type of network, associate to the 'paddlevideo/modeling/framework/' .
backbone: #Mandatory, indicate the type of backbone, associate to the 'paddlevideo/modeling/backbones/' .
name: "AGCN2s" #Mandatory, The name of backbone.
num_point: 25
num_person: 1
graph: "ntu_rgb_d"
graph_args:
labeling_mode: "spatial"
in_channels: 2
head:
name: "AGCN2sHead" #Mandatory, indicate the type of head, associate to the 'paddlevideo/modeling/heads'
num_classes: 60 #Optional, the number of classes to be classified.
in_channels: 64 #output the number of classes.
M: 1 #number of people.
DATASET: #DATASET field
batch_size: 64 #Mandatory, bacth size
num_workers: 4 #Mandatory, the number of subprocess on each GPU.
test_batch_size: 64
test_num_workers: 0
train:
format: "SkeletonDataset" #Mandatory, indicate the type of dataset, associate to the 'paddlevidel/loader/dateset'
file_path: "data/fsd10/FSD_train_data.npy" #Mandatory, train data index file path
label_path: "data/fsd10/FSD_train_label.npy"
test:
format: "SkeletonDataset" #Mandatory, indicate the type of dataset, associate to the 'paddlevidel/loader/dateset'
file_path: "data/fsd10/test_A_data.npy" #Mandatory, valid data index file path
test_mode: True
PIPELINE: #PIPELINE field
train: #Mandotary, indicate the pipeline to deal with the training data, associate to the 'paddlevideo/loader/pipelines/'
sample:
name: "AutoPadding"
window_size: 300
transform: #Mandotary, image transfrom operator
- SkeletonNorm:
test: #Mandotary, indicate the pipeline to deal with the training data, associate to the 'paddlevideo/loader/pipelines/'
sample:
name: "AutoPadding"
window_size: 300
transform: #Mandotary, image transfrom operator
- SkeletonNorm:
OPTIMIZER: #OPTIMIZER field
name: 'Momentum'
momentum: 0.9
learning_rate:
iter_step: True
name: 'CustomWarmupAdjustDecay'
step_base_lr: 0.1
warmup_epochs: 5
lr_decay_rate: 0.1
boundaries: [ 30, 40 ]
weight_decay:
name: 'L2'
value: 1e-4
use_nesterov: True
METRIC:
name: 'SkeletonMetric'
out_file: 'submission.csv'
INFERENCE:
name: 'STGCN_Inference_helper'
num_channels: 2
window_size: 350
vertex_nums: 25
person_nums: 1
model_name: "AGCN2s"
log_interval: 10 #Optional, the interal of logger, default:10
epochs: 50 #Mandatory, total epoch
save_interval: 10
针对 FSD 数据集，需要进行如下修改，才能与该配置文件相适配：
MODEL 字段
在 MODEL 字段中，将 num_classes 修改为 30，对应 FSD 数据集中的 30 个类别。
i n w a r d _ o r i _ i n d e x inward\_ori\_index inward_ori_index 是一个变量，用于存储 每个节点与向心节点的连接对 。向心节点是指 从其他节点指向该节点的节点 。例如，( 1 , 2 ) (1, 2) (1,2) 表示节点1与节点2相连，且节点2是向心节点 。这个变量是用于创建 NTU RGB-D 数据集对应的图结构的，其中每个节点代表一个人体关节，每条边代表一个人体骨骼。
i n w a r d _ o r i _ i n d e x inward\_ori\_index inward_ori_index 是根据下图定义的，左图显示了 Kinetics-Skeleton 数据集的关节标签，右图显示了 NTU-RGBD 数据集的关节标签(21是中心关节)。
File "/home/aistudio/work/FigureSkating/paddlevideo/modeling/backbones/agcn2s.py", line 218, in forward
x = self.data_bn(x)
ValueError: (InvalidArgument) ShapeError: the shape of scale must equal to [75]But received: the shape of scale is [50]
[Hint: Expected scale_dim[0] == C, but received scale_dim[0]:50 != C:75.] (at /paddle/paddle/phi/infermeta/multiary.cc:593)
def forward(self, x):
N, C, T, V, M = x.shape
print(N,C,T,V,M) # 打印
x = x.transpose([0, 4, 3, 1, 2]).reshape_([N, M * V * C, T])
x = self.data_bn(x)
...
DATASET: #DATASET field
batch_size: 64 #Mandatory, bacth size
num_workers: 4 #Mandatory, the number of subprocess on each GPU.
test_batch_size: 1
test_num_workers: 0
train:
format: "SkeletonDataset"
file_path: "/home/aistudio/work/dataset/train_data.npy"
label_path: "/home/aistudio/work/dataset/train_label.npy"
...
这是节点流的 DATASET 配置，骨骼流的 DATASET 配置也一模一样。
batch_size: 64，这和输出中的 N 的大小是 64 一致，说明每个批次确实是 64 个样本。
import os.path as osp
import copy
import random
import numpy as np
import pickle
from ..registry import DATASETS
from .base import BaseDataset
from ...utils import get_logger
logger = get_logger("paddlevideo")
@DATASETS.register()
class SkeletonDataset(BaseDataset):
"""
Skeleton dataset for action recognition.
The dataset loads skeleton feature, and apply norm operatations.
Args:
file_path (str): Path to the index file.
pipeline(obj): Define the pipeline of data preprocessing.
data_prefix (str): directory path of the data. Default: None.
test_mode (bool): Whether to bulid the test dataset. Default: False.
"""
def __init__(self, file_path, pipeline, label_path=None, test_mode=False):
self.label_path = label_path
super().__init__(file_path, pipeline, test_mode=test_mode)
def load_file(self):
"""Load feature file to get skeleton information."""
logger.info("Loading data, it will take some moment...")
self.data = np.load(self.file_path)
if self.label_path:
if self.label_path.endswith('npy'):
self.label = np.load(self.label_path)
elif self.label_path.endswith('pkl'):
with open(self.label_path, 'rb') as f:
sample_name, self.label = pickle.load(f)
else:
logger.info(
"Label path not provided when test_mode={}, here just output predictions."
.format(self.test_mode))
logger.info("Data Loaded!")
return self.data # used for __len__
def prepare_train(self, idx):
"""Prepare the feature for training/valid given index. """
results = dict()
results['data'] = copy.deepcopy(self.data[idx])
results['label'] = copy.deepcopy(self.label[idx])
results = self.pipeline(results)
return results['data'], results['label']
def prepare_test(self, idx):
"""Prepare the feature for test given index. """
results = dict()
results['data'] = copy.deepcopy(self.data[idx])
if self.label_path:
results['label'] = copy.deepcopy(self.label[idx])
results = self.pipeline(results)
return results['data'], results['label']
else:
results = self.pipeline(results)
return [results['data']]
ResourceExhaustedError:
Out of memory error on GPU 0.
Cannot allocate 976.562500MB memory on GPU 0,
31.607422GB memory has been allocated and available memory is only 144.500000MB.
ResourceExhaustedError:
Out of memory error on GPU 0.
Cannot allocate 488.281250MB memory on GPU 0,
31.472656GB memory has been allocated and available memory is only 282.500000MB.
File "/home/aistudio/work/FigureSkating/paddlevideo/solver/custom_lr.py", line 322, in step
self.last_epoch += 1 / self.num_iters # update step with iters
TypeError: unsupported operand type(s) for /: 'int' and 'NoneType'
问题定位，
python复制代码
class CustomWarmupAdjustDecay(LRScheduler):
r"""
We combine warmup and stepwise-cosine which is used in slowfast model.
Args:
step_base_lr (float): start learning rate used in warmup stage.
warmup_epochs (int): the number epochs of warmup.
lr_decay_rate (float|int, optional): base learning rate decay rate.
step (int): step in change learning rate.
last_epoch (int, optional): The index of last epoch. Can be set to restart training. Default: -1, means initial learning rate.
verbose (bool, optional): If ``True``, prints a message to stdout for each update. Default: ``False`` .
Returns:
``CosineAnnealingDecay`` instance to schedule learning rate.
"""
def __init__(self,
step_base_lr,
warmup_epochs,
lr_decay_rate,
boundaries,
num_iters=None,
last_epoch=-1,
verbose=False):
self.step_base_lr = step_base_lr
self.warmup_epochs = warmup_epochs
self.lr_decay_rate = lr_decay_rate
self.boundaries = boundaries
self.num_iters = num_iters
#call step() in base class, last_lr/last_epoch/base_lr will be update
super(CustomWarmupAdjustDecay, self).__init__(last_epoch=last_epoch,
verbose=verbose)
def step(self, epoch=None):
"""
``step`` should be called after ``optimizer.step`` . It will update the learning rate in optimizer according to current ``epoch`` .
The new learning rate will take effect on next ``optimizer.step`` .
Args:
epoch (int, None): specify current epoch. Default: None. Auto-increment from last_epoch=-1.
Returns:
None
"""
if epoch is None:
if self.last_epoch == -1:
self.last_epoch += 1
else:
self.last_epoch += 1 / self.num_iters # update step with iters
else:
self.last_epoch = epoch
self.last_lr = self.get_lr()
if self.verbose:
print('Epoch {}: {} set learning rate to {}.'.format(
self.last_epoch, self.__class__.__name__, self.last_lr))
def parse_args():
parser = argparse.ArgumentParser("PaddleVideo train script")
parser.add_argument('-c',
'--config',
type=str,
default='configs/example.yaml',
help='config file path')
parser.add_argument('-o',
'--override',
action='append',
default=[],
help='config options to be overridden')
parser.add_argument('--test',
action='store_true',
help='whether to test a model')
parser.add_argument('--train_dali',
action='store_true',
help='whether to use dali to speed up training')
parser.add_argument('--multigrid',
action='store_true',
help='whether to use multigrid training')
parser.add_argument('-w',
'--weights',
type=str,
help='weights for finetuning or testing')
parser.add_argument('--fleet',
action='store_true',
help='whether to use fleet run distributed training')
parser.add_argument('--amp',
action='store_true',
help='whether to open amp training.')
parser.add_argument(
'--validate',
action='store_true',
help='whether to evaluate the checkpoint during training')
args = parser.parse_args()
return args
import os
import re
import numpy as np
import csv
def softmax(X):
m = np.max(X, axis=1, keepdims=True)
exp_X = np.exp(X - m)
exp_X = np.exp(X)
prob = exp_X / np.sum(exp_X, axis=1, keepdims=True)
return prob
def is_Mb(file_name):
pattern = 'CTRGCN_Mb_fold\d+\.npy'
return re.match(pattern, file_name) is not None
output_prob = None
folder = './logits'
for logits_file in os.listdir(folder):
logits = np.load(os.path.join(folder, logits_file))
prob = softmax(logits)
if is_Mb(logits_file):
prob *= 0.7
if output_prob is None:
output_prob = prob
else:
output_prob = output_prob + prob
pred = np.argmax(output_prob, axis=1)
with open('./submission_ensemble.csv', 'w') as f:
writer = csv.writer(f)
writer.writerow(('sample_index', 'predict_category'))
for i, p in enumerate(pred):
writer.writerow((i, p))
configs 文件夹
里面是以下7种特征的配置 .yaml 文件：
Joint（J）的配置文件
ctrgcn_fsd_J_fold0.yaml
python复制代码
MODEL: #MODEL field
framework: "RecognizerGCN" #Mandatory, indicate the type of network, associate to the 'paddlevideo/modeling/framework/'.
backbone: #Mandatory, indicate the type of backbone, associate to the 'paddlevideo/modeling/backbones/' .
name: "CTRGCN" #Mandatory, The name of backbone.
in_channels: 2
head:
name: "CTRGCNHead" #Mandatory, indicate the type of head, associate to the 'paddlevideo/modeling/heads'
num_classes: 30 #Optional, the number of classes to be classified.
ls_eps: 0.1
DATASET: #DATASET field
batch_size: 16 #Mandatory, bacth size
num_workers: 2 #Mandatory, the number of subprocess on each GPU.
test_batch_size: 1
test_num_workers: 0
train:
format: "SkeletonDataset" #Mandatory, indicate the type of dataset, associate to the 'paddlevidel/loader/dateset'
file_path: "../dataset/train/J_fold0.npy" #Mandatory, train data index file path
label_path: "../dataset/train/fold0_label.npy"
valid:
format: "SkeletonDataset" #Mandatory, indicate the type of dataset, associate to the 'paddlevidel/loader/dateset'
file_path: "../dataset/valid/J_fold0.npy" #Mandatory, train data index file path
label_path: "../dataset/valid/fold0_label.npy"
test:
format: "SkeletonDataset" #Mandatory, indicate the type of dataset, associate to the 'paddlevidel/loader/dateset'
file_path: "../dataset/test/J.npy" #Mandatory, valid data index file path
test_mode: True
PIPELINE: #PIPELINE field
train: #Mandotary, indicate the pipeline to deal with the training data, associate to the 'paddlevideo/loader/pipelines/'
sample:
name: "UniformSampleFrames"
window_size: 350
transform: #Mandotary, image transfrom operator
- SkeletonNorm_J:
valid: #Mandotary, indicate the pipeline to deal with the training data, associate to the 'paddlevideo/loader/pipelines/'
sample:
name: "UniformSampleFrames"
window_size: 350
test_mode: True
transform: #Mandotary, image transfrom operator
- SkeletonNorm_J:
test: #Mandatory, indicate the pipeline to deal with the validing data. associate to the 'paddlevideo/loader/pipelines/'
sample:
name: "UniformSampleFrames"
window_size: 350
test_mode: True
transform: #Mandotary, image transfrom operator
- SkeletonNorm_J:
OPTIMIZER: #OPTIMIZER field
name: 'Momentum'
momentum: 0.9
learning_rate:
iter_step: False
name: 'CustomWarmupCosineDecay'
max_epoch: 90
warmup_epochs: 10
warmup_start_lr: 0.01
cosine_base_lr: 0.1
weight_decay:
name: 'L2'
value: 4e-4
METRIC:
name: 'SkeletonMetric'
out_file: 'submission.csv'
INFERENCE:
name: 'STGCN_Inference_helper'
num_channels: 5
window_size: 350
vertex_nums: 25
person_nums: 1
model_name: "CTRGCN_J_fold0"
save_interval: 10
val_interval: 1
log_interval: 20 #Optional, the interal of logger, default:10
epochs: 90 #Mandatory, total epoch
MODEL: #MODEL field
framework: "RecognizerGCN" #Mandatory, indicate the type of network, associate to the 'paddlevideo/modeling/framework/'.
backbone: #Mandatory, indicate the type of backbone, associate to the 'paddlevideo/modeling/backbones/' .
name: "CTRGCN" #Mandatory, The name of backbone.
in_channels: 9
head:
name: "CTRGCNHead" #Mandatory, indicate the type of head, associate to the 'paddlevideo/modeling/heads'
num_classes: 30 #Optional, the number of classes to be classified.
ls_eps: 0.1
DATASET: #DATASET field
batch_size: 16 #Mandatory, bacth size
num_workers: 2 #Mandatory, the number of subprocess on each GPU.
test_batch_size: 1
test_num_workers: 0
train:
format: "SkeletonDataset" #Mandatory, indicate the type of dataset, associate to the 'paddlevidel/loader/dateset'
file_path: "../dataset/train/JA_fold0.npy" #Mandatory, train data index file path
label_path: "../dataset/train/fold0_label.npy"
valid:
format: "SkeletonDataset" #Mandatory, indicate the type of dataset, associate to the 'paddlevidel/loader/dateset'
file_path: "../dataset/valid/JA_fold0.npy" #Mandatory, train data index file path
label_path: "../dataset/valid/fold0_label.npy"
test:
format: "SkeletonDataset" #Mandatory, indicate the type of dataset, associate to the 'paddlevidel/loader/dateset'
file_path: "../dataset/test/JA.npy" #Mandatory, valid data index file path
test_mode: True
PIPELINE: #PIPELINE field
train: #Mandotary, indicate the pipeline to deal with the training data, associate to the 'paddlevideo/loader/pipelines/'
sample:
name: "UniformSampleFrames"
window_size: 350
transform: #Mandotary, image transfrom operator
- SkeletonNorm_JA:
valid: #Mandotary, indicate the pipeline to deal with the training data, associate to the 'paddlevideo/loader/pipelines/'
sample:
name: "UniformSampleFrames"
window_size: 350
test_mode: True
transform: #Mandotary, image transfrom operator
- SkeletonNorm_JA:
test: #Mandatory, indicate the pipeline to deal with the validing data. associate to the 'paddlevideo/loader/pipelines/'
sample:
name: "UniformSampleFrames"
window_size: 350
test_mode: True
transform: #Mandotary, image transfrom operator
- SkeletonNorm_JA:
OPTIMIZER: #OPTIMIZER field
name: 'Momentum'
momentum: 0.9
learning_rate:
iter_step: False
name: 'CustomWarmupCosineDecay'
max_epoch: 90
warmup_epochs: 10
warmup_start_lr: 0.01
cosine_base_lr: 0.1
weight_decay:
name: 'L2'
value: 4e-4
METRIC:
name: 'SkeletonMetric'
out_file: 'submission.csv'
INFERENCE:
name: 'STGCN_Inference_helper'
num_channels: 5
window_size: 350
vertex_nums: 25
person_nums: 1
model_name: "CTRGCN_JA_fold0"
save_interval: 10
val_interval: 1
log_interval: 20 #Optional, the interal of logger, default:10
epochs: 90 #Mandatory, total epoch
print("This is my package.")
from .module1 import foo
from .module2 import bar
__all__ = ["foo", "bar"]
module1.py 文件的内容是：
bash复制代码
print("This is module1.")
def foo():
print("This is foo.")
module2.py 文件的内容是：
bash复制代码
print("This is module2.")
def bar():
print("This is bar.")
现在，如果你在 Python 解释器中输入：
python复制代码
>>> import my_package
你会看到输出：
bash复制代码
This is my package.
This is module1.
This is module2.
这说明，当你导入 my_package 这个包 时，它的 __init__.py 文件被隐式地执行 了，它打印了一句话，并且从 module1.py 和 module2.py 文件中导入了 f o o foo foo 和 b a r bar bar 这两个函数，并将它们添加到了 my_package 这个包的命名空间中 。所以，你可以直接使用 my_package.foo() 和 my_package.bar() 来调用这两个函数。
这说明，当你使用 from my_package import * 时，它只导入了 __all__ 中指定的名称，即 f o o foo foo 和 b a r bar bar 这两个函数，并将它们添加到了当前的命名空间 中。所以，你可以直接使用 foo() 和 bar() 来调用这两个函数。
registry.py
python复制代码
class Registry(object):
"""
The registry that provides name -> object mapping, to support third-party users' custom modules.
To register an object:
.. code-block:: python
BACKBONES = Registry('backbone')
@BACKBONES.register()
class ResNet:
pass
Or:
.. code-block:: python
BACKBONES = Registry('backbone')
class ResNet:
pass
BACKBONES.register(ResNet)
Usage: To build a module.
.. code-block:: python
backbone_name = "ResNet"
b = BACKBONES.get(backbone_name)()
"""
def __init__(self, name):
"""
Args:
name (str): the name of this registry
"""
self._name = name
self._obj_map = {}
def __contains__(self, key):
return self._obj_map.get(key) is not None
def _do_register(self, name, obj):
assert (
name not in self._obj_map
), "An object named '{}' was already registered in '{}' registry!".format(
name, self._name)
self._obj_map[name] = obj
def register(self, obj=None, name=None):
"""
Register the given object under the the name `obj.__name__`.
Can be used as either a decorator or not. See docstring of this class for usage.
"""
if obj is None:
# used as a decorator
def deco(func_or_class, name=name):
if name is None:
name = func_or_class.__name__
self._do_register(name, func_or_class)
return func_or_class
return deco
# used as a function call
if name is None:
name = obj.__name__
self._do_register(name, obj)
def get(self, name):
"""Get the registry record.
Args:
name (str): The class name.
Returns:
ret: The class.
"""
ret = self._obj_map.get(name)
if ret is None:
raise KeyError(
"No object named '{}' found in '{}' registry!".format(
name, self._name))
return ret
_do_register(self, name, obj)：私有方法，用来将一个对象 o b j obj obj 注册到一个名称 n a m e name name 上，如果该名称已经被注册过，就抛出断言错误。
register(self, obj=None, name=None)：公开方法，用来注册一个对象或者作为装饰器使用 。如果传入了 o b j obj obj 参数，就将它注册到 n a m e name name 参数指定的名称上（如果没有指定 n a m e name name 参数，就使用 obj.__name__ 作为名称）。如果没有传入 o b j obj obj 参数，就返回一个装饰器函数，用来装饰一个类或者函数，并将它注册到指定的名称上。
.. code-block:: python
backbone_name = "ResNet"
b = BACKBONES.get(backbone_name)()
这个类可以用来实现一种插件机制，让不同的模块可以向注册器中添加自己的对象，并通过名称来访问它们。
build_utils.py
python复制代码
def build(cfg, registry, key='name'):
"""Build a module from config dict.
Args:
cfg (dict): Config dict. It should at least contain the key.
registry (XXX): The registry to search the type from.
key (str): the key.
Returns:
obj: The constructed object.
"""
assert isinstance(cfg, dict) and key in cfg
cfg_copy = cfg.copy()
obj_type = cfg_copy.pop(key)
obj_cls = registry.get(obj_type)
if obj_cls is None:
raise KeyError('{} is not in the {} registry'.format(
obj_type, registry.name))
return obj_cls(**cfg_copy)
这段代码是定义了一个 b u i l d build build 函数，它的作用是根据一个配置字典和一个注册器，构建一个模块对象。它的参数和返回值如下：
c f g cfg cfg ( d i c t dict dict)：配置字典，它至少应该包含一个 k e y key key ，表示要构建的模块的类型。
c f g cfg cfg 字典可以有多个键 ，只要其中有一个键是 n a m e name name，用来指定要从注册器中获取的类 。其他的键和值都会作为参数传递给类的构造函数 。
例如，如果想要创建一个 T h i n g 3 Thing3 Thing3 的实例，而 T h i n g 3 Thing3 Thing3 的构造函数需要三个参数， a r g 1 arg1 arg1， a r g 2 arg2 arg2 和 a r g 3 arg3 arg3，可以使用以下代码：
这段代码创建一个名为 t h i n g thing thing 的注册器 ，然后向注册器中注册两个类 ， T h i n g 1 Thing1 Thing1 和 T h i n g 2 Thing2 Thing2（ T h i n g 1 Thing1 Thing1 和 T h i n g 2 Thing2 Thing2 是两个自定义的类），并给它们分别指定一个字符串作为键。
那么调用 build(cfg, registry) 就相当于调用 Thing1(arg1=1, arg2=2)（这是因为 c f g cfg cfg 中的 'name': 'Thing1' 指定了调用 b u i l d build build 要创建 T h i n g 1 Thing1 Thing1 类），并返回一个 T h i n g 1 Thing1 Thing1 的实例。
注册器是一个用于存储和查找类的容器 ，可以根据键来获取对应的类。
例如，如果想要创建一个 T h i n g 1 Thing1 Thing1 的实例，可以使用以下代码：thing1 = registry.get('Thing1')() 或者 thing1 = registry['Thing1']()。
如果想要创建 T h i n g 1 Thing1 Thing1 和 T h i n g 2 Thing2 Thing2 的实例，可以使用两个不同的 c f g cfg cfg 字典，分别指定 n a m e name name 键的值为 ′ T h i n g 1 ′ 'Thing1' ′Thing1′ 和 ′ T h i n g 2 ′ 'Thing2' ′Thing2′，然后分别调用 build(cfg, registry) 函数。例如，可以使用以下代码：
import os
import yaml
from paddlevideo.utils.logger import coloring, get_logger, setup_logger
__all__ = ['get_config']
logger = setup_logger("./", name="paddlevideo", level="INFO")
class AttrDict(dict):
def __getattr__(self, key):
return self[key]
def __setattr__(self, key, value):
if key in self.__dict__:
self.__dict__[key] = value
else:
self[key] = value
def create_attr_dict(yaml_config):
from ast import literal_eval
for key, value in yaml_config.items():
if type(value) is dict:
yaml_config[key] = value = AttrDict(value)
if isinstance(value, str):
try:
value = literal_eval(value)
except BaseException:
pass
if isinstance(value, AttrDict):
create_attr_dict(yaml_config[key])
else:
yaml_config[key] = value
def parse_config(cfg_file):
"""Load a config file into AttrDict"""
with open(cfg_file, 'r') as fopen:
yaml_config = AttrDict(yaml.load(fopen, Loader=yaml.SafeLoader))
create_attr_dict(yaml_config)
return yaml_config
def print_dict(d, delimiter=0):
"""
Recursively visualize a dict and
indenting acrrording by the relationship of keys.
"""
placeholder = "-" * 60
for k, v in sorted(d.items()):
if isinstance(v, dict):
logger.info("{}{} : ".format(delimiter * " ", coloring(k,
"HEADER")))
print_dict(v, delimiter + 4)
elif isinstance(v, list) and len(v) >= 1 and isinstance(v[0], dict):
logger.info("{}{} : ".format(delimiter * " ",
coloring(str(k), "HEADER")))
for value in v:
print_dict(value, delimiter + 4)
else:
logger.info("{}{} : {}".format(delimiter * " ",
coloring(k, "HEADER"),
coloring(v, "OKGREEN")))
if k.isupper():
logger.info(placeholder)
def print_config(config):
"""
visualize configs
Arguments:
config: configs
"""
print_dict(config)
def check_config(config):
"""
Check config
"""
pass
def override(dl, ks, v):
"""
Recursively replace dict of list
Args:
dl(dict or list): dict or list to be replaced
ks(list): list of keys
v(str): value to be replaced
"""
def str2num(v):
try:
return eval(v)
except Exception:
return v
assert isinstance(dl, (list, dict)), ("{} should be a list or a dict")
assert len(ks) > 0, ('lenght of keys should larger than 0')
if isinstance(dl, list):
k = str2num(ks[0])
if len(ks) == 1:
assert k < len(dl), ('index({}) out of range({})'.format(k, dl))
dl[k] = str2num(v)
else:
override(dl[k], ks[1:], v)
else:
if len(ks) == 1:
#assert ks[0] in dl, ('{} is not exist in {}'.format(ks[0], dl))
if not ks[0] in dl:
logger.warning('A new filed ({}) detected!'.format(ks[0], dl))
dl[ks[0]] = str2num(v)
else:
assert ks[0] in dl, (
'({}) doesn\'t exist in {}, a new dict field is invalid'.format(
ks[0], dl))
override(dl[ks[0]], ks[1:], v)
def override_config(config, options=None):
"""
Recursively override the config
Args:
config(dict): dict to be replaced
options(list): list of pairs(key0.key1.idx.key2=value)
such as: [
epochs=20',
'PIPELINE.train.transform.1.ResizeImage.resize_short=300'
]
Returns:
config(dict): replaced config
"""
if options is not None:
for opt in options:
assert isinstance(opt,
str), ("option({}) should be a str".format(opt))
assert "=" in opt, (
"option({}) should contain a ="
"to distinguish between key and value".format(opt))
pair = opt.split('=')
assert len(pair) == 2, ("there can be only a = in the option")
key, value = pair
keys = key.split('.')
override(config, keys, value)
return config
def get_config(fname, overrides=None, show=True):
"""
Read config from file
"""
assert os.path.exists(fname), ('config file({}) is not exist'.format(fname))
config = parse_config(fname)
override_config(config, overrides)
if show:
print_config(config)
check_config(config)
return config
import os
import os.path as osp
import time
import pickle
from tqdm import tqdm
import paddle
import paddle.nn.functional as F
from paddlevideo.utils import get_logger
from paddlevideo.utils import main_only
def pretrain_vit_param_trans(model, state_dicts, num_patches, seg_num, attention_type):
"""
Convert ViT's pre-trained model parameters to a parameter dictionary that matches the existing model
"""
if 'head' + '.weight' in state_dicts:
del state_dicts['head' + '.weight']
if 'head' + '.bias' in state_dicts:
del state_dicts['head' + '.bias']
total_len = len(model.state_dict())
if num_patches + 1 != state_dicts['pos_embed'].shape[1]:
pos_embed = state_dicts['pos_embed']
cls_pos_embed = pos_embed[0, 0, :].unsqueeze(0).unsqueeze(1)
other_pos_embed = pos_embed[0, 1:, :].unsqueeze(0).unsqueeze(1).transpose((0, 1, 3, 2))
new_pos_embed = F.interpolate(
other_pos_embed,
size=(other_pos_embed.shape[-2], num_patches),
mode='nearest'
)
new_pos_embed = new_pos_embed.squeeze(0).transpose((0, 2, 1))
new_pos_embed = paddle.concat((cls_pos_embed, new_pos_embed), axis=1)
state_dicts['pos_embed'] = new_pos_embed
time.sleep(0.01)
if 'time_embed' in state_dicts and seg_num != state_dicts['time_embed'].shape[1]:
time_embed = state_dicts['time_embed'].transpose((0, 2, 1)).unsqueeze(0)
new_time_embed = F.interpolate(
time_embed,
size=(time_embed.shape[-2], seg_num),
mode='nearest'
)
state_dicts['time_embed'] = new_time_embed.squeeze(0).transpose((0, 2, 1))
time.sleep(0.01)
with tqdm(total=total_len, position=1, bar_format='{desc}', desc="Loading weights") as desc:
if attention_type == 'divided_space_time':
new_state_dicts = state_dicts.copy()
for key in tqdm(state_dicts):
if 'blocks' in key and 'attn' in key:
desc.set_description("Loading %s" % key)
new_key = key.replace('attn', 'temporal_attn')
if not new_key in state_dicts:
new_state_dicts[new_key] = state_dicts[key]
else:
new_state_dicts[new_key] = state_dicts[new_key]
if 'blocks' in key and 'norm1' in key:
desc.set_description("Loading %s" % key)
new_key = key.replace('norm1', 'temporal_norm1')
if not new_key in state_dicts:
new_state_dicts[new_key] = state_dicts[key]
else:
new_state_dicts[new_key] = state_dicts[new_key]
time.sleep(0.01)
ret_str = "loading {:<20d} weights completed.".format(len(model.state_dict()))
desc.set_description(ret_str)
return new_state_dicts
#XXX(shipping): maybe need load N times because of different cards have different params.
@main_only
def load_ckpt(model,
weight_path,
**kargs):
"""
1. Load pre-trained model parameters
2. Extract and convert from the pre-trained model to the parameters
required by the existing model
3. Load the converted parameters of the existing model
"""
#model.set_state_dict(state_dict)
if not osp.isfile(weight_path):
raise IOError(f'{weight_path} is not a checkpoint file')
#state_dicts = load(weight_path)
logger = get_logger("paddlevideo")
state_dicts = paddle.load(weight_path)
if "VisionTransformer" in str(model): # For TimeSformer case
tmp = pretrain_vit_param_trans(model, state_dicts, kargs['num_patches'], kargs['seg_num'], kargs['attention_type'])
else:
tmp = {}
total_len = len(model.state_dict())
with tqdm(total=total_len, position=1, bar_format='{desc}', desc="Loading weights") as desc:
for item in tqdm(model.state_dict(), total=total_len, position=0):
name = item
desc.set_description('Loading %s' % name)
if name not in state_dicts: # Convert from non-parallel model
if str('backbone.' + name) in state_dicts:
tmp[name] = state_dicts['backbone.' + name]
else: # Convert from parallel model
tmp[name] = state_dicts[name]
time.sleep(0.01)
ret_str = "loading {:<20d} weights completed.".format(len(model.state_dict()))
desc.set_description(ret_str)
model.set_state_dict(tmp)
def mkdir(dir):
if not os.path.exists(dir):
# avoid error when train with multiple gpus
try:
os.makedirs(dir)
except:
pass
@main_only
def save(obj, path):
paddle.save(obj, path)
def load(file_name):
if not osp.isfile(file_name):
raise IOError(f'{file_name} not exist')
return paddle.load(file_name)
import paddle
import itertools
from paddlevideo.utils import get_logger
logger = get_logger("paddlevideo")
"""
Implement precise bn, which is useful for improving accuracy.
"""
@paddle.no_grad() # speed up and save CUDA memory
def do_preciseBN(model, data_loader, parallel, num_iters=200):
"""
Recompute and update the batch norm stats to make them more precise. During
training both BN stats and the weight are changing after every iteration, so
the running average can not precisely reflect the actual stats of the
current model.
In this function, the BN stats are recomputed with fixed weights, to make
the running average more precise. Specifically, it computes the true average
of per-batch mean/variance instead of the running average.
This is useful to improve validation accuracy.
Args:
model: the model whose bn stats will be recomputed
data_loader: an iterator. Produce data as input to the model
num_iters: number of iterations to compute the stats.
Return:
the model with precise mean and variance in bn layers.
"""
bn_layers_list = [
m for m in model.sublayers()
if any((isinstance(m, bn_type)
for bn_type in (paddle.nn.BatchNorm1D, paddle.nn.BatchNorm2D,
paddle.nn.BatchNorm3D))) and m.training
]
if len(bn_layers_list) == 0:
return
# moving_mean=moving_mean*momentum+batch_mean*(1.−momentum)
# we set momentum=0. to get the true mean and variance during forward
momentum_actual = [bn._momentum for bn in bn_layers_list]
for bn in bn_layers_list:
bn._momentum = 0.
running_mean = [paddle.zeros_like(bn._mean)
for bn in bn_layers_list] #pre-ignore
running_var = [paddle.zeros_like(bn._variance) for bn in bn_layers_list]
ind = -1
for ind, data in enumerate(itertools.islice(data_loader, num_iters)):
logger.info("doing precise BN {} / {}...".format(ind + 1, num_iters))
if parallel:
model._layers.train_step(data)
else:
model.train_step(data)
for i, bn in enumerate(bn_layers_list):
# Accumulates the bn stats.
running_mean[i] += (bn._mean - running_mean[i]) / (ind + 1)
running_var[i] += (bn._variance - running_var[i]) / (ind + 1)
assert ind == num_iters - 1, (
"update_bn_stats is meant to run for {} iterations, but the dataloader stops at {} iterations."
.format(num_iters, ind))
# Sets the precise bn stats.
for i, bn in enumerate(bn_layers_list):
bn._mean.set_value(running_mean[i])
bn._variance.set_value(running_var[i])
bn._momentum = momentum_actual[i]
from .builder import build_dataset, build_dataloader, build_batch_pipeline
from .dataset import SkeletonDataset
from .dali_loader import TSN_Dali_loader, get_input_data
__all__ = [
'build_dataset', 'build_dataloader', 'build_batch_pipeline', 'SkeletonDataset',
'TSN_Dali_loader', 'get_input_data'
]
skeleton.py
文件路径：loader/dataset/skeleton.py
python复制代码
import os.path as osp
import copy
import random
import numpy as np
import pickle
from ..registry import DATASETS
from .base import BaseDataset
from ...utils import get_logger
logger = get_logger("paddlevideo")
# #set random seed
# random.seed(0)
# np.random.seed(0)
# copyright (c) 2021 PaddlePaddle Authors. All Rights Reserve.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os.path as osp
import copy
import random
import numpy as np
import pickle
import gc
from ..registry import DATASETS
from .base import BaseDataset
from ...utils import get_logger
logger = get_logger("paddlevideo")
@DATASETS.register()
class KFoldSkeletonDataset(BaseDataset):
"""
Skeleton dataset for action recognition.
The dataset loads skeleton feature, and apply norm operatations.
Args:
file_path (str): Path to the index file.
pipeline(obj): Define the pipeline of data preprocessing.
data_prefix (str): directory path of the data. Default: None.
test_mode (bool): Whether to bulid the test dataset. Default: False.
"""
def __init__(self, index_path, pipeline, file_path='/home/zhaorifa/competition/home/aistudio/work/data/train_data_padd.npy',
label_path='/home/zhaorifa/competition/home/aistudio/work/data/train_label.npy', test_mode=False):
self.file_path = file_path
if not test_mode:
self.label_path = label_path
self.index_path = index_path
super().__init__(file_path, pipeline, test_mode=test_mode)
def load_file(self):
"""Load feature file to get skeleton information."""
logger.info("Loading data, it will take some moment...")
self.idx = np.load(self.index_path)
tmp_data = np.load(self.file_path)
self.data = tmp_data[self.idx]
del tmp_data
gc.collect()
if self.label_path:
if self.label_path.endswith('npy'):
self.label = np.load(self.label_path)
elif self.label_path.endswith('pkl'):
with open(self.label_path, 'rb') as f:
sample_name, self.label = pickle.load(f)
self.label = self.label[self.idx]
else:
logger.info(
"Label path not provided when test_mode={}, here just output predictions."
.format(self.test_mode))
logger.info("Data Loaded!")
return self.data # used for __len__
def prepare_train(self, idx):
"""Prepare the feature for training/valid given index. """
results = dict()
results['data'] = copy.deepcopy(self.data[idx])
results['label'] = copy.deepcopy(self.label[idx])
results = self.pipeline(results)
return results['data'], results['label']
def prepare_test(self, idx):
"""Prepare the feature for test given index. """
results = dict()
results['data'] = copy.deepcopy(self.data[idx])
if self.label_path:
results['label'] = copy.deepcopy(self.label[idx])
results = self.pipeline(results)
return results['data'], results['label']
else:
results = self.pipeline(results)
return [results['data']]
@DATASETS.register()
class SkeletonDataset(BaseDataset):
"""
Skeleton dataset for action recognition.
The dataset loads skeleton feature, and apply norm operatations.
Args:
file_path (str): Path to the index file.
pipeline(obj): Define the pipeline of data preprocessing.
data_prefix (str): directory path of the data. Default: None.
test_mode (bool): Whether to bulid the test dataset. Default: False.
"""
def __init__(self, file_path, pipeline, label_path=None, test_mode=False):
self.label_path = label_path
super().__init__(file_path, pipeline, test_mode=test_mode)
def load_file(self):
"""Load feature file to get skeleton information."""
logger.info("Loading data, it will take some moment...")
self.data = np.load(self.file_path)
if self.label_path:
if self.label_path.endswith('npy'):
self.label = np.load(self.label_path)
elif self.label_path.endswith('pkl'):
with open(self.label_path, 'rb') as f:
sample_name, self.label = pickle.load(f)
else:
logger.info(
"Label path not provided when test_mode={}, here just output predictions."
.format(self.test_mode))
logger.info("Data Loaded!")
return self.data # used for __len__
def prepare_train(self, idx):
"""Prepare the feature for training/valid given index. """
results = dict()
results['data'] = copy.deepcopy(self.data[idx])
results['label'] = copy.deepcopy(self.label[idx])
results = self.pipeline(results)
return results['data'], results['label']
def prepare_test(self, idx):
"""Prepare the feature for test given index. """
results = dict()
results['data'] = copy.deepcopy(self.data[idx])
if self.label_path:
results['label'] = copy.deepcopy(self.label[idx])
results = self.pipeline(results)
return results['data'], results['label']
else:
results = self.pipeline(results)
return [results['data']]
from .optimizer import build_optimizer
from .lr import build_lr
custom_lr.py
python复制代码
import math
from paddle.optimizer.lr import *
"""
PaddleVideo Learning Rate Schedule:
You can use paddle.optimizer.lr
or define your custom_lr in this file.
"""
class CustomWarmupCosineDecay(LRScheduler):
r"""
We combine warmup and stepwise-cosine which is used in slowfast model.
Args:
warmup_start_lr (float): start learning rate used in warmup stage.
warmup_epochs (int): the number epochs of warmup.
cosine_base_lr (float|int, optional): base learning rate in cosine schedule.
max_epoch (int): total training epochs.
num_iters(int): number iterations of each epoch.
last_epoch (int, optional): The index of last epoch. Can be set to restart training. Default: -1, means initial learning rate.
verbose (bool, optional): If ``True``, prints a message to stdout for each update. Default: ``False`` .
Returns:
``CosineAnnealingDecay`` instance to schedule learning rate.
"""
def __init__(self,
warmup_start_lr,
warmup_epochs,
cosine_base_lr,
max_epoch,
num_iters,
last_epoch=-1,
verbose=False):
self.warmup_start_lr = warmup_start_lr
self.warmup_epochs = warmup_epochs
self.cosine_base_lr = cosine_base_lr
self.max_epoch = max_epoch
self.num_iters = num_iters
#call step() in base class, last_lr/last_epoch/base_lr will be update
super(CustomWarmupCosineDecay, self).__init__(last_epoch=last_epoch,
verbose=verbose)
def step(self, epoch=None):
"""
``step`` should be called after ``optimizer.step`` . It will update the learning rate in optimizer according to current ``epoch`` .
The new learning rate will take effect on next ``optimizer.step`` .
Args:
epoch (int, None): specify current epoch. Default: None. Auto-increment from last_epoch=-1.
Returns:
None
"""
if epoch is None:
if self.last_epoch == -1:
self.last_epoch += 1
else:
self.last_epoch += 1 / self.num_iters # update step with iters
else:
self.last_epoch = epoch
self.last_lr = self.get_lr()
if self.verbose:
print('Epoch {}: {} set learning rate to {}.'.format(
self.last_epoch, self.__class__.__name__, self.last_lr))
def _lr_func_cosine(self, cur_epoch, cosine_base_lr, max_epoch):
return cosine_base_lr * (math.cos(math.pi * cur_epoch / max_epoch) +
1.0) * 0.5
def get_lr(self):
"""Define lr policy"""
lr = self._lr_func_cosine(self.last_epoch, self.cosine_base_lr,
self.max_epoch)
lr_end = self._lr_func_cosine(self.warmup_epochs, self.cosine_base_lr,
self.max_epoch)
# Perform warm up.
if self.last_epoch < self.warmup_epochs:
lr_start = self.warmup_start_lr
alpha = (lr_end - lr_start) / self.warmup_epochs
lr = self.last_epoch * alpha + lr_start
return lr
class CustomWarmupPiecewiseDecay(LRScheduler):
r"""
This op combine warmup and stepwise-cosine which is used in slowfast model.
Args:
warmup_start_lr (float): start learning rate used in warmup stage.
warmup_epochs (int): the number epochs of warmup.
step_base_lr (float|int, optional): base learning rate in step schedule.
max_epoch (int): total training epochs.
num_iters(int): number iterations of each epoch.
last_epoch (int, optional): The index of last epoch. Can be set to restart training. Default: -1, means initial learning rate.
verbose (bool, optional): If ``True``, prints a message to stdout for each update. Default: ``False`` .
Returns:
``CustomWarmupPiecewiseDecay`` instance to schedule learning rate.
"""
def __init__(self,
warmup_start_lr,
warmup_epochs,
step_base_lr,
lrs,
gamma,
steps,
max_epoch,
num_iters,
last_epoch=0,
verbose=False):
self.warmup_start_lr = warmup_start_lr
self.warmup_epochs = warmup_epochs
self.step_base_lr = step_base_lr
self.lrs = lrs
self.gamma = gamma
self.steps = steps
self.max_epoch = max_epoch
self.num_iters = num_iters
self.last_epoch = last_epoch
self.last_lr = self.warmup_start_lr # used in first iter
self.verbose = verbose
self._var_name = None
def step(self, epoch=None, rebuild=False):
"""
``step`` should be called after ``optimizer.step`` . It will update the learning rate in optimizer according to current ``epoch`` .
The new learning rate will take effect on next ``optimizer.step`` .
Args:
epoch (int, None): specify current epoch. Default: None. Auto-increment from last_epoch=-1.
Returns:
None
"""
if epoch is None:
if not rebuild:
self.last_epoch += 1 / self.num_iters # update step with iters
else:
self.last_epoch = epoch
self.last_lr = self.get_lr()
if self.verbose:
print('Epoch {}: {} set learning rate to {}.'.format(
self.last_epoch, self.__class__.__name__, self.last_lr))
def _lr_func_steps_with_relative_lrs(self, cur_epoch, lrs, base_lr, steps,
max_epoch):
# get step index
steps = steps + [max_epoch]
for ind, step in enumerate(steps):
if cur_epoch < step:
break
return lrs[ind - 1] * base_lr
def get_lr(self):
"""Define lr policy"""
lr = self._lr_func_steps_with_relative_lrs(
self.last_epoch,
self.lrs,
self.step_base_lr,
self.steps,
self.max_epoch,
)
lr_end = self._lr_func_steps_with_relative_lrs(
self.warmup_epochs,
self.lrs,
self.step_base_lr,
self.steps,
self.max_epoch,
)
# Perform warm up.
if self.last_epoch < self.warmup_epochs:
lr_start = self.warmup_start_lr
alpha = (lr_end - lr_start) / self.warmup_epochs
lr = self.last_epoch * alpha + lr_start
return lr
class CustomPiecewiseDecay(PiecewiseDecay):
def __init__(self, **kargs):
kargs.pop('num_iters')
super().__init__(**kargs)
import paddle
import paddle.nn as nn
from .base import BaseHead
from ..registry import HEADS
from ..weight_init import weight_init_
@HEADS.register()
class CTRGCNHead(BaseHead):
"""
Head for ST-GCN model.
Args:
in_channels: int, input feature channels. Default: 256.
num_classes: int, number classes. Default: 10.
"""
def __init__(self, in_channels=256, num_classes=10, **kwargs):
super().__init__(num_classes, in_channels, **kwargs)
def forward(self, x):
"""Define how the head is going to run.
"""
return x
cross_entropy_loss.py
文件路径：modeling/losses/cross_entropy_loss.py
python复制代码
import paddle
import paddle.nn.functional as F
from ..registry import LOSSES
from .base import BaseWeightedLoss
@LOSSES.register()
class CrossEntropyLoss(BaseWeightedLoss):
"""Cross Entropy Loss."""
def _forward(self, score, labels, **kwargs):
"""Forward function.
Args:
score (paddle.Tensor): The class score.
labels (paddle.Tensor): The ground truth labels.
kwargs: Any keyword argument to be used to calculate
CrossEntropy loss.
Returns:
loss (paddle.Tensor): The returned CrossEntropy loss.
"""
loss = F.cross_entropy(score, labels, **kwargs)
return loss
