Deep Learning with PyTorch: Tensors (张量)

Deep Learning with PyTorch: Tensors {张量}

[1. Tensor Initialization](#1. Tensor Initialization)
[2. Tensor Attributes](#2. Tensor Attributes)
[3. Tensor Operations](#3. Tensor Operations)
[4. Bridge with NumPy](#4. Bridge with NumPy)
- [4.1. Tensor to NumPy array](#4.1. Tensor to NumPy array)
- [4.2. NumPy array to Tensor](#4.2. NumPy array to Tensor)
References

https://docs.pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html

Tensors are a specialized data structure that are very similar to arrays and matrices. In PyTorch, we use tensors to encode the inputs and outputs of a model, as well as the model's parameters.

Tensors are similar to NumPy's ndarrays, except that tensors can run on GPUs or other specialized hardware to accelerate computing.

复制代码

blitz [blɪts]
n. 闪电战；集中力量的行动；闪击式行动；突袭
adj. 闪电战的
v. 用闪电战空袭或毁坏

1. Tensor Initialization

Tensors can be initialized in various ways.

0 维张量：标量

1 维张量：向量

2 维张量：矩阵

3 维张量：图像 (height, width, channel)

4 维张量：视频 (batch, height, width, channel)

Directly from data

Tensors can be created directly from data.

复制代码

# !/usr/bin/env python
# coding=utf-8

import torch
import numpy as np

data = [[1, 2], [3, 4]]
x_data = torch.tensor(data, dtype=torch.float32)

print(f"Shape of tensor: {x_data.shape}")
print(f"Datatype of tensor: {x_data.dtype}")
print(f"Device tensor is stored on: {x_data.device}")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py 
Shape of tensor: torch.Size([2, 2])
Datatype of tensor: torch.float32
Device tensor is stored on: cpu

Process finished with exit code 0

From a NumPy array

Tensors can be created from NumPy arrays.

复制代码

# !/usr/bin/env python
# coding=utf-8

import torch
import numpy as np

data = [[1, 2], [3, 4]]
np_array = np.array(data, dtype=np.float32)
x_np = torch.from_numpy(np_array)

print(f"Shape of tensor: {x_np.shape}")
print(f"Datatype of tensor: {x_np.dtype}")
print(f"Device tensor is stored on: {x_np.device}")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py 
Shape of tensor: torch.Size([2, 2])
Datatype of tensor: torch.float32
Device tensor is stored on: cpu

Process finished with exit code 0

From another tensor

The new tensor retains the properties (shape, data type) of the argument tensor, unless explicitly overridden.

复制代码

# !/usr/bin/env python
# coding=utf-8

import torch
import numpy as np

data = [[1, 2], [3, 4]]
x_data = torch.tensor(data, dtype=torch.float32)

x_ones = torch.ones_like(x_data)  # retains the properties of x_data
print(f"Ones Tensor: \n {x_ones} \n")

x_rand = torch.rand_like(x_data, dtype=torch.float)  # overrides the datatype of x_data
print(f"Random Tensor: \n {x_rand} \n")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py 
Ones Tensor: 
 tensor([[1., 1.],
        [1., 1.]]) 

Random Tensor: 
 tensor([[0.4697, 0.2215],
        [0.9592, 0.5815]]) 


Process finished with exit code 0

With random or constant values

shape is a tuple of tensor dimensions. In the functions below, it determines the dimensionality of the output tensor.

复制代码

# !/usr/bin/env python
# coding=utf-8

import torch
import numpy as np

shape = (2, 3)
rand_tensor = torch.rand(shape, dtype=torch.float32)
ones_tensor = torch.ones(shape, dtype=torch.int32)
zeros_tensor = torch.zeros(shape, dtype=torch.float32)

print(f"Random Tensor: \n {rand_tensor} \n")
print(f"Ones Tensor: \n {ones_tensor} \n")
print(f"Zeros Tensor: \n {zeros_tensor}")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py 
Random Tensor: 
 tensor([[0.0894, 0.9370, 0.5483],
        [0.8597, 0.1212, 0.4357]]) 

Ones Tensor: 
 tensor([[1, 1, 1],
        [1, 1, 1]], dtype=torch.int32) 

Zeros Tensor: 
 tensor([[0., 0., 0.],
        [0., 0., 0.]])

Process finished with exit code 0

2. Tensor Attributes

Tensor attributes describe their shape, data type, and the device on which they are stored.

复制代码

# !/usr/bin/env python
# coding=utf-8

import torch
import numpy as np

shape = (2, 3)
rand_tensor = torch.rand(shape, dtype=torch.float32)

print(f"Random Tensor: \n {rand_tensor} \n")
print(f"Shape of tensor: {rand_tensor.shape}")
print(f"Datatype of tensor: {rand_tensor.dtype}")
print(f"Device tensor is stored on: {rand_tensor.device}")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py 
Random Tensor: 
 tensor([[0.9206, 0.1489, 0.6908],
        [0.9211, 0.9240, 0.9722]]) 

Shape of tensor: torch.Size([2, 3])
Datatype of tensor: torch.float32
Device tensor is stored on: cpu

Process finished with exit code 0

可以通过 torch.zeros() 构造一个矩阵全为 0，并且通过 dtype 设置数据类型。还可以通过 torch.zero_() 和 torch.zeros_like() 将现有矩阵转换为全 0 矩阵。

复制代码

# !/usr/bin/env python
# coding=utf-8

import torch
import numpy as np

x = torch.ones(1, 2, dtype=torch.double)
print(f"x: \n {x} \n")

x = x.new_ones(2, 3, dtype=torch.float32)
print(f"x: \n {x} \n")

y = torch.randn_like(x, dtype=torch.float32)
print(f"y: \n {y} \n")

print(f"y.size(): {y.size()}")
print(f"y.shape: {y.shape}")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py 
x: 
 tensor([[1., 1.]], dtype=torch.float64) 

x: 
 tensor([[1., 1., 1.],
        [1., 1., 1.]]) 

y: 
 tensor([[-0.5393,  0.2279,  0.6721],
        [ 0.8600,  1.0513,  1.0447]]) 

y.size(): torch.Size([2, 3])
y.shape: torch.Size([2, 3])

Process finished with exit code 0

3. Tensor Operations

Over 100 tensor operations, including transposing, indexing, slicing, mathematical operations, linear algebra, random sampling, and more are comprehensively described here (https://docs.pytorch.org/docs/stable/torch.html).

Each of them can be run on the GPU (at typically higher speeds than on a CPU).

复制代码

# !/usr/bin/env python
# coding=utf-8

import torch
import numpy as np

shape = (2, 3)
rand_tensor = torch.rand(shape, dtype=torch.float32)

print(f"Random Tensor: \n {rand_tensor} \n")
print(f"Shape of tensor: {rand_tensor.shape}")
print(f"Datatype of tensor: {rand_tensor.dtype}")
print(f"Device tensor is stored on: {rand_tensor.device}")

# We move our tensor to the GPU if available
if torch.cuda.is_available():
    tensor = rand_tensor.to('cuda')
    print(f"Device tensor is stored on: {tensor.device}")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py 
Random Tensor: 
 tensor([[0.5417, 0.3594, 0.4347],
        [0.7247, 0.6159, 0.4751]]) 

Shape of tensor: torch.Size([2, 3])
Datatype of tensor: torch.float32
Device tensor is stored on: cpu
Device tensor is stored on: cuda:0

Process finished with exit code 0

Standard numpy-like indexing and slicing

!/usr/bin/env python

coding=utf-8

import torch
import numpy as np

ones_tensor = torch.ones(4, 4, dtype=torch.float32)
ones_tensor[:, 1] = 0

print(f"Ones Tensor: \n {ones_tensor} \n")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py
Ones Tensor
tensor([[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.]])

Process finished with exit code 0

索引出来的结果与原数据共享内存，修改一个，另一个会跟着修改。如果不想修改，可以考虑使用 copy() 等方法。

复制代码

# !/usr/bin/env python
# coding=utf-8

import torch
import numpy as np

x = torch.rand(2, 3, dtype=torch.float32)
print(f"x: \n {x} \n")
y = x[0, :]
print(f"y: \n {y} \n")

y += 1
print(f"x: \n {x} \n")
print(f"y: \n {y} \n")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py 
x: 
 tensor([[0.4515, 0.7297, 0.3031],
        [0.7768, 0.8343, 0.0476]]) 

y: 
 tensor([0.4515, 0.7297, 0.3031]) 

x: 
 tensor([[1.4515, 1.7297, 1.3031],
        [0.7768, 0.8343, 0.0476]]) 

y: 
 tensor([1.4515, 1.7297, 1.3031]) 


Process finished with exit code 0

Joining tensors

You can use torch.cat to concatenate a sequence of tensors along a given dimension.

复制代码

# !/usr/bin/env python
# coding=utf-8

import torch
import numpy as np

ones_tensor = torch.ones(4, 4, dtype=torch.float32)
ones_tensor[:, 1] = 0

torch_cat = torch.cat([ones_tensor, ones_tensor, ones_tensor], dim=1)
print(f"Tensor: \n {torch_cat} \n")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py 
Tensor: 
 tensor([[1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],
        [1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],
        [1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],
        [1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.]]) 


Process finished with exit code 0

Multiplying tensors

!/usr/bin/env python

coding=utf-8

import torch
import numpy as np

ones_tensor = torch.ones(4, 4, dtype=torch.float32)
ones_tensor[:, 1] = 0

This computes the element-wise product

print(f"ones_tensor.mul(ones_tensor): \n {ones_tensor.mul(ones_tensor)} \n")

Alternative syntax

print(f"ones_tensor * ones_tensor: \n {ones_tensor * ones_tensor}")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py
ones_tensor.mul(ones_tensor):
tensor([[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.]])

ones_tensor * ones_tensor:
tensor([[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.]])

Process finished with exit code 0

!/usr/bin/env python

coding=utf-8

import torch
import numpy as np

ones_tensor = torch.ones(4, 4, dtype=torch.float32)
ones_tensor[:, 1] = 0

This computes the matrix multiplication between two tensors

print(f"ones_tensor.matmul(ones_tensor.T): \n {ones_tensor.matmul(ones_tensor.T)} \n")

Alternative syntax

print(f"ones_tensor @ ones_tensor.T: \n {ones_tensor @ ones_tensor.T}")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py
ones_tensor.matmul(ones_tensor.T):
tensor([[3., 3., 3., 3.],
[3., 3., 3., 3.],
[3., 3., 3., 3.],
[3., 3., 3., 3.]])

ones_tensor @ ones_tensor.T:
tensor([[3., 3., 3., 3.],
[3., 3., 3., 3.],
[3., 3., 3., 3.],
[3., 3., 3., 3.]])

Process finished with exit code 0
In-place operations

Operations that have a _ suffix are in-place. For example: x.copy_(y), x.t_(), will change x.

复制代码

# !/usr/bin/env python
# coding=utf-8

import torch
import numpy as np

ones_tensor = torch.ones(4, 4, dtype=torch.float32)
ones_tensor[:, 1] = 0
print(f"Ones Tensor: \n {ones_tensor} \n")

ones_tensor.add_(5)
print(f"Ones Tensor: \n {ones_tensor} \n")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py 
Ones Tensor: 
 tensor([[1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.]]) 

Ones Tensor: 
 tensor([[6., 5., 6., 6.],
        [6., 5., 6., 6.],
        [6., 5., 6., 6.],
        [6., 5., 6., 6.]]) 


Process finished with exit code 0

Note

In-place operations save some memory, but can be problematic when computing derivatives because of an immediate loss of history. Hence, their use is discouraged.

复制代码

problematic [ˌprɒbləˈmætɪk]
adj. 造成困难的；产生问题的

torch.view() 返回的新 tensor 与源 tensor 共享内存 (其实是同一个 tensor)，更改其中的一个，另外一个也会跟着改变。view() 仅仅是改变了对这个张量的观察角度。

torch.reshape() 同样可以改变张量的形状，但是此函数并不能保证返回的是其拷贝值，所以官方不推荐使用。推荐的方法是我们先用 clone() 创造一个张量副本然后再使用 torch.view() 进行函数维度变换。

使用 clone() 是会被记录在计算图中，即梯度回传到副本时也会传到源 Tensor。

复制代码

# !/usr/bin/env python
# coding=utf-8

import torch
import numpy as np

x = torch.rand(2, 3, dtype=torch.float32)
print(f"x: \n {x} \n")
y = x.view(-1)
print(f"y: \n {y} \n")

y += 1
print(f"x: \n {x} \n")
print(f"y: \n {y} \n")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py 
x: 
 tensor([[0.0200, 0.8498, 0.9486],
        [0.2756, 0.5103, 0.0668]]) 

y: 
 tensor([0.0200, 0.8498, 0.9486, 0.2756, 0.5103, 0.0668]) 

x: 
 tensor([[1.0200, 1.8498, 1.9486],
        [1.2756, 1.5103, 1.0668]]) 

y: 
 tensor([1.0200, 1.8498, 1.9486, 1.2756, 1.5103, 1.0668]) 


Process finished with exit code 0

如果我们有一个元素 tensor ，我们可以使用 .item() 来获得这个 value，而不获得其他性质。

复制代码

# !/usr/bin/env python
# coding=utf-8

import torch
import numpy as np

x = torch.rand(1, dtype=torch.float32)
print(f"type(x): type(x)")
print(f"type(x.item()): type(x.item())")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py 
type(x): type(x)
type(x.item()): type(x.item())

Process finished with exit code 0

4. Bridge with NumPy

Tensors on the CPU and NumPy arrays can share their underlying memory locations, and changing one will change the other.

4.1. Tensor to NumPy array

复制代码

# !/usr/bin/env python
# coding=utf-8

import torch
import numpy as np

pytorch_tensor = torch.ones(4, 4, dtype=torch.float32)
pytorch_tensor[:, 1] = 0
print(f"pytorch_tensor: \n {pytorch_tensor} \n")

numpy_tensor = pytorch_tensor.numpy()
print(f"numpy_tensor: \n {numpy_tensor} \n")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py 
pytorch_tensor: 
 tensor([[1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.]]) 

numpy_tensor: 
 [[1. 0. 1. 1.]
 [1. 0. 1. 1.]
 [1. 0. 1. 1.]
 [1. 0. 1. 1.]] 


Process finished with exit code 0

A change in the tensor reflects in the NumPy array.

复制代码

# !/usr/bin/env python
# coding=utf-8

import torch
import numpy as np

pytorch_tensor = torch.ones(4, 4, dtype=torch.float32)
pytorch_tensor[:, 1] = 0
print(f"pytorch_tensor: \n {pytorch_tensor} \n")

numpy_tensor = pytorch_tensor.numpy()
print(f"numpy_tensor: \n {numpy_tensor} \n")

pytorch_tensor.add_(1)
print(f"pytorch_tensor: \n {pytorch_tensor} \n")
print(f"numpy_tensor: \n {numpy_tensor} \n")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py 
pytorch_tensor: 
 tensor([[1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.]]) 

numpy_tensor: 
 [[1. 0. 1. 1.]
 [1. 0. 1. 1.]
 [1. 0. 1. 1.]
 [1. 0. 1. 1.]] 

pytorch_tensor: 
 tensor([[2., 1., 2., 2.],
        [2., 1., 2., 2.],
        [2., 1., 2., 2.],
        [2., 1., 2., 2.]]) 

numpy_tensor: 
 [[2. 1. 2. 2.]
 [2. 1. 2. 2.]
 [2. 1. 2. 2.]
 [2. 1. 2. 2.]] 


Process finished with exit code 0

4.2. NumPy array to Tensor

Changes in the NumPy array reflects in the tensor.

复制代码

# !/usr/bin/env python
# coding=utf-8

import torch
import numpy as np

numpy_tensor = np.ones((2, 3), dtype=np.float32)
print(f"numpy_tensor: \n {numpy_tensor} \n")

pytorch_tensor = torch.from_numpy(numpy_tensor)
print(f"pytorch_tensor: \n {pytorch_tensor} \n")

np.add(numpy_tensor, 1, out=numpy_tensor)
print(f"numpy_tensor: \n {numpy_tensor} \n")
print(f"pytorch_tensor: \n {pytorch_tensor} \n")

/home/yongqiang/miniconda3/bin/python /home/yongqiang/quantitative_analysis/tensors.py 
numpy_tensor: 
 [[1. 1. 1.]
 [1. 1. 1.]] 

pytorch_tensor: 
 tensor([[1., 1., 1.],
        [1., 1., 1.]]) 

numpy_tensor: 
 [[2. 2. 2.]
 [2. 2. 2.]] 

pytorch_tensor: 
 tensor([[2., 2., 2.],
        [2., 2., 2.]]) 


Process finished with exit code 0

References

1\] Yongqiang Cheng (程永强), \[2\] Deep Learning with PyTorch: A 60 Minute Blitz,

Deep Learning with PyTorch: Tensors (张量)

Deep Learning with PyTorch: Tensors {张量}

1. Tensor Initialization

2. Tensor Attributes

3. Tensor Operations

!/usr/bin/env python

coding=utf-8

!/usr/bin/env python

coding=utf-8

This computes the element-wise product

Alternative syntax

!/usr/bin/env python

coding=utf-8

This computes the matrix multiplication between two tensors

Alternative syntax

4. Bridge with NumPy

4.1. Tensor to NumPy array

4.2. NumPy array to Tensor

References