Ubuntu 22.04 x86_64 源码编译 pytorch-v2.0.1 笔记【2】编译成功

20230831继续：

当前状态

复制代码

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch$ pwd
/home/yeqiang/Downloads/src/pytorch
(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch$ python3 -V
Python 3.10.6
(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch$

故障1

复制代码

CMake Error at cmake/Dependencies.cmake:1554 (add_subdirectory):
  The source directory

    /home/yeqiang/Downloads/src/pytorch/third_party/foxi

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)

目录是空的

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch$ ll third_party/foxi

总计 0

手动checkout

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch/third_party/foxi$ git pull

已经是最新的。

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch/third_party/foxi$ git checkout master

已经位于 'master'

您的分支与上游分支 'origin/master' 一致。

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch/third_party/foxi$ ll

总计 12

drwxrwxr-x 2 yeqiang yeqiang 104 2023-08-31 09:44:29 cmake

-rw-rw-r-- 1 yeqiang yeqiang 4077 2023-08-31 09:44:29 CMakeLists.txt

drwxrwxr-x 2 yeqiang yeqiang 168 2023-08-31 09:44:29 foxi

-rw-rw-r-- 1 yeqiang yeqiang 1064 2023-08-31 09:44:29 LICENSE

-rw-rw-r-- 1 yeqiang yeqiang 6 2023-08-31 09:44:29 VERSION_NUMBER

故障2

复制代码

Traceback (most recent call last):
  File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/yeqiang/Downloads/src/pytorch/torchgen/gen.py", line 22, in <module>
    from typing_extensions import Literal  # Python 3.8+
ModuleNotFoundError: No module named 'typing_extensions'
CMake Error at cmake/Codegen.cmake:197 (message):
  Failed to get generated_headers list
Call Stack (most recent call first):
  caffe2/CMakeLists.txt:2 (include)

手动安装依赖

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch$ pip --version

pip 23.2 from /home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/site-packages/pip (python 3.10)

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch$ pip install typing-extensions

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple

Collecting typing-extensions

Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ec/6b/63cc3df74987c36fe26157ee12e09e8f9db4de771e0f3404263117e75b95/typing_extensions-4.7.1-py3-none-any.whl (33 kB)

Installing collected packages: typing-extensions

Successfully installed typing-extensions-4.7.1

构建

make -j6

构建到90%报错

90%\] Linking CXX shared library ../lib/libtorch_cpu.so /usr/bin/ld: ../lib/libgloo.a(store.cc.o):(.rodata+0x0): multiple definition of \`gloo::rendezvous::Store::kDefaultTimeout'; CMakeFiles/torch_cpu.dir/__/torch/csrc/distributed/c10d/ProcessGroupGloo.cpp.o:(.rodata._ZN4gloo10rendezvous5Store15kDefaultTimeoutE\[_ZN4gloo10rendezvous5Store15kDefaultTimeoutE\]+0x0): first defined here collect2: error: ld returned 1 exit status make\[3\]: \*\*\* \[caffe2/CMakeFiles/torch_cpu.dir/build.make:17403：lib/libtorch_cpu.so\] 错误 1 make\[3\]: 离开目录"/home/yeqiang/Downloads/src/pytorch/build" make\[2\]: \*\*\* \[CMakeFiles/Makefile2:4511：caffe2/CMakeFiles/torch_cpu.dir/all\] 错误 2 make\[2\]: 离开目录"/home/yeqiang/Downloads/src/pytorch/build" make\[1\]: \*\*\* \[Makefile:146：all\] 错误 2 make\[1\]: 离开目录"/home/yeqiang/Downloads/src/pytorch/build" make: \*\*\* \[Makefile:6：all\] 错误 2 conda install mkl mkl-include # CUDA only: Add LAPACK support for the GPU if needed conda install -c pytorch magma-cuda110 # or the magma-cuda\* that matches your CUDA version from https://anaconda.org/pytorch/repo # (optional) If using torch.compile with inductor/triton, install the matching version of triton # Run from the pytorch directory after cloning make triton 更新子模块 (pytorch-build) yeqiang@yeqiang-MS-7B23:\~/Downloads/src/pytorch$ git submodule update --init --recursive (pytorch-build) yeqiang@yeqiang-MS-7B23:\~/Downloads/src/pytorch$ export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"} python setup.py develop Building wheel torch-2.0.0a0+gite9ebda2 -- Building version 2.0.0a0+gite9ebda2 Could not find any of CMakeLists.txt, Makefile, setup.py, LICENSE, LICENSE.md, LICENSE.txt in /home/yeqiang/Downloads/src/pytorch/third_party/ios-cmake Did you run 'git submodule update --init --recursive'? × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─\> \[46 lines of output

Traceback (most recent call last):

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/urllib/request.py", line 1348, in do_open

h.request(req.get_method(), req.selector, req.data, headers,

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/http/client.py", line 1282, in request

self._send_request(method, url, body, headers, encode_chunked)

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/http/client.py", line 1328, in _send_request

self.endheaders(body, encode_chunked=encode_chunked)

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/http/client.py", line 1277, in endheaders

self._send_output(message_body, encode_chunked=encode_chunked)

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/http/client.py", line 1037, in _send_output

self.send(msg)

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/http/client.py", line 975, in send

self.connect()

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/http/client.py", line 1454, in connect

self.sock = self._context.wrap_socket(self.sock,

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/ssl.py", line 513, in wrap_socket

return self.sslsocket_class._create(

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/ssl.py", line 1071, in _create

self.do_handshake()

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/ssl.py", line 1342, in do_handshake

self._sslobj.do_handshake()

ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "<string>", line 2, in <module>

File "<pip-setuptools-caller>", line 34, in <module>

File "/tmp/pip-req-build-75vmmdis/python/setup.py", line 203, in <module>

download_and_copy_ptxas()

File "/tmp/pip-req-build-75vmmdis/python/setup.py", line 113, in download_and_copy_ptxas

ftpstream = urllib.request.urlopen(url)

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/urllib/request.py", line 216, in urlopen

return opener.open(url, data, timeout)

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/urllib/request.py", line 519, in open

response = self._open(req, data)

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/urllib/request.py", line 536, in _open

result = self._call_chain(self.handle_open, protocol, protocol +

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/urllib/request.py", line 496, in _call_chain

result = func(*args)

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/urllib/request.py", line 1391, in https_open

return self.do_open(http.client.HTTPSConnection, req,

File "/home/yeqiang/miniconda3/envs/pytorch-build/lib/python3.10/urllib/request.py", line 1351, in do_open

raise URLError(err)

urllib.error.URLError: <urlopen error TLS/SSL connection has been closed (EOF) (_ssl.c:997)>

downloading and extracting https://conda.anaconda.org/nvidia/label/cuda-12.0.0/linux-64/cuda-nvcc-12.0.76-0.tar.bz2 ...

end of output

cuda-12.0.0 期望的是cuda-11.8

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch$ python3 setup.py build

Building wheel torch-2.0.0a0+gite9ebda2

-- Building version 2.0.0a0+gite9ebda2

Could not find any of CMakeLists.txt, Makefile, setup.py, LICENSE, LICENSE.md, LICENSE.txt in /home/yeqiang/Downloads/src/pytorch/third_party/ios-cmake

Did you run 'git submodule update --init --recursive'?

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch$ cd third_party/ios-cmake/

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch/third_party/ios-cmake$ ll

总计 0

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch/third_party/ios-cmake$ git remote -v

origin https://github.com/Yangqing/ios-cmake.git (fetch)

origin https://github.com/Yangqing/ios-cmake.git (push)

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch/third_party/ios-cmake$ du -sh

4.0K .

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch/third_party/ios-cmake$ git checkout master

已经位于 'master'

您的分支与上游分支 'origin/master' 一致。

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch/third_party/ios-cmake$ ll

总计 8

-rw-rw-r-- 1 yeqiang yeqiang 1621 2023-08-31 13:47:03 LICENSE

-rw-rw-r-- 1 yeqiang yeqiang 887 2023-08-31 13:47:03 README.md

drwxrwxr-x 4 yeqiang yeqiang 52 2023-08-31 13:47:03 samples

drwxrwxr-x 2 yeqiang yeqiang 31 2023-08-31 13:47:03 toolchain

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch/third_party/ios-cmake$ du -sh

160K .

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch$ export USE_CUDA=1

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch$ python3 setup.py build

Building wheel torch-2.0.0a0+gite9ebda2

-- Building version 2.0.0a0+gite9ebda2

Could not find any of CMakeLists.txt, Makefile, setup.py, LICENSE, LICENSE.md, LICENSE.txt in /home/yeqiang/Downloads/src/pytorch/third_party/python-enum

Did you run 'git submodule update --init --recursive'?

同样的问题，源码没有git checkout master

(pytorch-build) yeqiang@yeqiang-MS-7B23:~/Downloads/src/pytorch$ python3 setup.py build

Building wheel torch-2.0.0a0+gite9ebda2

-- Building version 2.0.0a0+gite9ebda2

cmake --build . --target install --config Release -- -j 6