在ESPnet使用Makefile安装PyTorch和相关依赖的详细教程

在ESPnet使用Makefile安装PyTorch和相关依赖的详细教程

介绍

在数据科学和机器学习项目中，经常需要安装和配置多个软件包和库。手动安装这些依赖项可能会非常繁琐和耗时。为了简化这个过程，我们可以使用Makefile来自动化安装过程。本教程将详细介绍如何使用Makefile来安装PyTorch和其他相关依赖项。

环境要求

• 操作系统：Linux 或 macOS
• Python 版本：3.7 及以上
• Conda 或 pip（如果你使用的是Anaconda）

项目结构

假设你的espnet项目结构如下：

espnet/
│
tools/
│
├── installers/
│   ├── install_ffmpeg.sh
│   ├── install_sctk.sh
│   ├── install_sph2pipe.sh
│   ├── install_torch.sh
│   ├── install_warp-transducer.sh
│   ├── install_chainer.sh
│   ├── install_kenlm.sh
│   └── ...
├── activate_python.sh
├── Makefile
└── ...

Makefile 详解

Makefile 是一个自动化构建工具，通过定义目标（targets）和依赖关系，可以方便地管理构建任务。

基本语法

在Makefile中，每个目标都有一组依赖项和一组命令。例如：

target: dependenciescommand

主要目标

在我们的Makefile中，主要目标包括：

• 安装基本工具和库（如 cmake、ffmpeg、sox 等）
• 安装 Python 依赖项（如 numpy、torch、espnet 等）
• 安装特定工具（如 sctk、sph2pipe 等）

示例Makefile

以下是一个示例Makefile，包含所有主要目标和命令：

TH_VERSION := 2.3.0# Use pip for pytorch installation even if you have anaconda
ifneq ($(shell test -f ./activate_python.sh && grep 'conda activate' ./activate_python.sh),)
USE_CONDA := 1
else
USE_CONDA :=
endif# Set if install binaries on CPU mode e.g. make CPU_ONLY=0
# If you don't have nvcc, this value will be set automatically
ifneq ($(shell which nvcc 2>/dev/null),)
CPU_ONLY :=
# Derive CUDA version from nvcc
CUDA_VERSION := $(shell nvcc --version | grep "Cuda compilation tools" | cut -d" " -f5 | sed s/,//)
CUDA_VERSION_WITHOUT_DOT := $(strip $(subst .,,$(CUDA_VERSION)))
else
CPU_ONLY := 0
CUDA_VERSION :=
CUDA_VERSION_WITHOUT_DOT :=
endif
WITH_OMP=ON.PHONY: all cleanall: showenv conda_packages.done python ffmpeg.done sctk check_installpython: activate_python.sh setuptools.done packaging.done espnet.done pytorch.done chainer.done fairscale.done torch_optimizer.done flash_attn.doneactivate_python.sh:test -f activate_python.sh || { echo "Error: Run ./setup_python.sh or ./setup_anaconda.sh"; exit 1; }################ Logging ################
showenv: activate_python.sh
ifeq ($(strip $(CPU_ONLY)),)@echo CUDA_VERSION=$(CUDA_VERSION)
else@echo Perform on CPU mode: CPU_ONLY=$(CPU_ONLY)
endif@echo PYTHON=$(shell . ./activate_python.sh && command -v python3)@echo PYTHON_VERSION=$(shell . ./activate_python.sh && python3 --version)@echo USE_CONDA=$(USE_CONDA)@echo TH_VERSION=$(TH_VERSION)@echo WITH_OMP=$(WITH_OMP)#########################################bc.done: activate_python.sh. ./activate_python.sh && { command -v bc || conda install -y bc -c conda-forge; }touch bc.donecmake.done: activate_python.sh. ./activate_python.sh && { command -v cmake || conda install -y cmake; }touch cmake.doneflac.done: activate_python.sh. ./activate_python.sh && { command -v flac || conda install -y libflac -c conda-forge; }touch flac.donesox.done: activate_python.sh. ./activate_python.sh && { command -v sox || conda install -y sox -c conda-forge; }touch sox.donesndfile.done: activate_python.sh. ./activate_python.sh && { conda install -y libsndfile -c conda-forge; }touch sndfile.doneifneq ($(strip $(USE_CONDA)),)
conda_packages.done: bc.done cmake.done flac.done sox.done sndfile.done
else
conda_packages.done:
endiftouch conda_packages.doneffmpeg.done: activate_python.sh
ifneq ($(strip $(USE_CONDA)),). ./activate_python.sh && { command -v ffmpeg || conda install -y ffmpeg -c conda-forge; }
else. ./activate_python.sh && { command -v ffmpeg || ./installers/install_ffmpeg.sh; }
endiftouch ffmpeg.donesctk: sctk/bin/sclite
sctk/bin/sclite:./installers/install_sctk.shsph2pipe: sph2pipe/sph2pipe
sph2pipe/sph2pipe:./installers/install_sph2pipe.shsetuptools.done: activate_python.sh. ./activate_python.sh && { python -m ensurepip --upgrade; pip install setuptools; }touch setuptools.donepackaging.done: setuptools.done. ./activate_python.sh && python3 -m pip install packagingtouch packaging.donenumpy.done: activate_python.sh
ifeq ($(strip $(USE_CONDA)),). ./activate_python.sh && python3 -m pip install numpy
else. ./activate_python.sh && conda install -y numpy
endiftouch numpy.donenumba.done: numpy.done. ./activate_python.sh && python3 -m pip install -U numbatouch numba.donepytorch.done: packaging.done numba.done
ifeq ($(strip $(USE_CONDA)),). ./activate_python.sh && ./installers/install_torch.sh "false" "${TH_VERSION}" "${CUDA_VERSION}"
else. ./activate_python.sh && ./installers/install_torch.sh "true" "${TH_VERSION}" "${CUDA_VERSION}"
endiftouch pytorch.doneespnet.done: pytorch.done conda_packages.done. ./activate_python.sh && python3 -m pip install -e "..[train, recipe]". ./activate_python.sh && python -m nltk.downloader averaged_perceptron_tagger_engtouch espnet.donesounfile_test: espnet.done. ./activate_python.sh && python3 ./test_soundfile.pychainer.done: espnet.done. ./activate_python.sh && ./installers/install_chainer.sh "${CUDA_VERSION}"touch chainer.donewarp-transducer.done: pytorch.done conda_packages.done
ifeq ($(strip $(CPU_ONLY)),)[ -n "${CUDA_HOME}" ] || { echo "Error: CUDA_HOME is not set."; exit 1; }
endif. ./activate_python.sh && ./installers/install_warp-transducer.sh ${WITH_OMP}touch warp-transducer.donechainer_ctc.done: chainer.done
ifeq ($(strip $(CPU_ONLY)),)[ -n "${CUDA_HOME}" ] || { echo "Error: CUDA_HOME is not set."; exit 1; }
endif. ./activate_python.sh && ./installers/install_chainer_ctc.sh ${WITH_OMP}touch chainer_ctc.donenkf.done:./installers/install_nkf.shtouch nkf.donepyopenjtalk.done: espnet.done conda_packages.done. ./activate_python.sh && ./installers/install_pyopenjtalk.shtouch pyopenjtalk.donephonemizer.done: espnet.done conda_packages.done
ifeq ($(WITH_OMP),ON). ./activate_python.sh && ./installers/install_phonemizer.shtouch phonemizer.done
elseecho "Warning: ./installers/install_phonemizer.sh requires WITH_OMP=ON"
endifmoses.done:git clone --depth 1 https://github.com/moses-smt/mosesdecoder.git mosestouch moses.donemwerSegmenter.done:./installers/install_mwerSegmenter.shtouch mwerSegmenter.donekenlm.done: espnet.done conda_packages.done. ./activate_python.sh && ./installers/install_kenlm.shtouch kenlm.donepesq: PESQ/P862_annex_A_2005_CD/source/PESQ
PESQ/P862_annex_A_2005_CD/source/PESQ:./installers/install_pesq.shpy3mmseg.done: espnet.done. ./activate_python.sh && ./installers/install_py3mmseg.shtouch py3mmseg.donebeamformit.done:./installers/install_beamformit.shtouch beamformit.donetorch_optimizer.done: espnet.done. ./activate_python.sh && ./installers/install_torch_optimizer.shtouch torch_optimizer.donefairscale.done: espnet.done. ./activate_python.sh && ./installers/install_fairscale.shtouch fairscale.donefairseq.done: espnet.done. ./activate_python.sh && ./installers/install_fairseq.shtouch fairseq.dones3prl.done: espnet.done. ./activate_python.sh && ./installers/install_s3prl.shtouch s3prl.donewhisper.done: espnet.done. ./activate_python.sh && ./installers/install_whisper.shtouch whisper.donelora.done: espnet.done. ./activate_python.sh && ./installers/install_lora.shtouch lora.donek2.done: espnet.done. ./activate_python.sh && ./installers/install_k2.shtouch k2.donetransformers.done: espnet.done. ./activate_python.sh && ./installers/install_transformers.shtouch transformers.doneparallel-wavegan.done: espnet.done. ./activate_python.sh && ./installers/install_parallel-wavegan.shtouch parallel-wavegan.donelongformer.done: espnet.done. ./activate_python.sh && ./installers/install_longformer.shtouch longformer.doneflash_attn.done: espnet.done. ./activate_python.sh && ./installers/install_flash_attn.shtouch flash_attn.donemuskits.done: espnet.done. ./activate_python.sh && ./installers/install_muskits.shtouch muskits.donecheck_install: python. ./activate_python.sh; . ./extra_path.sh; python3 check_install.pyclean: clean_extrarm -rf warp-transducer chainer_ctcrm -rf *.donefind . -iname "*.pyc" -deleteclean_python:rm -rf warp-transducer chainer_ctcrm -f chainer_ctc.done espnet.done chainer.done pytorch.done warp-transducer.donefind . -iname "*.pyc" -deleteclean_extra:rm -rf nkf.done swig.done moses.done mwerSegmenter.donerm -rf hts_engine_API.done open_jtalk.done pyopenjtalk.donerm -rf muskits.donerm -rf rvad_fast.donerm -rf chainer espeak-ng festival MBROLA ParallelWaveGAN versarm -rf py3mmseg sctk* speech_tools sph2pipe* ._mwerSegmenterrm -rf nkf mecab swig moses mwerSegmenterrm -rf PESQ PESQ.zip

安装和使用步骤

1. 准备环境

   • 安装 conda 或者确保有 pip 。
   • 克隆或下载你的项目。

2. 激活Python环境

   • 创建并激活Python环境：

     ./setup_anaconda.sh  # 或者 ./setup_python.sh
     

3. 运行Makefile

   • 安装所有依赖项：

     make all
     

   • 你可以使用特定目标来安装某些依赖项，例如：

     make pytorch.done
     

4. 检查安装

   • 确保所有依赖项都已正确安装：

     make check_install
     

常见问题

• 问题：安装过程中遇到权限问题。

  • 解决方案：确保你有足够的权限，或者在命令前添加 sudo。

• 问题：某些依赖项未能安装。

  • 解决方案：查看具体的错误信息，并根据提示手动安装依赖项。

清理

• 可以使用以下命令来清理安装的文件和依赖项：

  make clean
  

结论

通过使用Makefile，可以大大简化复杂项目的安装过程。希望本教程能帮助你更好地理解和使用Makefile来管理项目依赖。如果有任何问题或建议，请随时留言。