240919-Pip先在线下载不安装+再离线安装

A. 最终效果

# 使用modelscope sdk下载模型
import os
os.environ['MODELSCOPE_CACHE'] = '您希望的下载路径'from modelscope import snapshot_download
model_dir = snapshot_download('opendatalab/PDF-Extract-Kit')
print(f"模型文件下载路径为：{model_dir}/models")

B. 需求背景

最近接触下面这个项目：

• MinerU/docs/README_Ubuntu_CUDA_Acceleration_en_US.md at master · opendatalab/MinerU

需要安装Python环境

pip install -U magic-pdf[full] --extra-index-url https://wheels.myhloli.com

然而，公司不能访问：https://wheels.myhloli.com

C. 解决思路

• 在有网可访问的环境中pip download所有的安装包
• 将所有安装包拷贝到离线的服务器上
• 再pip 安装

D. 理论说明

要使用 pip install 命令来 只下载 包而不安装，可以通过添加 --no-install 选项来实现。然而，--no-install 选项已被弃用。现在，你应该使用 pip download 命令来完成这个任务。

如果你仍想利用 pip install 下载包而不实际安装，直接的方法并不存在。你应使用 pip download 命令，如下所示：

pip download <package-name> -d <directory>

此命令只会下载包到指定的目录，而不会安装它们。例如：

pip download numpy -d ./packages

这将会把 numpy 包下载到 ./packages 目录中，而不会安装它。

E. 实战操作

E.1 在线下载

• 下面这段命令有误

pip download -d pip_download_mineru -U magic-pdf[full] --extra-index-url https://wheels.myhloli.com

上面提供的命令有几个地方需要注意：

1. -U 参数（升级包）通常在 pip install 中使用，而 pip download 的作用是下载包，而不是安装或升级包，所以不需要 -U 参数。
2. 目录 pip_download_mineru 应该存在，或者在当前路径中会自动创建。
3. magic-pdf[full] 的形式正确，这种语法用于指定包的扩展依赖。

以下是修改后的命令：

pip download -d pip_download_mineru magic-pdf[full] --extra-index-url https://wheels.myhloli.com

这条命令会在 pip_download_mineru 目录下下载 magic-pdf 包及其扩展依赖，使用你指定的 --extra-index-url。

E.2 查看大小

在 Linux 中，你可以使用 du 命令来查看指定文件夹的大小。常见的命令格式如下：

du -sh /path/to/folder

解释：

• -s：只显示总计（summary）。
• -h：以人类可读的格式显示（例如 KB、MB、GB）。

示例：

du -sh /home/user/documents

这会返回 /home/user/documents 文件夹的总大小，使用友好的单位显示。

E.3 离线安装

下载完后，你可以通过以下命令安装下载的包：

pip install --no-index --find-links=./pip_download_mineru magic-pdf[full]

解释：

1. --no-index：告诉 pip 不从 PyPI 仓库下载，避免网络请求。
2. --find-links=./pip_download_mineru：指定你之前下载的包所在目录。
3. magic-pdf[full]：安装指定的包以及其扩展依赖。

确保你在执行该命令时当前路径能正确找到 pip_download_mineru 目录，或者提供该目录的绝对路径。

F. 补充操作

F.1 Linux压缩文件夹

在Linux系统中，使用zip命令可以压缩文件夹及其内容。以下是压缩文件夹的步骤：

• 压缩文件夹

1. 打开终端。

2. 使用以下命令来压缩文件夹：

   zip -r 压缩文件名.zip 文件夹名/
   

   其中：

   • -r 参数表示递归压缩，也就是将文件夹中的所有内容（包括子文件夹）都压缩。
   • 压缩文件名.zip 是你希望生成的压缩文件名。
   • 文件夹名/ 是你要压缩的文件夹路径。

• 示例
  假设你有一个名为 my_folder 的文件夹，想要压缩为 my_folder.zip：

zip -r my_folder.zip my_folder/

• 解压文件
  如果你想解压这个压缩文件，可以使用以下命令：

unzip my_folder.zip

这样可以将压缩文件夹恢复到当前目录下。

如果你需要更多帮助，请告诉我！

F.2 Linux分割大文件

在Linux系统中，可以使用split命令将大文件分割为指定大小的小文件。以下是分割大文件的步骤：

• 使用 split 命令分割大文件

1. 打开终端。

2. 使用以下命令将文件分割成指定大小的小文件：

   split -b [大小] [大文件名] [输出文件前缀]
   

   其中：

   • -b [大小]：指定每个分割文件的大小。你可以使用字节（B）、千字节（K）、兆字节（M）或千兆字节（G）作为单位。例如，10M 表示 10 MB，500K 表示 500 KB。
   • [大文件名]：要分割的大文件的文件名。
   • [输出文件前缀]：分割文件的前缀名，后面会自动添加分割文件的编号或字母。

• 示例

假设你有一个名为 large_file.txt 的大文件，并且想要将其分割成每个 100 MB 的小文件，使用如下命令：

split -b 100M large_file.txt small_part_

这将生成名为 small_part_aa、small_part_ab、small_part_ac 等的分割文件，每个文件大小为 100 MB。

• 重新合并分割的文件

如果你想将分割的文件重新合并成一个大文件，可以使用 cat 命令：

cat small_part_* > merged_file.txt

这将所有以 small_part_ 开头的文件合并为 merged_file.txt。

如果你有其他问题，随时告诉我！

G. 问题汇总

G.1 报错1

• 终端显示

Processing ./pip_download_mineru/fairscale-0.4.13.tar.gz (from unimernet==0.1.6->magic-pdf[full])Installing build dependencies ... errorerror: subprocess-exited-with-error× pip subprocess to install build dependencies did not run successfully.│ exit code: 1╰─> [4 lines of output]Looking in links: ./pip_download_mineruProcessing ./pip_download_mineru/setuptools-75.1.0-py3-none-any.whlERROR: Could not find a version that satisfies the requirement wheel>=0.30.0 (from versions: none)ERROR: No matching distribution found for wheel>=0.30.0[end of output]note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.note: This error originates from a subprocess, and is likely not a problem with pip.

• 解决方法
  The error you’re encountering stems from missing dependencies for the package you’re trying to install. Specifically, the error message says that wheel>=0.30.0 could not be found. Here’s how you can resolve this issue:

1. Ensure wheel is installed:
   Manually install wheel before trying again.

   pip install wheel
   

2. Update setuptools and pip:
   It’s also a good idea to make sure both setuptools and pip are up to date:

   pip install --upgrade pip setuptools
   

3. Check local repository:
   You’re pointing to a local link (./pip_download_mineru) for installation. This could indicate that the dependencies are stored locally, but the required version of wheel isn’t there. Ensure that the wheel package is available in the local directory or use PyPI as a fallback:

   pip install wheel>=0.30.0 --no-index --find-links ./pip_download_mineru
   

4. Try installing from PyPI:
   If the local link is missing some dependencies, try installing the packages directly from PyPI:

   pip install fairscale==0.4.13
   

After these steps, attempt the installation of unimernet again:

pip install unimernet==0.1.6

Let me know if you run into further issues!

G.2 报错2

Building wheels for collected packages: wordninja, fire, fvcore, antlr4-python3-runtime, langdetectBuilding wheel for wordninja (setup.py) ... errorerror: subprocess-exited-with-error× python setup.py bdist_wheel did not run successfully.│ exit code: 1╰─> [18 lines of output]Traceback (most recent call last):File "<string>", line 2, in <module>File "<pip-setuptools-caller>", line 34, in <module>File "/tmp/pip-install-gkp67rkk/wordninja_89b2c7b1604a41e4af4e95471e7a9145/setup.py", line 10, in <module>setup(name='wordninja',File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 170, in setupok = dist.parse_command_line()File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 472, in parse_command_lineargs = self._parse_command_opts(parser, args)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/dist.py", line 852, in _parse_command_optsnargs = _Distribution._parse_command_opts(self, parser, args)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 531, in _parse_command_optscmd_class = self.get_command_class(command)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/dist.py", line 691, in get_command_classfrom .command.bdist_wheel import bdist_wheelFile "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/command/bdist_wheel.py", line 26, in <module>from wheel.wheelfile import WheelFileModuleNotFoundError: No module named 'wheel.wheelfile'[end of output]note: This error originates from a subprocess, and is likely not a problem with pip.ERROR: Failed building wheel for wordninjaRunning setup.py clean for wordninjaBuilding wheel for fire (setup.py) ... errorerror: subprocess-exited-with-error× python setup.py bdist_wheel did not run successfully.│ exit code: 1╰─> [22 lines of output]/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/dist.py:261: UserWarning: Unknown distribution option: 'tests_require'warnings.warn(msg)Traceback (most recent call last):File "<string>", line 2, in <module>File "<pip-setuptools-caller>", line 34, in <module>File "/tmp/pip-install-gkp67rkk/fire_dc658731467749cbb9438498fd528508/setup.py", line 46, in <module>setup(File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/__init__.py", line 117, in setupreturn distutils.core.setup(**attrs)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 170, in setupok = dist.parse_command_line()File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 472, in parse_command_lineargs = self._parse_command_opts(parser, args)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/dist.py", line 852, in _parse_command_optsnargs = _Distribution._parse_command_opts(self, parser, args)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 531, in _parse_command_optscmd_class = self.get_command_class(command)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/dist.py", line 691, in get_command_classfrom .command.bdist_wheel import bdist_wheelFile "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/command/bdist_wheel.py", line 26, in <module>from wheel.wheelfile import WheelFileModuleNotFoundError: No module named 'wheel.wheelfile'[end of output]note: This error originates from a subprocess, and is likely not a problem with pip.ERROR: Failed building wheel for fireRunning setup.py clean for fireBuilding wheel for fvcore (setup.py) ... errorerror: subprocess-exited-with-error× python setup.py bdist_wheel did not run successfully.│ exit code: 1╰─> [20 lines of output]Traceback (most recent call last):File "<string>", line 2, in <module>File "<pip-setuptools-caller>", line 34, in <module>File "/tmp/pip-install-gkp67rkk/fvcore_1565d5fa326e4480a955279e9cb6c3fa/setup.py", line 36, in <module>setup(File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/__init__.py", line 117, in setupreturn distutils.core.setup(**attrs)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 170, in setupok = dist.parse_command_line()File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 472, in parse_command_lineargs = self._parse_command_opts(parser, args)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/dist.py", line 852, in _parse_command_optsnargs = _Distribution._parse_command_opts(self, parser, args)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 531, in _parse_command_optscmd_class = self.get_command_class(command)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/dist.py", line 691, in get_command_classfrom .command.bdist_wheel import bdist_wheelFile "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/command/bdist_wheel.py", line 26, in <module>from wheel.wheelfile import WheelFileModuleNotFoundError: No module named 'wheel.wheelfile'[end of output]note: This error originates from a subprocess, and is likely not a problem with pip.ERROR: Failed building wheel for fvcoreRunning setup.py clean for fvcoreBuilding wheel for antlr4-python3-runtime (setup.py) ... errorerror: subprocess-exited-with-error× python setup.py bdist_wheel did not run successfully.│ exit code: 1╰─> [20 lines of output]Traceback (most recent call last):File "<string>", line 2, in <module>File "<pip-setuptools-caller>", line 34, in <module>File "/tmp/pip-install-gkp67rkk/antlr4-python3-runtime_2f0618cd1c4145a0bd498a4cb967badf/setup.py", line 3, in <module>setup(File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/__init__.py", line 117, in setupreturn distutils.core.setup(**attrs)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 170, in setupok = dist.parse_command_line()File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 472, in parse_command_lineargs = self._parse_command_opts(parser, args)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/dist.py", line 852, in _parse_command_optsnargs = _Distribution._parse_command_opts(self, parser, args)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 531, in _parse_command_optscmd_class = self.get_command_class(command)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/dist.py", line 691, in get_command_classfrom .command.bdist_wheel import bdist_wheelFile "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/command/bdist_wheel.py", line 26, in <module>from wheel.wheelfile import WheelFileModuleNotFoundError: No module named 'wheel.wheelfile'[end of output]note: This error originates from a subprocess, and is likely not a problem with pip.ERROR: Failed building wheel for antlr4-python3-runtimeRunning setup.py clean for antlr4-python3-runtimeBuilding wheel for langdetect (setup.py) ... errorerror: subprocess-exited-with-error× python setup.py bdist_wheel did not run successfully.│ exit code: 1╰─> [20 lines of output]Traceback (most recent call last):File "<string>", line 2, in <module>File "<pip-setuptools-caller>", line 34, in <module>File "/tmp/pip-install-gkp67rkk/langdetect_59538374717e4b6e841b1667ce4bc473/setup.py", line 11, in <module>setup(File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/__init__.py", line 117, in setupreturn distutils.core.setup(**attrs)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 170, in setupok = dist.parse_command_line()File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 472, in parse_command_lineargs = self._parse_command_opts(parser, args)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/dist.py", line 852, in _parse_command_optsnargs = _Distribution._parse_command_opts(self, parser, args)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 531, in _parse_command_optscmd_class = self.get_command_class(command)File "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/dist.py", line 691, in get_command_classfrom .command.bdist_wheel import bdist_wheelFile "/home/lgk/.conda/envs/MinerU/lib/python3.10/site-packages/setuptools/command/bdist_wheel.py", line 26, in <module>from wheel.wheelfile import WheelFileModuleNotFoundError: No module named 'wheel.wheelfile'[end of output]note: This error originates from a subprocess, and is likely not a problem with pip.ERROR: Failed building wheel for langdetectRunning setup.py clean for langdetect
Failed to build wordninja fire fvcore antlr4-python3-runtime langdetect
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (wordninja, fire, fvcore, antlr4-python3-runtime, langdetect)

• 解决方法

The errors you’re encountering are all related to a missing wheel module. Specifically, the wheel.wheelfile module is required for building wheels from the source. Here’s how you can troubleshoot and resolve the issue:

• Steps to resolve:

1. Install or upgrade wheel:
   Make sure wheel is properly installed and up to date in your environment:

   pip install --upgrade wheel
   

2. Ensure setuptools and pip are up to date:
   Updating setuptools and pip often helps with build-related errors:

   pip install --upgrade setuptools pip
   

3. Install dependencies using --no-build-isolation:
   Sometimes the default build isolation can cause issues. You can try installing the packages without build isolation:

   pip install --no-build-isolation wordninja fire fvcore antlr4-python3-runtime langdetect
   

4. Attempt to install pre-built wheels from PyPI:
   If you’re building from source and it’s failing, check if pre-built wheels are available on PyPI for the specific packages:

   pip install wordninja fire fvcore antlr4-python3-runtime langdetect --prefer-binary
   

5. Use conda-forge channel (if using conda):
   If you’re using a conda environment, try using the conda-forge channel to install pre-built versions of the libraries:

   conda install -c conda-forge wordninja fire fvcore antlr4-python3-runtime langdetect
   

6. Check your environment:
   If you have multiple Python environments, ensure that the correct one is being used. Activate the specific environment and try the installation again.

Once you’ve tried these steps, re-run the package installation:

pip install unimernet==0.1.6

Let me know how it goes or if you encounter any further issues!

• 最终验证

H. 参考文献

• ModelScope模型下载默认位置及如何修改为指定路径 – 通塔师AI导航