问题:使用Requirements.txt安装时,避免在单个软件包上出现故障

我正在从安装软件包 requirements.txt

pip install -r requirements.txt

requirements.txt文件显示为:

Pillow
lxml
cssselect
jieba
beautifulsoup
nltk

lxml是唯一无法安装的软件包,这将导致一切失败(larsk在注释中指出了预期的结果)。但是,lxml失败后pip仍会继续运行并下载其余软件包。

据我了解,pip install -r requirements.txt如果requirements.txt无法安装中列出的任何软件包,该命令将失败。

我在运行时可以传递任何参数pip install -r requirements.txt来告诉它安装可以执行的操作并跳过不能执行的程序包,或者在看到失败后立即退出吗?

I am installing packages from requirements.txt

pip install -r requirements.txt

The requirements.txt file reads:

Pillow
lxml
cssselect
jieba
beautifulsoup
nltk

lxml is the only package failing to install and this leads to everything failing (expected results as pointed out by larsks in the comments). However, after lxml fails pip still runs through and downloads the rest of the packages.

From what I understand the pip install -r requirements.txt command will fail if any of the packages listed in the requirements.txt fail to install.

Is there any argument I can pass when running pip install -r requirements.txt to tell it to install what it can and skip the packages that it cannot, or to exit as soon as it sees something fail?


回答 0

运行每一行pip install可能是一种解决方法。

cat requirements.txt | xargs -n 1 pip install

注意:该-a参数在MacOS下不可用,因此老猫更便携。

Running each line with pip install may be a workaround.

cat requirements.txt | xargs -n 1 pip install

Note: -a parameter is not available under MacOS, so old cat is more portable.


回答 1

该解决方案处理您的requirements.txt中的空行,空白行,#注释行,whitespace-then-#注释行。

cat requirements.txt | sed -e '/^\s*#.*$/d' -e '/^\s*$/d' | xargs -n 1 pip install

帽尖到这个答案的sed的魔法。

This solution handles empty lines, whitespace lines, # comment lines, whitespace-then-# comment lines in your requirements.txt.

cat requirements.txt | sed -e '/^\s*#.*$/d' -e '/^\s*$/d' | xargs -n 1 pip install

Hat tip to this answer for the sed magic.


回答 2

对于Windows:

点版本> = 18

import sys
from pip._internal import main as pip_main

def install(package):
    pip_main(['install', package])

if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        for line in f:
            install(line)

点版本<18

import sys
import pip

def install(package):
    pip.main(['install', package])

if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        for line in f:
            install(line)

For Windows:

pip version >=18

import sys
from pip._internal import main as pip_main

def install(package):
    pip_main(['install', package])

if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        for line in f:
            install(line)

pip version <18

import sys
import pip

def install(package):
    pip.main(['install', package])

if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        for line in f:
            install(line)

回答 3

xargs解决方案有效,但可能会存在可移植性问题(BSD / GNU),并且/或者如果您的需求文件中有注释或空白行,则可能会很麻烦。

至于需要这种行为的用例,例如,我使用了两个单独的需求文件,一个仅列出需要始终安装的核心依赖关系,而另一个文件则列出了90%的具有非核心依赖关系的文件。大多数用例都不需要。这相当于Recommendsdebian软件包的这一部分。

我使用以下shell脚本(requires sed)安装可选的依赖项

#!/bin/sh

while read dependency; do
    dependency_stripped="$(echo "${dependency}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
    # Skip comments
    if [[ $dependency_stripped == \#* ]]; then
        continue
    # Skip blank lines
    elif [ -z "$dependency_stripped" ]; then
        continue
    else
        if pip install "$dependency_stripped"; then
            echo "$dependency_stripped is installed"
        else
            echo "Could not install $dependency_stripped, skipping"
        fi
    fi
done < recommends.txt

The xargs solution works but can have portability issues (BSD/GNU) and/or be cumbersome if you have comments or blank lines in your requirements file.

As for the usecase where such a behavior would be required, I use for instance two separate requirement files, one which is only listing core dependencies that need to be always installed and another file with non-core dependencies that are in 90% of the cases not needed for most usecases. That would be an equivalent of the Recommends section of a debian package.

I use the following shell script (requires sed) to install optional dependencies:

#!/bin/sh

while read dependency; do
    dependency_stripped="$(echo "${dependency}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
    # Skip comments
    if [[ $dependency_stripped == \#* ]]; then
        continue
    # Skip blank lines
    elif [ -z "$dependency_stripped" ]; then
        continue
    else
        if pip install "$dependency_stripped"; then
            echo "$dependency_stripped is installed"
        else
            echo "Could not install $dependency_stripped, skipping"
        fi
    fi
done < recommends.txt

回答 4

谢谢Etienne Prothon提供的Windows保护套。

但是,升级到pip 18后,pip包不会将main公开。因此,您可能需要像这样更改代码。

 # This code install line by line a list of pip package 
 import sys
 from pip._internal import main as pip_main

 def install(package):
    pip_main(['install', package])

 if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        for line in f:
            install(line)

Thanks, Etienne Prothon for windows cases.

But, after upgrading to pip 18, pip package don’t expose main to public. So you may need to change code like this.

 # This code install line by line a list of pip package 
 import sys
 from pip._internal import main as pip_main

 def install(package):
    pip_main(['install', package])

 if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        for line in f:
            install(line)

回答 5

对于Windows:

import os
from pip.__main__ import _main as main

error_log = open('error_log.txt', 'w')

def install(package):
    try:
        main(['install'] + [str(package)])
    except Exception as e:
        error_log.write(str(e))

if __name__ == '__main__':
    f = open('requirements1.txt', 'r')
    for line in f:
        install(line)
    f.close()
    error_log.close()
  1. 创建一个本地目录,然后将requirements.txt文件放入其中。
  2. 复制上面的代码,并将其另存为python文件在同一目录中。.py例如,请记住使用扩展名,install_packages.py
  3. 使用cmd运行此文件: python install_packages.py
  4. 提到的所有软件包都将一口气安装起来,而无需停止。:)

您可以在安装功能中添加其他参数。喜欢: main(['install'] + [str(package)] + ['--update'])

For Windows:

import os
from pip.__main__ import _main as main

error_log = open('error_log.txt', 'w')

def install(package):
    try:
        main(['install'] + [str(package)])
    except Exception as e:
        error_log.write(str(e))

if __name__ == '__main__':
    f = open('requirements1.txt', 'r')
    for line in f:
        install(line)
    f.close()
    error_log.close()
  1. Create a local directory, and put your requirements.txt file in it.
  2. Copy the code above and save it as a python file in the same directory. Remember to use .py extension, for instance, install_packages.py
  3. Run this file using a cmd: python install_packages.py
  4. All the packages mentioned will be installed in one go without stopping at all. :)

You can add other parameters in install function. Like: main(['install'] + [str(package)] + ['--update'])


声明:本站所有文章,如无特殊说明或标注,均为本站原创发布。任何个人或组织,在未征得本站同意时,禁止复制、盗用、采集、发布本站内容到任何网站、书籍等各类媒体平台。如若本站内容侵犯了原著者的合法权益,可联系我们进行处理。