检查对象是否为数字的最有效方法是什么?

问题:检查对象是否为数字的最有效方法是什么?

给定一个任意的python对象,确定它是否为数字的最佳方法是什么?这里is定义为acts like a number in certain circumstances

例如,假设您正在编写向量类。如果给定另一个向量,则要查找点积。如果给出标量,则要缩放整个矢量。

检查,如果事情是intfloatlongbool很烦人,不包括可能像数字用户定义的对象。但是,__mul__例如,检查并不够好,因为我刚刚描述的向量类将定义__mul__,但它不是我想要的那种类型。

Given an arbitrary python object, what’s the best way to determine whether it is a number? Here is is defined as acts like a number in certain circumstances.

For example, say you are writing a vector class. If given another vector, you want to find the dot product. If given a scalar, you want to scale the whole vector.

Checking if something is int, float, long, bool is annoying and doesn’t cover user-defined objects that might act like numbers. But, checking for __mul__, for example, isn’t good enough because the vector class I just described would define __mul__, but it wouldn’t be the kind of number I want.


回答 0

使用Numbernumbers模块测试isinstance(n, Number)(因为2.6可用)。

>>> from numbers import Number
... from decimal import Decimal
... from fractions import Fraction
... for n in [2, 2.0, Decimal('2.0'), complex(2, 0), Fraction(2, 1), '2']:
...     print(f'{n!r:>14} {isinstance(n, Number)}')
              2 True
            2.0 True
 Decimal('2.0') True
         (2+0j) True
 Fraction(2, 1) True
            '2' False

当然,这与鸭子的打字相反。如果你更关心的对象如何行为,而不是它什么,执行您的操作,如果你有一个号码,使用异常,否则告诉你。

Use Number from the numbers module to test isinstance(n, Number) (available since 2.6).

>>> from numbers import Number
... from decimal import Decimal
... from fractions import Fraction
... for n in [2, 2.0, Decimal('2.0'), complex(2, 0), Fraction(2, 1), '2']:
...     print(f'{n!r:>14} {isinstance(n, Number)}')
              2 True
            2.0 True
 Decimal('2.0') True
         (2+0j) True
 Fraction(2, 1) True
            '2' False

This is, of course, contrary to duck typing. If you are more concerned about how an object acts rather than what it is, perform your operations as if you have a number and use exceptions to tell you otherwise.


回答 1

您要检查是否有物体

在某些情况下像数字一样

如果您使用的是Python 2.5或更早版本,则唯一的真实方法是检查某些“特定情况”并查看。

在2.6或更好的,你可以使用isinstancenumbers.Number -一个抽象基类(ABC)存在正是为了这个目的(其它更多的ABC中存在的collections模块为各种形式的集合/容器,重新开始与2.6;以及同样仅在这些发行版中,如果需要,您可以轻松地添加自己的抽象基类。

0在某些情况下,Bach达到2.5或更早版本时,“可以添加但不能迭代”可能是一个很好的定义。但是,您确实需要问自己,您要问的是,您要考虑的“数字”一定一定能够做的,而它绝对不能做的是什么,然后检查。

在2.6或更高版本中也可能需要这样做,也许是出于进行自己的注册以添加您尚未注册的您所关心的类型的目的numbers.Numbers-如果您想排除某些声称其为数字的类型,但是您只是无法处理,这需要更多的注意,因为ABC没有unregister方法[[例如,您可以制作自己的ABC WeirdNum并在其中注册所有此类怪异类型,然后isinstance在继续进行之前先检查其保释金检查isinstance正常numbers.Number是否继续成功。

顺便说一句,是否以及何时需要检查是否x可以做某事,通常必须尝试以下操作:

try: 0 + x
except TypeError: canadd=False
else: canadd=True

__add__本身的存在告诉您没有什么用处,因为例如所有序列都具有将其与其他序列连接的目的。例如,此检查等效于定义“数字是某种东西,使得这样的事物的序列是内置函数的有效单个参数sum”。完全怪异的类型(例如,总和为0时引发“错误的”异常的类型,例如a ZeroDivisionErrorValueError&c)将传播异常,但这没关系,让用户尽快知道这样的疯狂类型根本不能接受公司;-); 但是,一个可乘以标量的“向量”(Python的标准库没有),但是在这里它当然是作为第三方扩展而流行的),在这里也会给出错误的结果,因此(例如“不允许迭代”(例如,检查是否iter(x)加注TypeError,或者是否存在特殊方法__iter__-如果您的年龄在2.5或更早,因此需要您自己进行检查)。

简要了解一下此类复杂性可能足以激励您在可行的情况下改为依赖抽象基类。

You want to check if some object

acts like a number in certain circumstances

If you’re using Python 2.5 or older, the only real way is to check some of those “certain circumstances” and see.

In 2.6 or better, you can use isinstance with numbers.Number — an abstract base class (ABC) that exists exactly for this purpose (lots more ABCs exist in the collections module for various forms of collections/containers, again starting with 2.6; and, also only in those releases, you can easily add your own abstract base classes if you need to).

Bach to 2.5 and earlier, “can be added to 0 and is not iterable” could be a good definition in some cases. But, you really need to ask yourself, what it is that you’re asking that what you want to consider “a number” must definitely be able to do, and what it must absolutely be unable to do — and check.

This may also be needed in 2.6 or later, perhaps for the purpose of making your own registrations to add types you care about that haven’t already be registered onto numbers.Numbers — if you want to exclude some types that claim they’re numbers but you just can’t handle, that takes even more care, as ABCs have no unregister method [[for example you could make your own ABC WeirdNum and register there all such weird-for-you types, then first check for isinstance thereof to bail out before you proceed to checking for isinstance of the normal numbers.Number to continue successfully.

BTW, if and when you need to check if x can or cannot do something, you generally have to try something like:

try: 0 + x
except TypeError: canadd=False
else: canadd=True

The presence of __add__ per se tells you nothing useful, since e.g all sequences have it for the purpose of concatenation with other sequences. This check is equivalent to the definition “a number is something such that a sequence of such things is a valid single argument to the builtin function sum“, for example. Totally weird types (e.g. ones that raise the “wrong” exception when summed to 0, such as, say, a ZeroDivisionError or ValueError &c) will propagate exception, but that’s OK, let the user know ASAP that such crazy types are just not acceptable in good company;-); but, a “vector” that’s summable to a scalar (Python’s standard library doesn’t have one, but of course they’re popular as third party extensions) would also give the wrong result here, so (e.g.) this check should come after the “not allowed to be iterable” one (e.g., check that iter(x) raises TypeError, or for the presence of special method __iter__ — if you’re in 2.5 or earlier and thus need your own checks).

A brief glimpse at such complications may be sufficient to motivate you to rely instead on abstract base classes whenever feasible…;-).


回答 2

这是一个exceptions真正发光的好例子。只需执行对数字类型的处理,然后TypeError从其他所有类型中捕获即可。

但是显然,这只会检查操作是否有效,而不是是否有意义!唯一真正的解决方案是永远不要混合类型,并且始终确切地知道您的值属于什么类型类。

This is a good example where exceptions really shine. Just do what you would do with the numeric types and catch the TypeError from everything else.

But obviously, this only checks if a operation works, not whether it makes sense! The only real solution for that is to never mix types and always know exactly what typeclass your values belong to.


回答 3

将对象乘以零。任何数字乘以零就是零。其他任何结果均表示该对象不是数字(包括异常)

def isNumber(x):
    try:
        return bool(0 == x*0)
    except:
        return False

因此,使用isNumber将给出以下输出:

class A: pass 

def foo(): return 1

for x in [1,1.4, A(), range(10), foo, foo()]:
    answer = isNumber(x)
    print('{answer} == isNumber({x})'.format(**locals()))

输出:

True == isNumber(1)
True == isNumber(1.4)
False == isNumber(<__main__.A instance at 0x7ff52c15d878>)
False == isNumber([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
False == isNumber(<function foo at 0x7ff52c121488>)
True == isNumber(1)

世界上可能有一些非数字对象定义__mul__为乘以零时返回零,但这是一个极端的exceptions。该解决方案应涵盖您生成/诱骗的所有正常代码和健全代码。

numpy.array示例:

import numpy as np

def isNumber(x):
    try:
        return bool(x*0 == 0)
    except:
        return False

x = np.array([0,1])

answer = isNumber(x)
print('{answer} == isNumber({x})'.format(**locals()))

输出:

False == isNumber([0 1])

Multiply the object by zero. Any number times zero is zero. Any other result means that the object is not a number (including exceptions)

def isNumber(x):
    try:
        return bool(0 == x*0)
    except:
        return False

Using isNumber thusly will give the following output:

class A: pass 

def foo(): return 1

for x in [1,1.4, A(), range(10), foo, foo()]:
    answer = isNumber(x)
    print('{answer} == isNumber({x})'.format(**locals()))

Output:

True == isNumber(1)
True == isNumber(1.4)
False == isNumber(<__main__.A instance at 0x7ff52c15d878>)
False == isNumber([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
False == isNumber(<function foo at 0x7ff52c121488>)
True == isNumber(1)

There probably are some non-number objects in the world that define __mul__ to return zero when multiplied by zero but that is an extreme exception. This solution should cover all normal and sane code that you generate/encouter.

numpy.array example:

import numpy as np

def isNumber(x):
    try:
        return bool(x*0 == 0)
    except:
        return False

x = np.array([0,1])

answer = isNumber(x)
print('{answer} == isNumber({x})'.format(**locals()))

output:

False == isNumber([0 1])

回答 4

为了改写您的问题,您正在尝试确定某物是集合还是单个值。试图比较某物是矢量还是数字,就是将苹果与橘子进行比较-我可以使用字符串或数字的矢量,也可以使用单个字符串或数字。您对拥有的数量(1个或更多)感兴趣,而不是实际的类型感兴趣

我对此问题的解决方案是通过检查是否存在来检查输入是单个值还是集合__len__。例如:

def do_mult(foo, a_vector):
    if hasattr(foo, '__len__'):
        return sum([a*b for a,b in zip(foo, a_vector)])
    else:
        return [foo*b for b in a_vector]

或者,对于鸭子输入法,您可以先尝试迭代foo

def do_mult(foo, a_vector):
    try:
        return sum([a*b for a,b in zip(foo, a_vector)])
    except TypeError:
        return [foo*b for b in a_vector]

最终,测试某事物是否类似于矢量比测试某事物是否类似于标量要容易。如果您遇到不同类型的值(例如,字符串,数字等),那么程序的逻辑可能需要做些工作-您最终是如何尝试将字符串乘以数字向量的?

To rephrase your question, you are trying to determine whether something is a collection or a single value. Trying to compare whether something is a vector or a number is comparing apples to oranges – I can have a vector of strings or numbers, and I can have a single string or single number. You are interested in how many you have (1 or more), not what type you actually have.

my solution for this problem is to check whether the input is a single value or a collection by checking the presence of __len__. For example:

def do_mult(foo, a_vector):
    if hasattr(foo, '__len__'):
        return sum([a*b for a,b in zip(foo, a_vector)])
    else:
        return [foo*b for b in a_vector]

Or, for the duck-typing approach, you can try iterating on foo first:

def do_mult(foo, a_vector):
    try:
        return sum([a*b for a,b in zip(foo, a_vector)])
    except TypeError:
        return [foo*b for b in a_vector]

Ultimately, it is easier to test whether something is vector-like than to test whether something is scalar-like. If you have values of different type (i.e. string, numeric, etc.) coming through, then the logic of your program may need some work – how did you end up trying to multiply a string by a numeric vector in the first place?


回答 5

总结/评估现有方法:

Candidate    | type                      | delnan | mat | shrewmouse | ant6n
-------------------------------------------------------------------------
0            | <type 'int'>              |      1 |   1 |          1 |     1
0.0          | <type 'float'>            |      1 |   1 |          1 |     1
0j           | <type 'complex'>          |      1 |   1 |          1 |     0
Decimal('0') | <class 'decimal.Decimal'> |      1 |   0 |          1 |     1
True         | <type 'bool'>             |      1 |   1 |          1 |     1
False        | <type 'bool'>             |      1 |   1 |          1 |     1
''           | <type 'str'>              |      0 |   0 |          0 |     0
None         | <type 'NoneType'>         |      0 |   0 |          0 |     0
'0'          | <type 'str'>              |      0 |   0 |          0 |     1
'1'          | <type 'str'>              |      0 |   0 |          0 |     1
[]           | <type 'list'>             |      0 |   0 |          0 |     0
[1]          | <type 'list'>             |      0 |   0 |          0 |     0
[1, 2]       | <type 'list'>             |      0 |   0 |          0 |     0
(1,)         | <type 'tuple'>            |      0 |   0 |          0 |     0
(1, 2)       | <type 'tuple'>            |      0 |   0 |          0 |     0

(我是通过这个问题来到这里的)

#!/usr/bin/env python

"""Check if a variable is a number."""

import decimal


def delnan_is_number(candidate):
    import numbers
    return isinstance(candidate, numbers.Number)


def mat_is_number(candidate):
    return isinstance(candidate, (int, long, float, complex))


def shrewmouse_is_number(candidate):
    try:
        return 0 == candidate * 0
    except:
        return False


def ant6n_is_number(candidate):
    try:
        float(candidate)
        return True
    except:
        return False

# Test
candidates = (0, 0.0, 0j, decimal.Decimal(0),
              True, False, '', None, '0', '1', [], [1], [1, 2], (1, ), (1, 2))

methods = [delnan_is_number, mat_is_number, shrewmouse_is_number, ant6n_is_number]

print("Candidate    | type                      | delnan | mat | shrewmouse | ant6n")
print("-------------------------------------------------------------------------")
for candidate in candidates:
    results = [m(candidate) for m in methods]
    print("{:<12} | {:<25} | {:>6} | {:>3} | {:>10} | {:>5}"
          .format(repr(candidate), type(candidate), *results))

To summarize / evaluate existing methods:

Candidate    | type                      | delnan | mat | shrewmouse | ant6n
-------------------------------------------------------------------------
0            | <type 'int'>              |      1 |   1 |          1 |     1
0.0          | <type 'float'>            |      1 |   1 |          1 |     1
0j           | <type 'complex'>          |      1 |   1 |          1 |     0
Decimal('0') | <class 'decimal.Decimal'> |      1 |   0 |          1 |     1
True         | <type 'bool'>             |      1 |   1 |          1 |     1
False        | <type 'bool'>             |      1 |   1 |          1 |     1
''           | <type 'str'>              |      0 |   0 |          0 |     0
None         | <type 'NoneType'>         |      0 |   0 |          0 |     0
'0'          | <type 'str'>              |      0 |   0 |          0 |     1
'1'          | <type 'str'>              |      0 |   0 |          0 |     1
[]           | <type 'list'>             |      0 |   0 |          0 |     0
[1]          | <type 'list'>             |      0 |   0 |          0 |     0
[1, 2]       | <type 'list'>             |      0 |   0 |          0 |     0
(1,)         | <type 'tuple'>            |      0 |   0 |          0 |     0
(1, 2)       | <type 'tuple'>            |      0 |   0 |          0 |     0

(I came here by this question)

Code

#!/usr/bin/env python

"""Check if a variable is a number."""

import decimal


def delnan_is_number(candidate):
    import numbers
    return isinstance(candidate, numbers.Number)


def mat_is_number(candidate):
    return isinstance(candidate, (int, long, float, complex))


def shrewmouse_is_number(candidate):
    try:
        return 0 == candidate * 0
    except:
        return False


def ant6n_is_number(candidate):
    try:
        float(candidate)
        return True
    except:
        return False

# Test
candidates = (0, 0.0, 0j, decimal.Decimal(0),
              True, False, '', None, '0', '1', [], [1], [1, 2], (1, ), (1, 2))

methods = [delnan_is_number, mat_is_number, shrewmouse_is_number, ant6n_is_number]

print("Candidate    | type                      | delnan | mat | shrewmouse | ant6n")
print("-------------------------------------------------------------------------")
for candidate in candidates:
    results = [m(candidate) for m in methods]
    print("{:<12} | {:<25} | {:>6} | {:>3} | {:>10} | {:>5}"
          .format(repr(candidate), type(candidate), *results))

回答 6

最好以相反的方式进行操作:检查它是否是向量。如果是,则进行点积运算,在所有其他情况下,将尝试进行标量乘法。

检查向量很容易,因为它应该是向量类类型(或从其继承)。您也可以先尝试做一个点积,如果失败了(=它实际上不是一个向量),然后退回到标量乘法。

Probably it’s better to just do it the other way around: You check if it’s a vector. If it is, you do a dot product and in all other cases you attempt scalar multiplication.

Checking for the vector is easy, since it should of your vector class type (or inherited from it). You could also just try first to do a dot-product, and if that fails (= it wasn’t really a vector), then fall back to scalar multiplication.


回答 7

只是为了补充。也许我们可以如下结合使用isinstance和isdigit来确定值是否为数字(int,float等)

如果isinstance(num1,int)或isinstance(num1,float)或num1.isdigit():

Just to add upon. Perhaps we can use a combination of isinstance and isdigit as follows to find whether a value is a number (int, float, etc)

if isinstance(num1, int) or isinstance(num1 , float) or num1.isdigit():


回答 8

对于假设的向量类:

假设v是一个向量,我们将其乘以x。如果是有意义的繁衍每个组件v通过x,我们或许意味着,所以尝试,第一。如果没有,也许我们可以点吗?否则是类型错误。

编辑 -以下代码不起作用,因为2*[0]==[0,0]而不是引发TypeError。我将其保留,因为它已被评论。

def __mul__( self, x ):
    try:
        return [ comp * x for comp in self ]
    except TypeError:
        return [ x * y for x, y in itertools.zip_longest( self, x, fillvalue = 0 )

For the hypothetical vector class:

Suppose v is a vector, and we are multiplying it by x. If it makes sense to multiply each component of v by x, we probably meant that, so try that first. If not, maybe we can dot? Otherwise it’s a type error.

EDIT — the below code doesn’t work, because 2*[0]==[0,0] instead of raising a TypeError. I leave it because it was commented-upon.

def __mul__( self, x ):
    try:
        return [ comp * x for comp in self ]
    except TypeError:
        return [ x * y for x, y in itertools.zip_longest( self, x, fillvalue = 0 )

回答 9

在实现某种矢量类时,我遇到了类似的问题。检查数字的一种方法是只转换为一个,即使用

float(x)

这应该拒绝x不能转换为数字的情况;但也可能会拒绝其他可能有效的类似数字的结构,例如复数。

I had a similar issue, when implementing a sort of vector class. One way to check for a number is to just convert to one, i.e. by using

float(x)

This should reject cases where x cannot be converted to a number; but may also reject other kinds of number-like structures that could be valid, for example complex numbers.


回答 10

如果要根据参数类型调用不同的方法,请查看multipledispatch

例如,假设您正在编写向量类。如果给定另一个向量,则要查找点积。如果给出标量,则要缩放整个矢量。

from multipledispatch import dispatch

class Vector(list):

    @dispatch(object)
    def __mul__(self, scalar):
        return Vector( x*scalar for x in self)

    @dispatch(list)
    def __mul__(self, other):
        return sum(x*y for x,y in zip(self, other))


>>> Vector([1,2,3]) * Vector([2,4,5])   # Vector time Vector is dot product
25
>>> Vector([1,2,3]) * 2                 # Vector times scalar is scaling
[2, 4, 6]

不幸的是,(据我所知)我们无法编写代码,@dispatch(Vector)因为我们仍在定义type Vector,因此尚未定义类型名称。相反,我使用的是基类型list,它甚至允许您找到a Vector和a 的点积list

If you want to call different methods depending on the argument type(s), look into multipledispatch.

For example, say you are writing a vector class. If given another vector, you want to find the dot product. If given a scalar, you want to scale the whole vector.

from multipledispatch import dispatch

class Vector(list):

    @dispatch(object)
    def __mul__(self, scalar):
        return Vector( x*scalar for x in self)

    @dispatch(list)
    def __mul__(self, other):
        return sum(x*y for x,y in zip(self, other))


>>> Vector([1,2,3]) * Vector([2,4,5])   # Vector time Vector is dot product
25
>>> Vector([1,2,3]) * 2                 # Vector times scalar is scaling
[2, 4, 6]

Unfortunately, (to my knowledge) we can’t write @dispatch(Vector) since we are still defining the type Vector, so that type name is not yet defined. Instead, I’m using the base type list, which allows you to even find the dot product of a Vector and a list.


回答 11

简短的方法:

obj = 12345
print(isinstance(obj,int))

输出:

True

如果对象是字符串,则将返回’False’:

obj = 'some string'
print(isinstance(obj,int))

输出:

False

Short and simple way :

obj = 12345
print(isinstance(obj,int))

Output :

True

If the object is a string, ‘False’ will be returned :

obj = 'some string'
print(isinstance(obj,int))

Output :

False

回答 12

您有一个数据项,说rec_day当写入文件时将是一个float。但程序处理期间,可以是floatintstr类型(str初始化一个新的记录时被使用并且包含一个伪标记的值)。

然后,您可以检查一下是否有此号码

                type(rec_day) != str 

我已经以这种方式构造了一个python程序,然后将其作为数字检查放入“维护补丁”中。这是Python方式吗?很可能没有,因为我以前使用COBOL编程。

You have a data item, say rec_day that when written to a file will be a float. But during program processing it can be either float, int or str type (the str is used when initializing a new record and contains a dummy flag value).

You can then check to see if you have a number with this

                type(rec_day) != str 

I’ve structured a python program this way and just put in ‘maintenance patch’ using this as a numeric check. Is it the Pythonic way? Most likely not since I used to program in COBOL.


回答 13

您可以使用isdigit()函数。

>>> x = "01234"
>>> a.isdigit()
True
>>> y = "1234abcd"
>>> y.isdigit()
False

You could use the isdigit() function.

>>> x = "01234"
>>> a.isdigit()
True
>>> y = "1234abcd"
>>> y.isdigit()
False

在matplotlib中动态更新图

问题:在matplotlib中动态更新图

我正在用Python开发一个应用程序,该应用程序从串行端口收集数据并针对到达时间绘制收集的数据图。数据的到达时间不确定。我希望在收到数据时更新绘图。我搜索了如何执行此操作,发现了两种方法:

  1. 清除绘图,然后重新绘制所有点的绘图。
  2. 通过在特定间隔后对其进行更改来对图进行动画处理。

我不喜欢第一个程序,因为该程序会运行很长时间(例如一天),并且会收集数据,因此重绘该图将非常慢。第二种方法也不可取,因为数据的到达时间不确定,我希望仅在接收到数据时更新绘图。

有没有一种方法可以仅在收到数据后才通过添加更多点来更新图?

I am making an application in Python which collects data from a serial port and plots a graph of the collected data against arrival time. The time of arrival for the data is uncertain. I want the plot to be updated when data is received. I searched on how to do this and found two methods:

  1. Clear the plot and re-draw the plot with all the points again.
  2. Animate the plot by changing it after a particular interval.

I do not prefer the first one as the program runs and collects data for a long time (a day for example), and redrawing the plot will be pretty slow. The second one is also not preferable as time of arrival of data is uncertain and I want the plot to update only when the data is received.

Is there a way in which I can update the plot just by adding more points to it only when the data is received?


回答 0

有没有一种方法可以通过添加更多点来更新图…

matplotlib中有多种动画数据的方式,具体取决于您使用的版本。您看过matplotlib食谱示例吗?另外,请在matplotlib文档中查看更现代的动画示例。最后,动画API定义了一个函数FuncAnimation,该函数可以对一个函数进行实时动画处理。该功能可能只是您用来获取数据的功能。

每种方法基本上都设置了data要绘制的对象的属性,因此不需要清除屏幕或图形。data可以简单地扩展该属性,因此您可以保留先前的点,而只需继续添加到线(或图像或所绘制的任何东西)中。

假设您说数据到达时间不确定,那么最好的选择就是执行以下操作:

import matplotlib.pyplot as plt
import numpy

hl, = plt.plot([], [])

def update_line(hl, new_data):
    hl.set_xdata(numpy.append(hl.get_xdata(), new_data))
    hl.set_ydata(numpy.append(hl.get_ydata(), new_data))
    plt.draw()

然后,当您从串行端口接收数据时,只需调用update_line

Is there a way in which I can update the plot just by adding more point[s] to it…

There are a number of ways of animating data in matplotlib, depending on the version you have. Have you seen the matplotlib cookbook examples? Also, check out the more modern animation examples in the matplotlib documentation. Finally, the animation API defines a function FuncAnimation which animates a function in time. This function could just be the function you use to acquire your data.

Each method basically sets the data property of the object being drawn, so doesn’t require clearing the screen or figure. The data property can simply be extended, so you can keep the previous points and just keep adding to your line (or image or whatever you are drawing).

Given that you say that your data arrival time is uncertain your best bet is probably just to do something like:

import matplotlib.pyplot as plt
import numpy

hl, = plt.plot([], [])

def update_line(hl, new_data):
    hl.set_xdata(numpy.append(hl.get_xdata(), new_data))
    hl.set_ydata(numpy.append(hl.get_ydata(), new_data))
    plt.draw()

Then when you receive data from the serial port just call update_line.


回答 1

为了在不使用FuncAnimation的情况下执行此操作(例如,要在生成图时执行代码的其他部分,或者要同时更新多个图),draw单独调用不会生成图(至少使用qt后端)。

以下对我有用:

import matplotlib.pyplot as plt
plt.ion()
class DynamicUpdate():
    #Suppose we know the x range
    min_x = 0
    max_x = 10

    def on_launch(self):
        #Set up plot
        self.figure, self.ax = plt.subplots()
        self.lines, = self.ax.plot([],[], 'o')
        #Autoscale on unknown axis and known lims on the other
        self.ax.set_autoscaley_on(True)
        self.ax.set_xlim(self.min_x, self.max_x)
        #Other stuff
        self.ax.grid()
        ...

    def on_running(self, xdata, ydata):
        #Update data (with the new _and_ the old points)
        self.lines.set_xdata(xdata)
        self.lines.set_ydata(ydata)
        #Need both of these in order to rescale
        self.ax.relim()
        self.ax.autoscale_view()
        #We need to draw *and* flush
        self.figure.canvas.draw()
        self.figure.canvas.flush_events()

    #Example
    def __call__(self):
        import numpy as np
        import time
        self.on_launch()
        xdata = []
        ydata = []
        for x in np.arange(0,10,0.5):
            xdata.append(x)
            ydata.append(np.exp(-x**2)+10*np.exp(-(x-7)**2))
            self.on_running(xdata, ydata)
            time.sleep(1)
        return xdata, ydata

d = DynamicUpdate()
d()

In order to do this without FuncAnimation (eg you want to execute other parts of the code while the plot is being produced or you want to be updating several plots at the same time), calling draw alone does not produce the plot (at least with the qt backend).

The following works for me:

import matplotlib.pyplot as plt
plt.ion()
class DynamicUpdate():
    #Suppose we know the x range
    min_x = 0
    max_x = 10

    def on_launch(self):
        #Set up plot
        self.figure, self.ax = plt.subplots()
        self.lines, = self.ax.plot([],[], 'o')
        #Autoscale on unknown axis and known lims on the other
        self.ax.set_autoscaley_on(True)
        self.ax.set_xlim(self.min_x, self.max_x)
        #Other stuff
        self.ax.grid()
        ...

    def on_running(self, xdata, ydata):
        #Update data (with the new _and_ the old points)
        self.lines.set_xdata(xdata)
        self.lines.set_ydata(ydata)
        #Need both of these in order to rescale
        self.ax.relim()
        self.ax.autoscale_view()
        #We need to draw *and* flush
        self.figure.canvas.draw()
        self.figure.canvas.flush_events()

    #Example
    def __call__(self):
        import numpy as np
        import time
        self.on_launch()
        xdata = []
        ydata = []
        for x in np.arange(0,10,0.5):
            xdata.append(x)
            ydata.append(np.exp(-x**2)+10*np.exp(-(x-7)**2))
            self.on_running(xdata, ydata)
            time.sleep(1)
        return xdata, ydata

d = DynamicUpdate()
d()

回答 2

这是一种允许在绘制一定数量的点之后删除点的方法:

import matplotlib.pyplot as plt
# generate axes object
ax = plt.axes()

# set limits
plt.xlim(0,10) 
plt.ylim(0,10)

for i in range(10):        
     # add something to axes    
     ax.scatter([i], [i]) 
     ax.plot([i], [i+1], 'rx')

     # draw the plot
     plt.draw() 
     plt.pause(0.01) #is necessary for the plot to update for some reason

     # start removing points if you don't want all shown
     if i>2:
         ax.lines[0].remove()
         ax.collections[0].remove()

Here is a way which allows to remove points after a certain number of points plotted:

import matplotlib.pyplot as plt
# generate axes object
ax = plt.axes()

# set limits
plt.xlim(0,10) 
plt.ylim(0,10)

for i in range(10):        
     # add something to axes    
     ax.scatter([i], [i]) 
     ax.plot([i], [i+1], 'rx')

     # draw the plot
     plt.draw() 
     plt.pause(0.01) #is necessary for the plot to update for some reason

     # start removing points if you don't want all shown
     if i>2:
         ax.lines[0].remove()
         ax.collections[0].remove()

回答 3

我知道我来不及回答这个问题,但是对于您的问题,您可以考虑使用“操纵杆”包装。我设计它用于从串行端口绘制数据流,但它适用于任何流。它还允许交互式文本记录或图像绘图(除了图形绘图外)。无需在单独的线程中执行自己的循环,程序包会照顾它,只需提供您希望的更新频率即可。另外,终端在绘制时仍可用于监视命令。参见http://www.github.com/ceyzeriat/joystick/https://pypi.python.org/pypi/joystick(使用pip install游戏杆进行安装)

只需将np.random.random()替换为您从以下代码中从串行端口读取的实际数据点:

import joystick as jk
import numpy as np
import time

class test(jk.Joystick):
    # initialize the infinite loop decorator
    _infinite_loop = jk.deco_infinite_loop()

    def _init(self, *args, **kwargs):
        """
        Function called at initialization, see the doc
        """
        self._t0 = time.time()  # initialize time
        self.xdata = np.array([self._t0])  # time x-axis
        self.ydata = np.array([0.0])  # fake data y-axis
        # create a graph frame
        self.mygraph = self.add_frame(jk.Graph(name="test", size=(500, 500), pos=(50, 50), fmt="go-", xnpts=10000, xnptsmax=10000, xylim=(None, None, 0, 1)))

    @_infinite_loop(wait_time=0.2)
    def _generate_data(self):  # function looped every 0.2 second to read or produce data
        """
        Loop starting with the simulation start, getting data and
    pushing it to the graph every 0.2 seconds
        """
        # concatenate data on the time x-axis
        self.xdata = jk.core.add_datapoint(self.xdata, time.time(), xnptsmax=self.mygraph.xnptsmax)
        # concatenate data on the fake data y-axis
        self.ydata = jk.core.add_datapoint(self.ydata, np.random.random(), xnptsmax=self.mygraph.xnptsmax)
        self.mygraph.set_xydata(t, self.ydata)

t = test()
t.start()
t.stop()

I know I’m late to answer this question, but for your issue you could look into the “joystick” package. I designed it for plotting a stream of data from the serial port, but it works for any stream. It also allows for interactive text logging or image plotting (in addition to graph plotting). No need to do your own loops in a separate thread, the package takes care of it, just give the update frequency you wish. Plus the terminal remains available for monitoring commands while plotting. See http://www.github.com/ceyzeriat/joystick/ or https://pypi.python.org/pypi/joystick (use pip install joystick to install)

Just replace np.random.random() by your real data point read from the serial port in the code below:

import joystick as jk
import numpy as np
import time

class test(jk.Joystick):
    # initialize the infinite loop decorator
    _infinite_loop = jk.deco_infinite_loop()

    def _init(self, *args, **kwargs):
        """
        Function called at initialization, see the doc
        """
        self._t0 = time.time()  # initialize time
        self.xdata = np.array([self._t0])  # time x-axis
        self.ydata = np.array([0.0])  # fake data y-axis
        # create a graph frame
        self.mygraph = self.add_frame(jk.Graph(name="test", size=(500, 500), pos=(50, 50), fmt="go-", xnpts=10000, xnptsmax=10000, xylim=(None, None, 0, 1)))

    @_infinite_loop(wait_time=0.2)
    def _generate_data(self):  # function looped every 0.2 second to read or produce data
        """
        Loop starting with the simulation start, getting data and
    pushing it to the graph every 0.2 seconds
        """
        # concatenate data on the time x-axis
        self.xdata = jk.core.add_datapoint(self.xdata, time.time(), xnptsmax=self.mygraph.xnptsmax)
        # concatenate data on the fake data y-axis
        self.ydata = jk.core.add_datapoint(self.ydata, np.random.random(), xnptsmax=self.mygraph.xnptsmax)
        self.mygraph.set_xydata(t, self.ydata)

t = test()
t.start()
t.stop()

保存和加载对象以及使用泡菜

问题:保存和加载对象以及使用泡菜

我正在尝试使用pickle模块保存和加载对象。
首先,我声明我的对象:

>>> class Fruits:pass
...
>>> banana = Fruits()

>>> banana.color = 'yellow'
>>> banana.value = 30

之后,我打开一个名为“ Fruits.obj”的文件(以前,我创建了一个新的.txt文件,并将其重命名为“ Fruits.obj”):

>>> import pickle
>>> filehandler = open(b"Fruits.obj","wb")
>>> pickle.dump(banana,filehandler)

完成此操作后,我关闭了会话,开始了一个新会话,然后放入下一个会话(尝试访问应该保存的对象):

file = open("Fruits.obj",'r')
object_file = pickle.load(file)

但是我有这个信息:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python31\lib\pickle.py", line 1365, in load
encoding=encoding, errors=errors).load()
ValueError: read() from the underlying stream did notreturn bytes

我不知道该怎么办,因为我不了解此消息。有人知道我如何加载对象“香蕉”吗?谢谢!

编辑: 正如你们中的一些人所说的那样:

>>> import pickle
>>> file = open("Fruits.obj",'rb')

没问题,但是我要讲的是:

>>> object_file = pickle.load(file)

我有错误:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python31\lib\pickle.py", line 1365, in load
encoding=encoding, errors=errors).load()
EOFError

I´m trying to save and load objects using pickle module.
First I declare my objects:

>>> class Fruits:pass
...
>>> banana = Fruits()

>>> banana.color = 'yellow'
>>> banana.value = 30

After that I open a file called ‘Fruits.obj'(previously I created a new .txt file and I renamed ‘Fruits.obj’):

>>> import pickle
>>> filehandler = open(b"Fruits.obj","wb")
>>> pickle.dump(banana,filehandler)

After do this I close my session and I began a new one and I put the next (trying to access to the object that it supposed to be saved):

file = open("Fruits.obj",'r')
object_file = pickle.load(file)

But I have this message:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python31\lib\pickle.py", line 1365, in load
encoding=encoding, errors=errors).load()
ValueError: read() from the underlying stream did notreturn bytes

I don´t know what to do because I don´t understand this message. Does anyone know How I can load my object ‘banana’? Thank you!

EDIT: As some of you have sugested I put:

>>> import pickle
>>> file = open("Fruits.obj",'rb')

There were no problem, but the next I put was:

>>> object_file = pickle.load(file)

And I have error:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python31\lib\pickle.py", line 1365, in load
encoding=encoding, errors=errors).load()
EOFError

回答 0

至于第二个问题:

 Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "C:\Python31\lib\pickle.py", line
 1365, in load encoding=encoding,
 errors=errors).load() EOFError

读取文件内容后,文件指针将位于文件末尾-不再有其他数据可读取。您必须倒带该文件,以便从头开始再次读取它:

file.seek(0)

但是,您通常要使用上下文管理器打开文件并从中读取数据。这样,文件将在块执行完后自动关闭,这也将帮助您将文件操作组织为有意义的块。

最后,cPickle是C语言中pickle模块的更快实现。因此:

In [1]: import cPickle

In [2]: d = {"a": 1, "b": 2}

In [4]: with open(r"someobject.pickle", "wb") as output_file:
   ...:     cPickle.dump(d, output_file)
   ...:

# pickle_file will be closed at this point, preventing your from accessing it any further

In [5]: with open(r"someobject.pickle", "rb") as input_file:
   ...:     e = cPickle.load(input_file)
   ...:

In [7]: print e
------> print(e)
{'a': 1, 'b': 2}

As for your second problem:

 Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "C:\Python31\lib\pickle.py", line
 1365, in load encoding=encoding,
 errors=errors).load() EOFError

After you have read the contents of the file, the file pointer will be at the end of the file – there will be no further data to read. You have to rewind the file so that it will be read from the beginning again:

file.seek(0)

What you usually want to do though, is to use a context manager to open the file and read data from it. This way, the file will be automatically closed after the block finishes executing, which will also help you organize your file operations into meaningful chunks.

Finally, cPickle is a faster implementation of the pickle module in C. So:

In [1]: import cPickle

In [2]: d = {"a": 1, "b": 2}

In [4]: with open(r"someobject.pickle", "wb") as output_file:
   ...:     cPickle.dump(d, output_file)
   ...:

# pickle_file will be closed at this point, preventing your from accessing it any further

In [5]: with open(r"someobject.pickle", "rb") as input_file:
   ...:     e = cPickle.load(input_file)
   ...:

In [7]: print e
------> print(e)
{'a': 1, 'b': 2}

回答 1

以下对我有用:

class Fruits: pass

banana = Fruits()

banana.color = 'yellow'
banana.value = 30

import pickle

filehandler = open("Fruits.obj","wb")
pickle.dump(banana,filehandler)
filehandler.close()

file = open("Fruits.obj",'rb')
object_file = pickle.load(file)
file.close()

print(object_file.color, object_file.value, sep=', ')
# yellow, 30

The following works for me:

class Fruits: pass

banana = Fruits()

banana.color = 'yellow'
banana.value = 30

import pickle

filehandler = open("Fruits.obj","wb")
pickle.dump(banana,filehandler)
filehandler.close()

file = open("Fruits.obj",'rb')
object_file = pickle.load(file)
file.close()

print(object_file.color, object_file.value, sep=', ')
# yellow, 30

回答 2

您也忘记将其读取为二进制文件。

在您的写作部分中,您有:

open(b"Fruits.obj","wb") # Note the wb part (Write Binary)

在阅读部分中,您有:

file = open("Fruits.obj",'r') # Note the r part, there should be a b too

因此,将其替换为:

file = open("Fruits.obj",'rb')

它将起作用:)


至于第二个错误,很可能是由于未正确关闭/同步文件而引起的。

尝试这段代码来编写:

>>> import pickle
>>> filehandler = open(b"Fruits.obj","wb")
>>> pickle.dump(banana,filehandler)
>>> filehandler.close()

这(不变)为:

>>> import pickle
>>> file = open("Fruits.obj",'rb')
>>> object_file = pickle.load(file)

更整洁的版本将使用该with语句。

写作:

>>> import pickle
>>> with open('Fruits.obj', 'wb') as fp:
>>>     pickle.dump(banana, fp)

阅读:

>>> import pickle
>>> with open('Fruits.obj', 'rb') as fp:
>>>     banana = pickle.load(fp)

You’re forgetting to read it as binary too.

In your write part you have:

open(b"Fruits.obj","wb") # Note the wb part (Write Binary)

In the read part you have:

file = open("Fruits.obj",'r') # Note the r part, there should be a b too

So replace it with:

file = open("Fruits.obj",'rb')

And it will work :)


As for your second error, it is most likely cause by not closing/syncing the file properly.

Try this bit of code to write:

>>> import pickle
>>> filehandler = open(b"Fruits.obj","wb")
>>> pickle.dump(banana,filehandler)
>>> filehandler.close()

And this (unchanged) to read:

>>> import pickle
>>> file = open("Fruits.obj",'rb')
>>> object_file = pickle.load(file)

A neater version would be using the with statement.

For writing:

>>> import pickle
>>> with open('Fruits.obj', 'wb') as fp:
>>>     pickle.dump(banana, fp)

For reading:

>>> import pickle
>>> with open('Fruits.obj', 'rb') as fp:
>>>     banana = pickle.load(fp)

回答 3

在这种情况下,始终以二进制模式打开

file = open("Fruits.obj",'rb')

Always open in binary mode, in this case

file = open("Fruits.obj",'rb')

回答 4

您没有以二进制模式打开文件。

open("Fruits.obj",'rb')

应该管用。

对于第二个错误,文件很可能为空,这意味着您无意中清空了文件或使用了错误的文件名或其他名称。

(这是假设您确实确实关闭了会话。如果没有,则是因为您没有在写入和读取之间关闭文件)。

我测试了您的代码,它可以正常工作。

You didn’t open the file in binary mode.

open("Fruits.obj",'rb')

Should work.

For your second error, the file is most likely empty, which mean you inadvertently emptied it or used the wrong filename or something.

(This is assuming you really did close your session. If not, then it’s because you didn’t close the file between the write and the read).

I tested your code, and it works.


回答 5

看来您想跨会话保存您的类实例,并且使用pickle是一种不错的方法。但是,有一个名为的程序包klepto,将对象的保存抽象到字典接口,因此您可以选择腌制对象并将其保存到文件(如下所示),或腌制对象并将其保存到数据库,或者选择使用pickle使用json或其他许多选项。有趣的klepto是,通过抽象到通用接口,它很容易,因此您不必记住如何通过酸洗保存到文件等其他底层细节。

请注意,它适用于动态添加的类属性,而pickle无法做到这一点…

dude@hilbert>$ python
Python 2.7.6 (default, Nov 12 2013, 13:26:39) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from klepto.archives import file_archive 
>>> db = file_archive('fruits.txt')
>>> class Fruits: pass
... 
>>> banana = Fruits()
>>> banana.color = 'yellow'
>>> banana.value = 30
>>> 
>>> db['banana'] = banana 
>>> db.dump()
>>> 

然后我们重新启动…

dude@hilbert>$ python
Python 2.7.6 (default, Nov 12 2013, 13:26:39) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from klepto.archives import file_archive
>>> db = file_archive('fruits.txt')
>>> db.load()
>>> 
>>> db['banana'].color
'yellow'
>>> 

Klepto 适用于python2和python3。

在此处获取代码:https : //github.com/uqfoundation

It seems you want to save your class instances across sessions, and using pickle is a decent way to do this. However, there’s a package called klepto that abstracts the saving of objects to a dictionary interface, so you can choose to pickle objects and save them to a file (as shown below), or pickle the objects and save them to a database, or instead of use pickle use json, or many other options. The nice thing about klepto is that by abstracting to a common interface, it makes it easy so you don’t have to remember the low-level details of how to save via pickling to a file, or otherwise.

Note that It works for dynamically added class attributes, which pickle cannot do…

dude@hilbert>$ python
Python 2.7.6 (default, Nov 12 2013, 13:26:39) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from klepto.archives import file_archive 
>>> db = file_archive('fruits.txt')
>>> class Fruits: pass
... 
>>> banana = Fruits()
>>> banana.color = 'yellow'
>>> banana.value = 30
>>> 
>>> db['banana'] = banana 
>>> db.dump()
>>> 

Then we restart…

dude@hilbert>$ python
Python 2.7.6 (default, Nov 12 2013, 13:26:39) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from klepto.archives import file_archive
>>> db = file_archive('fruits.txt')
>>> db.load()
>>> 
>>> db['banana'].color
'yellow'
>>> 

Klepto works on python2 and python3.

Get the code here: https://github.com/uqfoundation


回答 6

您可以使用anycache为您完成这项工作。假设您有一个myfunc创建实例的函数:

from anycache import anycache

class Fruits:pass

@anycache(cachedir='/path/to/your/cache')    
def myfunc()
    banana = Fruits()
    banana.color = 'yellow'
    banana.value = 30
return banana

Anycache会myfunc在第一次调用时,cachedir使用唯一的标识符(取决于函数名和参数)作为文件名,将结果腌制到文件中。在任何连续运行中,将加载已腌制的对象。

如果在cachedir两次python运行之间保留了,则腌制的对象将从先前的python运行中获取。

函数参数也被考虑在内。重构的实现也是如此:

from anycache import anycache

class Fruits:pass

@anycache(cachedir='/path/to/your/cache')    
def myfunc(color, value)
    fruit = Fruits()
    fruit.color = color
    fruit.value = value
return fruit

You can use anycache to do the job for you. Assuming you have a function myfunc which creates the instance:

from anycache import anycache

class Fruits:pass

@anycache(cachedir='/path/to/your/cache')    
def myfunc()
    banana = Fruits()
    banana.color = 'yellow'
    banana.value = 30
return banana

Anycache calls myfunc at the first time and pickles the result to a file in cachedir using an unique identifier (depending on the the function name and the arguments) as filename. On any consecutive run, the pickled object is loaded.

If the cachedir is preserved between python runs, the pickled object is taken from the previous python run.

The function arguments are also taken into account. A refactored implementation works likewise:

from anycache import anycache

class Fruits:pass

@anycache(cachedir='/path/to/your/cache')    
def myfunc(color, value)
    fruit = Fruits()
    fruit.color = color
    fruit.value = value
return fruit

Python方式打印列表项

问题:Python方式打印列表项

我想知道是否有比这更好的方法来打印Python列表中的所有对象:

myList = [Person("Foo"), Person("Bar")]
print("\n".join(map(str, myList)))
Foo
Bar

我读这种方式不是很好:

myList = [Person("Foo"), Person("Bar")]
for p in myList:
    print(p)

是否没有类似的东西:

print(p) for p in myList

如果没有,我的问题是…为什么?如果我们可以使用综合列表来完成此类工作,为什么不将其作为列表之外的简单语句呢?

I would like to know if there is a better way to print all objects in a Python list than this :

myList = [Person("Foo"), Person("Bar")]
print("\n".join(map(str, myList)))
Foo
Bar

I read this way is not really good :

myList = [Person("Foo"), Person("Bar")]
for p in myList:
    print(p)

Isn’t there something like :

print(p) for p in myList

If not, my question is… why ? If we can do this kind of stuff with comprehensive lists, why not as a simple statement outside a list ?


回答 0

假设您正在使用Python 3.x:

print(*myList, sep='\n')

您可以使用from __future__ import print_function,在Python 2.x上获得相同的行为,如mgilson在评论中所述。

使用Python 2.x上的print语句,您将需要某种形式的迭代,关于您的print(p) for p in myList不工作问题,您可以使用以下代码做同样的事情,并且仍然是一行:

for p in myList: print p

对于使用的解决方案'\n'.join(),我更喜欢列表推导和生成器,map()因此我可能会使用以下内容:

print '\n'.join(str(p) for p in myList) 

Assuming you are using Python 3.x:

print(*myList, sep='\n')

You can get the same behavior on Python 2.x using from __future__ import print_function, as noted by mgilson in comments.

With the print statement on Python 2.x you will need iteration of some kind, regarding your question about print(p) for p in myList not working, you can just use the following which does the same thing and is still one line:

for p in myList: print p

For a solution that uses '\n'.join(), I prefer list comprehensions and generators over map() so I would probably use the following:

print '\n'.join(str(p) for p in myList) 

回答 1

我经常用这个 :

#!/usr/bin/python

l = [1,2,3,7] 
print "".join([str(x) for x in l])

I use this all the time :

#!/usr/bin/python

l = [1,2,3,7] 
print "".join([str(x) for x in l])

回答 2

[print(a) for a in list] 尽管会打印出所有项目,但最后会给出一堆None类型

[print(a) for a in list] will give a bunch of None types at the end though it prints out all the items


回答 3

对于Python 2. *:

如果为您的Person类重载了函数__str __(),则可以省略带有map(str,…)的部分。另一种方法是创建一个函数,就像您写的那样:

def write_list(lst):
    for item in lst:
        print str(item) 

...

write_list(MyList)

Python 3. * 中有print()函数的参数sep。看一下文档。

For Python 2.*:

If you overload the function __str__() for your Person class, you can omit the part with map(str, …). Another way for this is creating a function, just like you wrote:

def write_list(lst):
    for item in lst:
        print str(item) 

...

write_list(MyList)

There is in Python 3.* the argument sep for the print() function. Take a look at documentation.


回答 4

扩展@lucasg的答案(受其收到的评论启发):

要获得格式化的列表输出,可以按照以下步骤进行操作:

l = [1,2,5]
print ", ".join('%02d'%x for x in l)

01, 02, 05

现在,", "提供分隔符(仅在项目之间,而不是末尾),并且'02d'结合使用%x格式字符串为每个项目提供格式化的字符串x-在这种情况下,格式为具有两位数字的整数,并用零填充。

Expanding @lucasg’s answer (inspired by the comment it received):

To get a formatted list output, you can do something along these lines:

l = [1,2,5]
print ", ".join('%02d'%x for x in l)

01, 02, 05

Now the ", " provides the separator (only between items, not at the end) and the formatting string '02d'combined with %x gives a formatted string for each item x – in this case, formatted as an integer with two digits, left-filled with zeros.


回答 5

要显示每个内容,我使用:

mylist = ['foo', 'bar']
indexval = 0
for i in range(len(mylist)):     
    print(mylist[indexval])
    indexval += 1

在函数中使用的示例:

def showAll(listname, startat):
   indexval = startat
   try:
      for i in range(len(mylist)):
         print(mylist[indexval])
         indexval = indexval + 1
   except IndexError:
      print('That index value you gave is out of range.')

希望我能帮上忙。

To display each content, I use:

mylist = ['foo', 'bar']
indexval = 0
for i in range(len(mylist)):     
    print(mylist[indexval])
    indexval += 1

Example of using in a function:

def showAll(listname, startat):
   indexval = startat
   try:
      for i in range(len(mylist)):
         print(mylist[indexval])
         indexval = indexval + 1
   except IndexError:
      print('That index value you gave is out of range.')

Hope I helped.


回答 6

我认为如果您只想查看列表中的内容,这是最方便的:

myList = ['foo', 'bar']
print('myList is %s' % str(myList))

简单,易读,可与格式字符串一起使用。

I think this is the most convenient if you just want to see the content in the list:

myList = ['foo', 'bar']
print('myList is %s' % str(myList))

Simple, easy to read and can be used together with format string.


回答 7

OP的问题是:是否存在类似以下内容的内容,如果不存在,为什么?

print(p) for p in myList # doesn't work, OP's intuition

答案是,它确实存在,它是:

[p for p in myList] #works perfectly

基本上,[]用于列表理解并print避免避免打印None。看看为什么print打印None看到这个

OP’s question is: does something like following exists, if not then why

print(p) for p in myList # doesn't work, OP's intuition

answer is, it does exist which is:

[p for p in myList] #works perfectly

Basically, use [] for list comprehension and get rid of print to avoiding printing None. To see why print prints None see this


回答 8

我最近制作了一个密码生成器,尽管我对python还是很陌生,但我还是想把它作为一种显示列表中所有项目的方式(进行一些小的修改即可满足您的需要…

    x = 0
    up = 0
    passwordText = ""
    password = []
    userInput = int(input("Enter how many characters you want your password to be: "))
    print("\n\n\n") # spacing

    while x <= (userInput - 1): #loops as many times as the user inputs above
            password.extend([choice(groups.characters)]) #adds random character from groups file that has all lower/uppercase letters and all numbers
            x = x+1 #adds 1 to x w/o using x ++1 as I get many errors w/ that
            passwordText = passwordText + password[up]
            up = up+1 # same as x increase


    print(passwordText)

就像我说的,IM对Python非常新,我相信这对于专家来说是笨拙的方式,但是我在这里只是另一个例子

I recently made a password generator and although I’m VERY NEW to python, I whipped this up as a way to display all items in a list (with small edits to fit your needs…

    x = 0
    up = 0
    passwordText = ""
    password = []
    userInput = int(input("Enter how many characters you want your password to be: "))
    print("\n\n\n") # spacing

    while x <= (userInput - 1): #loops as many times as the user inputs above
            password.extend([choice(groups.characters)]) #adds random character from groups file that has all lower/uppercase letters and all numbers
            x = x+1 #adds 1 to x w/o using x ++1 as I get many errors w/ that
            passwordText = passwordText + password[up]
            up = up+1 # same as x increase


    print(passwordText)

Like I said, IM VERY NEW to Python and I’m sure this is way to clunky for a expert, but I’m just here for another example


回答 9

假设您可以很好地打印列表[1,2,3],那么Python3中的一种简单方法是:

mylist=[1,2,3,'lorem','ipsum','dolor','sit','amet']

print(f"There are {len(mylist):d} items in this lorem list: {str(mylist):s}")

运行此命令将产生以下输出:

此lorem列表中有8个项目:[1、2、3,’lorem’,’ipsum’,’dolor’,’sit’,’amet’]

Assuming you are fine with your list being printed [1,2,3], then an easy way in Python3 is:

mylist=[1,2,3,'lorem','ipsum','dolor','sit','amet']

print(f"There are {len(mylist):d} items in this lorem list: {str(mylist):s}")

Running this produces the following output:

There are 8 items in this lorem list: [1, 2, 3, ‘lorem’, ‘ipsum’, ‘dolor’, ‘sit’, ‘amet’]


Python,在目录字符串中添加尾部斜线,独立于操作系统

问题:Python,在目录字符串中添加尾部斜线,独立于操作系统

如果尾部斜杠尚不存在,如何在目录字符串中添加尾部斜杠(/对于* nix,\对于win32)?谢谢!

How can I add a trailing slash (/ for *nix, \ for win32) to a directory string, if the tailing slash is not already there? Thanks!


回答 0

os.path.join(path, '') 如果末尾不存在斜线,则会添加该斜线。

你可以做os.path.join(path, '', '')os.path.join(path_with_a_trailing_slash, '')你仍然只会得到一个尾部的斜杠。

os.path.join(path, '') will add the trailing slash if it’s not already there.

You can do os.path.join(path, '', '') or os.path.join(path_with_a_trailing_slash, '') and you will still only get one trailing slash.


回答 1

由于要连接目录和文件名,请使用

os.path.join(directory, filename)

如果要摆脱.\..\..\blah\路径,请使用

os.path.join(os.path.normpath(directory), filename)

Since you want to connect a directory and a filename, use

os.path.join(directory, filename)

If you want to get rid of .\..\..\blah\ paths, use

os.path.join(os.path.normpath(directory), filename)

回答 2

您可以通过以下方式手动进行操作:

path = ...

import os
if not path.endswith(os.path.sep):
    path += os.path.sep

但是,通常使用起来更清洁os.path.join

You can do it manually by:

path = ...

import os
if not path.endswith(os.path.sep):
    path += os.path.sep

However, it is usually much cleaner to use os.path.join.


回答 3

您可以使用如下形式:

os.path.normcase(path)
    Normalize the case of a pathname. On Unix and Mac OS X, this returns the path unchanged; on case-insensitive filesystems, it converts the path to lowercase. On Windows, it also converts forward slashes to backward slashes.

否则,您可以在页面上寻找其他内容

You could use something like this:

os.path.normcase(path)
    Normalize the case of a pathname. On Unix and Mac OS X, this returns the path unchanged; on case-insensitive filesystems, it converts the path to lowercase. On Windows, it also converts forward slashes to backward slashes.

Else you could look for something else on this page


Python子进程。Popen“ OSError:[Errno 12]无法分配内存”

问题:Python子进程。Popen“ OSError:[Errno 12]无法分配内存”

注意:此问题最初是在此处提出的但赏金时间已过,即使实际上未找到可接受的答案。我正在重新询问这个问题,包括原始问题中提供的所有详细信息。

python脚本使用sched模块每60秒运行一组类函数:

# sc is a sched.scheduler instance
sc.enter(60, 1, self.doChecks, (sc, False))

该脚本使用此处的代码作为守护进程运行。

在doChecks中调用的许多类方法使用子过程模块来调用系统函数,以获取系统统计信息:

ps = subprocess.Popen(['ps', 'aux'], stdout=subprocess.PIPE).communicate()[0]

在整个脚本崩溃并出现以下错误之前,它可以正常运行一段时间:

File "/home/admin/sd-agent/checks.py", line 436, in getProcesses
File "/usr/lib/python2.4/subprocess.py", line 533, in __init__
File "/usr/lib/python2.4/subprocess.py", line 835, in _get_handles
OSError: [Errno 12] Cannot allocate memory

脚本崩溃后,服务器上的free -m输出为:

$ free -m
                  total       used       free     shared     buffers    cached
Mem:                894        345        549          0          0          0
-/+ buffers/cache:  345        549
Swap:                 0          0          0

服务器正在运行CentOS 5.3。我无法在自己的CentOS盒子上或任何其他报告相同问题的用户上进行复制。

我已经尝试了许多方法来调试此问题,如原始问题中所建议:

  1. 在Popen调用之前和之后记录free -m的输出。内存使用没有显着变化,即,脚本运行时内存不会逐渐消耗完。

  2. 我在Popen调用中添加了close_fds = True,但这没有什么不同-脚本仍然因相同的错误而崩溃。建议在这里这里

  3. 我检查了这所建议双方RLIMIT_DATA和RLIMIT_AS显示(-1,-1)的rlimits 这里

  4. 一篇文章建议没有交换空间可能是原因,但是交换实际上是按需提供的(根据Web主机),这在这里也被认为是虚假的原因。

  5. 进程已关闭,因为这是使用.communicate()的行为,该行为由Python源代码和此处的注释支持。

可以在GitHub上的第442行定义的getProcesses函数中找到整个检查。此操作由从第520行开始的doChecks()调用。

崩溃前,该脚本使用strace运行,输出如下:

recv(4, "Total Accesses: 516662\nTotal kBy"..., 234, 0) = 234
gettimeofday({1250893252, 887805}, NULL) = 0
write(3, "2009-08-21 17:20:52,887 - checks"..., 91) = 91
gettimeofday({1250893252, 888362}, NULL) = 0
write(3, "2009-08-21 17:20:52,888 - checks"..., 74) = 74
gettimeofday({1250893252, 888897}, NULL) = 0
write(3, "2009-08-21 17:20:52,888 - checks"..., 67) = 67
gettimeofday({1250893252, 889184}, NULL) = 0
write(3, "2009-08-21 17:20:52,889 - checks"..., 81) = 81
close(4)                                = 0
gettimeofday({1250893252, 889591}, NULL) = 0
write(3, "2009-08-21 17:20:52,889 - checks"..., 63) = 63
pipe([4, 5])                            = 0
pipe([6, 7])                            = 0
fcntl64(7, F_GETFD)                     = 0
fcntl64(7, F_SETFD, FD_CLOEXEC)         = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f12708) = -1 ENOMEM (Cannot allocate memory)
write(2, "Traceback (most recent call last"..., 35) = 35
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/agent."..., 52) = 52
open("/home/admin/sd-agent/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/home/admin/sd-agent/dae"..., 60) = 60
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/agent."..., 54) = 54
open("/usr/lib/python2.4/sched.py", O_RDONLY|O_LARGEFILE) = 8
write(2, "  File \"/usr/lib/python2.4/sched"..., 55) = 55
fstat64(8, {st_mode=S_IFREG|0644, st_size=4054, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
read(8, "\"\"\"A generally useful event sche"..., 4096) = 4054
write(2, "    ", 4)                     = 4
write(2, "void = action(*argument)\n", 25) = 25
close(8)                                = 0
munmap(0xb7d28000, 4096)                = 0
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/checks"..., 60) = 60
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/checks"..., 64) = 64
open("/usr/lib/python2.4/subprocess.py", O_RDONLY|O_LARGEFILE) = 8
write(2, "  File \"/usr/lib/python2.4/subpr"..., 65) = 65
fstat64(8, {st_mode=S_IFREG|0644, st_size=39931, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
read(8, "# subprocess - Subprocesses with"..., 4096) = 4096
read(8, "lso, the newlines attribute of t"..., 4096) = 4096
read(8, "code < 0:\n        print >>sys.st"..., 4096) = 4096
read(8, "alse does not exist on 2.2.0\ntry"..., 4096) = 4096
read(8, " p2cread\n        # c2pread    <-"..., 4096) = 4096
write(2, "    ", 4)                     = 4
write(2, "errread, errwrite)\n", 19)    = 19
close(8)                                = 0
munmap(0xb7d28000, 4096)                = 0
open("/usr/lib/python2.4/subprocess.py", O_RDONLY|O_LARGEFILE) = 8
write(2, "  File \"/usr/lib/python2.4/subpr"..., 71) = 71
fstat64(8, {st_mode=S_IFREG|0644, st_size=39931, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
read(8, "# subprocess - Subprocesses with"..., 4096) = 4096
read(8, "lso, the newlines attribute of t"..., 4096) = 4096
read(8, "code < 0:\n        print >>sys.st"..., 4096) = 4096
read(8, "alse does not exist on 2.2.0\ntry"..., 4096) = 4096
read(8, " p2cread\n        # c2pread    <-"..., 4096) = 4096
read(8, "table(self, handle):\n           "..., 4096) = 4096
read(8, "rrno using _sys_errlist (or siml"..., 4096) = 4096
read(8, " p2cwrite = None, None\n         "..., 4096) = 4096
write(2, "    ", 4)                     = 4
write(2, "self.pid = os.fork()\n", 21)  = 21
close(8)                                = 0
munmap(0xb7d28000, 4096)                = 0
write(2, "OSError", 7)                  = 7
write(2, ": ", 2)                       = 2
write(2, "[Errno 12] Cannot allocate memor"..., 33) = 33
write(2, "\n", 1)                       = 1
unlink("/var/run/sd-agent.pid")         = 0
close(3)                                = 0
munmap(0xb7e0d000, 4096)                = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x589978}, {0xb89a60, [], SA_RESTORER, 0x589978}, 8) = 0
brk(0xa022000)                          = 0xa022000
exit_group(1)                           = ?

Note: This question was originally asked here but the bounty time expired even though an acceptable answer was not actually found. I am re-asking this question including all details provided in the original question.

A python script is running a set of class functions every 60 seconds using the sched module:

# sc is a sched.scheduler instance
sc.enter(60, 1, self.doChecks, (sc, False))

The script is running as a daemonised process using the code here.

A number of class methods that are called as part of doChecks use the subprocess module to call system functions in order to get system statistics:

ps = subprocess.Popen(['ps', 'aux'], stdout=subprocess.PIPE).communicate()[0]

This runs fine for a period of time before the entire script crashing with the following error:

File "/home/admin/sd-agent/checks.py", line 436, in getProcesses
File "/usr/lib/python2.4/subprocess.py", line 533, in __init__
File "/usr/lib/python2.4/subprocess.py", line 835, in _get_handles
OSError: [Errno 12] Cannot allocate memory

The output of free -m on the server once the script has crashed is:

$ free -m
                  total       used       free     shared     buffers    cached
Mem:                894        345        549          0          0          0
-/+ buffers/cache:  345        549
Swap:                 0          0          0

The server is running CentOS 5.3. I am unable to reproduce on my own CentOS boxes nor with any other user reporting the same problem.

I have tried a number of things to debug this as suggested in the original question:

  1. Logging the output of free -m before and after the Popen call. There is no significant change in memory usage i.e. memory is not gradually being used up as the script runs.

  2. I added close_fds=True to the Popen call but this made no difference – the script still crashed with the same error. Suggested here and here.

  3. I checked the rlimits which showed (-1, -1) on both RLIMIT_DATA and RLIMIT_AS as suggested here.

  4. An article suggested the having no swap space might be the cause but swap is actually available on demand (according to the web host) and this was also suggested as a bogus cause here.

  5. The processes are being closed because that is the behaviour of using .communicate() as backed up by the Python source code and comments here.

The entire checks can be found at on GitHub here with the getProcesses function defined from line 442. This is called by doChecks() starting at line 520.

The script was run with strace with the following output before the crash:

recv(4, "Total Accesses: 516662\nTotal kBy"..., 234, 0) = 234
gettimeofday({1250893252, 887805}, NULL) = 0
write(3, "2009-08-21 17:20:52,887 - checks"..., 91) = 91
gettimeofday({1250893252, 888362}, NULL) = 0
write(3, "2009-08-21 17:20:52,888 - checks"..., 74) = 74
gettimeofday({1250893252, 888897}, NULL) = 0
write(3, "2009-08-21 17:20:52,888 - checks"..., 67) = 67
gettimeofday({1250893252, 889184}, NULL) = 0
write(3, "2009-08-21 17:20:52,889 - checks"..., 81) = 81
close(4)                                = 0
gettimeofday({1250893252, 889591}, NULL) = 0
write(3, "2009-08-21 17:20:52,889 - checks"..., 63) = 63
pipe([4, 5])                            = 0
pipe([6, 7])                            = 0
fcntl64(7, F_GETFD)                     = 0
fcntl64(7, F_SETFD, FD_CLOEXEC)         = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f12708) = -1 ENOMEM (Cannot allocate memory)
write(2, "Traceback (most recent call last"..., 35) = 35
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/agent."..., 52) = 52
open("/home/admin/sd-agent/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/home/admin/sd-agent/dae"..., 60) = 60
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/agent."..., 54) = 54
open("/usr/lib/python2.4/sched.py", O_RDONLY|O_LARGEFILE) = 8
write(2, "  File \"/usr/lib/python2.4/sched"..., 55) = 55
fstat64(8, {st_mode=S_IFREG|0644, st_size=4054, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
read(8, "\"\"\"A generally useful event sche"..., 4096) = 4054
write(2, "    ", 4)                     = 4
write(2, "void = action(*argument)\n", 25) = 25
close(8)                                = 0
munmap(0xb7d28000, 4096)                = 0
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/checks"..., 60) = 60
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/checks"..., 64) = 64
open("/usr/lib/python2.4/subprocess.py", O_RDONLY|O_LARGEFILE) = 8
write(2, "  File \"/usr/lib/python2.4/subpr"..., 65) = 65
fstat64(8, {st_mode=S_IFREG|0644, st_size=39931, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
read(8, "# subprocess - Subprocesses with"..., 4096) = 4096
read(8, "lso, the newlines attribute of t"..., 4096) = 4096
read(8, "code < 0:\n        print >>sys.st"..., 4096) = 4096
read(8, "alse does not exist on 2.2.0\ntry"..., 4096) = 4096
read(8, " p2cread\n        # c2pread    <-"..., 4096) = 4096
write(2, "    ", 4)                     = 4
write(2, "errread, errwrite)\n", 19)    = 19
close(8)                                = 0
munmap(0xb7d28000, 4096)                = 0
open("/usr/lib/python2.4/subprocess.py", O_RDONLY|O_LARGEFILE) = 8
write(2, "  File \"/usr/lib/python2.4/subpr"..., 71) = 71
fstat64(8, {st_mode=S_IFREG|0644, st_size=39931, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
read(8, "# subprocess - Subprocesses with"..., 4096) = 4096
read(8, "lso, the newlines attribute of t"..., 4096) = 4096
read(8, "code < 0:\n        print >>sys.st"..., 4096) = 4096
read(8, "alse does not exist on 2.2.0\ntry"..., 4096) = 4096
read(8, " p2cread\n        # c2pread    <-"..., 4096) = 4096
read(8, "table(self, handle):\n           "..., 4096) = 4096
read(8, "rrno using _sys_errlist (or siml"..., 4096) = 4096
read(8, " p2cwrite = None, None\n         "..., 4096) = 4096
write(2, "    ", 4)                     = 4
write(2, "self.pid = os.fork()\n", 21)  = 21
close(8)                                = 0
munmap(0xb7d28000, 4096)                = 0
write(2, "OSError", 7)                  = 7
write(2, ": ", 2)                       = 2
write(2, "[Errno 12] Cannot allocate memor"..., 33) = 33
write(2, "\n", 1)                       = 1
unlink("/var/run/sd-agent.pid")         = 0
close(3)                                = 0
munmap(0xb7e0d000, 4096)                = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x589978}, {0xb89a60, [], SA_RESTORER, 0x589978}, 8) = 0
brk(0xa022000)                          = 0xa022000
exit_group(1)                           = ?

回答 0

作为一般规则(即香草内核),fork/ clone有故障ENOMEM 发生的具体原因的任何一个诚实的神了内存不足的条件dup_mmdup_task_structalloc_pidmpol_dupmm_init等呱呱叫),或者是因为security_vm_enough_memory_mm你失望实施过载策略

首先,在尝试进行分叉时,检查未能分叉的进程的vmsize,然后将其与过量使用策略相关的可用内存(物理和交换)量进行比较(插入数字)。

在您的特定情况下,请注意,Virtuozzo 在过量使用执法方面还有其他检查。而且,我不确定您容器内部交换和过量使用配置真正拥有多少控制权(以影响执行结果)。

现在,为了真正前进,我想告诉您,您还有两个选择

  • 切换到更大的实例,或者
  • 投入一些编码工作来更有效地控制脚本的内存占用量

注意,如果事实证明不是您自己,而是与您运行amock在同一台服务器上的其他实例并置在另一个实例中,那么编码工作可能就一事无成。

在内存方面,我们已经知道subprocess.Popen使用fork/ clone 在幕后,这意味着每次调用它时,您都在请求与Python已经耗尽的内存一样多的内存,即增加数百MB,以便exec很小的10kB可执行文件,例如freeps。如果出现不利的过量使用政策,您很快就会看到ENOMEM

替代方法fork没有此父页面表等。复制问题为vforkposix_spawn。但是,如果您不想subprocess.Popenvfork/ 重写大块的代码posix_spawn,请考虑suprocess.Popen在脚本开始时(Python的内存占用最小的情况下)仅使用一次,以生成一个shell脚本,然后再运行free/ ps/ sleep以及其他与脚本并行循环;轮询脚本的输出或同步读取它,如果您还有其他要异步处理的内容,则可能从一个单独的线程中读取它-使用Python处理数据,但将分叉交给下级进程处理。

无论其,在您的特定情况下,你可以跳过调用psfree干脆; 无论您选择自己亲自还是通过现有的库和/或程序包访问这些信息,都可以直接在Python中直接从中procfs使用。如果和是你正在运行的唯一的实用工具,那么你就可以弄死完全psfreesubprocess.Popen

最后,无论您做什么subprocess.Popen,如果脚本泄漏内存,您最终还是会碰壁。密切注意它,并检查是否有内存泄漏

As a general rule (i.e. in vanilla kernels), fork/clone failures with ENOMEM occur specifically because of either an honest to God out-of-memory condition (dup_mm, dup_task_struct, alloc_pid, mpol_dup, mm_init etc. croak), or because security_vm_enough_memory_mm failed you while enforcing the overcommit policy.

Start by checking the vmsize of the process that failed to fork, at the time of the fork attempt, and then compare to the amount of free memory (physical and swap) as it relates to the overcommit policy (plug the numbers in.)

In your particular case, note that Virtuozzo has additional checks in overcommit enforcement. Moreover, I’m not sure how much control you truly have, from within your container, over swap and overcommit configuration (in order to influence the outcome of the enforcement.)

Now, in order to actually move forward I’d say you’re left with two options:

  • switch to a larger instance, or
  • put some coding effort into more effectively controlling your script’s memory footprint

NOTE that the coding effort may be all for naught if it turns out that it’s not you, but some other guy collocated in a different instance on the same server as you running amock.

Memory-wise, we already know that subprocess.Popen uses fork/clone under the hood, meaning that every time you call it you’re requesting once more as much memory as Python is already eating up, i.e. in the hundreds of additional MB, all in order to then exec a puny 10kB executable such as free or ps. In the case of an unfavourable overcommit policy, you’ll soon see ENOMEM.

Alternatives to fork that do not have this parent page tables etc. copy problem are vfork and posix_spawn. But if you do not feel like rewriting chunks of subprocess.Popen in terms of vfork/posix_spawn, consider using suprocess.Popen only once, at the beginning of your script (when Python’s memory footprint is minimal), to spawn a shell script that then runs free/ps/sleep and whatever else in a loop parallel to your script; poll the script’s output or read it synchronously, possibly from a separate thread if you have other stuff to take care of asynchronously — do your data crunching in Python but leave the forking to the subordinate process.

HOWEVER, in your particular case you can skip invoking ps and free altogether; that information is readily available to you in Python directly from procfs, whether you choose to access it yourself or via existing libraries and/or packages. If ps and free were the only utilities you were running, then you can do away with subprocess.Popen completely.

Finally, whatever you do as far as subprocess.Popen is concerned, if your script leaks memory you will still hit the wall eventually. Keep an eye on it, and check for memory leaks.


回答 1

free -m我看来,从输出看,您实际上没有可用的交换内存。我不确定在Linux中交换是否总是可以按需自动进行,但是我遇到了同样的问题,这里的答案都没有真正帮助我。但是,添加一些交换内存可以解决我的问题,因为这可能会帮助其他面临相同问题的人,所以我发布了有关如何添加1GB交换的答案(在Ubuntu 12.04上,但对于其他发行版也应类似地工作)。

您可以首先检查是否启用了任何交换内存。

$sudo swapon -s

如果为空,则表示您没有启用任何交换。要添加1GB交换空间:

$sudo dd if=/dev/zero of=/swapfile bs=1024 count=1024k
$sudo mkswap /swapfile
$sudo swapon /swapfile

将以下行添加到中,fstab以使交换永久生效。

$sudo vim /etc/fstab

     /swapfile       none    swap    sw      0       0 

来源和更多信息可以在这里找到。

Looking at the output of free -m it seems to me that you actually do not have swap memory available. I am not sure if in Linux the swap always will be available automatically on demand, but I was having the same problem and none of the answers here really helped me. Adding some swap memory however, fixed the problem in my case so since this might help other people facing the same problem, I post my answer on how to add a 1GB swap (on Ubuntu 12.04 but it should work similarly for other distributions.)

You can first check if there is any swap memory enabled.

$sudo swapon -s

if it is empty, it means you don’t have any swap enabled. To add a 1GB swap:

$sudo dd if=/dev/zero of=/swapfile bs=1024 count=1024k
$sudo mkswap /swapfile
$sudo swapon /swapfile

Add the following line to the fstab to make the swap permanent.

$sudo vim /etc/fstab

     /swapfile       none    swap    sw      0       0 

Source and more information can be found here.


回答 2

swap可能不是以前建议的红色鲱鱼。之前的python进程有多大ENOMEM

在内核2.6下,/proc/sys/vm/swappiness控制内核将如何积极地进行交换,并overcommit*归档内核可以眨眨一下头来分配多少内存以及如何精确分配内存。就像您的Facebook关系状态一样,这很复杂

…但是交换实际上是按需提供的(根据Web主机)…

但不是根据free(1)命令的输出,该命令的输出不显示服务器实例识别的交换空间。现在,您的Web主机肯定比我对这个主题了解更多,但是我使用的虚拟RHEL / CentOS系统报告了可用于来宾OS的交换。

改编Red Hat KB第15252条

只要匿名内存和系统V共享内存的总和少于RAM的3/4,红帽企业Linux 5系统就可以很好地运行,根本没有交换空间。….内存小于或等于4GB的系统 [建议]至少具有2GB的交换空间。

将您的/proc/sys/vm设置与普通的CentOS 5.3安装进行比较。添加交换文件。棘轮下来swappiness,看看你是否再活下去。

swap may not be the red herring previously suggested. How big is the python process in question just before the ENOMEM?

Under kernel 2.6, /proc/sys/vm/swappiness controls how aggressively the kernel will turn to swap, and overcommit* files how much and how precisely the kernel may apportion memory with a wink and a nod. Like your facebook relationship status, it’s complicated.

…but swap is actually available on demand (according to the web host)…

but not according to the output of your free(1) command, which shows no swap space recognized by your server instance. Now, your web host may certainly know much more than I about this topic, but virtual RHEL/CentOS systems I’ve used have reported swap available to the guest OS.

Adapting Red Hat KB Article 15252:

A Red Hat Enterprise Linux 5 system will run just fine with no swap space at all as long as the sum of anonymous memory and system V shared memory is less than about 3/4 the amount of RAM. …. Systems with 4GB of ram or less [are recommended to have] a minimum of 2GB of swap space.

Compare your /proc/sys/vm settings to a plain CentOS 5.3 installation. Add a swap file. Ratchet down swappiness and see if you live any longer.


回答 3

为了轻松解决,您可以

echo 1 > /proc/sys/vm/overcommit_memory

如果您确定系统有足够的内存。参见Linux以上的提交启发式方法

For an easy fix, you could

echo 1 > /proc/sys/vm/overcommit_memory

if your’re sure that your system has enough memory. See Linux over commit heuristic.


回答 4

我仍然怀疑您的客户/用户已加载了某些内核模块或驱动程序,从而干扰了clone()系统调用(可能是一些晦涩的安全性增强功能,例如LIDS,但更为晦涩吗?),或者是以某种方式填充了某些内核数据结构对fork()/来说是必需的clone()(进程表,页面表,文件描述符表等)。

这是fork(2)手册页的相关部分:

错误
       EAGAIN fork()无法分配足够的内存来复制父级的页表并为该任务分配任务结构
              儿童。

       EAGAIN无法创建新进程,因为遇到了调用者的RLIMIT_NPROC资源限制。至
              超过此限制,该进程必须具有CAP_SYS_ADMIN或CAP_SYS_RESOURCE能力。

       由于内存紧张,ENOMEM fork()无法分配必要的内核结构。

我建议让用户在引导到普通的通用内核之后,并仅加载最少的一组模块和驱动程序(运行应用程序/脚本的最低要求)之后尝试一下。从那里开始,假设它在该配置下有效,则他们可以在显示该问题的配置和配置之间执行二进制搜索。这是标准的系统管理员故障排除101。

您中的相关行strace是:

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f12708) = -1 ENOMEM (Cannot allocate memory)

…我知道其他人已经讨论过交换和内存的可用性(我建议您至少设置一个小的交换分区,即使该分区位于RAM磁盘上,也具有讽刺意味的是…与可用零交换的那些交换(exceptions处理路径)相比,甚至很少的可用交换也被广泛地执行。

但是,我怀疑这仍然是鲱鱼。

free报告高速缓存和缓冲区正在使用的0(零)内存的事实非常令人不安。我怀疑free输出…甚至可能是您的应用程序问题,是由某种专有内核模块引起的,该模块以某种方式干扰了内存分配。

根据fork()/ clone()的手册页,如果您的调用会导致资源限制冲突(RLIMIT_NPROC),则fork()系统调用应返回EAGAIN …但是,它没有说是否要返回EAGAIN违反其他RLIMIT *。无论如何,如果目标/主机具有某种奇怪的Vormetric或其他安全设置(或者即使您的进程在某种奇怪的SELinux策略下运行),也可能导致此-ENOMEM故障。

不太可能是正常的Linux / UNIX问题。您正在那里进行一些非标准的操作。

I continue to suspect that your customer/user has some kernel module or driver loaded which is interfering with the clone() system call (perhaps some obscure security enhancement, something like LIDS but more obscure?) or is somehow filling up some of the kernel data structures that are necessary for fork()/clone() to operate (process table, page tables, file descriptor tables, etc).

Here’s the relevant portion of the fork(2) man page:

ERRORS
       EAGAIN fork() cannot allocate sufficient memory to copy the parent's page tables and allocate a task  structure  for  the
              child.

       EAGAIN It  was not possible to create a new process because the caller's RLIMIT_NPROC resource limit was encountered.  To
              exceed this limit, the process must have either the CAP_SYS_ADMIN or the CAP_SYS_RESOURCE capability.

       ENOMEM fork() failed to allocate the necessary kernel structures because memory is tight.

I suggest having the user try this after booting into a stock, generic kernel and with only a minimal set of modules and drivers loaded (minimum necessary to run your application/script). From there, assuming it works in that configuration, they can perform a binary search between that and the configuration which exhibits the issue. This is standard sysadmin troubleshooting 101.

The relevant line in your strace is:

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f12708) = -1 ENOMEM (Cannot allocate memory)

… I know others have talked about swap and memory availability (and I would recommend that you set up at least a small swap partition, ironically even if it’s on a RAM disk … the code paths through the Linux kernel when it has even a tiny bit of swap available have been exercised far more extensively than those (exception handling paths) in which there is zero swap available.

However I suspect that this is still a red herring.

The fact that free is reporting 0 (ZERO) memory in use by the cache and buffers is very disturbing. I suspect that the free output … and possibly your application issue here, are caused by some proprietary kernel module which is interfering with the memory allocation in some way.

According to the man pages for fork()/clone() the fork() system call should return EAGAIN if your call would cause a resource limit violation (RLIMIT_NPROC) … however, it doesn’t say if EAGAIN is to be returned by other RLIMIT* violations. In any event if your target/host has some sort of weird Vormetric or other security settings (or even if your process is running under some weird SELinux policy) then it might be causing this -ENOMEM failure.

It’s pretty unlikely to be a normal run-of-the-mill Linux/UNIX issue. You’ve got something non-standard going on there.


回答 5

您是否尝试过使用:

(status,output) = commands.getstatusoutput("ps aux")

我认为这为我解决了完全相同的问题。但是后来我的过程最终被杀死,而不是没有产生,这更糟。

经过一些测试,我发现这仅在旧版本的python上发生:它在2.6.5中发生,而在2.7.2中不发生

我的搜索将我引到了这里python-close_fds-issue,但是取消设置closed_fds并不能解决问题。仍然值得一读。

我发现python通过仅关注它就泄漏了文件描述符:

watch "ls /proc/$PYTHONPID/fd | wc -l"

像您一样,我确实想捕获命令的输出,并且确实要避免OOM错误……但是看来,唯一的方法是让人们使用没有太多错误的Python版本。不理想…

Have you tried using:

(status,output) = commands.getstatusoutput("ps aux")

I thought this had fixed the exact same problem for me. But then my process ended up getting killed instead of failing to spawn, which is even worse..

After some testing I found that this only occurred on older versions of python: it happens with 2.6.5 but not with 2.7.2

My search had led me here python-close_fds-issue, but unsetting closed_fds had not solved the issue. It is still well worth a read.

I found that python was leaking file descriptors by just keeping an eye on it:

watch "ls /proc/$PYTHONPID/fd | wc -l"

Like you, I do want to capture the command’s output, and I do want to avoid OOM errors… but it looks like the only way is for people to use a less buggy version of Python. Not ideal…


回答 6

munmap(0xb7d28000,4096)= 0
写(2,“ OSError”,7)= 7

我看过草率的代码,看起来像这样:

serrno = errno;
some_Syscall(...)
if (serrno != errno)
/* sound alarm: CATROSTOPHIC ERROR !!! */

您应该检查这是否是python代码中正在发生的事情。Errno仅在进行中的系统调用失败时才有效。

编辑添加:

您没有说这个过程可以持续多久。可能的内存使用者

  • 分叉过程
  • 未使用的数据结构
  • 共享库
  • 内存映射文件

munmap(0xb7d28000, 4096) = 0
write(2, “OSError”, 7) = 7

I’ve seen sloppy code that looks like this:

serrno = errno;
some_Syscall(...)
if (serrno != errno)
/* sound alarm: CATROSTOPHIC ERROR !!! */

You should check to see if this is what is happening in the python code. Errno is only valid if the proceeding system call failed.

Edited to add:

You don’t say how long this process lives. Possible consumers of memory

  • forked processes
  • unused data structures
  • shared libraries
  • memory mapped files

回答 7

也许你可以简单地

$ sudo bash -c "echo vm.overcommit_memory=1 >> /etc/sysctl.conf"
$ sudo sysctl -p

它适用于我的情况。

参考:https : //github.com/openai/gym/issues/110#issuecomment-220672405

Maybe you can simply

$ sudo bash -c "echo vm.overcommit_memory=1 >> /etc/sysctl.conf"
$ sudo sysctl -p

It works for my case.

Reference: https://github.com/openai/gym/issues/110#issuecomment-220672405


我应该在Python 3中使用编码声明吗?

问题:我应该在Python 3中使用编码声明吗?

默认情况下,Python 3对源代码文件使用UTF-8编码。我还应该在每个源文件的开头使用编码声明吗?喜欢# -*- coding: utf-8 -*-

Python 3 uses UTF-8 encoding for source-code files by default. Should I still use the encoding declaration at the beginning of every source file? Like # -*- coding: utf-8 -*-


回答 0

因为默认值为 UTF-8,所以仅在偏离默认值时或者在依赖其他工具(例如IDE或文本编辑器)来使用该信息时,才需要使用该声明。

换句话说,就Python而言,仅当您要使用不同的编码时,才需要使用该声明。

其他工具(例如您的编辑器)也可以支持类似的语法,这就是PEP 263规范在语法上具有相当大的灵活性的原因(它必须是注释,文本coding必须在其中,后跟a :=字符以及可选的空白,然后是公认的编解码器)。

请注意,它仅适用于Python如何读取源代码。它不适用于执行该代码,因此不适用于打印,打开文件或字节与Unicode之间的任何其他I / O操作转换。有关Python,Unicode和编码的更多详细信息,强烈建议您阅读Python Unicode HOWTO或Ned Batchelder撰写的非常详尽的Pragmatic Unicode演讲

Because the default is UTF-8, you only need to use that declaration when you deviate from the default, or if you rely on other tools (like your IDE or text editor) to make use of that information.

In other words, as far as Python is concerned, only when you want to use an encoding that differs do you have to use that declaration.

Other tools, such as your editor, can support similar syntax, which is why the PEP 263 specification allows for considerable flexibility in the syntax (it must be a comment, the text coding must be there, followed by either a : or = character and optional whitespace, followed by a recognised codec).

Note that it only applies to how Python reads the source code. It doesn’t apply to executing that code, so not to how printing, opening files, or any other I/O operations translate between bytes and Unicode. For more details on Python, Unicode, and encodings, I strongly urge you to read the Python Unicode HOWTO, or the very thorough Pragmatic Unicode talk by Ned Batchelder.


回答 1

不,如果:

  • 整个项目仅使用UTF-8,这是默认设置。
  • 并且您确定您的IDE工具不需要每个文件中的编码声明。

是的,如果

  • 您的项目依赖于不同的编码
  • 或依赖于许多编码。

对于多编码项目:

如果某些文件在中进行了编码non-utf-8,那么即使对于这些文件,UTF-8您也应该添加编码声明,因为黄金法则是Explicit is better than implicit.

参考:

  • PyCharm不需要该声明:

在pycharm中为特定文件配置编码

  • vim不需要该声明,但是:
# vim: set fileencoding=<encoding name> :

No, if:

  • entire project use only the UTF-8, which is a default.
  • and you’re sure your IDE tool doesn’t need that encoding declaration in each file.

Yes, if

  • your project relies on different encoding
  • or relies on many encodings.

For multi-encodings projects:

If some files are encoded in the non-utf-8, then even for these encoded in UTF-8 you should add encoding declaration too, because the golden rule is Explicit is better than implicit.

Reference:

  • PyCharm doesn’t need that declaration:

configuring encoding for specific file in pycharm

  • vim doesn’t need that declaration, but:
# vim: set fileencoding=<encoding name> :

Python正则表达式返回true / false

问题:Python正则表达式返回true / false

使用Python正则表达式如何获得True/ False返回?所有Python回报是:

<_sre.SRE_Match object at ...>

Using Python regular expressions how can you get a True/False returned? All Python returns is:

<_sre.SRE_Match object at ...>

回答 0

Match对象始终为true,None如果不匹配,则返回。只是测试真实性。

if re.match(...):

Match objects are always true, and None is returned if there is no match. Just test for trueness.

if re.match(...):

回答 1

如果您确实需要TrueFalse,请使用bool

>>> bool(re.search("hi", "abcdefghijkl"))
True
>>> bool(re.search("hi", "abcdefgijkl"))
False

正如其他答案所指出的那样,如果您只是将其用作ifor 的条件while,则可以直接使用它而无需将其包装bool()

If you really need True or False, just use bool

>>> bool(re.search("hi", "abcdefghijkl"))
True
>>> bool(re.search("hi", "abcdefgijkl"))
False

As other answers have pointed out, if you are just using it as a condition for an if or while, you can use it directly without wrapping in bool()


回答 2

伊格纳西奥·巴斯克斯(Ignacio Vazquez-Abrams)是正确的。但要详细说明,re.match()将返回None,其结果为False,或者返回一个匹配对象,该对象将始终True如他所说。仅当您需要有关与正则表达式匹配的部分的信息时,才需要签出匹配对象的内容。

Ignacio Vazquez-Abrams is correct. But to elaborate, re.match() will return either None, which evaluates to False, or a match object, which will always be True as he said. Only if you want information about the part(s) that matched your regular expression do you need to check out the contents of the match object.


回答 3

一种方法是仅针对返回值进行测试。因为您得到<_sre.SRE_Match object at ...>它意味着它将评估为true。当正则表达式不匹配时,您将返回无值,其结果为false。

import re

if re.search("c", "abcdef"):
    print "hi"

产生hi为输出。

One way to do this is just to test against the return value. Because you’re getting <_sre.SRE_Match object at ...> it means that this will evaluate to true. When the regular expression isn’t matched you’ll the return value None, which evaluates to false.

import re

if re.search("c", "abcdef"):
    print "hi"

Produces hi as output.


回答 4

这是我的方法:

import re
# Compile
p = re.compile(r'hi')
# Match and print
print bool(p.match("abcdefghijkl"))

Here is my method:

import re
# Compile
p = re.compile(r'hi')
# Match and print
print bool(p.match("abcdefghijkl"))

回答 5

您可以使用re.match()re.search()。Python提供了两种基于正则表达式的原始操作:re.match()仅在字符串的开头re.search()检查匹配项,而在字符串中的任意位置检查匹配项(这是Perl的默认设置)。参考这个

You can use re.match() or re.search(). Python offers two different primitive operations based on regular expressions: re.match() checks for a match only at the beginning of the string, while re.search() checks for a match anywhere in the string (this is what Perl does by default). refer this


sqlalchemy flush()并获取ID?

问题:sqlalchemy flush()并获取ID?

我想做这样的事情:

f = Foo(bar='x')
session.add(f)
session.flush()

# do additional queries using f.id before commit()
print f.id # should be not None

session.commit()

但是f.idNone我尝试的时候。我怎样才能使它工作?

I want to do something like this:

f = Foo(bar='x')
session.add(f)
session.flush()

# do additional queries using f.id before commit()
print f.id # should be not None

session.commit()

But f.id is None when I try it. How can I get this to work?


回答 0

您的示例代码应该已经按原样工作。SQLAlchemy应该为其提供一个值f.id,并假设其为自动生成的主键列。主键属性在flush()生成时立即在过程中填充,并且不需要调用commit()。因此,这里的答案在于以下一项或多项:

  1. 映射的详细信息
  2. 如果使用的后端有任何奇怪现象(例如,SQLite不会为复合主键生成整数值)
  3. 打开echo时发出的SQL表示什么

Your sample code should have worked as it is. SQLAlchemy should be providing a value for f.id, assuming its an autogenerating primary-key column. Primary-key attributes are populated immediately within the flush() process as they are generated, and no call to commit() should be required. So the answer here lies in one or more of the following:

  1. The details of your mapping
  2. If there are any odd quirks of the backend in use (such as, SQLite doesn’t generate integer values for a composite primary key)
  3. What the emitted SQL says when you turn on echo

回答 1

我遇到了同样的问题,经过测试,我发现这些答案中没有一个是足够的。

当前,或者从sqlalchemy .6+开始,有一个非常简单的解决方案(我不知道它是否存在于以前的版本中,尽管我认为确实存在):

session.refresh()

因此,您的代码将如下所示:

f = Foo(bar=x)
session.add(f)
session.flush()
# At this point, the object f has been pushed to the DB, 
# and has been automatically assigned a unique primary key id

f.id
# is None

session.refresh(f)
# refresh updates given object in the session with its state in the DB
# (and can also only refresh certain attributes - search for documentation)

f.id
# is the automatically assigned primary key ID given in the database.

就是这样。

I’ve just run across the same problem, and after testing I have found that NONE of these answers are sufficient.

Currently, or as of sqlalchemy .6+, there is a very simple solution (I don’t know if this exists in prior version, though I imagine it does):

session.refresh()

So, your code would look something like this:

f = Foo(bar=x)
session.add(f)
session.flush()
# At this point, the object f has been pushed to the DB, 
# and has been automatically assigned a unique primary key id

f.id
# is None

session.refresh(f)
# refresh updates given object in the session with its state in the DB
# (and can also only refresh certain attributes - search for documentation)

f.id
# is the automatically assigned primary key ID given in the database.

That’s how to do it.


回答 2

谢谢大家 我通过修改列映射解决了我的问题。对我来说,autoincrement=True是必需的。

起源:

id = Column('ID', Integer, primary_key=True, nullable=False)

修改后:

id = Column('ID', Integer, primary_key=True, autoincrement=True, nullable=True)

然后

session.flush()  
print(f.id)

没关系!

Thanks for everybody. I solved my problem by modifying the column mapping. For me, autoincrement=True is required.

origin:

id = Column('ID', Integer, primary_key=True, nullable=False)

after modified:

id = Column('ID', Integer, primary_key=True, autoincrement=True, nullable=True)

then

session.flush()  
print(f.id)

is ok!


回答 3

与dpb给出的答案不同,不需要刷新。刷新后,您可以访问id字段,sqlalchemy会自动刷新在后端自动生成的id

我遇到了这个问题,经过一番调查后找出了确切的原因,我的模型是用id作为integerfield创建的,而在我的表单中,id用hiddenfield表示(因为我不想在表单中显示id)。默认情况下,隐藏字段表示为文本。一旦我使用widget = hiddenInput()将表单更改为integerfield,问题就解决了。

unlike the answer given by dpb, a refresh is not necessary. once you flush, you can access the id field, sqlalchemy automatically refreshes the id which is auto generated at the backend

I encountered this problem and figured the exact reason after some investigation, my model was created with id as integerfield and in my form the id was represented with hiddenfield( since i did not wanted to show the id in my form). The hidden field is by default represented as a text. once I changed the form to integerfield with widget=hiddenInput()) the problem was solved.


回答 4

我曾经0在调用session.add方法之前分配给ID时遇到问题。该ID已由数据库正确分配,但在之后的会话中未检索到正确的ID session.flush()

I once had a problem with having assigned 0 to id before calling session.add method. The id was correctly assigned by the database but the correct id was not retrieved from the session after session.flush().


回答 5

您应该尝试使用session.save_or_update(f)而不是session.add(f)

You should try using session.save_or_update(f) instead of session.add(f).


pytest无法导入模块,而python可以

问题:pytest无法导入模块,而python可以

我正在使用Python开发包。我使用virtualenv。我在virtualenv的.pth路径中将模块的根目录设置为路径,以便在开发代码并进行测试时可以导入软件包的模块(问题1:这是一个好方法吗?)。这工作正常(这是一个示例,这是我想要的行为):

(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ python
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from rc import ns
>>> exit()
(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ python tests/test_ns.py 
issued command: echo hello
command output: hello

但是,如果我尝试使用PyTest,则会收到一些导入错误消息:

(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ pytest
=========================================== test session starts ============================================
platform linux2 -- Python 2.7.12, pytest-3.0.5, py-1.4.31, pluggy-0.4.0
rootdir: /home/zz/Desktop/GitFolders/rc, inifile: 
collected 0 items / 1 errors 

================================================== ERRORS ==================================================
________________________________ ERROR collecting tests/test_ns.py ________________________________
ImportError while importing test module '/home/zz/Desktop/GitFolders/rc/tests/test_ns.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_ns.py:2: in <module>
    from rc import ns
E   ImportError: cannot import name ns
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================= 1 error in 0.09 seconds ==========================================
(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ which pytest
/home/zz/Desktop/VirtualEnvs/VEnvTestRc/bin/pytest

我有点疑惑,它看起来像是一个导入错误,但是Python很好,所以为什么PyTest特有问题?对原因/补救措施有什么建议(问题2)?我用谷歌搜索并堆栈溢出了PyTest的“ ImportError:无法导入”错误,但是我得到的结果与缺少python路径有关,并且对此有补救措施,这似乎不是这里的问题。有什么建议?

I am working on a package in Python. I use virtualenv. I set the path to the root of the module in a .pth path in my virtualenv, so that I can import modules of the package while developing the code and do testing (Question 1: is it a good way to do?). This works fine (here is an example, this is the behavior I want):

(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ python
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from rc import ns
>>> exit()
(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ python tests/test_ns.py 
issued command: echo hello
command output: hello

However, if I try to use PyTest, I get some import error messages:

(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ pytest
=========================================== test session starts ============================================
platform linux2 -- Python 2.7.12, pytest-3.0.5, py-1.4.31, pluggy-0.4.0
rootdir: /home/zz/Desktop/GitFolders/rc, inifile: 
collected 0 items / 1 errors 

================================================== ERRORS ==================================================
________________________________ ERROR collecting tests/test_ns.py ________________________________
ImportError while importing test module '/home/zz/Desktop/GitFolders/rc/tests/test_ns.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_ns.py:2: in <module>
    from rc import ns
E   ImportError: cannot import name ns
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================= 1 error in 0.09 seconds ==========================================
(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ which pytest
/home/zz/Desktop/VirtualEnvs/VEnvTestRc/bin/pytest

I am a bit puzzled, it looks like this indicates an import error, but Python does it fine so why is there a problem specifically with PyTest? Any suggestion to the reason / remedy (Question 2)? I googled and stack-overflowed the ‘ImportError: cannot import’ error for PyTest, but the hits I got were related to missing python path and remedy to this, which does not seem to be the problem here. Any suggestions?


回答 0

找到了答案:

__init__.py如果计划使用pytest,请勿将文件放在包含TESTS的文件夹中。我有一个这样的文件,删除它可以解决问题。

这实际上是在pytest“ ImportError:没有名为YadaYadaYada的模块”PATH问题的第二个答案的注释中掩埋的,所以我没有看到它,希望它在这里能够引起更多的关注。

Found the answer:

DO NOT put a __init__.py file in a folder containing TESTS if you plan on using pytest. I had one such file, deleting it solved the problem.

This was actually buried in the comments to the second answer of PATH issue with pytest ‘ImportError: No module named YadaYadaYada’ so I did not see it, hope it gets more visibility here.


回答 1

我不能说我理解为什么会这样,但是我遇到了同样的问题,如果运行,测试也可以正常工作python -m pytest

我在virtualenv中,pytest也可以在全球范围内使用:

(proj)tom@neon ~/dev/proj$ type -a python
python is /home/tom/.virtualenvs/proj/bin/python
python is /usr/bin/python

(proj)tom@neon ~/dev/proj$ python -V
Python 3.5.2

(proj)tom@neon ~/dev/proj$ type -a pytest
pytest is /home/tom/.virtualenvs/proj/bin/pytest
pytest is /usr/bin/pytest

(proj)tom@neon ~/dev/proj$ pytest --version
This is pytest version 3.5.0, imported from /home/tom/.virtualenvs/proj/lib/python3.5/site-packages/pytest.py

I can’t say I understand why this works, but I had the same problem and the tests work fine if I run python -m pytest.

I’m in a virtualenv, with pytest also available globally:

(proj)tom@neon ~/dev/proj$ type -a python
python is /home/tom/.virtualenvs/proj/bin/python
python is /usr/bin/python

(proj)tom@neon ~/dev/proj$ python -V
Python 3.5.2

(proj)tom@neon ~/dev/proj$ type -a pytest
pytest is /home/tom/.virtualenvs/proj/bin/pytest
pytest is /usr/bin/pytest

(proj)tom@neon ~/dev/proj$ pytest --version
This is pytest version 3.5.0, imported from /home/tom/.virtualenvs/proj/lib/python3.5/site-packages/pytest.py

回答 2

我只是通过删除项目根目录中的__init__.py来解决此问题:

.
├── __init__.py <--- removed
├── models
   ├── __init__.py
   ├── address.py
   ├── appointment.py
   └── client.py
├── requirements.txt
├── setup.cfg
├── tests
   ├── __init__.py
   ├── models
      ├── __init__.py
      ├── appointment_test.py
      └── client_test.py
   └── other_test.py
└── script.py

I just solved this by removing the __init__.py in my project root:

.
├── __init__.py <--- removed
├── models
│   ├── __init__.py
│   ├── address.py
│   ├── appointment.py
│   └── client.py
├── requirements.txt
├── setup.cfg
├── tests
│   ├── __init__.py
│   ├── models
│   │   ├── __init__.py
│   │   ├── appointment_test.py
│   │   └── client_test.py
│   └── other_test.py
└── script.py

回答 3

我遇到了同样的问题,但是由于上述原因之外的另一个原因:

我在全局安装了py.test,而软件包则安装在虚拟环境中。

解决方案是pytest在虚拟环境中安装。(如果您的shell像Bash一样哈希可执行文件,请使用hash -r,或使用的完整路径py.test

I had the same problem but for another reason than the ones mentioned:

I had py.test installed globally, while the packages were installed in a virtual environment.

The solution was to install pytest in the virtual environment. (In case your shell hashes executables, as Bash does, use hash -r, or use the full path to py.test)


回答 4

如果您有一个tests.py文件和一个带有的测试文件夹,则会发生此问题tests/__init__.py

在收集过程中,pytest会找到文件夹,但是当尝试从文件夹中导入测试文件时,tests.py文件会导致导入问题。

要解决此问题,只需删除tests.py文件,然后将所有测试放在tests/文件夹中即可。

对于您的特定情况,修复将精确地是:

  • 删除文件 /home/zz/Desktop/GitFolders/rc/tests.py
  • 确保/home/zz/Desktop/GitFolders/rc/tests/__init__.py存在

This problem will happen if you have a tests.py file and a tests folder with tests/__init__.py.

During the collection pytest finds the folder, but when it tries to import the test files from the folder, tests.py file will cause the import problem.

To fix simply remove the tests.py file and put all your tests inside the tests/ folder.

For your specific case the fix will be precisely:

  • Remove the file /home/zz/Desktop/GitFolders/rc/tests.py
  • Make sure /home/zz/Desktop/GitFolders/rc/tests/__init__.py is present

回答 5

我有一个类似的问题,完全相同的错误,但原因不同。我正在运行测试代码,但是使用的是模块的旧版本。在我的代码的先前版本中,一个类存在,而另一个不存在。更新代码后,我应该运行以下命令进行安装。

sudo pip install ./ --upgrade

安装更新的模块后,运行pytest会产生正确的结果(因为我使用的是正确的代码库)。

I had a similar issue, exact same error, but different cause. I was running the test code just fine, but against an old version of the module. In the previous version of my code one class existed, while the other did not. After updating my code, I should have run the following to install it.

sudo pip install ./ --upgrade

Upon installing the updated module running pytest produced the correct results (because i was using the correct code base).


回答 6

在我的情况下,发生导入错误是因为程序包指向另一个具有相同名称的程序包/目录,并且其路径在我实际想要的文件夹上方一层。我认为这也解释了为什么有些人需要删除_ init _.py而另一些人需要重新添加。

我只是在控制台和脚本中print(the_root_package.__path__)(在之后import the_root_package)放置以比较差异pythonpytest

底线:执行此操作时python,导入的包可能与运行时的包不同pytest

In my case, the import error occurred because the package is pointing to another package/directory with the same name and its path is one level above the folder I actually wanted. I think this also explains why some people need to remove _ init _.py while others need to add back.

I just put print(the_root_package.__path__) (after import the_root_package) in both python console and pytest scripts to compare the difference

BOTTOM LINE: When you do python, the package you import may be different from the package when you run pytest.


回答 7

将软件包安装到您的虚拟环境中。
然后启动一个新的Shell并再次获取您的虚拟环境。

Install the packages into Your virtual environment.
Then start a new shell and source Your virtual environment again.


回答 8

上面的答案对我不起作用。我只是通过将未找到的模块的绝对路径附加到(您的测试模块)的sys.path顶部来解决它test_xxx.py,例如:

import sys
sys.path.append('path')

The answer above not work for me. I just solved it by appending the absolute path of the module which not found to the sys.path at top of the test_xxx.py (your test module), like:

import sys
sys.path.append('path')

回答 9

如果它与最初在python 2.7中开发,现在迁移到python 3.x的python代码有关,则该问题可能与导入问题有关。

例如,从base同一目录中的文件导入对象时,它将在python 2.x中工作:

from base import MyClass

在python 3.x中,您应该替换为base完整路径,否则.base 将导致上述问题。所以尝试:

from .base import MyClass

If it is related to python code that was originally developed in python 2.7 and now migrated into python 3.x than the problem is probably related to an import issue.

e.g. when importing an object from a file: base that is located in the same directory this will work in python 2.x:

from base import MyClass

in python 3.x you should replace with base full path or .base not doing so will cause the above problem. so try:

from .base import MyClass

回答 10

我今天遇到了这个问题,并通过python -m pytest从项目目录的根目录进行调用来解决了。

pytest从同一位置拨打电话仍然会引起问题。

我的项目目录组织为:

api/
 - server/
  - tests/
      - test_routes.py
  - routes/
      - routes.py
 - app.py

该模块routes以我的test_routes.py身份导入:from server.routes.routes import Routes

希望有帮助!

I was experiencing this issue today and solved it by calling python -m pytest from the root of my project directory.

Calling pytest from the same location still caused issues.

My Project dir is organized as:

api/
 - server/
  - tests/
      - test_routes.py
  - routes/
      - routes.py
 - app.py

The module routes was imported in my test_routes.py as: from server.routes.routes import Routes

Hope that helps!


回答 11

另一个特殊情况:

我在使用毒药时遇到了问题。因此,我的程序运行良好,但是通过Tox进行的单元测试一直在抱怨。安装软件包(程序需要)之后,您需要另外在tox.ini中指定单元测试中使用的软件包。

[testenv]
deps =
    package1
    package2 
...

Another special case:

I had the problem using tox. So my program ran fine, but unittests via tox kept complaining. After installing packages (needed for the program) you need to additionally specify the packages used in the unittests in the tox.ini

[testenv]
deps =
    package1
    package2 
...

回答 12

我正在使用VSCode来获得它。我有一个conda环境。我不认为VScode python扩展可以看到我所做的更新。

python c:\Users\brig\.vscode\extensions\ms-python.python-2019.9.34911\pythonFiles\testing_tools\run_adapter.py discover pytest -- -s --cache-clear test
Test Discovery failed:

我必须跑步 pip install ./ --upgrade

I was getting this using VSCode. I have a conda environment. I don’t think the VScode python extension could see the updates I was making.

python c:\Users\brig\.vscode\extensions\ms-python.python-2019.9.34911\pythonFiles\testing_tools\run_adapter.py discover pytest -- -s --cache-clear test
Test Discovery failed:

I had to run pip install ./ --upgrade


回答 13

编辑您的conftest.py并添加以下代码行:

import os, sys
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(file), '..')))

如果尝试通过终端运行测试用例,请使用以下示例:

python -m pytest test_some_step_file_steps.py --html=HTML_step_file_output.html --self-contained-html

Edit your conftest.py and add following lines of code:

import os, sys
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(file), '..')))

And if trying to run the test case through terminal, use following ex:

python -m pytest test_some_step_file_steps.py --html=HTML_step_file_output.html --self-contained-html

回答 14

Python的导入系统又获得了一次巨大的胜利。我认为,没有达成共识的原因是,有效的方法可能取决于您的环境以及在其上使用的工具。

我正在VS Code中使用它,它是在conda环境下Windows的Python 3.8中的测试资源管理器中使用的。

我要工作的设置是:

mypkg/
    __init__.py
    app.py
    view.py
tests/
    test_app.py
    test_view.py

在这种设置下,智能感知将起作用,测试发现也将起作用。

请注意,我最初试图以下,推荐这里

src/
    mypkg/
        __init__.py
        app.py
        view.py
tests/
    test_app.py
    test_view.py

我找不到从VS Code上src使它起作用的方法,因为该文件夹使导入系统无所适从。我可以想象有一种方法可以从命令行运行它。作为一种相对较新的Python编程转换,它带给我怀旧的使用COM的感觉,但是却没有那么有趣。

Yet another massive win for Python’s import system. I think the reason there is no consensus is that what works probably depends on your environment and the tools you are using on top of it.

I’m using this from VS Code, in the test explorer under Windows in a conda environment, Python 3.8.

The setup I have got to work is:

mypkg/
    __init__.py
    app.py
    view.py
tests/
    test_app.py
    test_view.py

Under this setup intellisense works and so does test discovery.

Note that I originally tried the following, as recommended here.

src/
    mypkg/
        __init__.py
        app.py
        view.py
tests/
    test_app.py
    test_view.py

I could find no way of getting this to work from VS Code because the src folder just blew the mind of the import system. I can imagine there is a way of getting this to work from the command line. As a relatively new convert to Python programming it gives me a nostalgic feeling of working with COM, but being slightly less fun.


回答 15

如果不使用虚拟环境,我为此付出的2美分:pytest会偶然失败。有时它会工作,有时则不会。

因此,解决方案是:

  • 用pip卸载删除pytest
  • 创建您的venv
  • 激活你的venv
  • pip以可编辑模式安装项目路径,因此pytest会将其视为模块(否则,pytest将找不到您的内部导入)。您将需要一个setup.py文件
  • 安装您的软件包,包括pytest
  • 最后,运行测试

使用Windows PowerShell的代码:

pip uninstall pytest
python.exe -m venv my_env
.\my_env\Scripts\activate
(my_env) pip install -e .
(my_env) pip install pytest pytest-html pandas numpy

然后最后

(my_env) pytest --html="my_testing_report.html"

用于pip install -e的setup.py示例:

import setuptools

setuptools.setup(
    name='my_package',
    version='devel',
    author='erickfis',
    author_email='erickfis@gmail.com',
    description='My package',
    long_description='My gooood package',
    packages=setuptools.find_packages(),
    classifiers=[
        'Programming Language :: Python :: 3',
        'Operating System :: OS Independent',
    ],
    include_package_data=True
)

My 2 cents on this: pytest will fail at chance if you are not using virtual environments. Sometimes it will just work, sometimes not.

Therefore, the solution is:

  • remove pytest with pip uninstall
  • create your venv
  • activate your venv
  • pip install your project path in editable mode, so it will be treated by pytest as a module (otherwise, pytest wont find your internal imports). You will need a setup.py file for that
  • install your packages, including pytest
  • finally, run your tests

The code, using windows PowerShell:

pip uninstall pytest
python.exe -m venv my_env
.\my_env\Scripts\activate
(my_env) pip install -e .
(my_env) pip install pytest pytest-html pandas numpy

Then finally

(my_env) pytest --html="my_testing_report.html"

An example of setup.py, for pip install -e:

import setuptools

setuptools.setup(
    name='my_package',
    version='devel',
    author='erickfis',
    author_email='erickfis@gmail.com',
    description='My package',
    long_description='My gooood package',
    packages=setuptools.find_packages(),
    classifiers=[
        'Programming Language :: Python :: 3',
        'Operating System :: OS Independent',
    ],
    include_package_data=True
)

回答 16

我不同意这些帖子,说您必须删除所有__init__.py文件。您必须要做的是更改sys.path

sys.path正常运行代码的情况下,在您进行打印的地方进行实验。然后sys.path在通过pytest运行代码的同时进行打印。我认为您会发现这两条路径之间存在差异,因此pytest为什么会中断。

要解决此问题,请将第一个实验的路径插入第二个实验的第0个索引。

我们'/usr/exampleUser/Documents/foo'是第一个元素print(sys.path)的实验1。

以下是应该解决您问题的代码:

import sys sys.path[0] = '/usr/exampleUser/Documents/foo'

在实际导入语句之前,将此内容放在文件顶部。

资料来源:我本人正在处理,上述过程解决了它。

I disagree with the posts saying that you must remove any __init__.py files. What you must instead do is alter the sys.path.

Run an experiment where you print sys.path when running the code normally. Then print sys.path while running the code via pytest. I think you will find that there is a difference between these two paths, hence why pytest breaks.

To fix this, insert the path from the first experiment at the 0th index of the second.

Let '/usr/exampleUser/Documents/foo' be the first element of print(sys.path) for experiment 1.

Below is code that should fix your issue:

import sys sys.path[0] = '/usr/exampleUser/Documents/foo'

Put this at the top of your file, before your actual import statement.

Source: I was dealing with this myself and the above process solved it.


回答 17

一切都保持相同,只是增加了在根文件夹中的空白测试文件..解决它

这是调查结果,这个问题确实困扰了我一段时间。我的文件夹结构是

mathapp/
    - server.py  
    - configuration.py 
    - __init__.py 
    - static/ 
       - home.html  
tests/            
    - functional 
       - test_errors.py 
    - unit  
       - test_add.py

pytest会抱怨ModuleNotFoundError并给出提示-确保您的测试模块/软件包具有有效的Python名称。

我在与mathsapp和tests目录相同的级别引入了模拟测试文件。该文件不包含任何内容。现在pytest不会抱怨。

没有文件的结果

$ pytest
============================= test session starts =============================
platform win32 -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: C:\mak2006\workspace\0github\python-rest-app-cont
collected 1 item / 1 error

=================================== ERRORS ====================================
_______________ ERROR collecting tests/functional/test_func.py ________________
ImportError while importing test module 'C:\mainak\workspace\0github\python-rest-app-cont\tests\functional\test_func.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests\functional\test_func.py:4: in <module>
    from mathapp.service import sum
E   ModuleNotFoundError: No module named 'mathapp'
=========================== short test summary info ===========================
ERROR tests/functional/test_func.py
!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
============================== 1 error in 0.24s ===============================

结果与文件

$ pytest
============================= test session starts =============================
platform win32 -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: C:\mak2006\workspace\0github\python-rest-app-cont
collected 2 items

tests\functional\test_func.py .                                          [ 50%]
tests\unit\test_unit.py .                                                [100%]

============================== 2 passed in 0.11s ==============================

Kept everything same and just added a blank test file at the root folder .. Solved it

Here are the findings, this problem really bugged me for a while. My folder structure was

mathapp/
    - server.py  
    - configuration.py 
    - __init__.py 
    - static/ 
       - home.html  
tests/            
    - functional 
       - test_errors.py 
    - unit  
       - test_add.py

and pytest would complain with the ModuleNotFoundError and gives the HINT – make sure your test modules/packages have valid Python names.

I introduced a mock test file at the same level as mathsapp and tests directory. The file contained nothing. Now pytest does not complain.

Result without the file

$ pytest
============================= test session starts =============================
platform win32 -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: C:\mak2006\workspace\0github\python-rest-app-cont
collected 1 item / 1 error

=================================== ERRORS ====================================
_______________ ERROR collecting tests/functional/test_func.py ________________
ImportError while importing test module 'C:\mainak\workspace\0github\python-rest-app-cont\tests\functional\test_func.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests\functional\test_func.py:4: in <module>
    from mathapp.service import sum
E   ModuleNotFoundError: No module named 'mathapp'
=========================== short test summary info ===========================
ERROR tests/functional/test_func.py
!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
============================== 1 error in 0.24s ===============================

Results with the file

$ pytest
============================= test session starts =============================
platform win32 -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: C:\mak2006\workspace\0github\python-rest-app-cont
collected 2 items

tests\functional\test_func.py .                                          [ 50%]
tests\unit\test_unit.py .                                                [100%]

============================== 2 passed in 0.11s ==============================

回答 18

我通过PYTHONPATH为运行测试的特定配置设置环境变量解决了我的问题。

在PyCharm上查看测试文件时:

  1. Ctrl+ Shift+A
  2. 类型 Edit Configurations
  3. PYTHONPATH在环境>环境变量下设置。

I solved my problem by setting the PYTHONPATH in Environment Variables for the specific configuration I’m running my tests with.

While you’re viewing the test file on PyCharm:

  1. Ctrl+Shift+A
  2. Type Edit Configurations
  3. Set your PYTHONPATH under Environment > Environment variables.

回答 19

只需将一个空conftest.py文件放在项目根目录中,因为当pytest发现conftest.py时,它将修改sys.path以便可以从conftest模块中导入内容。常规目录结构可以是:

Root
├── conftest.py
├── module1
   ├── __init__.py
   └── sample.py
└── tests
    └── test_sample.py

Simply put an empty conftest.py file in the project root directory, because when pytest discovers a conftest.py, it modifies sys.path so it can import stuff from the conftest module. A general directory structure can be:

Root
├── conftest.py
├── module1
│   ├── __init__.py
│   └── sample.py
└── tests
    └── test_sample.py

回答 20

有一个类似的问题,当我__init__.py在tests目录下添加文件时,它可以工作。

Had a similar issue and it worked when I added __init__.py file under tests directory.


回答 21

可能是因为Pytest并未将Python包作为Python模块读取(可能是由于路径问题)。尝试更改pytest脚本的目录或将模块显式添加到PYTHONPATH。

也可能是您的计算机上安装了两个版本的Python。检查您的Python源是否有pytest和您运行的python shell。如果它们不同(例如,Python 2 vs 3),请使用source activate来确保您正在为安装模块的相同python运行pytest。

It could be that Pytest is not reading the package as a Python module while Python is (likely due to path issues). Try changing the directory of the pytest script or adding the module explicitly to your PYTHONPATH.

Or it could be that you have two versions of Python installed on your machine. Check your Python source for pytest and for the python shell that you run. If they are different (i.e. Python 2 vs 3), use source activate to make sure that you are running the pytest installed for the same python that the module is installed in.


回答 22

对于任何尝试了一切但仍然出错的人,我都有解决方法。

安装pytest文件夹中,转到pytest-env文件夹。

打开 pyvenv.cfg文件。

在文件中,将include-system-site-packagesfalse更改为true

home = /usr/bin
include-system-site-packages = true
version = 3.6.6

希望它能起作用。不要忘记投票。

For anyone who tried everything and still getting error,I have a work around.

In the folder where pytest is installed,go to pytest-env folder.

Open pyvenv.cfg file.

In the file change include-system-site-packages from false to true.

home = /usr/bin
include-system-site-packages = true
version = 3.6.6

Hope it works .Don’t forget to up vote.


回答 23

如果您已经有.pyc文件,请尝试将其删除。

今天,我遇到了这个问题,这是发生了什么:

首先,我在Mac中运行pytest(这将生成pyc文件),然后启动安装了项目目录的docker容器(操作系统为alpine),然后尝试在容器中运行pytest时,发生ImportError。清除所有pyc文件后,再也没有错误。

希望这会有所帮助。

If you already have .pyc files, try to delete them.

Today, I encounter this problem, here is what happend:

first I run pytest in mac (this will generate pyc files) then I launch a docker container (the os is alpine), with project dir mounted, and then when I try to run pytest in container, ImportError happens. after cleaning all pyc files, no error any more.

Hope this may be helpful.


回答 24

如果您的文件夹中需要一个init .py文件,请制作该文件夹的副本,然后删除其中的init .py来运行您的测试,该测试适用于本地项目。如果需要定期运行测试,请查看是否可以将init .py移到单独的文件中。

if you need a init.py file in your folder make a copy of the folder and delete init.py in that one to run your tests it works for local projects. If you need to run test regularly see if you can move your init.py to a separate file.


回答 25

[解决]在直接进入删除/添加的解决方案之前__init__.py,我们可能还想看一下如何在您的类中完成导入。实际上,我只是__init__.py想着这可能是问题的原因,所以迷失了一天。)但是,那确实很有用。

就我而言,这是将类从一个python类调用到另一个正在抛出的python类的错误方式 ImportError。修复了类/模块的调用方式,它就像魅力一样工作。希望这对其他人也有帮助。

是的,对于类似的错误,根据代码的编写方式,我们可能有不同的解决方案。最好花更多时间进行自我调试。获得的经验:)编码愉快!!!

[Resolved] Before directly jumping into the solution of removing/ adding __init__.py, we might also want to look at how the imports are been done in your classes. Actually, I lost a day playing around with just __init__.py thinking that might be the problem :) However, that was quite informative.

In my case, it was the wrong way of calling classes from one python class to another python class which was throwing ImportError. Fixed the way classes/ modules to be called and it worked like charm. Hope, this helps others as well.

And yes, for similar error, we might have different solutions depending on how the code is written. Better to spend more time on self debugging. Lesson Learnt :) Happy coding!!!


回答 26

就我而言,我在一个容器中工作,不幸的是pytest倾向于使用python2.7而不是我选择的python3解释器。

就我而言,这可行:

python3 -m pytest

我的文件夹结构

/
app/
-module1.py
-module2.py
-tests/
--test_module1.py
--test_module2.py
requirements.txt
README.md

In my case I am working in a container and unfortuantely pytest has the tendency to use python2.7 rather than my python3 interpreter of choice.

In my case this worked:

python3 -m pytest

My folder structure

/
app/
-module1.py
-module2.py
-tests/
--test_module1.py
--test_module2.py
requirements.txt
README.md

回答 27

我已经将所有测试放在一个tests文件夹中,并且得到了相同的错误。我通过在该文件夹中添加init .py 来解决此问题,如下所示:

.
|-- Pipfile
|-- Pipfile.lock
|-- README.md
|-- api
|-- app.py
|-- config.py
|-- migrations
|-- pull_request_template.md
|-- settings.py
`-- tests
    |-- __init__.py <------
    |-- conftest.py
    `-- test_sample.py

I had placed all my tests in a tests folder and was getting the same error. I solved this by adding an init.py in that folder like so:

.
|-- Pipfile
|-- Pipfile.lock
|-- README.md
|-- api
|-- app.py
|-- config.py
|-- migrations
|-- pull_request_template.md
|-- settings.py
`-- tests
    |-- __init__.py <------
    |-- conftest.py
    `-- test_sample.py

有趣好用的Python教程

退出移动版
微信支付
请使用 微信 扫码支付