Python 实用宝典

Question 1

I know @ is for decorators, but what is @= for in Python? Is it just reservation for some future idea?

This is just one of my many questions while reading tokenizer.py.

Question 2

The @ (at) operator is intended to be used for matrix multiplication. No builtin Python types implement this operator.

The @ operator was introduced in Python 3.5. @= is matrix multiplication followed by assignment, as you would expect. They map to __matmul__, __rmatmul__ or __imatmul__ similar to how + and += map to __add__, __radd__ or __iadd__.

The operator and the rationale behind it are discussed in detail in PEP 465.

Question 3

@= and @ are new operators introduced in Python 3.5 performing matrix multiplication. They are meant to clarify the confusion which existed so far with the operator * which was used either for element-wise multiplication or matrix multiplication depending on the convention employed in that particular library/code. As a result, in the future, the operator * is meant to be used for element-wise multiplication only.

As explained in PEP0465, two operators were introduced:

A new binary operator A @ B, used similarly as A * B
An in-place version A @= B, used similarly as A *= B

Matrix Multiplication vs Element-wise Multiplication

To quickly highlight the difference, for two matrices:

A = [[1, 2],    B = [[11, 12],
     [3, 4]]         [13, 14]]

Element-wise multiplication will yield:

A * B = [[1 * 11,   2 * 12], 
         [3 * 13,   4 * 14]]

Matrix multiplication will yield:

A @ B  =  [[1 * 11 + 2 * 13,   1 * 12 + 2 * 14],
           [3 * 11 + 4 * 13,   3 * 12 + 4 * 14]]

Usage in Numpy

So far, Numpy used the following convention:

the * operator (and arithmetic operators in general) were defined as element-wise operations on ndarrays and as matrix-multiplication on numpy.matrix type.
method/function dot was used for matrix multiplication of ndarrays

Introduction of the @ operator makes the code involving matrix multiplications much easier to read. PEP0465 gives us an example:

# Current implementation of matrix multiplications using dot function
S = np.dot((np.dot(H, beta) - r).T,
            np.dot(inv(np.dot(np.dot(H, V), H.T)), np.dot(H, beta) - r))

# Current implementation of matrix multiplications using dot method
S = (H.dot(beta) - r).T.dot(inv(H.dot(V).dot(H.T))).dot(H.dot(beta) - r)

# Using the @ operator instead
S = (H @ beta - r).T @ inv(H @ V @ H.T) @ (H @ beta - r)

Clearly, the last implementation is much easier to read and interpret as an equation.

Question 4

@ is the new operator for Matrix Multiplication added in Python3.5

Reference: https://docs.python.org/3/whatsnew/3.5.html#whatsnew-pep-465

Example

C = A @ B

Question 5

I need to know what += does in python. It’s that simple. I also would appreciate links to definitions of other short hand tools in python.

Question 6

In Python, += is sugar coating for the __iadd__ special method, or __add__ or __radd__ if __iadd__ isn’t present. The __iadd__ method of a class can do anything it wants. The list object implements it and uses it to iterate over an iterable object appending each element to itself in the same way that the list’s extend method does.

Here’s a simple custom class that implements the __iadd__ special method. You initialize the object with an int, then can use the += operator to add a number. I’ve added a print statement in __iadd__ to show that it gets called. Also, __iadd__ is expected to return an object, so I returned the addition of itself plus the other number which makes sense in this case.

>>> class Adder(object):
        def __init__(self, num=0):
            self.num = num

        def __iadd__(self, other):
            print 'in __iadd__', other
            self.num = self.num + other
            return self.num

>>> a = Adder(2)
>>> a += 3
in __iadd__ 3
>>> a
5

Hope this helps.

Question 7

+= adds another value with the variable’s value and assigns the new value to the variable.

>>> x = 3
>>> x += 2
>>> print x
5

-=, *=, /= does similar for subtraction, multiplication and division.

Question 8

x += 5 is not exactly same as saying x = x + 5 in Python.

Note here:

In [1]: x = [2,3,4]    
In [2]: y = x    
In [3]: x += 7,8,9    
In [4]: x
Out[4]: [2, 3, 4, 7, 8, 9]    
In [5]: y
Out[5]: [2, 3, 4, 7, 8, 9]    
In [6]: x += [44,55]    
In [7]: x
Out[7]: [2, 3, 4, 7, 8, 9, 44, 55]    
In [8]: y
Out[8]: [2, 3, 4, 7, 8, 9, 44, 55]    
In [9]: x = x + [33,22]    
In [10]: x
Out[10]: [2, 3, 4, 7, 8, 9, 44, 55, 33, 22]    
In [11]: y
Out[11]: [2, 3, 4, 7, 8, 9, 44, 55]

See for reference: Why does += behave unexpectedly on lists?

Question 9

+= adds a number to a variable, changing the variable itself in the process (whereas + would not). Similar to this, there are the following that also modifies the variable:

-=, subtracts a value from variable, setting the variable to the result
*=, multiplies the variable and a value, making the outcome the variable
/=, divides the variable by the value, making the outcome the variable
%=, performs modulus on the variable, with the variable then being set to the result of it

There may be others. I am not a Python programmer.

Question 10

It adds the right operand to the left. x += 2 means x = x + 2

It can also add elements to a list — see this SO thread.

Question 11

It is not a mere syntactic shortcut. Try this:

x=[]                   # empty list
x += "something"       # iterates over the string and appends to list
print(x)               # ['s', 'o', 'm', 'e', 't', 'h', 'i', 'n', 'g']

versus

x=[]                   # empty list
x = x + "something"    # TypeError: can only concatenate list (not "str") to list

This illustrates that += invokes the iadd list method but + invokes add, which do different things with lists.

Question 12

Notionally a += b “adds” b to a storing the result in a. This simplistic description would describe the += operator in many languages.

However the simplistic description raises a couple of questions.

What exactly do we mean by “adding”?
What exactly do we mean by “storing the result in a”? python variables don’t store values directly they store references to objects.

In python the answers to both of these questions depend on the data type of a.

So what exactly does “adding” mean?

For numbers it means numeric addition.
For lists, tuples, strings etc it means concatenation.

Note that for lists += is more flexible than +, the + operator on a list requires another list, but the += operator will accept any iterable.

So what does “storing the value in a” mean?

If the object is mutable then it is encouraged (but not required) to perform the modification in-place. So a points to the same object it did before but that object now has different content.

If the object is immutable then it obviously can’t perform the modification in-place. Some mutable objects may also not have an implementation of an in-place “add” operation . In this case the variable “a” will be updated to point to a new object containing the result of an addition operation.

Technically this is implemented by looking for __IADD__ first, if that is not implemented then __ADD__ is tried and finally __RADD__.

Care is required when using += in python on variables where we are not certain of the exact type and in particular where we are not certain if the type is mutable or not. For example consider the following code.

def dostuff(a):
    b = a
    a += (3,4)
    print(repr(a)+' '+repr(b))

dostuff((1,2))
dostuff([1,2])

When we invoke dostuff with a tuple then the tuple is copied as part of the += operation and so b is unaffected. However when we invoke it with a list the list is modified in place, so both a and b are affected.

In python 3, similar behaviour is observed with the “bytes” and “bytearray” types.

Finally note that reassignment happens even if the object is not replaced. This doesn’t matter much if the left hand side is simply a variable but it can cause confusing behaviour when you have an immutable collection referring to mutable collections for example:

a = ([1,2],[3,4])
a[0] += [5]

In this case [5] will successfully be added to the list referred to by a[0] but then afterwards an exception will be raised when the code tries and fails to reassign a[0].

Question 13

The short answer is += can be translated as “add whatever is to the right of the += to the variable on the left of the +=”.

Ex. If you have a = 10 then a += 5 would be: a = a + 5

So, “a” now equal to 15.

Question 14

Note x += y is not the same as x = x + y in some situations where an additional operator is included because of the operator precedence combined with the fact that the right hand side is always evaluated first, e.g.

>>> x = 2
>>> x += 2 and 1
>>> x
3

>>> x = 2
>>> x = x + 2 and 1
>>> x
1

Note the first case expand to:

>>> x = 2
>>> x = x + (2 and 1)
>>> x
3

You are more likely to encounter this in the ‘real world’ with other operators, e.g.

x *= 2 + 1 == x = x * (2 + 1) != x = x * 2 + 1

Question 15

+= is just a shortcut for writing

number = 4
number = number + 1

So instead you would write

numbers = 4
numbers += 1

Both ways are correct but example two helps you write a little less code

Question 16

As others also said, the += operator is a shortcut. An example:

var = 1;
var = var + 1;
#var = 2

It could also be written like so:

var = 1;
var += 1;
#var = 2

So instead of writing the first example, you can just write the second one, which would work just fine.

Question 17

Remember when you used to sum, for example 2 & 3, in your old calculator and every time you hit the = you see 3 added to the total, the += does similar job. Example:

>>> orange = 2
>>> orange += 3
>>> print(orange)
5
>>> orange +=3
>>> print(orange)
8

Question 18

I’m seeing a lot of answers that don’t bring up using += with multiple integers.

One example:

x -= 1 + 3

This would be similar to:

x = x - (1 + 3)

and not:

x = (x - 1) + 3

Question 19

According to the documentation

x += y is equivalent to x = operator.iadd(x, y). Another way to put it is to say that z = operator.iadd(x, y) is equivalent to the compound statement z = x; z += y.

So x += 3 is the same as x = x + 3.

x = 2

x += 3

print(x)

will output 5.

Notice that there’s also

&=
//=
<<=
%=
*=
@=
|=
**=
>>=
-=
/=
^=

Question 20

I’m using itertools.chain to “flatten” a list of lists in this fashion:

uniqueCrossTabs = list(itertools.chain(*uniqueCrossTabs))

how is this different than saying:

uniqueCrossTabs = list(itertools.chain(uniqueCrossTabs))

Question 21

* is the “splat” operator: It takes a list as input, and expands it into actual positional arguments in the function call.

So if uniqueCrossTabs was [ [ 1, 2 ], [ 3, 4 ] ], then itertools.chain(*uniqueCrossTabs) is the same as saying itertools.chain([ 1, 2 ], [ 3, 4 ])

This is obviously different from passing in just uniqueCrossTabs. In your case, you have a list of lists that you wish to flatten; what itertools.chain() does is return an iterator over the concatenation of all the positional arguments you pass to it, where each positional argument is iterable in its own right.

In other words, you want to pass each list in uniqueCrossTabs as an argument to chain(), which will chain them together, but you don’t have the lists in separate variables, so you use the * operator to expand the list of lists into several list arguments.

As Jochen Ritzel has pointed out in the comments, chain.from_iterable() is better-suited for this operation, as it assumes a single iterable of iterables to begin with. Your code then becomes simply:

uniqueCrossTabs = list(itertools.chain.from_iterable(uniqueCrossTabs))

Question 22

It splits the sequence into separate arguments for the function call.

>>> def foo(a, b=None, c=None):
...   print a, b, c
... 
>>> foo([1, 2, 3])
[1, 2, 3] None None
>>> foo(*[1, 2, 3])
1 2 3
>>> def bar(*a):
...   print a
... 
>>> bar([1, 2, 3])
([1, 2, 3],)
>>> bar(*[1, 2, 3])
(1, 2, 3)

Question 23

Just an alternative way of explaining the concept/using it.

import random

def arbitrary():
    return [x for x in range(1, random.randint(3,10))]

a, b, *rest = arbitrary()

# a = 1
# b = 2
# rest = [3,4,5]

Question 24

I ran across the caret operator in python today and trying it out, I got the following output:

>>> 8^3
11
>>> 8^4
12
>>> 8^1
9
>>> 8^0
8
>>> 7^1
6
>>> 7^2
5
>>> 7^7
0
>>> 7^8
15
>>> 9^1
8
>>> 16^1
17
>>> 15^1
14
>>>

It seems to be based on 8, so I’m guessing some sort of byte operation? I can’t seem to find much about this searching sites other than it behaves oddly for floats, does anybody have a link to what this operator does or can you explain it here?

Question 25

It’s a bitwise XOR (exclusive OR).

It results to true if one (and only one) of the operands (evaluates to) true.

To demonstrate:

>>> 0^0
0
>>> 1^1
0
>>> 1^0
1
>>> 0^1
1

To explain one of your own examples:

>>> 8^3
11

Think about it this way:

1000  # 8 (binary)
0011  # 3 (binary)
----  # APPLY XOR ('vertically')
1011  # result = 11 (binary)

Question 26

It invokes the __xor__() or __rxor__() method of the object as needed, which for integer types does a bitwise exclusive-or.

Question 27

It’s a bit-by-bit exclusive-or. Binary bitwise operators are documented in chapter 5 of the Python Language Reference.

Question 28

Generally speaking, the symbol ^ is an infix version of the __xor__ or __rxor__ methods. Whatever data types are placed to the right and left of the symbol must implement this function in a compatible way. For integers, it is the common XOR operation, but for example there is not a built-in definition of the function for type float with type int:

In [12]: 3 ^ 4
Out[12]: 7

In [13]: 3.3 ^ 4
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-13-858cc886783d> in <module>()
----> 1 3.3 ^ 4

TypeError: unsupported operand type(s) for ^: 'float' and 'int'

One neat thing about Python is that you can override this behavior in a class of your own. For example, in some languages the ^ symbol means exponentiation. You could do that this way, just as one example:

class Foo(float):
    def __xor__(self, other):
        return self ** other

Then something like this will work, and now, for instances of Foo only, the ^ symbol will mean exponentiation.

In [16]: x = Foo(3)

In [17]: x
Out[17]: 3.0

In [18]: x ^ 4
Out[18]: 81.0

Question 29

When you use the ^ operator, behind the curtains the method __xor__ is called.

a^b is equivalent to a.__xor__(b).

Also, a ^= b is equivalent to a = a.__ixor__(b) (where __xor__ is used as a fallback when __ixor__ is implicitly called via using ^= but does not exist).

In principle, what __xor__ does is completely up to its implementation. Common use cases in Python are:

Symmetric Difference of sets (all elements present in exactly one of two sets)

Demo:

>>> a = {1, 2, 3}
>>> b = {1, 4, 5}
>>> a^b
{2, 3, 4, 5}
>>> a.symmetric_difference(b)
{2, 3, 4, 5}

Bitwise Non-Equal for the bits of two integers

Demo:

>>> a = 5
>>> b = 6
>>> a^b
3

Explanation:

    101 (5 decimal)
XOR 110 (6 decimal)
-------------------
    011 (3 decimal)

Question 30

I have a class where I want to override the __eq__ method. It seems to make sense that I should override the __ne__ method as well, but does it make sense to implement __ne__ in terms of __eq__ as such?

class A:

    def __init__(self, attr):
        self.attr = attr

    def __eq__(self, other):
        return self.attr == other.attr
    
    def __ne__(self, other):
        return not self.__eq__(other)

Or is there something that I am missing with the way Python uses these methods that makes this not a good idea?

Question 31

Yes, that’s perfectly fine. In fact, the documentation urges you to define __ne__ when you define __eq__:

There are no implied relationships among the comparison operators. The truth of x==y does not imply that x!=y is false. Accordingly, when defining __eq__(), one should also define __ne__() so that the operators will behave as expected.

In a lot of cases (such as this one), it will be as simple as negating the result of __eq__, but not always.

Question 32

Python, should I implement __ne__() operator based on __eq__?

Short Answer: Don’t implement it, but if you must, use `==`, not `eq`

In Python 3, != is the negation of == by default, so you are not even required to write a __ne__, and the documentation is no longer opinionated on writing one.

Generally speaking, for Python 3-only code, don’t write one unless you need to overshadow the parent implementation, e.g. for a builtin object.

That is, keep in mind Raymond Hettinger’s comment:

The __ne__ method follows automatically from __eq__ only if __ne__ isn’t already defined in a superclass. So, if you’re inheriting from a builtin, it’s best to override both.

If you need your code to work in Python 2, follow the recommendation for Python 2 and it will work in Python 3 just fine.

In Python 2, Python itself does not automatically implement any operation in terms of another – therefore, you should define the __ne__ in terms of == instead of the __eq__. E.G.

class A(object):
    def __eq__(self, other):
        return self.value == other.value

    def __ne__(self, other):
        return not self == other # NOT `return not self.__eq__(other)`

See proof that

implementing __ne__() operator based on __eq__ and
not implementing __ne__ in Python 2 at all

provides incorrect behavior in the demonstration below.

Long Answer

The documentation for Python 2 says:

There are no implied relationships among the comparison operators. The truth of x==y does not imply that x!=y is false. Accordingly, when defining __eq__(), one should also define __ne__() so that the operators will behave as expected.

So that means that if we define __ne__ in terms of the inverse of __eq__, we can get consistent behavior.

This section of the documentation has been updated for Python 3:

By default, __ne__() delegates to __eq__() and inverts the result unless it is NotImplemented.

and in the “what’s new” section, we see this behavior has changed:

!= now returns the opposite of ==, unless == returns NotImplemented.

For implementing __ne__, we prefer to use the == operator instead of using the __eq__ method directly so that if self.__eq__(other) of a subclass returns NotImplemented for the type checked, Python will appropriately check other.__eq__(self) From the documentation:

The NotImplemented object

This type has a single value. There is a single object with this value. This object is accessed through the built-in name NotImplemented. Numeric methods and rich comparison methods may return this value if they do not implement the operation for the operands provided. (The interpreter will then try the reflected operation, or some other fallback, depending on the operator.) Its truth value is true.

When given a rich comparison operator, if they’re not the same type, Python checks if the other is a subtype, and if it has that operator defined, it uses the other‘s method first (inverse for <, <=, >= and >). If NotImplemented is returned, then it uses the opposite’s method. (It does not check for the same method twice.) Using the == operator allows for this logic to take place.

Expectations

Semantically, you should implement __ne__ in terms of the check for equality because users of your class will expect the following functions to be equivalent for all instances of A.:

def negation_of_equals(inst1, inst2):
    """always should return same as not_equals(inst1, inst2)"""
    return not inst1 == inst2

def not_equals(inst1, inst2):
    """always should return same as negation_of_equals(inst1, inst2)"""
    return inst1 != inst2

That is, both of the above functions should always return the same result. But this is dependent on the programmer.

Demonstration of unexpected behavior when defining `ne` based on `eq`:

First the setup:

class BaseEquatable(object):
    def __init__(self, x):
        self.x = x
    def __eq__(self, other):
        return isinstance(other, BaseEquatable) and self.x == other.x

class ComparableWrong(BaseEquatable):
    def __ne__(self, other):
        return not self.__eq__(other)

class ComparableRight(BaseEquatable):
    def __ne__(self, other):
        return not self == other

class EqMixin(object):
    def __eq__(self, other):
        """override Base __eq__ & bounce to other for __eq__, e.g. 
        if issubclass(type(self), type(other)): # True in this example
        """
        return NotImplemented

class ChildComparableWrong(EqMixin, ComparableWrong):
    """__ne__ the wrong way (__eq__ directly)"""

class ChildComparableRight(EqMixin, ComparableRight):
    """__ne__ the right way (uses ==)"""

class ChildComparablePy3(EqMixin, BaseEquatable):
    """No __ne__, only right in Python 3."""

Instantiate non-equivalent instances:

right1, right2 = ComparableRight(1), ChildComparableRight(2)
wrong1, wrong2 = ComparableWrong(1), ChildComparableWrong(2)
right_py3_1, right_py3_2 = BaseEquatable(1), ChildComparablePy3(2)

Expected Behavior:

(Note: while every second assertion of each of the below is equivalent and therefore logically redundant to the one before it, I’m including them to demonstrate that order does not matter when one is a subclass of the other.)

These instances have __ne__ implemented with ==:

assert not right1 == right2
assert not right2 == right1
assert right1 != right2
assert right2 != right1

These instances, testing under Python 3, also work correctly:

assert not right_py3_1 == right_py3_2
assert not right_py3_2 == right_py3_1
assert right_py3_1 != right_py3_2
assert right_py3_2 != right_py3_1

And recall that these have __ne__ implemented with __eq__ – while this is the expected behavior, the implementation is incorrect:

assert not wrong1 == wrong2         # These are contradicted by the
assert not wrong2 == wrong1         # below unexpected behavior!

Unexpected Behavior:

Note that this comparison contradicts the comparisons above (not wrong1 == wrong2).

>>> assert wrong1 != wrong2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError

and,

>>> assert wrong2 != wrong1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError

Don’t skip `ne` in Python 2

For evidence that you should not skip implementing __ne__ in Python 2, see these equivalent objects:

>>> right_py3_1, right_py3_1child = BaseEquatable(1), ChildComparablePy3(1)
>>> right_py3_1 != right_py3_1child # as evaluated in Python 2!
True

The above result should be False!

Python 3 source

The default CPython implementation for __ne__ is in typeobject.c in object_richcompare:

case Py_NE:
    /* By default, __ne__() delegates to __eq__() and inverts the result,
       unless the latter returns NotImplemented. */
    if (Py_TYPE(self)->tp_richcompare == NULL) {
        res = Py_NotImplemented;
        Py_INCREF(res);
        break;
    }
    res = (*Py_TYPE(self)->tp_richcompare)(self, other, Py_EQ);
    if (res != NULL && res != Py_NotImplemented) {
        int ok = PyObject_IsTrue(res);
        Py_DECREF(res);
        if (ok < 0)
            res = NULL;
        else {
            if (ok)
                res = Py_False;
            else
                res = Py_True;
            Py_INCREF(res);
        }
    }
    break;

But the default `ne` uses `eq`?

Python 3’s default __ne__ implementation detail at the C level uses __eq__ because the higher level == (PyObject_RichCompare) would be less efficient – and therefore it must also handle NotImplemented.

If __eq__ is correctly implemented, then the negation of == is also correct – and it allows us to avoid low level implementation details in our __ne__.

Using == allows us to keep our low level logic in one place, and avoid addressing NotImplemented in __ne__.

One might incorrectly assume that == may return NotImplemented.

It actually uses the same logic as the default implementation of __eq__, which checks for identity (see do_richcompare and our evidence below)

class Foo:
    def __ne__(self, other):
        return NotImplemented
    __eq__ = __ne__

f = Foo()
f2 = Foo()

And the comparisons:

>>> f == f
True
>>> f != f
False
>>> f2 == f
False
>>> f2 != f
True

Performance

Don’t take my word for it, let’s see what’s more performant:

class CLevel:
    "Use default logic programmed in C"

class HighLevelPython:
    def __ne__(self, other):
        return not self == other

class LowLevelPython:
    def __ne__(self, other):
        equal = self.__eq__(other)
        if equal is NotImplemented:
            return NotImplemented
        return not equal

def c_level():
    cl = CLevel()
    return lambda: cl != cl

def high_level_python():
    hlp = HighLevelPython()
    return lambda: hlp != hlp

def low_level_python():
    llp = LowLevelPython()
    return lambda: llp != llp

I think these performance numbers speak for themselves:

>>> import timeit
>>> min(timeit.repeat(c_level()))
0.09377292497083545
>>> min(timeit.repeat(high_level_python()))
0.2654011140111834
>>> min(timeit.repeat(low_level_python()))
0.3378178110579029

This makes sense when you consider that low_level_python is doing logic in Python that would otherwise be handled on the C level.

Response to some critics

Another answerer writes:

Aaron Hall’s implementation not self == other of the __ne__ method is incorrect as it can never return NotImplemented (not NotImplemented is False) and therefore the __ne__ method that has priority can never fall back on the __ne__ method that does not have priority.

Having __ne__ never return NotImplemented does not make it incorrect. Instead, we handle prioritization with NotImplemented via the check for equality with ==. Assuming == is correctly implemented, we’re done.

not self == other used to be the default Python 3 implementation of the __ne__ method but it was a bug and it was corrected in Python 3.4 on January 2015, as ShadowRanger noticed (see issue #21408).

Well, let’s explain this.

As noted earlier, Python 3 by default handles __ne__ by first checking if self.__eq__(other) returns NotImplemented (a singleton) – which should be checked for with is and returned if so, else it should return the inverse. Here is that logic written as a class mixin:

class CStyle__ne__:
    """Mixin that provides __ne__ functionality equivalent to 
    the builtin functionality
    """
    def __ne__(self, other):
        equal = self.__eq__(other)
        if equal is NotImplemented:
            return NotImplemented
        return not equal

This is necessary for correctness for C level Python API, and it was introduced in Python 3, making

redundant. All relevant __ne__ methods were removed, including ones implementing their own check as well as ones that delegate to __eq__ directly or via == – and == was the most common way of doing so.

Is Symmetry Important?

Our persistent critic provides a pathological example to make the case for handling NotImplemented in __ne__, valuing symmetry above all else. Let’s steel-man the argument with a clear example:

class B:
    """
    this class has no __eq__ implementation, but asserts 
    any instance is not equal to any other object
    """
    def __ne__(self, other):
        return True

class A:
    "This class asserts instances are equivalent to all other objects"
    def __eq__(self, other):
        return True

>>> A() == B(), B() == A(), A() != B(), B() != A()
(True, True, False, True)

So, by this logic, in order to maintain symmetry, we need to write the complicated __ne__, regardless of Python version.

class B:
    def __ne__(self, other):
        return True

class A:
    def __eq__(self, other):
        return True
    def __ne__(self, other):
        result = other.__eq__(self)
        if result is NotImplemented:
            return NotImplemented
        return not result

>>> A() == B(), B() == A(), A() != B(), B() != A()
(True, True, True, True)

Apparently we should give no mind that these instances are both equal and not equal.

I propose that symmetry is less important than the presumption of sensible code and following the advice of the documentation.

However, if A had a sensible implementation of __eq__, then we could still follow my direction here and we would still have symmetry:

class B:
    def __ne__(self, other):
        return True

class A:
    def __eq__(self, other):
        return False         # <- this boolean changed... 

>>> A() == B(), B() == A(), A() != B(), B() != A()
(False, False, True, True)

Conclusion

For Python 2 compatible code, use == to implement __ne__. It is more:

correct
simple
performant

In Python 3 only, use the low-level negation on the C level – it is even more simple and performant (though the programmer is responsible for determining that it is correct).

Again, do not write low-level logic in high level Python.

Question 33

Just for the record, a canonically correct and cross Py2/Py3 portable __ne__ would look like:

import sys

class ...:
    ...
    def __eq__(self, other):
        ...

    if sys.version_info[0] == 2:
        def __ne__(self, other):
            equal = self.__eq__(other)
            return equal if equal is NotImplemented else not equal

This works with any __eq__ you might define:

Unlike not (self == other), doesn’t interfere with in some annoying/complex cases involving comparisons where one of the classes involved doesn’t imply that the result of __ne__ is the same as the result of not on __eq__ (e.g. SQLAlchemy’s ORM, where both __eq__ and __ne__ return special proxy objects, not True or False, and trying to not the result of __eq__ would return False, rather than the correct proxy object).
Unlike not self.__eq__(other), this correctly delegates to the __ne__ of the other instance when self.__eq__ returns NotImplemented (not self.__eq__(other) would be extra wrong, because NotImplemented is truthy, so when __eq__ didn’t know how to perform the comparison, __ne__ would return False, implying that the two objects were equal when in fact the only object asked had no idea, which would imply a default of not equal)

If your __eq__ doesn’t use NotImplemented returns, this works (with meaningless overhead), if it does use NotImplemented sometimes, this handles it properly. And the Python version check means that if the class is import-ed in Python 3, __ne__ is left undefined, allowing Python’s native, efficient fallback __ne__ implementation (a C version of the above) to take over.

Why this is needed

Python overloading rules

The explanation of why you do this instead of other solutions is somewhat arcane. Python has a couple general rules about overloading operators, and comparison operators in particular:

(Applies to all operators) When running LHS OP RHS, try LHS.__op__(RHS), and if that returns NotImplemented, try RHS.__rop__(LHS). Exception: If RHS is a subclass of LHS‘s class, then test RHS.__rop__(LHS) first. In the case of comparison operators, __eq__ and __ne__ are their own “rop”s (so the test order for __ne__ is LHS.__ne__(RHS), then RHS.__ne__(LHS), reversed if RHS is a subclass of LHS‘s class)
Aside from the idea of the “swapped” operator, there is no implied relationship between the operators. Even for instance of the same class, LHS.__eq__(RHS) returning True does not imply LHS.__ne__(RHS) returns False (in fact, the operators aren’t even required to return boolean values; ORMs like SQLAlchemy intentionally do not, allowing for a more expressive query syntax). As of Python 3, the default __ne__ implementation behaves this way, but it’s not contractual; you can override __ne__ in ways that aren’t strict opposites of __eq__.

How this applies to overloading comparators

So when you overload an operator, you have two jobs:

If you know how to implement the operation yourself, do so, using only your own knowledge of how to do the comparison (never delegate, implicitly or explicitly, to the other side of the operation; doing so risks incorrectness and/or infinite recursion, depending on how you do it)
If you don’t know how to implement the operation yourself, always return NotImplemented, so Python can delegate to the other operand’s implementation

The problem with `not self.eq(other)`

def __ne__(self, other):
    return not self.__eq__(other)

never delegates to the other side (and is incorrect if __eq__ properly returns NotImplemented). When self.__eq__(other) returns NotImplemented (which is “truthy”), you silently return False, so A() != something_A_knows_nothing_about returns False, when it should have checked if something_A_knows_nothing_about knew how to compare to instances of A, and if it doesn’t, it should have returned True (since if neither side knows how to compare to the other, they’re considered not equal to one another). If A.__eq__ is incorrectly implemented (returning False instead of NotImplemented when it doesn’t recognize the other side), then this is “correct” from A‘s perspective, returning True (since A doesn’t think it’s equal, so it’s not equal), but it might be wrong from something_A_knows_nothing_about‘s perspective, since it never even asked something_A_knows_nothing_about; A() != something_A_knows_nothing_about ends up True, but something_A_knows_nothing_about != A() could False, or any other return value.

The problem with `not self == other`

def __ne__(self, other):
    return not self == other

is more subtle. It’s going to be correct for 99% of classes, including all classes for which __ne__ is the logical inverse of __eq__. But not self == other breaks both of the rules mentioned above, which means for classes where __ne__ isn’t the logical inverse of __eq__, the results are once again non-symmetric, because one of the operands is never asked if it can implement __ne__ at all, even if the other operand can’t. The simplest example is a weirdo class which returns False for all comparisons, so A() == Incomparable() and A() != Incomparable() both return False. With a correct implementation of A.__ne__ (one which returns NotImplemented when it doesn’t know how to do the comparison), the relationship is symmetric; A() != Incomparable() and Incomparable() != A() agree on the outcome (because in the former case, A.__ne__ returns NotImplemented, then Incomparable.__ne__ returns False, while in the latter, Incomparable.__ne__ returns False directly). But when A.__ne__ is implemented as return not self == other, A() != Incomparable() returns True (because A.__eq__ returns, not NotImplemented, then Incomparable.__eq__ returns False, and A.__ne__ inverts that to True), while Incomparable() != A() returns False.

You can see an example of this in action here.

Obviously, a class that always returns False for both __eq__ and __ne__ is a little strange. But as mentioned before, __eq__ and __ne__ don’t even need to return True/False; the SQLAlchemy ORM has classes with comparators that returns a special proxy object for query building, not True/False at all (they’re “truthy” if evaluated in a boolean context, but they’re never supposed to be evaluated in such a context).

By failing to overload __ne__ properly, you will break classes of that sort, as the code:

 results = session.query(MyTable).filter(MyTable.fieldname != MyClassWithBadNE())

will work (assuming SQLAlchemy knows how to insert MyClassWithBadNE into a SQL string at all; this can be done with type adapters without MyClassWithBadNE having to cooperate at all), passing the expected proxy object to filter, while:

 results = session.query(MyTable).filter(MyClassWithBadNE() != MyTable.fieldname)

will end up passing filter a plain False, because self == other returns a proxy object, and not self == other just converts the truthy proxy object to False. Hopefully, filter throws an exception on being handled invalid arguments like False. While I’m sure many will argue that MyTable.fieldname should be consistently on the left hand side of the comparison, the fact remains that there is no programmatic reason to enforce this in the general case, and a correct generic __ne__ will work either way, while return not self == other only works in one arrangement.

Question 34

Correct `ne` implementation

@ShadowRanger’s implementation of the special method __ne__ is the correct one:

def __ne__(self, other):
    result = self.__eq__(other)
    if result is not NotImplemented:
        return not result
    return NotImplemented

It also happens to be the default implementation of the special method __ne__ since Python 3.4, as stated in the Python documentation:

By default, __ne__() delegates to __eq__() and inverts the result unless it is NotImplemented.

Also note that returning the value NotImplemented for unsupported operands is not specific to the special method __ne__. In fact, all the special comparison methods¹ and special numeric methods² should return the value NotImplemented for unsupported operands, as specified in the Python documentation:

NotImplemented

This type has a single value. There is a single object with this value. This object is accessed through the built-in name NotImplemented. Numeric methods and rich comparison methods should return this value if they do not implement the operation for the operands provided. (The interpreter will then try the reflected operation, or some other fallback, depending on the operator.) Its truth value is true.

An example for the special numeric methods is given in the Python documentation:

class MyIntegral(Integral):

    def __add__(self, other):
        if isinstance(other, MyIntegral):
            return do_my_adding_stuff(self, other)
        elif isinstance(other, OtherTypeIKnowAbout):
            return do_my_other_adding_stuff(self, other)
        else:
            return NotImplemented

    def __radd__(self, other):
        if isinstance(other, MyIntegral):
            return do_my_adding_stuff(other, self)
        elif isinstance(other, OtherTypeIKnowAbout):
            return do_my_other_adding_stuff(other, self)
        elif isinstance(other, Integral):
            return int(other) + int(self)
        elif isinstance(other, Real):
            return float(other) + float(self)
        elif isinstance(other, Complex):
            return complex(other) + complex(self)
        else:
            return NotImplemented

¹ The special comparison methods: __lt__, __le__, __eq__, __ne__, __gt__ and __ge__.

² The special numeric methods: __add__, __sub__, __mul__, __matmul__, __truediv__, __floordiv__, __mod__, __divmod__, __pow__, __lshift__, __rshift__, __and__, __xor__, __or__ and their __r*__ reflected and __i*__ in-place counterparts.

Incorrect `ne` implementation #1

@Falmarri’s implementation of the special method __ne__ is incorrect:

def __ne__(self, other):
    return not self.__eq__(other)

The problem with this implementation is that it does not fall back on the special method __ne__ of the other operand as it never returns the value NotImplemented (the expression not self.__eq__(other) evaluates to the value True or False, including when its subexpression self.__eq__(other) evaluates to the value NotImplemented since the expression bool(NotImplemented) evaluates to the value True). The Boolean evaluation of the value NotImplemented breaks the complement relationship between the comparison operators != and ==:

class Correct:

    def __ne__(self, other):
        result = self.__eq__(other)
        if result is not NotImplemented:
            return not result
        return NotImplemented


class Incorrect:

    def __ne__(self, other):
        return not self.__eq__(other)


x, y = Correct(), Correct()
assert (x != y) is not (x == y)

x, y = Incorrect(), Incorrect()
assert (x != y) is not (x == y)  # AssertionError

Incorrect `ne` implementation #2

@AaronHall’s implementation of the special method __ne__ is also incorrect:

def __ne__(self, other):
    return not self == other

The problem with this implementation is that it directly falls back on the special method __eq__ of the other operand, bypassing the special method __ne__ of the other operand as it never returns the value NotImplemented (the expression not self == other falls back on the special method __eq__ of the other operand and evaluates to the value True or False). Bypassing a method is incorrect because that method may have side effects like updating the state of the object:

class Correct:

    def __init__(self):
        self.counter = 0

    def __ne__(self, other):
        self.counter += 1
        result = self.__eq__(other)
        if result is not NotImplemented:
            return not result
        return NotImplemented


class Incorrect:

    def __init__(self):
        self.counter = 0

    def __ne__(self, other):
        self.counter += 1
        return not self == other


x, y = Correct(), Correct()
assert x != y
assert x.counter == y.counter

x, y = Incorrect(), Incorrect()
assert x != y
assert x.counter == y.counter  # AssertionError

Understanding comparison operations

In mathematics, a binary relation R over a set X is a set of ordered pairs (x, y) in X². The statement (x, y) in R reads “x is R-related to y” and is denoted by xRy.

Properties of a binary relation R over a set X:

R is reflexive when for all x in X, xRx.
R is irreflexive (also called strict) when for all x in X, not xRx.
R is symmetric when for all x and y in X, if xRy then yRx.
R is antisymmetric when for all x and y in X, if xRy and yRx then x = y.
R is transitive when for all x, y and z in X, if xRy and yRz then xRz.
R is connex (also called total) when for all x and y in X, xRy or yRx.
R is an equivalence relation when R is reflexive, symmetric and transitive.
For example, =. However ≠ is only symmetric.
R is an order relation when R is reflexive, antisymmetric and transitive.
For example, ≤ and ≥.
R is a strict order relation when R is irreflexive, antisymmetric and transitive.
For example, < and >. However ≠ is only irreflexive.

Operations on two binary relations R and S over a set X:

The converse of R is the binary relation R^T = {(y, x) | xRy} over X.
The complement of R is the binary relation ¬R = {(x, y) | not xRy} over X.
The union of R and S is the binary relation R ∪ S = {(x, y) | xRy or xSy} over X.

Relationships between comparison relations that are always valid:

2 complementary relationships: = and ≠ are each other’s complement;
6 converse relationships: = is the converse of itself, ≠ is the converse of itself, < and > are each other’s converse, and ≤ and ≥ are each other’s converse;
2 union relationships: ≤ is the union < and =, and ≥ is the union of > and =.

Relationships between comparison relations that are only valid for connex orders:

4 complementary relationships: < and ≥ are each other’s complement, and > and ≤ are each other’s complement.

So to correctly implement in Python the comparison operators ==, !=, <, >, <=, and >= corresponding to the comparison relations =, ≠, <, >, ≤, and ≥, all the above mathematical properties and relationships should hold.

A comparison operation x operator y calls the special comparison method __operator__ of the class of one of its operands:

class X:

    def __operator__(self, other):
        # implementation

Since R is reflexive implies xRx, a reflexive comparison operation x operator y (x == y, x <= y and x >= y) or reflexive special comparison method call x.__operator__(y) (x.__eq__(y), x.__le__(y) and x.__ge__(y)) should evaluate to the value True if x and y are identical, that is if the expression x is y evaluates to True. Since R is irreflexive implies not xRx, an irreflexive comparison operation x operator y (x != y, x < y and x > y) or irreflexive special comparison method call x.__operator__(y) (x.__ne__(y), x.__lt__(y) and x.__gt__(y)) should evaluate to the value False if x and y are identical, that is if the expression x is y evaluates to True. The reflexive property is considered by Python for the comparison operator == and associated special comparison method __eq__ but surprisingly not considered for the comparison operators <= and >= and associated special comparison methods __le__ and __ge__, and the irreflexive property is considered by Python for the comparison operator != and associated special comparison method __ne__ but surprisingly not considered for the comparison operators < and > and associated special comparison methods __lt__ and __gt__. The ignored comparison operators instead raise the exception TypeError (and associated special comparison methods instead return the value NotImplemented), as explained in the Python documentation:

The default behavior for equality comparison (== and !=) is based on the identity of the objects. Hence, equality comparison of instances with the same identity results in equality, and equality comparison of instances with different identities results in inequality. A motivation for this default behavior is the desire that all objects should be reflexive (i.e. x is y implies x == y).

A default order comparison (<, >, <=, and >=) is not provided; an attempt raises TypeError. A motivation for this default behavior is the lack of a similar invariant as for equality. [This is incorrect since <= and >= are reflexive like ==, and < and > are irreflexive like !=.]

The class object provides the default implementations of the special comparison methods which are inherited by all its subclasses, as explained in the Python documentation:

object.__lt__(self, other)
object.__le__(self, other)
object.__eq__(self, other)
object.__ne__(self, other)
object.__gt__(self, other)
object.__ge__(self, other)

These are the so-called “rich comparison” methods. The correspondence between operator symbols and method names is as follows: x<y calls x.__lt__(y), x<=y calls x.__le__(y), x==y calls x.__eq__(y), x!=y calls x.__ne__(y), x>y calls x.__gt__(y), and x>=y calls x.__ge__(y).

A rich comparison method may return the singleton NotImplemented if it does not implement the operation for a given pair of arguments.

[…]

There are no swapped-argument versions of these methods (to be used when the left argument does not support the operation but the right argument does); rather, __lt__() and __gt__() are each other’s reflection, __le__() and __ge__() are each other’s reflection, and __eq__() and __ne__() are their own reflection. If the operands are of different types, and right operand’s type is a direct or indirect subclass of the left operand’s type, the reflected method of the right operand has priority, otherwise the left operand’s method has priority. Virtual subclassing is not considered.

Since R = (R^T)^T, a comparison xRy is equivalent to the converse comparison yR^Tx (informally named “reflected” in the Python documentation). So there are two ways to compute the result of a comparison operation x operator y: calling either x.__operator__(y) or y.__operatorT__(x). Python uses the following computing strategy:

It calls x.__operator__(y) unless the right operand’s class is a descendant of the left operand’s class, in which case it calls y.__operatorT__(x) (allowing classes to override their ancestors’ converse special comparison method).
If the operands x and y are unsupported (indicated by the return value NotImplemented), it calls the converse special comparison method as a 1st fallback.
If the operands x and y are unsupported (indicated by the return value NotImplemented), it raises the exception TypeError except for the comparison operators == and != for which it tests respectively the identity and non-identity of the operands x and y as a 2nd fallback (leveraging the reflexivity property of == and irreflexivity property of !=).
It returns the result.

In CPython this is implemented in C code, which can be translated into Python code (with the names eq for ==, ne for !=, lt for <, gt for >, le for <= and ge for >=):

def eq(left, right):
    if type(left) != type(right) and isinstance(right, type(left)):
        result = right.__eq__(left)
        if result is NotImplemented:
            result = left.__eq__(right)
    else:
        result = left.__eq__(right)
        if result is NotImplemented:
            result = right.__eq__(left)
    if result is NotImplemented:
        result = left is right
    return result

def ne(left, right):
    if type(left) != type(right) and isinstance(right, type(left)):
        result = right.__ne__(left)
        if result is NotImplemented:
            result = left.__ne__(right)
    else:
        result = left.__ne__(right)
        if result is NotImplemented:
            result = right.__ne__(left)
    if result is NotImplemented:
        result = left is not right
    return result

def lt(left, right):
    if type(left) != type(right) and isinstance(right, type(left)):
        result = right.__gt__(left)
        if result is NotImplemented:
            result = left.__lt__(right)
    else:
        result = left.__lt__(right)
        if result is NotImplemented:
            result = right.__gt__(left)
    if result is NotImplemented:
        raise TypeError(
            f"'<' not supported between instances of '{type(left).__name__}' "
            f"and '{type(right).__name__}'"
        )
    return result

def gt(left, right):
    if type(left) != type(right) and isinstance(right, type(left)):
        result = right.__lt__(left)
        if result is NotImplemented:
            result = left.__gt__(right)
    else:
        result = left.__gt__(right)
        if result is NotImplemented:
            result = right.__lt__(left)
    if result is NotImplemented:
        raise TypeError(
            f"'>' not supported between instances of '{type(left).__name__}' "
            f"and '{type(right).__name__}'"
        )
    return result

def le(left, right):
    if type(left) != type(right) and isinstance(right, type(left)):
        result = right.__ge__(left)
        if result is NotImplemented:
            result = left.__le__(right)
    else:
        result = left.__le__(right)
        if result is NotImplemented:
            result = right.__ge__(left)
    if result is NotImplemented:
        raise TypeError(
            f"'<=' not supported between instances of '{type(left).__name__}' "
            f"and '{type(right).__name__}'"
        )
    return result

def ge(left, right):
    if type(left) != type(right) and isinstance(right, type(left)):
        result = right.__le__(left)
        if result is NotImplemented:
            result = left.__ge__(right)
    else:
        result = left.__ge__(right)
        if result is NotImplemented:
            result = right.__le__(left)
    if result is NotImplemented:
        raise TypeError(
            f"'>=' not supported between instances of '{type(left).__name__}' "
            f"and '{type(right).__name__}'"
        )
    return result

Since R = ¬(¬R), a comparison xRy is equivalent to the complement comparison ¬(x¬Ry). ≠ is the complement of =, so the special method __ne__ is implemented in terms of the special method __eq__ for supported operands by default, while the other special comparison methods are implemented independently by default (the fact that ≤ is the union of < and =, and ≥ is the union of > and = is surprisingly not considered, which means that currently the special methods __le__ and __ge__ should be user implemented), as explained in the Python documentation:

By default, __ne__() delegates to __eq__() and inverts the result unless it is NotImplemented. There are no other implied relationships among the comparison operators, for example, the truth of (x<y or x==y) does not imply x<=y.

In CPython this is implemented in C code, which can be translated into Python code:

def __eq__(self, other):
    return self is other or NotImplemented

def __ne__(self, other):
    result = self.__eq__(other)
    if result is not NotImplemented:
        return not result
    return NotImplemented

def __lt__(self, other):
    return NotImplemented

def __gt__(self, other):
    return NotImplemented

def __le__(self, other):
    return NotImplemented

def __ge__(self, other):
    return NotImplemented

So by default:

a comparison operation x operator y raises the exception TypeError except for the comparison operators == and != for which it returns respectively the identity and non-identity of the operands x and y;
a special comparison method call x.__operator__(y) returns the value NotImplemented except for the special comparison methods __eq__ and __ne__ for which it returns respectively True and False if the operands x and y are respectively identical and non-identical and the value NotImplemented otherwise.

Question 35

If all of __eq__, __ne__, __lt__, __ge__, __le__, and __gt__ make sense for the class, then just implement __cmp__ instead. Otherwise, do as you’re doing, because of the bit Daniel DiPaolo said (while I was testing it instead of looking it up ;) )

Question 36

I’m trying to learn python and came across some code that is nice and short but doesn’t totally make sense

the context was:

def fn(*args):
    return len(args) and max(args)-min(args)

I get what it’s doing, but why does python do this – ie return the value rather than True/False?

10 and 7-2

returns 5. Similarly, changing the and to or will result in a change in functionality. So

10 or 7 - 2

Would return 10.

Is this legit/reliable style, or are there any gotchas on this?

Question 37

TL;DR

We start by summarising the two behaviour of the two logical operators and and or. These idioms will form the basis of our discussion below.

and

Return the first Falsy value if there are any, else return the last value in the expression.

or

Return the first Truthy value if there are any, else return the last value in the expression.

The behaviour is also summarised in the docs, especially in this table:

The only operator returning a boolean value regardless of its operands is the not operator.

“Truthiness”, and “Truthy” Evaluations

The statement

len(args) and max(args) - min(args)

Is a very ~~pythonic~~ concise (and arguably less readable) way of saying “if args is not empty, return the result of max(args) - min(args)“, otherwise return 0. In general, it is a more concise representation of an if-else expression. For example,

exp1 and exp2

Should (roughly) translate to:

r1 = exp1
if r1:
    r1 = exp2

Or, equivalently,

r1 = exp2 if exp1 else exp1

Similarly,

exp1 or exp2

Should (roughly) translate to:

r1 = exp1
if not r1:
    r1 = exp2

Or, equivalently,

r1 = exp1 if exp1 else exp2

Where exp1 and exp2 are arbitrary python objects, or expressions that return some object. The key to understanding the uses of the logical and and or operators here is understanding that they are not restricted to operating on, or returning boolean values. Any object with a truthiness value can be tested here. This includes int, str, list, dict, tuple, set, NoneType, and user defined objects. Short circuiting rules still apply as well.

But what is truthiness?
It refers to how objects are evaluated when used in conditional expressions. @Patrick Haugh summarises truthiness nicely in this post.

All values are considered “truthy” except for the following, which are “falsy”:

None

False

0

0.0

0j

Decimal(0)

Fraction(0, 1)

[] – an empty list

{} – an empty dict

() – an empty tuple

'' – an empty str

b'' – an empty bytes

set() – an empty set

an empty range, like range(0)

objects for which

obj.__bool__() returns False

obj.__len__() returns 0

A “truthy” value will satisfy the check performed by if or while statements. We use “truthy” and “falsy” to differentiate from the bool values True and False.

How `and` Works

We build on OP’s question as a segue into a discussion on how these operators in these instances.

Given a function with the definition
def foo(*args):
    ...
How do I return the difference between the minimum and maximum value in a list of zero or more arguments?

Finding the minimum and maximum is easy (use the inbuilt functions!). The only snag here is appropriately handling the corner case where the argument list could be empty (for example, calling foo()). We can do both in a single line thanks to the and operator:

def foo(*args):
     return len(args) and max(args) - min(args)

foo(1, 2, 3, 4, 5)
# 4

foo()
# 0

Since and is used, the second expression must also be evaluated if the first is True. Note that, if the first expression is evaluated to be truthy, the return value is always the result of the second expression. If the first expression is evaluated to be Falsy, then the result returned is the result of the first expression.

In the function above, If foo receives one or more arguments, len(args) is greater than 0 (a positive number), so the result returned is max(args) - min(args). OTOH, if no arguments are passed, len(args) is 0 which is Falsy, and 0 is returned.

Note that an alternative way to write this function would be:

def foo(*args):
    if not len(args):
        return 0
    
    return max(args) - min(args)

Or, more concisely,

def foo(*args):
    return 0 if not args else max(args) - min(args)

If course, none of these functions perform any type checking, so unless you completely trust the input provided, do not rely on the simplicity of these constructs.

How `or` Works

I explain the working of or in a similar fashion with a contrived example.

Given a function with the definition
def foo(*args):
    ...
How would you complete foo to return all numbers over 9000?

We use or to handle the corner case here. We define foo as:

def foo(*args):
     return [x for x in args if x > 9000] or 'No number over 9000!'

foo(9004, 1, 2, 500)
# [9004]

foo(1, 2, 3, 4)
# 'No number over 9000!'

foo performs a filtration on the list to retain all numbers over 9000. If there exist any such numbers, the result of the list comprehension is a non-empty list which is Truthy, so it is returned (short circuiting in action here). If there exist no such numbers, then the result of the list comp is [] which is Falsy. So the second expression is now evaluated (a non-empty string) and is returned.

Using conditionals, we could re-write this function as,

def foo(*args):
    r = [x for x in args if x > 9000]
    if not r:
        return 'No number over 9000!' 
    
    return r

As before, this structure is more flexible in terms of error handling.

Question 38

Quoting from Python Docs

Note that neither and nor or restrict the value and type they return to False and True, but rather return the last evaluated argument. This is sometimes useful, e.g., if s is a string that should be replaced by a default value if it is empty, the expression s or 'foo' yields the desired value.

So, this is how Python was designed to evaluate the boolean expressions and the above documentation gives us an insight of why they did it so.

To get a boolean value just typecast it.

return bool(len(args) and max(args)-min(args))

Why?

Short-circuiting.

For example:

2 and 3 # Returns 3 because 2 is Truthy so it has to check 3 too
0 and 3 # Returns 0 because 0 is Falsey and there's no need to check 3 at all

The same goes for or too, that is, it will return the expression which is Truthy as soon as it finds it, cause evaluating the rest of the expression is redundant.

Instead of returning hardcore True or False, Python returns Truthy or Falsey, which are anyway going to evaluate to True or False. You could use the expression as is, and it will still work.

To know what’s Truthy and Falsey, check Patrick Haugh’s answer

Question 39

and and or perform boolean logic, but they return one of the actual values when they are comparing. When using and, values are evaluated in a boolean context from left to right. 0, ”, [], (), {}, and None are false in a boolean context; everything else is true.

If all values are true in a boolean context, and returns the last value.

>>> 2 and 5
5
>>> 2 and 5 and 10
10

If any value is false in a boolean context and returns the first false value.

>>> '' and 5
''
>>> 2 and 0 and 5
0

So the code

return len(args) and max(args)-min(args)

returns the value of max(args)-min(args) when there is args else it returns len(args) which is 0.

Question 40

Is this legit/reliable style, or are there any gotchas on this?

This is legit, it is a short circuit evaluation where the last value is returned.

You provide a good example. The function will return 0 if no arguments are passed, and the code doesn’t have to check for a special case of no arguments passed.

Another way to use this, is to default None arguments to a mutable primitive, like an empty list:

def fn(alist=None):
    alist = alist or []
    ....

If some non-truthy value is passed to alist it defaults to an empty list, handy way to avoid an if statement and the mutable default argument pitfall

Question 41

Gotchas

Yes, there are a few gotchas.

`fn() == fn(3) == fn(4, 4)`

First, if fn returns 0, you cannot know if it was called without any parameter, with one parameter or with multiple, equal parameters :

>>> fn()
0
>>> fn(3)
0
>>> fn(3, 3, 3)
0

What does `fn` mean?

Then, Python is a dynamic language. It’s not specified anywhere what fn does, what its input should be and what its output should look like. Therefore, it’s really important to name the function correctly. Similarly, arguments don’t have to be called args. delta(*numbers) or calculate_range(*numbers) might describe better what the function is supposed to do.

Argument errors

Finally, the logical and operator is supposed to prevent the function to fail if called without any argument. It still fails if some argument isn’t a number, though:

>>> fn('1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in fn
TypeError: unsupported operand type(s) for -: 'str' and 'str'
>>> fn(1, '2')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in fn
TypeError: '>' not supported between instances of 'str' and 'int'
>>> fn('a', 'b')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in fn
TypeError: unsupported operand type(s) for -: 'str' and 'str'

Possible alternative

Here’s a way to write the function according to the “Easier to ask for forgiveness than permission.” principle:

def delta(*numbers):
    try:
        return max(numbers) - min(numbers)
    except TypeError:
        raise ValueError("delta should only be called with numerical arguments") from None
    except ValueError:
        raise ValueError("delta should be called with at least one numerical argument") from None

As an example:

>>> delta()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 7, in delta
ValueError: delta should be called with at least one numerical argument
>>> delta(3)
0
>>> delta('a')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in delta
ValueError: delta should only be called with numerical arguments
>>> delta('a', 'b')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in delta
ValueError: delta should only be called with numerical arguments
>>> delta('a', 3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in delta
ValueError: delta should only be called with numerical arguments
>>> delta(3, 4.5)
1.5
>>> delta(3, 5, 7, 2)
5

If you really don’t want to raise an exception when delta is called without any argument, you could return some value which cannot be possible otherwise (e.g. -1 or None):

>>> def delta(*numbers):
...     try:
...         return max(numbers) - min(numbers)
...     except TypeError:
...         raise ValueError("delta should only be called with numerical arguments") from None
...     except ValueError:
...         return -1 # or None
... 
>>> 
>>> delta()
-1

Question 42

Is this legit/reliable style, or are there any gotchas on this?

I would like to add to this question that it not only legit and reliable but it also ultra practical. Here is a simple example:

>>>example_list = []
>>>print example_list or 'empty list'
empty list

Therefore you can really use it at your advantage. In order to be conscise this is how I see it:

Or operator

Python’s or operator returns the first Truth-y value, or the last value, and stops

And operator

Python’s and operator returns the first False-y value, or the last value, and stops

Behind the scenes

In python, all numbers are interpreted as True except for 0. Therefore, saying:

0 and 10

is the same as:

False and True

Which is clearly False. It is therefore logical that it returns 0

Question 43

Yes. This is the correct behaviour of and comparison.

At least in Python, A and B returns B if A is essentially True including if A is NOT Null, NOT None NOT an Empty container (such as an empty list, dict, etc). A is returned IFF A is essentially False or None or Empty or Null.

On the other hand, A or B returns A if A is essentially True including if A is NOT Null, NOT None NOT an Empty container (such as an empty list, dict, etc), otherwise it returns B.

It is easy to not notice (or to overlook) this behaviour because, in Python, any non-null non-empty object evaluates to True is treated like a boolean.

For example, all the following will print “True”

if [102]: 
    print "True"
else: 
    print "False"

if "anything that is not empty or None": 
    print "True"
else: 
    print "False"

if {1, 2, 3}: 
    print "True"
else: 
    print "False"

On the other hand, all the following will print “False”

if []: 
    print "True"
else: 
    print "False"

if "": 
    print "True"
else: 
    print "False"

if set ([]): 
    print "True"
else: 
    print "False"

Question 44

to understand in simple way,

AND : if first_val is False return first_val else second_value

eg:

1 and 2 # here it will return 2 because 1 is not False

but,

0 and 2 # will return 0 because first value is 0 i.e False

and => if anyone false, it will be false. if both are true then only it will become true

OR : if first_val is False return second_val else first_value

reason is, if first is false it check whether 2 is true or not.

eg:

1 or 2 # here it will return 1 because 1 is not False

but,

0 or 2 # will return 2 because first value is 0 i.e False

or => if anyone false, it will be true. so if first value is false no matter what 2 value suppose to be. so it returns second value what ever it can be.

if anyone is true then it will become true. if both are false then it will become false.

Question 45

When testing for membership, we can use:

x not in y

Or alternatively:

not x in y

There can be many possible contexts for this expression depending on x and y. It could be for a substring check, list membership, dict key existence, for example.

Are the two forms always equivalent?
Is there a preferred syntax?

Question 46

They always give the same result.

In fact, not 'ham' in 'spam and eggs' appears to be special cased to perform a single “not in” operation, rather than an “in” operation and then negating the result:

>>> import dis

>>> def notin():
    'ham' not in 'spam and eggs'
>>> dis.dis(notin)
  2           0 LOAD_CONST               1 ('ham')
              3 LOAD_CONST               2 ('spam and eggs')
              6 COMPARE_OP               7 (not in)
              9 POP_TOP             
             10 LOAD_CONST               0 (None)
             13 RETURN_VALUE    

>>> def not_in():
    not 'ham' in 'spam and eggs'
>>> dis.dis(not_in)
  2           0 LOAD_CONST               1 ('ham')
              3 LOAD_CONST               2 ('spam and eggs')
              6 COMPARE_OP               7 (not in)
              9 POP_TOP             
             10 LOAD_CONST               0 (None)
             13 RETURN_VALUE    

>>> def not__in():
    not ('ham' in 'spam and eggs')
>>> dis.dis(not__in)
  2           0 LOAD_CONST               1 ('ham')
              3 LOAD_CONST               2 ('spam and eggs')
              6 COMPARE_OP               7 (not in)
              9 POP_TOP             
             10 LOAD_CONST               0 (None)
             13 RETURN_VALUE        

>>> def noteq():
    not 'ham' == 'spam and eggs'
>>> dis.dis(noteq)
  2           0 LOAD_CONST               1 ('ham')
              3 LOAD_CONST               2 ('spam and eggs')
              6 COMPARE_OP               2 (==)
              9 UNARY_NOT           
             10 POP_TOP             
             11 LOAD_CONST               0 (None)
             14 RETURN_VALUE

I had thought at first that they always gave the same result, but that not on its own was simply a low precedence logical negation operator, which could be applied to a in b just as easily as any other boolean expression, whereas not in was a separate operator for convenience and clarity.

The disassembly above was revealing! It seems that while not obviously is a logical negation operator, the form not a in b is special cased so that it’s not actually using the general operator. This makes not a in b literally the same expression as a not in b, rather than merely an expression that results in the same value.

Question 47

No, there is no difference.

The operator not in is defined to have the inverse true value of in.

—Python documentation
I would assume not in is preferred because it is more obvious and they added a special case for it.

Question 48

They are identical in meaning, but the pycodestyle Python style guide checker (formerly called pep8) prefers the not in operator in rule E713:

E713: test for membership should be not in

See also “Python if x is not None or if not x is None?” for a very similar choice of style.

Question 49

Others have already made it very clear that the two statements are, down to a quite low level, equivalent.

However, I don’t think that anyone yet has stressed enough that since this leaves the choice up to you, you should

choose the form that makes your code as readable as possible.

And not necessarily as readable as possible to anyone, even if that’s of course a nice thing to aim for. No, make sure the code is as readable as possible to you, since you are the one who is the most likely to come back to this code later and try to read it.

Question 50

In Python, there is no difference. And there is no preference.

Question 51

Syntactically they’re the same statement. I would be quick to state that 'ham' not in 'spam and eggs' conveys clearer intent, but I’ve seen code and scenarios in which not 'ham' in 'spam and eggs' conveys a clearer meaning than the other.

Question 52

The is operator does not match the values of the variables, but the instances themselves.

What does it really mean?

I declared two variables named x and y assigning the same values in both variables, but it returns false when I use the is operator.

I need a clarification. Here is my code.

x = [1, 2, 3]
y = [1, 2, 3]

print(x is y)  # It prints false!

Question 53

You misunderstood what the is operator tests. It tests if two variables point the same object, not if two variables have the same value.

From the documentation for the is operator:

The operators is and is not test for object identity: x is y is true if and only if x and y are the same object.

Use the == operator instead:

print(x == y)

This prints True. x and y are two separate lists:

x[0] = 4
print(y)  # prints [1, 2, 3]
print(x == y)   # prints False

If you use the id() function you’ll see that x and y have different identifiers:

>>> id(x)
4401064560
>>> id(y)
4401098192

but if you were to assign y to x then both point to the same object:

>>> x = y
>>> id(x)
4401064560
>>> id(y)
4401064560
>>> x is y
True

and is shows both are the same object, it returns True.

Remember that in Python, names are just labels referencing values; you can have multiple names point to the same object. is tells you if two names point to one and the same object. == tells you if two names refer to objects that have the same value.

Question 54

Another duplicate was asking why two equal strings are generally not identical, which isn’t really answered here:

>>> x = 'a' 
>>> x += 'bc'
>>> y = 'abc'
>>> x == y
True
>>> x is y
False

So, why aren’t they the same string? Especially given this:

>>> z = 'abc'
>>> w = 'abc'
>>> z is w
True

Let’s put off the second part for a bit. How could the first one be true?

The interpreter would have to have an “interning table”, a table mapping string values to string objects, so every time you try to create a new string with the contents 'abc', you get back the same object. Wikipedia has a more detailed discussion on how interning works.

And Python has a string interning table; you can manually intern strings with the sys.intern method.

In fact, Python is allowed to automatically intern any immutable types, but not required to do so. Different implementations will intern different values.

CPython (the implementation you’re using if you don’t know which implementation you’re using) auto-interns small integers and some special singletons like False, but not strings (or large integers, or small tuples, or anything else). You can see this pretty easily:

>>> a = 0
>>> a += 1
>>> b = 1
>>> a is b
True
>>> a = False
>>> a = not a
>>> b = True
a is b
True
>>> a = 1000
>>> a += 1
>>> b = 1001
>>> a is b
False

OK, but why were z and w identical?

That’s not the interpreter automatically interning, that’s the compiler folding values.

If the same compile-time string appears twice in the same module (what exactly this means is hard to define—it’s not the same thing as a string literal, because r'abc', 'abc', and 'a' 'b' 'c' are all different literals but the same string—but easy to understand intuitively), the compiler will only create one instance of the string, with two references.

In fact, the compiler can go even further: 'ab' + 'c' can be converted to 'abc' by the optimizer, in which case it can be folded together with an 'abc' constant in the same module.

Again, this is something Python is allowed but not required to do. But in this case, CPython always folds small strings (and also, e.g., small tuples). (Although the interactive interpreter’s statement-by-statement compiler doesn’t run the same optimization as the module-at-a-time compiler, so you won’t see exactly the same results interactively.)

So, what should you do about this as a programmer?

Well… nothing. You almost never have any reason to care if two immutable values are identical. If you want to know when you can use a is b instead of a == b, you’re asking the wrong question. Just always use a == b except in two cases:

For more readable comparisons to the singleton values like x is None.
For mutable values, when you need to know whether mutating x will affect the y.

Question 55

is only returns true if they’re actually the same object. If they were the same, a change to one would also show up in the other. Here’s an example of the difference.

>>> x = [1, 2, 3]
>>> y = [1, 2, 3]
>>> print x is y
False
>>> z = y
>>> print y is z
True
>>> print x is z
False
>>> y[0] = 5
>>> print z
[5, 2, 3]

Question 56

Prompted by a duplicate question, this analogy might work:

# - Darling, I want some pudding!
# - There is some in the fridge.

pudding_to_eat = fridge_pudding
pudding_to_eat is fridge_pudding
# => True

# - Honey, what's with all the dirty dishes?
# - I wanted to eat pudding so I made some. Sorry about the mess, Darling.
# - But there was already some in the fridge.

pudding_to_eat = make_pudding(ingredients)
pudding_to_eat is fridge_pudding
# => False

Question 57

is and is not are the two identity operators in Python. is operator does not compare the values of the variables, but compares the identities of the variables. Consider this:

>>> a = [1,2,3]
>>> b = [1,2,3]
>>> hex(id(a))
'0x1079b1440'
>>> hex(id(b))
'0x107960878'
>>> a is b
False
>>> a == b
True
>>>

The above example shows you that the identity (can also be the memory address in Cpython) is different for both a and b (even though their values are the same). That is why when you say a is b it returns false due to the mismatch in the identities of both the operands. However when you say a == b, it returns true because the == operation only verifies if both the operands have the same value assigned to them.

Interesting example (for the extra grade):

>>> del a
>>> del b
>>> a = 132
>>> b = 132
>>> hex(id(a))
'0x7faa2b609738'
>>> hex(id(b))
'0x7faa2b609738'
>>> a is b
True
>>> a == b
True
>>>

In the above example, even though a and b are two different variables, a is b returned True. This is because the type of a is int which is an immutable object. So python (I guess to save memory) allocated the same object to b when it was created with the same value. So in this case, the identities of the variables matched and a is b turned out to be True.

This will apply for all immutable objects:

>>> del a
>>> del b
>>> a = "asd"
>>> b = "asd"
>>> hex(id(a))
'0x1079b05a8'
>>> hex(id(b))
'0x1079b05a8'
>>> a is b
True
>>> a == b
True
>>>

Hope that helps.

Question 58

x is y is same as id(x) == id(y), comparing identity of objects.

As @tomasz-kurgan pointed out in the comment below is operator behaves unusually with certain objects.

E.g.

>>> class A(object):
...   def foo(self):
...     pass
... 
>>> a = A()
>>> a.foo is a.foo
False
>>> id(a.foo) == id(a.foo)
True

Ref;
https://docs.python.org/2/reference/expressions.html#is-not
https://docs.python.org/2/reference/expressions.html#id24

Question 59

As you can check here to a small integers. Numbers above 257 are not an small ints, so it is calculated as a different object.

It is better to use == instead in this case.

Further information is here: http://docs.python.org/2/c-api/int.html

Question 60

X points to an array, Y points to a different array. Those arrays are identical, but the is operator will look at those pointers, which are not identical.

Question 61

It compares object identity, that is, whether the variables refer to the same object in memory. It’s like the == in Java or C (when comparing pointers).

Question 62

A simple example with fruits

fruitlist = [" apple ", " banana ", " cherry ", " durian "]
newfruitlist = fruitlist
verynewfruitlist = fruitlist [:]
print ( fruitlist is newfruitlist )
print ( fruitlist is verynewfruitlist )
print ( newfruitlist is verynewfruitlist )

Output:

True
False
False

If you try

fruitlist = [" apple ", " banana ", " cherry ", " durian "]
newfruitlist = fruitlist
verynewfruitlist = fruitlist [:]
print ( fruitlist == newfruitlist )
print ( fruitlist == verynewfruitlist )
print ( newfruitlist == verynewfruitlist )

The output is different:

True
True
True

That’s because the == operator compares just the content of the variable. To compare the identities of 2 variable use the is operator

To print the identification number:

print ( id( variable ) )

Question 63

The is operator is nothing but an English version of ==. Because the IDs of the two lists are different so the answer is false. You can try:

a=[1,2,3]
b=a
print(b is a )#True

*Because the IDs of both the list would be same

Question 64

Consider this code:

x = 1        # 0001
x << 2       # Shift left 2 bits: 0100
# Result: 4

x | 2        # Bitwise OR: 0011
# Result: 3

x & 1        # Bitwise AND: 0001
# Result: 1

I can understand the arithmetic operators in Python (and other languages), but I never understood ‘bitwise’ operators quite well. In the above example (from a Python book), I understand the left-shift but not the other two.

Also, what are bitwise operators actually used for? I’d appreciate some examples.

问题：Python中的’@ =’符号是什么？

回答 0

回答 1

矩阵乘法与按元素乘法

在Numpy中使用

Matrix Multiplication vs Element-wise Multiplication

Usage in Numpy

回答 2

问题：+ =在python中到底是做什么的？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

回答 11

回答 12

回答 13

问题：函数调用中的星号

回答 0

回答 1

回答 2

问题：Python中的插入符（^）有什么作用？

回答 0

回答 1

回答 2

回答 3

回答 4

问题：Python，我是否应该基于__eq__实现__ne __（）运算符？

回答 0

回答 1

简短的回答：不要实现它，但是如果必须的话，请使用==，而不是__eq__

长答案

该NotImplemented对象

期望

演示__ne__基于以下内容的意外行为__eq__：

预期行为：

意外行为：

不要__ne__在Python 2中跳过

Python 3源码

但是默认__ne__使用__eq__？

性能

对一些批评家的回应

对称重要吗？

结论

Short Answer: Don’t implement it, but if you must, use ==, not __eq__

Long Answer

The NotImplemented object

Expectations

Demonstration of unexpected behavior when defining __ne__ based on __eq__:

Expected Behavior:

Unexpected Behavior:

Don’t skip __ne__ in Python 2

Python 3 source

But the default __ne__ uses __eq__?

Performance

Response to some critics

Is Symmetry Important?

Conclusion

回答 2

为什么需要这个

Python重载规则

这如何适用于比较器过载

问题所在 not self.__eq__(other)

问题所在 not self == other

Why this is needed

Python overloading rules

How this applies to overloading comparators

The problem with not self.__eq__(other)

The problem with not self == other

回答 3

简短答案：是（但请阅读文档以正确完成操作）

比较运算符的实现

比较方法的默认实现

错误的实现 __ne__

问题：Python，我是否应该基于eq实现ne （）运算符？

简短的回答：不要实现它，但是如果必须的话，请使用`==`，而不是`eq`

该`NotImplemented`对象

演示`ne`基于以下内容的意外行为`eq`：

不要`ne`在Python 2中跳过

但是默认`ne`使用`eq`？

Short Answer: Don’t implement it, but if you must, use `==`, not `eq`

The `NotImplemented` object

Demonstration of unexpected behavior when defining `ne` based on `eq`:

Don’t skip `ne` in Python 2

But the default `ne` uses `eq`?

问题所在 `not self.eq(other)`

问题所在 `not self == other`

The problem with `not self.eq(other)`

The problem with `not self == other`

错误的实现 `ne`

`not self == other` 实施

Correct `ne` implementation

Incorrect `ne` implementation #1

Incorrect `ne` implementation #2

`and`

`or`

如何`and`运作

如何`or`运作

`and`

`or`

How `and` Works

How `or` Works

`fn() == fn(3) == fn(4, 4)`

什么`fn`意思

`fn() == fn(3) == fn(4, 4)`

What does `fn` mean?