Is there an efficient way to know how many elements are in an iterator in Python, in general, without iterating through each and counting?

No. It’s not possible.

Example:

import random

def gen(n):
    for i in xrange(n):
        if random.randint(0, 1) == 0:
            yield i

iterator = gen(10)

Length of iterator is unknown until you iterate through it.

This code should work:

>>> iter = (i for i in range(50))
>>> sum(1 for _ in iter)
50

Although it does iterate through each item and count them, it is the fastest way to do so.

It also works for when the iterator has no item:

>>> sum(1 for _ in range(0))
0

Of course, it runs forever for an infinite input, so remember that iterators can be infinite:

>>> sum(1 for _ in itertools.count())
[nothing happens, forever]

Also, be aware that the iterator will be exhausted by doing this, and further attempts to use it will see no elements. That’s an unavoidable consequence of the Python iterator design. If you want to keep the elements, you’ll have to store them in a list or something.

No, any method will require you to resolve every result. You can do

iter_length = len(list(iterable))

but running that on an infinite iterator will of course never return. It also will consume the iterator and it will need to be reset if you want to use the contents.

Telling us what real problem you’re trying to solve might help us find you a better way to accomplish your actual goal.

Edit: Using list() will read the whole iterable into memory at once, which may be undesirable. Another way is to do

sum(1 for _ in iterable)

as another person posted. That will avoid keeping it in memory.

You cannot (except the type of a particular iterator implements some specific methods that make it possible).

Generally, you may count iterator items only by consuming the iterator. One of probably the most efficient ways:

import itertools
from collections import deque

def count_iter_items(iterable):
    """
    Consume an iterable not reading it into memory; return the number of items.
    """
    counter = itertools.count()
    deque(itertools.izip(iterable, counter), maxlen=0)  # (consume at C speed)
    return next(counter)

(For Python 3.x replace itertools.izip with zip).

Kinda. You could check the __length_hint__ method, but be warned that (at least up to Python 3.4, as gsnedders helpfully points out) it’s a undocumented implementation detail (following message in thread), that could very well vanish or summon nasal demons instead.

Otherwise, no. Iterators are just an object that only expose the next() method. You can call it as many times as required and they may or may not eventually raise StopIteration. Luckily, this behaviour is most of the time transparent to the coder. :)

I like the cardinality package for this, it is very lightweight and tries to use the fastest possible implementation available depending on the iterable.

Usage:

>>> import cardinality
>>> cardinality.count([1, 2, 3])
3
>>> cardinality.count(i for i in range(500))
500
>>> def gen():
...     yield 'hello'
...     yield 'world'
>>> cardinality.count(gen())
2

The actual count() implementation is as follows:

def count(iterable):
    if hasattr(iterable, '__len__'):
        return len(iterable)

    d = collections.deque(enumerate(iterable, 1), maxlen=1)
    return d[0][0] if d else 0

So, for those who would like to know the summary of that discussion. The final top scores for counting a 50 million-lengthed generator expression using:

• len(list(gen)),
• len([_ for _ in gen]),
• sum(1 for _ in gen),
• ilen(gen) (from more_itertool),
• reduce(lambda c, i: c + 1, gen, 0),

sorted by performance of execution (including memory consumption), will make you surprised:

“`

1: test_list.py:8: 0.492 KiB

gen = (i for i in data*1000); t0 = monotonic(); len(list(gen))

(‘list, sec’, 1.9684218849870376)

2: test_list_compr.py:8: 0.867 KiB

gen = (i for i in data*1000); t0 = monotonic(); len([i for i in gen])

(‘list_compr, sec’, 2.5885991149989422)

3: test_sum.py:8: 0.859 KiB

gen = (i for i in data*1000); t0 = monotonic(); sum(1 for i in gen); t1 = monotonic()

(‘sum, sec’, 3.441088170016883)

4: more_itertools/more.py:413: 1.266 KiB

d = deque(enumerate(iterable, 1), maxlen=1)

test_ilen.py:10: 0.875 KiB
gen = (i for i in data*1000); t0 = monotonic(); ilen(gen)

(‘ilen, sec’, 9.812256851990242)

5: test_reduce.py:8: 0.859 KiB

gen = (i for i in data*1000); t0 = monotonic(); reduce(lambda counter, i: counter + 1, gen, 0)

(‘reduce, sec’, 13.436614598002052) “`

So, len(list(gen)) is the most frequent and less memory consumable

An iterator is just an object which has a pointer to the next object to be read by some kind of buffer or stream, it’s like a LinkedList where you don’t know how many things you have until you iterate through them. Iterators are meant to be efficient because all they do is tell you what is next by references instead of using indexing (but as you saw you lose the ability to see how many entries are next).

Regarding your original question, the answer is still that there is no way in general to know the length of an iterator in Python.

Given that you question is motivated by an application of the pysam library, I can give a more specific answer: I’m a contributer to PySAM and the definitive answer is that SAM/BAM files do not provide an exact count of aligned reads. Nor is this information easily available from a BAM index file. The best one can do is to estimate the approximate number of alignments by using the location of the file pointer after reading a number of alignments and extrapolating based on the total size of the file. This is enough to implement a progress bar, but not a method of counting alignments in constant time.

A quick benchmark:

import collections
import itertools

def count_iter_items(iterable):
    counter = itertools.count()
    collections.deque(itertools.izip(iterable, counter), maxlen=0)
    return next(counter)

def count_lencheck(iterable):
    if hasattr(iterable, '__len__'):
        return len(iterable)

    d = collections.deque(enumerate(iterable, 1), maxlen=1)
    return d[0][0] if d else 0

def count_sum(iterable):           
    return sum(1 for _ in iterable)

iter = lambda y: (x for x in xrange(y))

%timeit count_iter_items(iter(1000))
%timeit count_lencheck(iter(1000))
%timeit count_sum(iter(1000))

The results:

10000 loops, best of 3: 37.2 µs per loop
10000 loops, best of 3: 47.6 µs per loop
10000 loops, best of 3: 61 µs per loop

I.e. the simple count_iter_items is the way to go.

Adjusting this for python3:

61.9 µs ± 275 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
74.4 µs ± 190 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
82.6 µs ± 164 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

There are two ways to get the length of “something” on a computer.

The first way is to store a count – this requires anything that touches the file/data to modify it (or a class that only exposes interfaces — but it boils down to the same thing).

The other way is to iterate over it and count how big it is.

It’s common practice to put this type of information in the file header, and for pysam to give you access to this. I don’t know the format, but have you checked the API?

As others have said, you can’t know the length from the iterator.

This is against the very definition of an iterator, which is a pointer to an object, plus information about how to get to the next object.

An iterator does not know how many more times it will be able to iterate until terminating. This could be infinite, so infinity might be your answer.

Although it’s not possible in general to do what’s been asked, it’s still often useful to have a count of how many items were iterated over after having iterated over them. For that, you can use jaraco.itertools.Counter or similar. Here’s an example using Python 3 and rwt to load the package.

$ rwt -q jaraco.itertools -- -q
>>> import jaraco.itertools
>>> items = jaraco.itertools.Counter(range(100))
>>> _ = list(counted)
>>> items.count
100
>>> import random
>>> def gen(n):
...     for i in range(n):
...         if random.randint(0, 1) == 0:
...             yield i
... 
>>> items = jaraco.itertools.Counter(gen(100))
>>> _ = list(counted)
>>> items.count
48

def count_iter(iter):
    sum = 0
    for _ in iter: sum += 1
    return sum

Presumably, you want count the number of items without iterating through, so that the iterator is not exhausted, and you use it again later. This is possible with copy or deepcopy

import copy

def get_iter_len(iterator):
    return sum(1 for _ in copy.copy(iterator))

###############################################

iterator = range(0, 10)
print(get_iter_len(iterator))

if len(tuple(iterator)) > 1:
    print("Finding the length did not exhaust the iterator!")
else:
    print("oh no! it's all gone")

The output is “Finding the length did not exhaust the iterator!“

Optionally (and unadvisedly), you can shadow the built-in len function as follows:

import copy

def len(obj, *, len=len):
    try:
        if hasattr(obj, "__len__"):
            r = len(obj)
        elif hasattr(obj, "__next__"):
            r = sum(1 for _ in copy.copy(obj))
        else:
            r = len(obj)
    finally:
        pass
    return r

s = [1,2,3,4,5,6,7,8,9]
n = 3

zip(*[iter(s)]*n) # returns [(1,2,3),(4,5,6),(7,8,9)]

How does zip(*[iter(s)]*n) work? What would it look like if it was written with more verbose code?

iter() is an iterator over a sequence. [x] * n produces a list containing n quantity of x, i.e. a list of length n, where each element is x. *arg unpacks a sequence into arguments for a function call. Therefore you’re passing the same iterator 3 times to zip(), and it pulls an item from the iterator each time.

x = iter([1,2,3,4,5,6,7,8,9])
print zip(x, x, x)

The other great answers and comments explain well the roles of argument unpacking and zip().

As Ignacio and ujukatzel say, you pass to zip() three references to the same iterator and zip() makes 3-tuples of the integers—in order—from each reference to the iterator:

1,2,3,4,5,6,7,8,9  1,2,3,4,5,6,7,8,9  1,2,3,4,5,6,7,8,9
^                    ^                    ^            
      ^                    ^                    ^
            ^                    ^                    ^

And since you ask for a more verbose code sample:

chunk_size = 3
L = [1,2,3,4,5,6,7,8,9]

# iterate over L in steps of 3
for start in range(0,len(L),chunk_size): # xrange() in 2.x; range() in 3.x
    end = start + chunk_size
    print L[start:end] # three-item chunks

Following the values of start and end:

[0:3) #[1,2,3]
[3:6) #[4,5,6]
[6:9) #[7,8,9]

FWIW, you can get the same result with map() with an initial argument of None:

>>> map(None,*[iter(s)]*3)
[(1, 2, 3), (4, 5, 6), (7, 8, 9)]

For more on zip() and map(): http://muffinresearch.co.uk/archives/2007/10/16/python-transposing-lists-with-map-and-zip/

I think one thing that’s missed in all the answers (probably obvious to those familiar with iterators) but not so obvious to others is –

Since we have the same iterator, it gets consumed and the remaining elements are used by the zip. So if we simply used the list and not the iter eg.

l = range(9)
zip(*([l]*3)) # note: not an iter here, the lists are not emptied as we iterate 
# output 
[(0, 0, 0), (1, 1, 1), (2, 2, 2), (3, 3, 3), (4, 4, 4), (5, 5, 5), (6, 6, 6), (7, 7, 7), (8, 8, 8)]

Using iterator, pops the values and only keeps remaining available, so for zip once 0 is consumed 1 is available and then 2 and so on. A very subtle thing, but quite clever!!!

iter(s) returns an iterator for s.

[iter(s)]*n makes a list of n times the same iterator for s.

So, when doing zip(*[iter(s)]*n), it extracts an item from all the three iterators from the list in order. Since all the iterators are the same object, it just groups the list in chunks of n.

One word of advice for using zip this way. It will truncate your list if it’s length is not evenly divisible. To work around this you could either use itertools.izip_longest if you can accept fill values. Or you could use something like this:

def n_split(iterable, n):
    num_extra = len(iterable) % n
    zipped = zip(*[iter(iterable)] * n)
    return zipped if not num_extra else zipped + [iterable[-num_extra:], ]

Usage:

for ints in n_split(range(1,12), 3):
    print ', '.join([str(i) for i in ints])

Prints:

1, 2, 3
4, 5, 6
7, 8, 9
10, 11

It is probably easier to see what is happening in python interpreter or ipython with n = 2:

In [35]: [iter("ABCDEFGH")]*2
Out[35]: [<iterator at 0x6be4128>, <iterator at 0x6be4128>]

So, we have a list of two iterators which are pointing to the same iterator object. Remember that iter on a object returns an iterator object and in this scenario, it is the same iterator twice due to the *2 python syntactic sugar. Iterators also run only once.

Further, zip takes any number of iterables (sequences are iterables) and creates tuple from i’th element of each of the input sequences. Since both iterators are identical in our case, zip moves the same iterator twice for each 2-element tuple of output.

In [41]: help(zip)
Help on built-in function zip in module __builtin__:

zip(...)
    zip(seq1 [, seq2 [...]]) -> [(seq1[0], seq2[0] ...), (...)]

    Return a list of tuples, where each tuple contains the i-th element
    from each of the argument sequences.  The returned list is truncated
    in length to the length of the shortest argument sequence.

The unpacking (*) operator ensures that the iterators run to exhaustion which in this case is until there is not enough input to create a 2-element tuple.

This can be extended to any value of n and zip(*[iter(s)]*n) works as described.

Currently I’m doing this:

try:
    something = iterator.next()
    # ...
except StopIteration:
    # ...

But I would like an expression that I can place inside a simple if statement. Is there anything built-in which would make this code look less clumsy?

any() returns False if an iterable is empty, but it will potentially iterate over all the items if it’s not. I only need it to check the first item.

Someone asks what I’m trying to do. I have written a function which executes an SQL query and yields its results. Sometimes when I call this function I just want to know if the query returned anything and make a decision based on that.

any won’t go beyond the first element if it’s True. In case the iterator yields something false-ish you can write any(True for _ in iterator).

In Python 2.6+, if name sentinel is bound to a value which the iterator can’t possibly yield,

if next(iterator, sentinel) is sentinel:
    print('iterator was empty')

If you have no idea of what the iterator might possibly yield, make your own sentinel (e.g. at the top of your module) with

sentinel = object()

Otherwise, you could use, in the sentinel role, any value which you “know” (based on application considerations) that the iterator can’t possibly yield.

This isn’t really cleaner, but it shows a way to package it in a function losslessly:

def has_elements(iter):
  from itertools import tee
  iter, any_check = tee(iter)
  try:
    any_check.next()
    return True, iter
  except StopIteration:
    return False, iter

has_el, iter = has_elements(iter)
if has_el:
  # not empty

This isn’t really pythonic, and for particular cases, there are probably better (but less general) solutions, like the next default.

first = next(iter, None)
if first:
  # Do something

This isn’t general because None can be a valid element in many iterables.

you can use:

if zip([None], iterator):
    # ...
else:
    # ...

but it’s a bit nonexplanatory for the code reader

The best way to do that is with a peekable from more_itertools.

from more_itertools import peekable
iterator = peekable(iterator)
if iterator:
    # Iterator is non-empty.
else:
    # Iterator is empty.

Just beware if you kept refs to the old iterator, that iterator will get advanced. You have to use the new peekable iterator from then on. Really, though, peekable expects to be the only bit of code modifying that old iterator, so you shouldn’t be keeping refs to the old iterator lying around anyway.

What about:

In [1]: i=iter([])

In [2]: bool(next(i,False))
Out[2]: False

In [3]: i=iter([1])

In [4]: bool(next(i,False))
Out[4]: True

__length_hint__ estimates the length of list(it) – it’s private method, though:

x = iter( (1, 2, 3) )
help(x.__length_hint__)
      1 Help on built-in function __length_hint__:
      2 
      3 __length_hint__(...)
      4     Private method returning an estimate of len(list(it)).

This is an overkill iterator wrapper that generally allows to check whether there’s a next item (via conversion to boolean). Of course pretty inefficient.

class LookaheadIterator ():

    def __init__(self, iterator):
        self.__iterator = iterator
        try:
            self.__next      = next (iterator)
            self.__have_next = True
        except StopIteration:
            self.__have_next = False

    def __iter__(self):
        return self

    def next (self):
        if self.__have_next:
            result = self.__next
            try:
                self.__next      = next (self.__iterator)
                self.__have_next = True
            except StopIteration:
                self.__have_next = False

            return result

        else:
            raise StopIteration

    def __nonzero__(self):
        return self.__have_next

x = LookaheadIterator (iter ([]))
print bool (x)
print list (x)

x = LookaheadIterator (iter ([1, 2, 3]))
print bool (x)
print list (x)

Output:

False
[]
True
[1, 2, 3]

A little late, but… You could turn the iterator into a list and then work with that list:

# Create a list of objects but runs out the iterator.
l = [_ for _ in iterator]

# If the list is not empty then the iterator had elements; else it was empty.
if l :
    pass # Use the elements of the list (i.e. from the iterator)
else :
    pass # Iterator was empty, thus list is empty.

Haven’t Python iterators got a hasNext method?

No, there is no such method. The end of iteration is indicated by an exception. See the documentation.

There’s an alternative to the StopIteration by using next(iterator, default_value).

For exapmle:

>>> a = iter('hi')
>>> print next(a, None)
h
>>> print next(a, None)
i
>>> print next(a, None)
None

So you can detect for None or other pre-specified value for end of the iterator if you don’t want the exception way.

If you really need a has-next functionality (because you’re just faithfully transcribing an algorithm from a reference implementation in Java, say, or because you’re writing a prototype that will need to be easily transcribed to Java when it’s finished), it’s easy to obtain it with a little wrapper class. For example:

class hn_wrapper(object):
  def __init__(self, it):
    self.it = iter(it)
    self._hasnext = None
  def __iter__(self): return self
  def next(self):
    if self._hasnext:
      result = self._thenext
    else:
      result = next(self.it)
    self._hasnext = None
    return result
  def hasnext(self):
    if self._hasnext is None:
      try: self._thenext = next(self.it)
      except StopIteration: self._hasnext = False
      else: self._hasnext = True
    return self._hasnext

now something like

x = hn_wrapper('ciao')
while x.hasnext(): print next(x)

emits

c
i
a
o

as required.

Note that the use of next(sel.it) as a built-in requires Python 2.6 or better; if you’re using an older version of Python, use self.it.next() instead (and similarly for next(x) in the example usage). [[You might reasonably think this note is redundant, since Python 2.6 has been around for over a year now — but more often than not when I use Python 2.6 features in a response, some commenter or other feels duty-bound to point out that they are 2.6 features, thus I’m trying to forestall such comments for once;-)]]