What is the most "pythonic" way to iterate over a list in chunks?
PythonListLoopsOptimizationChunksPython Problem Overview
I have a Python script which takes as input a list of integers, which I need to work with four integers at a time. Unfortunately, I don't have control of the input, or I'd have it passed in as a list of four-element tuples. Currently, I'm iterating over it this way:
for i in range(0, len(ints), 4):
# dummy op for example code
foo += ints[i] * ints[i + 1] + ints[i + 2] * ints[i + 3]
It looks a lot like "C-think", though, which makes me suspect there's a more pythonic way of dealing with this situation. The list is discarded after iterating, so it needn't be preserved. Perhaps something like this would be better?
while ints:
foo += ints[0] * ints[1] + ints[2] * ints[3]
ints[0:4] = []
Still doesn't quite "feel" right, though. :-/
Related question: How do you split a list into evenly sized chunks in Python?
Python Solutions
Solution 1 - Python
def chunker(seq, size):
return (seq[pos:pos + size] for pos in range(0, len(seq), size))
# (in python 2 use xrange() instead of range() to avoid allocating a list)
Works with any sequence:
text = "I am a very, very helpful text"
for group in chunker(text, 7):
print(repr(group),)
# 'I am a ' 'very, v' 'ery hel' 'pful te' 'xt'
print '|'.join(chunker(text, 10))
# I am a ver|y, very he|lpful text
animals = ['cat', 'dog', 'rabbit', 'duck', 'bird', 'cow', 'gnu', 'fish']
for group in chunker(animals, 3):
print(group)
# ['cat', 'dog', 'rabbit']
# ['duck', 'bird', 'cow']
# ['gnu', 'fish']
Solution 2 - Python
Modified from the Recipes section of Python's itertools
docs:
from itertools import zip_longest
def grouper(iterable, n, fillvalue=None):
args = [iter(iterable)] * n
return zip_longest(*args, fillvalue=fillvalue)
Example
grouper('ABCDEFG', 3, 'x') # --> 'ABC' 'DEF' 'Gxx'
Note: on Python 2 use izip_longest
instead of zip_longest
.
Solution 3 - Python
chunk_size = 4
for i in range(0, len(ints), chunk_size):
chunk = ints[i:i+chunk_size]
# process chunk of size <= chunk_size
Solution 4 - Python
import itertools
def chunks(iterable,size):
it = iter(iterable)
chunk = tuple(itertools.islice(it,size))
while chunk:
yield chunk
chunk = tuple(itertools.islice(it,size))
# though this will throw ValueError if the length of ints
# isn't a multiple of four:
for x1,x2,x3,x4 in chunks(ints,4):
foo += x1 + x2 + x3 + x4
for chunk in chunks(ints,4):
foo += sum(chunk)
Another way:
import itertools
def chunks2(iterable,size,filler=None):
it = itertools.chain(iterable,itertools.repeat(filler,size-1))
chunk = tuple(itertools.islice(it,size))
while len(chunk) == size:
yield chunk
chunk = tuple(itertools.islice(it,size))
# x2, x3 and x4 could get the value 0 if the length is not
# a multiple of 4.
for x1,x2,x3,x4 in chunks2(ints,4,0):
foo += x1 + x2 + x3 + x4
Solution 5 - Python
If you don't mind using an external package you could use iteration_utilities.grouper
from iteration_utilties
1. It supports all iterables (not just sequences):
from iteration_utilities import grouper
seq = list(range(20))
for group in grouper(seq, 4):
print(group)
which prints:
(0, 1, 2, 3)
(4, 5, 6, 7)
(8, 9, 10, 11)
(12, 13, 14, 15)
(16, 17, 18, 19)
In case the length isn't a multiple of the groupsize it also supports filling (the incomplete last group) or truncating (discarding the incomplete last group) the last one:
from iteration_utilities import grouper
seq = list(range(17))
for group in grouper(seq, 4):
print(group)
# (0, 1, 2, 3)
# (4, 5, 6, 7)
# (8, 9, 10, 11)
# (12, 13, 14, 15)
# (16,)
for group in grouper(seq, 4, fillvalue=None):
print(group)
# (0, 1, 2, 3)
# (4, 5, 6, 7)
# (8, 9, 10, 11)
# (12, 13, 14, 15)
# (16, None, None, None)
for group in grouper(seq, 4, truncate=True):
print(group)
# (0, 1, 2, 3)
# (4, 5, 6, 7)
# (8, 9, 10, 11)
# (12, 13, 14, 15)
Benchmarks
I also decided to compare the run-time of a few of the mentioned approaches. It's a log-log plot grouping into groups of "10" elements based on a list of varying size. For qualitative results: Lower means faster:
At least in this benchmark the iteration_utilities.grouper
performs best. Followed by the approach of Craz.
The benchmark was created with simple_benchmark
1. The code used to run this benchmark was:
import iteration_utilities
import itertools
from itertools import zip_longest
def consume_all(it):
return iteration_utilities.consume(it, None)
import simple_benchmark
b = simple_benchmark.BenchmarkBuilder()
@b.add_function()
def grouper(l, n):
return consume_all(iteration_utilities.grouper(l, n))
def Craz_inner(iterable, n, fillvalue=None):
args = [iter(iterable)] * n
return zip_longest(*args, fillvalue=fillvalue)
@b.add_function()
def Craz(iterable, n, fillvalue=None):
return consume_all(Craz_inner(iterable, n, fillvalue))
def nosklo_inner(seq, size):
return (seq[pos:pos + size] for pos in range(0, len(seq), size))
@b.add_function()
def nosklo(seq, size):
return consume_all(nosklo_inner(seq, size))
def SLott_inner(ints, chunk_size):
for i in range(0, len(ints), chunk_size):
yield ints[i:i+chunk_size]
@b.add_function()
def SLott(ints, chunk_size):
return consume_all(SLott_inner(ints, chunk_size))
def MarkusJarderot1_inner(iterable,size):
it = iter(iterable)
chunk = tuple(itertools.islice(it,size))
while chunk:
yield chunk
chunk = tuple(itertools.islice(it,size))
@b.add_function()
def MarkusJarderot1(iterable,size):
return consume_all(MarkusJarderot1_inner(iterable,size))
def MarkusJarderot2_inner(iterable,size,filler=None):
it = itertools.chain(iterable,itertools.repeat(filler,size-1))
chunk = tuple(itertools.islice(it,size))
while len(chunk) == size:
yield chunk
chunk = tuple(itertools.islice(it,size))
@b.add_function()
def MarkusJarderot2(iterable,size):
return consume_all(MarkusJarderot2_inner(iterable,size))
@b.add_arguments()
def argument_provider():
for exp in range(2, 20):
size = 2**exp
yield size, simple_benchmark.MultiArgument([[0] * size, 10])
r = b.run()
1 Disclaimer: I'm the author of the libraries iteration_utilities
and simple_benchmark
.
Solution 6 - Python
The ideal solution for this problem works with iterators (not just sequences). It should also be fast.
This is the solution provided by the documentation for itertools:
def grouper(n, iterable, fillvalue=None):
#"grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
args = [iter(iterable)] * n
return itertools.izip_longest(fillvalue=fillvalue, *args)
Using ipython's %timeit
on my mac book air, I get 47.5 us per loop.
However, this really doesn't work for me since the results are padded to be even sized groups. A solution without the padding is slightly more complicated. The most naive solution might be:
def grouper(size, iterable):
i = iter(iterable)
while True:
out = []
try:
for _ in range(size):
out.append(i.next())
except StopIteration:
yield out
break
yield out
Simple, but pretty slow: 693 us per loop
The best solution I could come up with uses islice
for the inner loop:
def grouper(size, iterable):
it = iter(iterable)
while True:
group = tuple(itertools.islice(it, None, size))
if not group:
break
yield group
With the same dataset, I get 305 us per loop.
Unable to get a pure solution any faster than that, I provide the following solution with an important caveat: If your input data has instances of filldata
in it, you could get wrong answer.
def grouper(n, iterable, fillvalue=None):
#"grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
args = [iter(iterable)] * n
# itertools.zip_longest on Python 3
for x in itertools.izip_longest(*args, fillvalue=fillvalue):
if x[-1] is fillvalue:
yield tuple(v for v in x if v is not fillvalue)
else:
yield x
I really don't like this answer, but it is significantly faster. 124 us per loop
Solution 7 - Python
I needed a solution that would also work with sets and generators. I couldn't come up with anything very short and pretty, but it's quite readable at least.
def chunker(seq, size):
res = []
for el in seq:
res.append(el)
if len(res) == size:
yield res
res = []
if res:
yield res
List:
>>> list(chunker([i for i in range(10)], 3))
[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]
Set:
>>> list(chunker(set([i for i in range(10)]), 3))
[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]
Generator:
>>> list(chunker((i for i in range(10)), 3))
[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]
Solution 8 - Python
With Python 3.8 you can use the walrus operator and itertools.islice
.
from itertools import islice
list_ = [i for i in range(10, 100)]
def chunker(it, size):
iterator = iter(it)
while chunk := list(islice(iterator, size)):
print(chunk)
In [2]: chunker(list_, 10)
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
[20, 21, 22, 23, 24, 25, 26, 27, 28, 29]
[30, 31, 32, 33, 34, 35, 36, 37, 38, 39]
[40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
[50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
[60, 61, 62, 63, 64, 65, 66, 67, 68, 69]
[70, 71, 72, 73, 74, 75, 76, 77, 78, 79]
[80, 81, 82, 83, 84, 85, 86, 87, 88, 89]
[90, 91, 92, 93, 94, 95, 96, 97, 98, 99]
Solution 9 - Python
The more-itertools package has chunked method which does exactly that:
import more_itertools
for s in more_itertools.chunked(range(9), 4):
print(s)
Prints
[0, 1, 2, 3]
[4, 5, 6, 7]
[8]
chunked
returns the items in a list. If you'd prefer iterables, use ichunked.
Solution 10 - Python
from itertools import izip_longest
def chunker(iterable, chunksize, filler):
return izip_longest(*[iter(iterable)]*chunksize, fillvalue=filler)
Solution 11 - Python
Similar to other proposals, but not exactly identical, I like doing it this way, because it's simple and easy to read:
it = iter([1, 2, 3, 4, 5, 6, 7, 8, 9])
for chunk in zip(it, it, it, it):
print chunk
>>> (1, 2, 3, 4)
>>> (5, 6, 7, 8)
This way you won't get the last partial chunk. If you want to get (9, None, None, None)
as last chunk, just use izip_longest
from itertools
.
Solution 12 - Python
Since nobody's mentioned it yet here's a zip()
solution:
>>> def chunker(iterable, chunksize):
... return zip(*[iter(iterable)]*chunksize)
It works only if your sequence's length is always divisible by the chunk size or you don't care about a trailing chunk if it isn't.
Example:
>>> s = '1234567890'
>>> chunker(s, 3)
[('1', '2', '3'), ('4', '5', '6'), ('7', '8', '9')]
>>> chunker(s, 4)
[('1', '2', '3', '4'), ('5', '6', '7', '8')]
>>> chunker(s, 5)
[('1', '2', '3', '4', '5'), ('6', '7', '8', '9', '0')]
Or using itertools.izip to return an iterator instead of a list:
>>> from itertools import izip
>>> def chunker(iterable, chunksize):
... return izip(*[iter(iterable)]*chunksize)
Padding can be fixed using @ΤΖΩΤΖΙΟΥ's answer:
>>> from itertools import chain, izip, repeat
>>> def chunker(iterable, chunksize, fillvalue=None):
... it = chain(iterable, repeat(fillvalue, chunksize-1))
... args = [it] * chunksize
... return izip(*args)
Solution 13 - Python
Another approach would be to use the two-argument form of iter
:
from itertools import islice
def group(it, size):
it = iter(it)
return iter(lambda: tuple(islice(it, size)), ())
This can be adapted easily to use padding (this is similar to Markus Jarderot’s answer):
from itertools import islice, chain, repeat
def group_pad(it, size, pad=None):
it = chain(iter(it), repeat(pad))
return iter(lambda: tuple(islice(it, size)), (pad,) * size)
These can even be combined for optional padding:
_no_pad = object()
def group(it, size, pad=_no_pad):
if pad == _no_pad:
it = iter(it)
sentinel = ()
else:
it = chain(iter(it), repeat(pad))
sentinel = (pad,) * size
return iter(lambda: tuple(islice(it, size)), sentinel)
Solution 14 - Python
Using map() instead of zip() fixes the padding issue in J.F. Sebastian's answer:
>>> def chunker(iterable, chunksize):
... return map(None,*[iter(iterable)]*chunksize)
Example:
>>> s = '1234567890'
>>> chunker(s, 3)
[('1', '2', '3'), ('4', '5', '6'), ('7', '8', '9'), ('0', None, None)]
>>> chunker(s, 4)
[('1', '2', '3', '4'), ('5', '6', '7', '8'), ('9', '0', None, None)]
>>> chunker(s, 5)
[('1', '2', '3', '4', '5'), ('6', '7', '8', '9', '0')]
Solution 15 - Python
If the list is large, the highest-performing way to do this will be to use a generator:
def get_chunk(iterable, chunk_size):
result = []
for item in iterable:
result.append(item)
if len(result) == chunk_size:
yield tuple(result)
result = []
if len(result) > 0:
yield tuple(result)
for x in get_chunk([1,2,3,4,5,6,7,8,9,10], 3):
print x
(1, 2, 3)
(4, 5, 6)
(7, 8, 9)
(10,)
Solution 16 - Python
Using little functions and things really doesn't appeal to me; I prefer to just use slices:
data = [...]
chunk_size = 10000 # or whatever
chunks = [data[i:i+chunk_size] for i in xrange(0,len(data),chunk_size)]
for chunk in chunks:
...
Solution 17 - Python
To avoid all conversions to a list import itertools
and:
>>> for k, g in itertools.groupby(xrange(35), lambda x: x/10):
... list(g)
Produces:
...
0 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
1 [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
2 [20, 21, 22, 23, 24, 25, 26, 27, 28, 29]
3 [30, 31, 32, 33, 34]
>>>
I checked groupby
and it doesn't convert to list or use len
so I (think) this will delay resolution of each value until it is actually used. Sadly none of the available answers (at this time) seemed to offer this variation.
Obviously if you need to handle each item in turn nest a for loop over g:
for k,g in itertools.groupby(xrange(35), lambda x: x/10):
for i in g:
# do what you need to do with individual items
# now do what you need to do with the whole group
My specific interest in this was the need to consume a generator to submit changes in batches of up to 1000 to the gmail API:
messages = a_generator_which_would_not_be_smart_as_a_list
for idx, batch in groupby(messages, lambda x: x/1000):
batch_request = BatchHttpRequest()
for message in batch:
batch_request.add(self.service.users().messages().modify(userId='me', id=message['id'], body=msg_labels))
http = httplib2.Http()
self.credentials.authorize(http)
batch_request.execute(http=http)
Solution 18 - Python
One-liner, adhoc solution to iterate over a list x
in chunks of size 4
-
for a, b, c, d in zip(x[0::4], x[1::4], x[2::4], x[3::4]):
... do something with a, b, c and d ...
Solution 19 - Python
With NumPy it's simple:
ints = array([1, 2, 3, 4, 5, 6, 7, 8])
for int1, int2 in ints.reshape(-1, 2):
print(int1, int2)
output:
1 2
3 4
5 6
7 8
Solution 20 - Python
def chunker(iterable, n):
"""Yield iterable in chunk sizes.
>>> chunks = chunker('ABCDEF', n=4)
>>> chunks.next()
['A', 'B', 'C', 'D']
>>> chunks.next()
['E', 'F']
"""
it = iter(iterable)
while True:
chunk = []
for i in range(n):
try:
chunk.append(next(it))
except StopIteration:
yield chunk
raise StopIteration
yield chunk
if __name__ == '__main__':
import doctest
doctest.testmod()
Solution 21 - Python
Unless I misses something, the following simple solution with generator expressions has not been mentioned. It assumes that both the size and the number of chunks are known (which is often the case), and that no padding is required:
def chunks(it, n, m):
"""Make an iterator over m first chunks of size n.
"""
it = iter(it)
# Chunks are presented as tuples.
return (tuple(next(it) for _ in range(n)) for _ in range(m))
Solution 22 - Python
In your second method, I would advance to the next group of 4 by doing this:
ints = ints[4:]
However, I haven't done any performance measurement so I don't know which one might be more efficient.
Having said that, I would usually choose the first method. It's not pretty, but that's often a consequence of interfacing with the outside world.
Solution 23 - Python
Yet another answer, the advantages of which are:
- Easily understandable
- Works on any iterable, not just sequences (some of the above answers will choke on filehandles)
- Does not load the chunk into memory all at once
- Does not make a chunk-long list of references to the same iterator in memory
- No padding of fill values at the end of the list
That being said, I haven't timed it so it might be slower than some of the more clever methods, and some of the advantages may be irrelevant given the use case.
def chunkiter(iterable, size):
def inneriter(first, iterator, size):
yield first
for _ in xrange(size - 1):
yield iterator.next()
it = iter(iterable)
while True:
yield inneriter(it.next(), it, size)
In [2]: i = chunkiter('abcdefgh', 3)
In [3]: for ii in i:
for c in ii:
print c,
print ''
...:
a b c
d e f
g h
Update:
A couple of drawbacks due to the fact the inner and outer loops are pulling values from the same iterator:
- continue doesn't work as expected in the outer loop - it just continues on to the next item rather than skipping a chunk. However, this doesn't seem like a problem as there's nothing to test in the outer loop.
- break doesn't work as expected in the inner loop - control will wind up in the inner loop again with the next item in the iterator. To skip whole chunks, either wrap the inner iterator (ii above) in a tuple, e.g.
for c in tuple(ii)
, or set a flag and exhaust the iterator.
Solution 24 - Python
def group_by(iterable, size):
"""Group an iterable into lists that don't exceed the size given.
>>> group_by([1,2,3,4,5], 2)
[[1, 2], [3, 4], [5]]
"""
sublist = []
for index, item in enumerate(iterable):
if index > 0 and index % size == 0:
yield sublist
sublist = []
sublist.append(item)
if sublist:
yield sublist
Solution 25 - Python
You can use partition or chunks function from funcy library:
from funcy import partition
for a, b, c, d in partition(4, ints):
foo += a * b * c * d
These functions also has iterator versions ipartition
and ichunks
, which will be more efficient in this case.
You can also peek at their implementation.
Solution 26 - Python
About solution gave by J.F. Sebastian
here:
def chunker(iterable, chunksize):
return zip(*[iter(iterable)]*chunksize)
It's clever, but has one disadvantage - always return tuple. How to get string instead?
Of course you can write ''.join(chunker(...))
, but the temporary tuple is constructed anyway.
You can get rid of the temporary tuple by writing own zip
, like this:
class IteratorExhausted(Exception):
pass
def translate_StopIteration(iterable, to=IteratorExhausted):
for i in iterable:
yield i
raise to # StopIteration would get ignored because this is generator,
# but custom exception can leave the generator.
def custom_zip(*iterables, reductor=tuple):
iterators = tuple(map(translate_StopIteration, iterables))
while True:
try:
yield reductor(next(i) for i in iterators)
except IteratorExhausted: # when any of iterators get exhausted.
break
Then
def chunker(data, size, reductor=tuple):
return custom_zip(*[iter(data)]*size, reductor=reductor)
Example usage:
>>> for i in chunker('12345', 2):
... print(repr(i))
...
('1', '2')
('3', '4')
>>> for i in chunker('12345', 2, ''.join):
... print(repr(i))
...
'12'
'34'
Solution 27 - Python
Here is a chunker without imports that supports generators:
def chunks(seq, size):
it = iter(seq)
while True:
ret = tuple(next(it) for _ in range(size))
if len(ret) == size:
yield ret
else:
raise StopIteration()
Example of use:
>>> def foo():
... i = 0
... while True:
... i += 1
... yield i
...
>>> c = chunks(foo(), 3)
>>> c.next()
(1, 2, 3)
>>> c.next()
(4, 5, 6)
>>> list(chunks('abcdefg', 2))
[('a', 'b'), ('c', 'd'), ('e', 'f')]
Solution 28 - Python
I like this approach. It feels simple and not magical and supports all iterable types and doesn't require imports.
def chunk_iter(iterable, chunk_size):
it = iter(iterable)
while True:
chunk = tuple(next(it) for _ in range(chunk_size))
if not chunk:
break
yield chunk
Solution 29 - Python
Quite pythonic here (you may also inline the body of the split_groups
function)
import itertools
def split_groups(iter_in, group_size):
return ((x for _, x in item) for _, item in itertools.groupby(enumerate(iter_in), key=lambda x: x[0] // group_size))
for x, y, z, w in split_groups(range(16), 4):
foo += x * y + z * w
Solution 30 - Python
I never want my chunks padded, so that requirement is essential. I find that the ability to work on any iterable is also requirement. Given that, I decided to extend on the accepted answer, https://stackoverflow.com/a/434411/1074659.
Performance takes a slight hit in this approach if padding is not wanted due to the need to compare and filter the padded values. However, for large chunk sizes, this utility is very performant.
#!/usr/bin/env python3
from itertools import zip_longest
_UNDEFINED = object()
def chunker(iterable, chunksize, fillvalue=_UNDEFINED):
"""
Collect data into chunks and optionally pad it.
Performance worsens as `chunksize` approaches 1.
Inspired by:
https://docs.python.org/3/library/itertools.html#itertools-recipes
"""
args = [iter(iterable)] * chunksize
chunks = zip_longest(*args, fillvalue=fillvalue)
yield from (
filter(lambda val: val is not _UNDEFINED, chunk)
if chunk[-1] is _UNDEFINED
else chunk
for chunk in chunks
) if fillvalue is _UNDEFINED else chunks
Solution 31 - Python
here is my go works on lists,iters and range ... lazily :
def chunker(it,size):
rv = []
for i,el in enumerate(it,1) :
rv.append(el)
if i % size == 0 :
yield rv
rv = []
if rv : yield rv
almost made it one-liner ;(
In [95]: list(chunker(range(9),2) )
Out[95]: [[0, 1], [2, 3], [4, 5], [6, 7], [8]]
In [96]: list(chunker([1,2,3,4,5],2) )
Out[96]: [[1, 2], [3, 4], [5]]
In [97]: list(chunker(iter(range(9)),2) )
Out[97]: [[0, 1], [2, 3], [4, 5], [6, 7], [8]]
In [98]: list(chunker(range(9),25) )
Out[98]: [[0, 1, 2, 3, 4, 5, 6, 7, 8]]
In [99]: list(chunker(range(9),1) )
Out[99]: [[0], [1], [2], [3], [4], [5], [6], [7], [8]]
In [101]: %timeit list(chunker(range(101),2) )
11.3 µs ± 68.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
Solution 32 - Python
At first, I designed it to split strings into substrings to parse string containing hex.
Today I turned it into complex, but still simple generator.
def chunker(iterable, size, reductor, condition):
it = iter(iterable)
def chunk_generator():
return (next(it) for _ in range(size))
chunk = reductor(chunk_generator())
while condition(chunk):
yield chunk
chunk = reductor(chunk_generator())
Arguments:
Obvious ones
iterable
is any iterable / iterator / generator containg / generating / iterating over input data,size
is, of course, size of chunk you want get,
More interesting
-
reductor
is a callable, which receives generator iterating over content of chunk.
I'd expect it to return sequence or string, but I don't demand that.You can pass as this argument for example
list
,tuple
,set
,frozenset
,
or anything fancier. I'd pass this function, returning string
(provided thatiterable
contains / generates / iterates over strings):def concatenate(iterable): return ''.join(iterable)
Note that
reductor
can cause closing generator by raising exception. -
condition
is a callable which receives anything whatreductor
returned.
It decides to approve & yield it (by returning anything evaluating toTrue
),
or to decline it & finish generator's work (by returning anything other or raising exception).When number of elements in
iterable
is not divisible bysize
, whenit
gets exhausted,reductor
will receive generator generating less elements thansize
.
Let's call these elements lasts elements.I invited two functions to pass as this argument:
-
lambda x:x
- the lasts elements will be yielded. -
lambda x: len(x)==<size>
- the lasts elements will be rejected.
replace<size>
using number equal tosize
-
Solution 33 - Python
It is easy to make itertools.groupby
work for you to get an iterable of iterables, without creating any temporary lists:
groupby(iterable, (lambda x,y: (lambda z: x.next()/y))(count(),100))
Don't get put off by the nested lambdas, outer lambda runs just once to put count()
generator and the constant 100
into the scope of the inner lambda.
I use this to send chunks of rows to mysql.
for k,v in groupby(bigdata, (lambda x,y: (lambda z: x.next()/y))(count(),100))):
cursor.executemany(sql, v)
Solution 34 - Python
This answer splits a list of strings, f.ex. to achieve PEP8-line length compliance:
def split(what, target_length=79):
'''splits list of strings into sublists, each
having string length at most 79'''
out = [[]]
while what:
if len("', '".join(out[-1])) + len(what[0]) < target_length:
out[-1].append(what.pop(0))
else:
if not out[-1]: # string longer than target_length
out[-1] = [what.pop(0)]
out.append([])
return out
Use as
>>> split(['deferred_income', 'long_term_incentive', 'restricted_stock_deferred', 'shared_receipt_with_poi', 'loan_advances', 'from_messages', 'other', 'director_fees', 'bonus', 'total_stock_value', 'from_poi_to_this_person', 'from_this_person_to_poi', 'restricted_stock', 'salary', 'total_payments', 'exercised_stock_options'], 75)
[['deferred_income', 'long_term_incentive', 'restricted_stock_deferred'], ['shared_receipt_with_poi', 'loan_advances', 'from_messages', 'other'], ['director_fees', 'bonus', 'total_stock_value', 'from_poi_to_this_person'], ['from_this_person_to_poi', 'restricted_stock', 'salary', 'total_payments'], ['exercised_stock_options']]
Solution 35 - Python
I am hoping that by turning an iterator out of a list i am not simply copying a slice of the list. Generators can be sliced and they will automatically still be a generator, while lists will be sliced into huge chunks of 1000 entries, which is less efficient.
def iter_group(iterable, batch_size:int):
length = len(iterable)
start = batch_size*-1
end = 0
while(end < length):
start += batch_size
end += batch_size
if type(iterable) == list:
yield (iterable[i] for i in range(start,min(length-1,end)))
else:
yield iterable[start:end]
Usage:
items = list(range(1,1251))
for item_group in iter_group(items, 1000):
for item in item_group:
print(item)
Solution 36 - Python
There doesn't seem to be a pretty way to do this. Here is a page that has a number of methods, including:
def split_seq(seq, size):
newseq = []
splitsize = 1.0/size*len(seq)
for i in range(size):
newseq.append(seq[int(round(i*splitsize)):int(round((i+1)*splitsize))])
return newseq
Solution 37 - Python
If the lists are the same size, you can combine them into lists of 4-tuples with zip()
. For example:
# Four lists of four elements each.
l1 = range(0, 4)
l2 = range(4, 8)
l3 = range(8, 12)
l4 = range(12, 16)
for i1, i2, i3, i4 in zip(l1, l2, l3, l4):
...
Here's what the zip()
function produces:
>>> print l1
[0, 1, 2, 3]
>>> print l2
[4, 5, 6, 7]
>>> print l3
[8, 9, 10, 11]
>>> print l4
[12, 13, 14, 15]
>>> print zip(l1, l2, l3, l4)
[(0, 4, 8, 12), (1, 5, 9, 13), (2, 6, 10, 14), (3, 7, 11, 15)]
If the lists are large, and you don't want to combine them into a bigger list, use itertools.izip()
, which produces an iterator, rather than a list.
from itertools import izip
for i1, i2, i3, i4 in izip(l1, l2, l3, l4):
...
Solution 38 - Python
Why not use list comprehension
l = [1 , 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
n = 4
filler = 0
fills = len(l) % n
chunks = ((l + [filler] * fills)[x * n:x * n + n] for x in range(int((len(l) + n - 1)/n)))
print(chunks)
[[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 0]]