Multiprocessing a for loop?
PythonMultiprocessingPython Problem Overview
I have an array (called data_inputs
) containing the names of hundreds of astronomy images files. These images are then manipulated. My code works and takes a few seconds to process each image. However, it can only do one image at a time because I'm running the array through a for
loop:
for name in data_inputs:
sci=fits.open(name+'.fits')
#image is manipulated
There is no reason why I have to modify an image before any other, so is it possible to utilise all 4 cores on my machine with each core running through the for loop on a different image?
I've read about the multiprocessing
module but I'm unsure how to implement it in my case.
I'm keen to get multiprocessing
to work because eventually I'll have to run this on 10,000+ images.
Python Solutions
Solution 1 - Python
You can simply use multiprocessing.Pool
:
from multiprocessing import Pool
def process_image(name):
sci=fits.open('{}.fits'.format(name))
<process>
if __name__ == '__main__':
pool = Pool() # Create a multiprocessing Pool
pool.map(process_image, data_inputs) # process data_inputs iterable with pool
Solution 2 - Python
You can use multiprocessing.Pool
:
from multiprocessing import Pool
class Engine(object):
def __init__(self, parameters):
self.parameters = parameters
def __call__(self, filename):
sci = fits.open(filename + '.fits')
manipulated = manipulate_image(sci, self.parameters)
return manipulated
try:
pool = Pool(8) # on 8 processors
engine = Engine(my_parameters)
data_outputs = pool.map(engine, data_inputs)
finally: # To make sure processes are closed in the end, even if errors happen
pool.close()
pool.join()
Solution 3 - Python
Alternatively
with Pool() as pool:
pool.map(fits.open, [name + '.fits' for name in datainput])
Solution 4 - Python
I would suggest to use imap_unordered
with chunksize
if you are only using a for
loop to iterate over an iterable. It will return results from each loop as soon as they are calculated. map
waits for all results to be computed and hence is blocking.
Solution 5 - Python
Actually, there are 3 methods to replace for loop
in python, making your code more simple and effective.
0. for loop
- multiprocessing
- list comprehension
- map + lambda expression
we see this situation
# inp
data = [1,2,3,4,5]
def f(x):
return x**2
# processing (for loop)
result = []
for i in data:
result.append(f(i))
# out
print(result) # [1,4,9,16,25]
Method 0: multiprocessing (what you want~)
# inp
data = [1,2,3,4,5]
def f(x):
return x**2
# processing (multiprocessing)
with Pool(os.cpu_count()-2) as p:
result = p.map(f, data)
# out
print(result) # [1,4,9,16,25]
Method 2: list comprehensions
# inp
data = [1,2,3,4,5]
def f(x):
return x**2
# processing (list comprehension)
result = [f(i) for i in data]
# out
print(result) # [1,4,9,16,25]
Method 3: map + lambda expression
# inp
data = [1,2,3,4,5]
def f(x):
return x**2
# processing (lambda + map)
result = map(lambda x: f(x), data)
# out
print(result) # [1,4,9,16,25]