Looking at the [documentation for Pool.map](http://docs.python.org/release/2.6.6/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.map) it seems you&#39;re almost correct: the `chunksize` parameter will cause the iterable to be split into pieces of **approximately** that size, and each piece is submitted as a separate task.

So in your example, yes, `map` will take the first 10 (approximately), submit it as a task for a single processor... then the next 10 will be submitted as another task, and so on. Note that it doesn&#39;t mean that this will make the processors alternate every 10 files, it&#39;s quite possible that processor #1 ends up getting 1-10 AND 11-20, and processor #2 gets 21-30 and 31-40.

I need to get the video duration in Python. The video formats that I need to get are [MP4][3], Flash video, [AVI][2], and MOV... I have a shared hosting solution, so I have no [FFmpeg][1] support.

  [1]: http://en.wikipedia.org/wiki/FFmpeg
  [2]: http://en.wikipedia.org/wiki/Audio_Video_Interleave
  [3]: http://en.wikipedia.org/wiki/MPEG-4_Part_14


How to get the duration of a video in Python?

What is the difference (if any) between the two following preprocessor control statements.

    #if

and

    #ifdef

Difference between preprocessor directives #if and #ifdef

If I have a pool object with 2 processors for example:

    p=multiprocessing.Pool(2)

and I want to iterate over a list of files on directory and use the map function

could someone explain what is the chunksize of this function:

    p.map(func, iterable[, chunksize])

If I set the chunksize for example to 10 does that means every 10 files should be processed with one processor?




&quot;chunksize&quot; parameter in multiprocessing.Pool.map

If I have a pool object with 2 processors for example:
<pre><code class="hljs language-ini">p=multiprocessing.Pool(2)
</code></pre>
and I want to iterate over a list of files on directory and use the map function
could someone explain what is the chunksize of this function:
<pre><code class="hljs language-css">p.map(func, iterable[, chunksize])
</code></pre>
If I set the chunksize for example to 10 does that means every 10 files should be processed with one processor?

I&#39;m trying to display a grayscale image using **matplotlib.pyplot.imshow()**.  My problem is that the grayscale image is displayed as a colormap.  I need the grayscale because I want to draw on top of the image with color.

I read in the image and convert to grayscale using **PIL&#39;s Image.open().convert(&quot;L&quot;)**  
&lt;pre&gt;&lt;code&gt;image = Image.open(file).convert(&quot;L&quot;)&lt;/code&gt;&lt;/pre&gt;

Then I convert the image to a matrix so that I can easily do some image processing using  
&lt;pre&gt;&lt;code&gt;matrix = scipy.misc.fromimage(image, 0)&lt;/code&gt;&lt;/pre&gt;

However, when I do  
&lt;pre&gt;&lt;code&gt;figure()  
matplotlib.pyplot.imshow(matrix)  
show()&lt;/code&gt;&lt;/pre&gt; 
it displays the image using a colormap (i.e. it&#39;s not grayscale).

What am I doing wrong here?


Display image as grayscale using matplotlib

I am relatively new to Python. I am looking to create a &quot;settings&quot; module where various application-specific constants will be stored.

Here is how I am wanting to set up my code:

settings.py

    CONSTANT = &#39;value&#39;

script.py
    
    import settings

    def func():
        var = CONSTANT
        # do some more coding
        return var

I am getting a Python error stating: 

&gt; `global name &#39;CONSTANT&#39; is not defined.`

I have noticed on Django&#39;s source code their `settings.py` file has constants named just like I do. I am confused on how they can be imported to a script and referenced through the application.

**EDIT**

Thank you for all your answers! I tried the following:

    import settings

    print settings.CONSTANT

I get the same error

&gt; `ImportError: cannot import name CONSTANT`

Create constants using a &quot;settings&quot; module?

I have seen few py scripts which use this at the top of the script. In what cases one should use it?
      
    import sys
    reload(sys)
    sys.setdefaultencoding(&quot;utf-8&quot;)



Why should we NOT use sys.setdefaultencoding(&quot;utf-8&quot;) in a py script?

Suppose I have the following code in a Python unit test:

    aw = aps.Request(&quot;nv1&quot;)
    aw2 = aps.Request(&quot;nv2&quot;, aw)

Is there an easy way to assert that a particular method (in my case `aw.Clear()`) was called during the second line of the test? e.g. is there something like this:

    #pseudocode:
    assertMethodIsCalled(aw.Clear, lambda: aps.Request(&quot;nv2&quot;, aw))

Assert that a method was called in a Python unit test

I&#39;m trying to install `lxml`. I&#39;ve had a look at the website, and version 2.2.8 looked reasonable to me but when I did `easy_install lxml`, it installed version 2.3.beta1 which is not really what I want I presume.

What is the best way to fix this and how can I force easy_install to install a particular version?

(Mac os x 10.6)

Install particular version with easy_install

I have the following function:

    def copy_file(source_file, target_dir):
        pass

Now I would like to use `multiprocessing` to execute this function at once:

    p = Pool(12)
    p.map(lambda x: copy_file(x,target_dir), file_list)

The problem is, lambda&#39;s can&#39;t be pickled, so this fails. What is the most neat (pythonic) way to fix this?



How to let Pool.map take a lambda function

I&#39;m interested in running a Python program using a computer cluster. I have in the past been using Python MPI interfaces, but due to difficulties in compiling/installing these, I would prefer solutions which use built-in modules, such as Python&#39;s &lt;a href=&quot;http://docs.python.org/library/multiprocessing.html&quot;&gt;multiprocessing&lt;/a&gt; module.

What I would really like to do is just set up a `multiprocessing.Pool` instance that would span across the whole computer cluster, and run a `Pool.map(...)`. Is this something that is possible/easy to do?

If this is impossible, I&#39;d like to at least be able to start `Process` instances on any of the nodes from a central script with different parameters for each node.

Using the multiprocessing module for cluster computing

I&#39;m wondering about the way that python&#39;s Multiprocessing.Pool class works with map, imap, and map_async.  My particular problem is that I want to map on an iterator that creates memory-heavy objects, and don&#39;t want all these objects to be generated into memory at the same time.  I wanted to see if the various map() functions would wring my iterator dry, or intelligently call the next() function only as child processes slowly advanced, so I hacked up some tests as such:

    def g():
      for el in xrange(100):
        print el
        yield el

    def f(x):
      time.sleep(1)
      return x*x
  
    if __name__ == &#39;__main__&#39;:
      pool = Pool(processes=4)              # start 4 worker processes
      go = g()
      g2 = pool.imap(f, go)
      g2.next()

And so on with map, imap, and map_async.  This is the most flagrant example however, as simply calling next() a single time on g2 prints out all my elements from my generator g(), whereas if imap were doing this &#39;lazily&#39; I would expect it to only call go.next() once, and therefore print out only &#39;1&#39;.

Can someone clear up what is happening, and if there is some way to have the process pool &#39;lazily&#39; evaluate the iterator as needed?

Thanks,

Gabe


Python Multiprocessing.Pool lazy iteration

In the Python [`multiprocessing`][1] library, is there a variant of `pool.map` which supports multiple arguments?

    import multiprocessing

    text = &quot;test&quot;

    def harvester(text, case):
        X = case[0]
        text + str(X)
    
    if __name__ == &#39;__main__&#39;:
        pool = multiprocessing.Pool(processes=6)
        case = RAW_DATASET
        pool.map(harvester(text, case), case, 1)
        pool.close()
        pool.join()


  [1]: https://docs.python.org/3/library/multiprocessing.html

How to use multiprocessing pool.map with multiple arguments

The piece of code that I have looks some what like this:

    glbl_array = # a 3 Gb array
    
    def my_func( args, def_param = glbl_array):
        #do stuff on args and def_param

    if __name__ == &#39;__main__&#39;:
      pool = Pool(processes=4)
      pool.map(my_func, range(1000))

Is there a way to make sure (or encourage) that the different processes does not get a copy of glbl_array but shares it. If there is no way to stop the copy I will go with a memmapped array, but my access patterns are not very regular, so I expect memmapped arrays to be slower. The above seemed like the first thing to try. This is on Linux. I just wanted some advice from Stackoverflow and do not want to annoy the sysadmin. Do you think it will help if the the second parameter is a genuine immutable object like `glbl_array.tostring()`.

Content Type	Original Author	Original Content on Stackoverflow
Question	sergio	View Question on Stackoverflow
Solution 1 - Python	detly	View Answer on Stackoverflow

"chunksize" parameter in multiprocessing.Pool.map

Python Problem Overview

Python Solutions

Solution 1 - Python

Difference between preprocessor directives #if and #ifdef

How to get the duration of a video in Python?

Attributions