The [article Eric pointed](https://web.archive.org/web/20190714164001/http://infinitemonkeycorps.net/docs/pph/) to is awesome because it covers details of organising large Python code bases.

If you&#39;ve landed here from Google and are trying to find out how to split one large source file into multiple, more manageable, files I&#39;ll summarise the process briefly.

Assume you currently have everything in a file called `main.py`:

 * Create another source file in the same folder (let&#39;s call ours `utils.py` for this example)
 * Move whatever classes, functions, statements, etc you need from `main.py` into `utils.py`
 * In `main.py` add a single line at the top: `import utils`

Conceptually what this does is to create a new module called `utils` in another source file.  You can then import it wherever it&#39;s needed.

The way you should organise your code and tests is exactly the same you would for any OO language. 

Answers from the way I do it. It may not be right but works for me

1. Depends on how your functionality is split. For my main python app I have 1 file with classes for the entry points and then packages of different bits of functionality
2. I use PyDev for eclipse and organise it like I would for Java. 
 

&gt;     &gt;  Workspace
&gt;     &gt;     |
&gt;     &gt;     |-Src
&gt;     &gt;     |   |-Package1
&gt;     &gt;     |   |-Package2
&gt;     &gt;     |   |-main.py
&gt;     &gt;     |-Test
&gt;     &gt;         |-TestPackage1
&gt;     &gt;         |-TestPackage2

3. Use DocString everywhere to keep track of everything
4. After making sure that the relevant `__init__.py` files are in the folders. its just a simple case of `from module import class`

codepad.org allow you to run C,C++,D etc code online but not Java... is there a site that I can use for Java?

Run Java Code Online

I had some tests working fine. Then, I moved it to a different package, and am now getting errors. Here is the code:

    import static org.junit.Assert.*;
    import java.util.HashSet;
    import java.util.Map;
    import java.util.Set;
    
    import org.jgrapht.Graphs;
    import org.jgrapht.WeightedGraph;
    import org.jgrapht.graph.DefaultWeightedEdge;
    import org.jgrapht.graph.SimpleWeightedGraph;
    import org.junit.*;	
    
    @Test
    	public void testEccentricity() {
    		WeightedGraph&lt;String, DefaultWeightedEdge&gt; g = generateSimpleCaseGraph();
    		Map&lt;String, Double&gt; eccen = JGraphtUtilities.eccentricities(g);
    		
    		assertEquals(70, eccen.get(&quot;alpha&quot;));
    		assertEquals(80, eccen.get(&quot;l&quot;));
    		assertEquals(130, eccen.get(&quot;l-0&quot;));
    		assertEquals(100, eccen.get(&quot;l-1&quot;));
    		assertEquals(90, eccen.get(&quot;r&quot;));
    		assertEquals(120, eccen.get(&quot;r-0&quot;));
    		assertEquals(130, eccen.get(&quot;r-1&quot;));
    	}

The error message is this:

**The method assertEquals(Object, Object) is ambiguous for the type JGraphtUtilitiesTest**

How can I fix this? Why did this problem occur as I moved the class to a different package?

Java JUnit: The method X is ambiguous for type Y

I&#39;m getting started with Python (it&#39;s high time I give it a shot), and I&#39;m looking for some best practices.

My first project is a queue which runs command-line experiments in multiple threads. I&#39;m starting to get a very long `main.py` file, and I&#39;d like to break it up. In general, I&#39;m looking for: **How do python programmers organize multiple source files?** Is there a particular structure that works for you?

My specific questions include:

 1. Should each class be in a separate file?
 2. How should I organize unit tests relative to source code?
 3. Where should I put doc comments, specifically those for command-line operation?
 4. If I use multiple directories, how do I import classes between them?

I can probably draw some of my own conclusions here by trial and error, but I&#39;d rather start from something _good_.

How should I organize Python source code?

I'm getting started with Python (it's high time I give it a shot), and I'm looking for some best practices.
My first project is a queue which runs command-line experiments in multiple threads. I'm starting to get a very long <code>main.py</code> file, and I'd like to break it up. In general, I'm looking for: How do python programmers organize multiple source files? Is there a particular structure that works for you?
My specific questions include:
<ol>
<li>Should each class be in a separate file?</li>
<li>How should I organize unit tests relative to source code?</li>
<li>Where should I put doc comments, specifically those for command-line operation?</li>
<li>If I use multiple directories, how do I import classes between them?</li>
</ol>
I can probably draw some of my own conclusions here by trial and error, but I'd rather start from something good.

How can I create a zip archive of a directory structure in Python?

How to create a zip archive of a directory?

I wanted to know if there are any built-in ways to continue to next iteration in outer loop in python.  For example,  consider the code:

    for ii in range(200):
        for jj in range(200, 400):
            ...block0...
            if something:
                continue
        ...block1...

I want this continue statement to exit the jj loop and goto next item in the ii loop.  I can implement this logic in some other way (by setting a flag variable), but is there an easy way to do this, or is this like asking for too much?

Python: Continuing to next iteration in outer loop

When I have regular methods for calling another method in a class, I have to do this

    class test:
        def __init__(self):
            pass
        def dosomething(self):
            print &quot;do something&quot;
            self.dosomethingelse()
        def dosomethingelse(self):
            print &quot;do something else&quot;

but when I have static methods I can&#39;t write

    self.dosomethingelse()

because there is no instance. What do I have to do in Python for calling an static method from another static method of the same class?


Static methods - How to call a method from another method?

I&#39;m writing a script to check permissions of files in user&#39;s directories and if they&#39;re not acceptable I&#39;ll be warning them, but I want to check permissions of not just the logged in user, but also group and others. How can i do this? It seems to me that `os.access()` in Python can only check the permissions for the user running the script.

Checking File Permissions in Linux with Python

I have a dictionary that I declared in a particular order and want to keep it in that order all the time. The keys/values can&#39;t really be kept in order based on their value, I just want it in the order that I declared it.

So if I have the dictionary:

    d = {&#39;ac&#39;: 33, &#39;gw&#39;: 20, &#39;ap&#39;: 102, &#39;za&#39;: 321, &#39;bs&#39;: 10}

It isn&#39;t in that order if I view it or iterate through it, is there any way to make sure Python will keep the explicit order that I declared the keys/values in?

How to keep keys/values in same order as declared?

I have been developing an iphone application using a domain model, and have put off the persistence aspect of the app until now.  Core Data looks like a really good solution since I already have a well defined model but I am running into a snag with my existing unit tests.

Here is simple example of what I have now:

    - (void)test_full_name_returns_correct_string {
        Patient *patient = [[Patient alloc] init];  
        patient.firstName = @&quot;charlie&quot;;
        patient.lastName = @&quot;chaplin&quot;;
        STAssertTrue([[patient fullName] isEqualToString:@&quot;charlie chaplin&quot;], @&quot;should have matched full name&quot;);
    }  

How can I make this work once my Patient object extends from NSManagedObject and uses @dynamic for the firstName and lastName properties?

Has anyone else run into this type of this with Core Data?  Thanks.


How to unit test my models now that I am using Core Data?

I know you can run all the tests in a certain class using:


    mvn test -Dtest=classname

But I want to run an individual method and -Dtest=classname.methodname  doesn&#39;t seem to work.

Run a single test method with maven

Some code I am unit testing needs to load a resource file. It contains the following line:

    NSString *path = [[NSBundle mainBundle] pathForResource:@&quot;foo&quot; ofType:@&quot;txt&quot;];

In the app it runs just fine, but when run by the unit testing framework `pathForResource:` returns nil, meaning it could not locate `foo.txt`.

I&#39;ve made sure that `foo.txt` is included in the **Copy Bundle Resources** build phase of the unit test target, so why can&#39;t it find the file?

Why can&#39;t code inside unit tests find bundle resources?

I need to write a unit test for a method that takes a stream which comes from a text file. I would like to do do something like this:

    Stream s = GenerateStreamFromString(&quot;a,b \n c,d&quot;);


How do I generate a stream from a string?

The very common directory structure for even a simple Python module seems to be to separate the unit tests into their own ``test`` directory:

    new_project/
        antigravity/
            antigravity.py
        test/
            test_antigravity.py
        setup.py
        etc.

for example see this [Python project howto][1].

My question is simply **What&#39;s the usual way of actually running the tests?** I suspect this is obvious to everyone except me, but you can&#39;t just run ``python test_antigravity.py`` from the test directory as its ``import antigravity`` will fail as the module is not on the path.

I know I could modify PYTHONPATH and other search path related tricks, but I can&#39;t believe that&#39;s the simplest way - it&#39;s fine if you&#39;re the developer but not realistic to expect your users to use if they just want to check the tests are passing.

The other alternative is just to copy the test file into the other directory, but it seems a bit dumb and misses the point of having them in a separate directory to start with.

So, if you had just downloaded the source to my new project how would you run the unit tests? I&#39;d prefer an answer that would let me say to my users: &quot;To run the unit tests do X.&quot;


  [1]: http://infinitemonkeycorps.net/docs/pph/

Running unittest with typical test directory structure

JAVA - NETBEANS

This is an IDE question

I am always working with collapsed methods, because I want to be able to see my methods all together. This is a little time consuming because I have to use the mouse to scroll up to the declaration of the method and click on the - (minus) icon. And then respectively go to the method I want to work on and click on the + (plus) icon.

Is there a way through a keyboard shortcut to do the collapse (and respectively the expand)?


NetBeans shortcut key for collapsing/expanding a method

So, having spend many years in an object oriented world with code reuse, design patterns and best practices always taken into account, I find myself struggling somewhat with code organization and code reuse in world of Spark. 

If I try to write code in a reusable way, it nearly always comes with a performance cost and I end up rewriting it to whatever is optimal for my particular use case. This constant &quot;write what is optimal for this particular use case&quot; also affects code organization, because splitting code into different objects or modules is difficult when &quot;it all really belongs together&quot; and I thus end up with very few &quot;God&quot; object containing long chains of complex transformations. In fact, I frequently think that if I had taken a look at most of the Spark code I&#39;m writing now back when I was working in the object oriented world, I would have winced and dismissed it as &quot;spaghetti code&quot;. 

I have surfed the internet trying to find some sort of equivalent to the best practices of the object oriented world, but without much luck. I can find some &quot;best practices&quot; for functional programming but Spark just adds an extra layer, because performance is such a major factor here. 

So my question to you is, have any of you Spark gurus found some best practices for writing Spark code that you can recommend?

**EDIT**

As written in a comment, I did not actually expect anyone to post an answer on how to _solve_ this problem, but rather I was hoping that someone in this community had come across some Martin Fowler type, who had written som articles or blog posts somewhere on how to address problems with code organization in the world of Spark. 

@DanielDarabos suggested that I might put in an example of a situation where code organization and performance are conflicting. While I find that I frequently have issues with this in my everyday work, I find it a bit hard to boil it down to a good minimal example ;) but I will try.

In the object oriented world, I&#39;m a big fan of the Single Responsibility Principle, so I would make sure that my methods were only responsible for one thing. It makes them reusable and easily testable. So if I had to, say, calculate the sum of some numbers in a list (matching some criteria) and I had to calculate the average of the same number, I would most definitely create two methods - one that calculated the sum and one that calculated the average. Like this:

&lt;!-- language: scala --&gt;
    
    def main(implicit args: Array[String]): Unit = {
      val list = List((&quot;DK&quot;, 1.2), (&quot;DK&quot;, 1.4), (&quot;SE&quot;, 1.5))

      println(&quot;Summed weights for DK = &quot; + summedWeights(list, &quot;DK&quot;)
      println(&quot;Averaged weights for DK = &quot; + averagedWeights(list, &quot;DK&quot;)
    }

    def summedWeights(list: List, country: String): Double = {
      list.filter(_._1 == country).map(_._2).sum
    }

    def averagedWeights(list: List, country: String): Double = {
      val filteredByCountry = list.filter(_._1 == country) 
      filteredByCountry.map(_._2).sum/ filteredByCountry.length
    }


I can of course continue to honor SRP in Spark:


&lt;!-- language: scala --&gt;

    def main(implicit args: Array[String]): Unit = {
      val df = List((&quot;DK&quot;, 1.2), (&quot;DK&quot;, 1.4), (&quot;SE&quot;, 1.5)).toDF(&quot;country&quot;, &quot;weight&quot;)

      println(&quot;Summed weights for DK = &quot; + summedWeights(df, &quot;DK&quot;)
      println(&quot;Averaged weights for DK = &quot; + averagedWeights(df, &quot;DK&quot;)
    }


    def avgWeights(df: DataFrame, country: String, sqlContext: SQLContext): Double = {
      import org.apache.spark.sql.functions._
      import sqlContext.implicits._

      val countrySpecific = df.filter(&#39;country === country)
      val summedWeight = countrySpecific.agg(avg(&#39;weight))

      summedWeight.first().getDouble(0)
    }

    def summedWeights(df: DataFrame, country: String, sqlContext: SQLContext): Double = {
      import org.apache.spark.sql.functions._
      import sqlContext.implicits._

      val countrySpecific = df.filter(&#39;country === country)
      val summedWeight = countrySpecific.agg(sum(&#39;weight))

      summedWeight.first().getDouble(0)
    }

But because my `df` may contain billions of rows I would rather not have to perform the `filter` twice. In fact, performance is directly coupled to EMR cost, so I REALLY don&#39;t want that. To overcome it, I thus decide to violate SRP and simply put the two functions in one and make sure I call persist on the country-filtered `DataFrame`, like this:


&lt;!-- language: scala --&gt;

    def summedAndAveragedWeights(df: DataFrame, country: String, sqlContext: SQLContext): (Double, Double) = {
      import org.apache.spark.sql.functions._
      import sqlContext.implicits._

      val countrySpecific = df.filter(&#39;country === country).persist(StorageLevel.MEMORY_AND_DISK_SER)
      val summedWeights = countrySpecific.agg(sum(&#39;weight)).first().getDouble(0)
      val averagedWeights = summedWeights / countrySpecific.count()

      (summedWeights, averagedWeights)
    }

Now, this example if of course a huge simplification of what&#39;s encountered in real life. Here I could simply solve it by filtering and persisting `df` _before_ handing it to the sum and avg functions (which would also be more SRP), but in real life there may be a number of intermediate calculations going on that are needed again and again. In other words, the `filter` function here is merely an attempt to make a _simple_ example of something that will benefit from being persisted. In fact, I think calls to `persist` is a keyword here. Calling `persist` will vastly speed up my job, but the cost is that I have to tightly couple all code that depends on the persisted `DataFrame` - even if they are logically separate.   

Content Type	Original Author	Original Content on Stackoverflow
Question	Andres Jaan Tack	View Question on Stackoverflow
Solution 1 - Python	Drew Noakes	View Answer on Stackoverflow
Solution 2 - Python	AutomatedTester	View Answer on Stackoverflow

How should I organize Python source code?

Python Problem Overview

Python Solutions

Solution 1 - Python

Solution 2 - Python

Java JUnit: The method X is ambiguous for type Y

Run Java Code Online

Attributions