Where do the Python unit tests go?
PythonUnit TestingCode OrganizationPython Problem Overview
If you're writing a library, or an app, where do the unit test files go?
It's nice to separate the test files from the main app code, but it's awkward to put them into a "tests" subdirectory inside of the app root directory, because it makes it harder to import the modules that you'll be testing.
Is there a best practice here?
Python Solutions
Solution 1 - Python
For a file module.py, the unit test should normally be called test_module.py, following Pythonic naming conventions.
There are several commonly accepted places to put test_module.py:
- In the same directory as
module.py. - In
../tests/test_module.py(at the same level as the code directory). - In
tests/test_module.py(one level under the code directory).
I prefer #1 for its simplicity of finding the tests and importing them. Whatever build system you're using can easily be configured to run files starting with test_. Actually, the default unittest pattern used for test discovery is test*.py.
Solution 2 - Python
Only 1 test file
If there has only 1 test files, putting it in a top-level directory is recommended:
module/
lib/
__init__.py
module.py
test.py
Run the test in CLI
python test.py
Many test files
If has many test files, put it in a tests folder:
module/
lib/
__init__.py
module.py
tests/
test_module.py
test_module_function.py
# test_module.py
import unittest
from lib import module
class TestModule(unittest.TestCase):
def test_module(self):
pass
if __name__ == '__main__':
unittest.main()
Run the test in CLI
# In top-level /module/ folder
python -m tests.test_module
python -m tests.test_module_function
Use unittest discovery
unittest discovery will find all test in package folder.
Create a __init__.py in tests/ folder
module/
lib/
__init__.py
module.py
tests/
__init__.py
test_module.py
test_module_function.py
Run the test in CLI
# In top-level /module/ folder
# -s, --start-directory (default current directory)
# -p, --pattern (default test*.py)
python -m unittest discover
Reference
pytestGood Practices for test layoutunittest
Unit test framework
Solution 3 - Python
A common practice is to put the tests directory in the same parent directory as your module/package. So if your module was called foo.py your directory layout would look like:
parent_dir/
foo.py
tests/
Of course there is no one way of doing it. You could also make a tests subdirectory and import the module using absolute import.
Wherever you put your tests, I would recommend you use nose to run them. Nose searches through your directories for tests. This way, you can put tests wherever they make the most sense organizationally.
Solution 4 - Python
We had the very same question when writing Pythoscope (https://pypi.org/project/pythoscope/), which generates unit tests for Python programs. We polled people on the testing in python list before we chose a directory, there were many different opinions. In the end we chose to put a "tests" directory in the same directory as the source code. In that directory we generate a test file for each module in the parent directory.
Solution 5 - Python
I also tend to put my unit tests in the file itself, as Jeremy Cantrell above notes, although I tend to not put the test function in the main body, but rather put everything in an
if __name__ == '__main__':
do tests...
block. This ends up adding documentation to the file as 'example code' for how to use the python file you are testing.
I should add, I tend to write very tight modules/classes. If your modules require very large numbers of tests, you can put them in another, but even then, I'd still add:
if __name__ == '__main__':
import tests.thisModule
tests.thisModule.runtests
This lets anybody reading your source code know where to look for the test code.
Solution 6 - Python
Every once in a while I find myself checking out the topic of test placement, and every time the majority recommends a separate folder structure beside the library code, but I find that every time the arguments are the same and are not that convincing. I end up putting my test modules somewhere beside the core modules.
The main reason for doing this is: refactoring.
When I move things around I do want test modules to move with the code; it's easy to lose tests if they are in a separate tree. Let's be honest, sooner or later you end up with a totally different folder structure, like django, flask and many others. Which is fine if you don't care.
The main question you should ask yourself is this:
Am I writing:
- a) reusable library or
- b) building a project than bundles together some semi-separated modules?
If a:
A separate folder and the extra effort to maintain its structure may be better suited. No one will complain about your tests getting deployed to production.
But it's also just as easy to exclude tests from being distributed when they are mixed with the core folders; put this in the setup.py:
find_packages("src", exclude=["*.tests", "*.tests.*", "tests.*", "tests"])
If b:
You may wish — as every one of us do — that you are writing reusable libraries, but most of the time their life is tied to the life of the project. Ability to easily maintain your project should be a priority.
Then if you did a good job and your module is a good fit for another project, it will probably get copied — not forked or made into a separate library — into this new project, and moving tests that lay beside it in the same folder structure is easy in comparison to fishing up tests in a mess that a separate test folder had become. (You may argue that it shouldn't be a mess in the first place but let's be realistic here).
So the choice is still yours, but I would argue that with mixed up tests you achieve all the same things as with a separate folder, but with less effort on keeping things tidy.
Solution 7 - Python
I use a tests/ directory, and then import the main application modules using relative imports. So in MyApp/tests/foo.py, there might be:
from .. import foo
to import the MyApp.foo module.
Solution 8 - Python
I don't believe there is an established "best practice".
I put my tests in another directory outside of the app code. I then add the main app directory to sys.path (allowing you to import the modules from anywhere) in my test runner script (which does some other stuff as well) before running all the tests. This way I never have to remove the tests directory from the main code when I release it, saving me time and effort, if an ever so tiny amount.
Solution 9 - Python
From my experience in developing Testing frameworks in Python, I would suggest to put python unit tests in a separate directory. Maintain a symmetric directory structure. This would be helpful in packaging just the core libraries and not package the unit tests. Below is implemented through a schematic diagram.
<Main Package>
/ \
/ \
lib tests
/ \
[module1.py, module2.py, [ut_module1.py, ut_module2.py, module3.py module4.py, ut_module3.py, ut_module.py]
__init__.py]
In this way when you package these libraries using an rpm, you can just package the main library modules (only). This helps maintainability particularly in agile environment.
Solution 10 - Python
I recommend you check some main Python projects on GitHub and get some ideas.
When your code gets larger and you add more libraries it's better to create a test folder in the same directory you have setup.py and mirror your project directory structure for each test type (unittest, integration, ...)
For example if you have a directory structure like:
myPackage/
myapp/
moduleA/
__init__.py
module_A.py
moduleB/
__init__.py
module_B.py
setup.py
After adding test folder you will have a directory structure like:
myPackage/
myapp/
moduleA/
__init__.py
module_A.py
moduleB/
__init__.py
module_B.py
test/
unit/
myapp/
moduleA/
module_A_test.py
moduleB/
module_B_test.py
integration/
myapp/
moduleA/
module_A_test.py
moduleB/
module_B_test.py
setup.py
Many properly written Python packages uses the same structure. A very good example is the Boto package. Check https://github.com/boto/boto
Solution 11 - Python
How I do it...
Folder structure:
project/
src/
code.py
tests/
setup.py
Setup.py points to src/ as the location containing my projects modules, then i run:
setup.py develop
Which adds my project into site-packages, pointing to my working copy. To run my tests i use:
setup.py tests
Using whichever test runner I've configured.
Solution 12 - Python
I prefer toplevel tests directory. This does mean imports become a little more difficult. For that I have two solutions:
- Use setuptools. Then you can pass
test_suite='tests.runalltests.suite'intosetup(), and can run the tests simply:python setup.py test - Set PYTHONPATH when running the tests:
PYTHONPATH=. python tests/runalltests.py
Here's how that stuff is supported by code in M2Crypto:
- http://svn.osafoundation.org/m2crypto/trunk/setup.py
- http://svn.osafoundation.org/m2crypto/trunk/tests/alltests.py
If you prefer to run tests with nosetests you might need do something a little different.
Solution 13 - Python
I put my tests in the same directory as the code under test (CUT). In projects where I can tweak pytest with my plugin, for foo.py I use foo.pt for the tests which makes editing a particular module and its test together really easy: vi foo.*.
Where I can't do this, I use foo_ut.py or similar. You can still use vi foo* though that will also catch foobar.py and foobar_ut.py if those exist.
In either case I tweak the test discovery process to find these.
This puts the tests right beside the code in a directory listing, making it obvious that tests are there, and makes opening the tests as easy as it can possibly be when they're in a separate file. (For editors started from the command line, as described above; for GUI systems, click on the code file and the adjacent (or very nearly adjacent) test file.
As others have pointed out, this also makes it easier to refactor and to extract the code for use elsewhere should that ever be necessary.
I really dislike the idea of putting tests in a completely different directory tree; why make it harder than necessary for developers to open up the tests when they're opening the file with the CUT? It's not like the vast majority of developers are so keen on writing or tweaking tests that they'll ignore any barrier to doing that, instead of using the barrier as an excuse. (Quite the opposite, in my experience; even when you make it as easy as possible I know many developers who can't be bothered to write tests.)
Solution 14 - Python
We use
app/src/code.py
app/testing/code_test.py
app/docs/..
In each test file we insert ../src/ in sys.path. It's not the nicest solution but works. I think it would be great if someone came up w/ something like maven in java that gives you standard conventions that just work, no matter what project you work on.
Solution 15 - Python
In C#, I've generally separated the tests into a separate assembly.
In Python -- so far -- I've tended to either write doctests, where the test is in the docstring of a function, or put them in the if __name__ == "__main__" block at the bottom of the module.
Solution 16 - Python
If the tests are simple, simply put them in the docstring -- most of the test frameworks for Python will be able to use that:
>>> import module
>>> module.method('test')
'testresult'
For other more involved tests, I'd put them either in ../tests/test_module.py or in tests/test_module.py.
Solution 17 - Python
When writing a package called "foo", I will put unit tests into a separate package "foo_test". Modules and subpackages will then have the same name as the SUT package module. E.g. tests for a module foo.x.y are found in foo_test.x.y. The __init__.py files of each testing package then contain an AllTests suite that includes all test suites of the package. setuptools provides a convenient way to specify the main testing package, so that after "python setup.py develop" you can just use "python setup.py test" or "python setup.py test -s foo_test.x.SomeTestSuite" to the just a specific suite.
Solution 18 - Python
I've recently started to program in Python, so I've not really had chance to find out best practice yet. But, I've written a module that goes and finds all the tests and runs them.
So, I have:
app/ appfile.py test/ appfileTest.pyI'll have to see how it goes as I progress to larger projects.