Sphinx autodoc is not automatic enough

I'm trying to use Sphinx to document a 5,000+ line project in Python. It has about 7 base modules. As far as I know, In order to use autodoc I need to write code like this for each file in my project:

.. automodule:: mods.set.tests
    :members:
    :show-inheritance:

This is way too tedious because I have many files. It would be much easier if I could just specify that I wanted the 'mods' package to be documented. Sphinx could then recursively go through the package and make a page for each submodule.

Is there a feature like this? If not I could write a script to make all the .rst files, but that would take up a lot of time.

You can check this script that I've made. I think it can help you.

This script parses a directory tree looking for python modules and packages and creates ReST files appropriately to create code documentation with Sphinx. It also creates a modules index.

UPDATE

This script is now part of Sphinx 1.1 as apidoc.

From Sphinx version 3.1 (June 2020), sphinx.ext.autosummary (finally!) has automatic recursion.

So no need to hard code module names or rely on 3rd party libraries like Sphinx AutoAPI or Sphinx AutoPackageSummary for their automatic package detection any more.

Example Python 3.7 package to document (see code on Github and result on ReadTheDocs):

mytoolbox
|-- mypackage
|   |-- __init__.py
|   |-- foo.py
|   |-- mysubpackage
|       |-- __init__.py
|       |-- bar.py
|-- doc
|   |-- source
|       |--index.rst
|       |--conf.py
|       |-- _templates
|           |-- custom-module-template.rst
|           |-- custom-class-template.rst

conf.py:

import os
import sys
sys.path.insert(0, os.path.abspath('../..'))  # Source code dir relative to this file

extensions = [
    'sphinx.ext.autodoc',  # Core library for html generation from docstrings
    'sphinx.ext.autosummary',  # Create neat summary tables
]
autosummary_generate = True  # Turn on sphinx.ext.autosummary

# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

index.rst (note new :recursive: option):

Welcome to My Toolbox
=====================

Some words.

.. autosummary::
   :toctree: _autosummary
   :template: custom-module-template.rst
   :recursive:

   mypackage

This is sufficient to automatically summarise every module in the package, however deeply nested. For each module, it then summarises every attribute, function, class and exception in that module.

Oddly, though, the default sphinx.ext.autosummary templates don't go on to generate separate documentation pages for each attribute, function, class and exception, and link to them from the summary tables. It's possible to extend the templates to do this, as shown below, but I can't understand why this isn't the default behaviour - surely that's what most people would want..? I've raised it as a feature request.

I had to copy the default templates locally, and then add to them:

• Copy site-packages/sphinx/ext/autosummary/templates/autosummary/module.rst to mytoolbox/doc/source/_templates/custom-module-template.rst
• Copy site-packages/sphinx/ext/autosummary/templates/autosummary/class.rst to mytoolbox/doc/source/_templates/custom-class-template.rst

The hook into custom-module-template.rst is in index.rst above, using the :template: option. (Delete that line to see what happens using the default site-packages templates.)

custom-module-template.rst (additional lines noted on the right):

{{ fullname | escape | underline}}

.. automodule:: {{ fullname }}
  
   {% block attributes %}
   {% if attributes %}
   .. rubric:: Module Attributes

   .. autosummary::
      :toctree:                                          <-- add this line
   {% for item in attributes %}
      {{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

   {% block functions %}
   {% if functions %}
   .. rubric:: {{ _('Functions') }}

   .. autosummary::
      :toctree:                                          <-- add this line
   {% for item in functions %}
      {{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

   {% block classes %}
   {% if classes %}
   .. rubric:: {{ _('Classes') }}

   .. autosummary::
      :toctree:                                          <-- add this line
      :template: custom-class-template.rst               <-- add this line
   {% for item in classes %}
      {{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

   {% block exceptions %}
   {% if exceptions %}
   .. rubric:: {{ _('Exceptions') }}

   .. autosummary::
      :toctree:                                          <-- add this line
   {% for item in exceptions %}
      {{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

{% block modules %}
{% if modules %}
.. rubric:: Modules

.. autosummary::
   :toctree:
   :template: custom-module-template.rst                 <-- add this line
   :recursive:
{% for item in modules %}
   {{ item }}
{%- endfor %}
{% endif %}
{% endblock %}

custom-class-template.rst (additional lines noted on the right):

{{ fullname | escape | underline}}

.. currentmodule:: {{ module }}

.. autoclass:: {{ objname }}
   :members:                                    <-- add at least this line
   :show-inheritance:                           <-- plus I want to show inheritance...
   :inherited-members:                          <-- ...and inherited members too

   {% block methods %}
   .. automethod:: __init__

   {% if methods %}
   .. rubric:: {{ _('Methods') }}

   .. autosummary::
   {% for item in methods %}
      ~{{ name }}.{{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

   {% block attributes %}
   {% if attributes %}
   .. rubric:: {{ _('Attributes') }}

   .. autosummary::
   {% for item in attributes %}
      ~{{ name }}.{{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

I do not know whether Sphinx had had autosummary extension at the time original question was asked, but for now it is quite possible to set up automatic generation of that kind without using sphinx-apidoc or similar script. Below there are settings which work for one of my projects.

1. Enable autosummary extension (as well as autodoc) in conf.py file and set its autosummary_generate option to True. This may be enough if you're not using custom *.rst templates. Otherwise add your templates directory to exclude list, or autosummary will try to treat them as input files (which seems to be a bug).

    extensions = ['sphinx.ext.autodoc', 'sphinx.ext.autosummary']
    autosummary_generate = True
    templates_path = [ '_templates' ]
    exclude_patterns = ['_build', '_templates']
   

2. Use autosummary:: in TOC tree in your index.rst file. In this example documentation for modules project.module1 and project.module2 will be generated automatically and placed into _autosummary directory.

    PROJECT
    =======
    
    .. toctree::
    
    .. autosummary::
       :toctree: _autosummary
    
       project.module1
       project.module2
   

3. By default autosummary will generate only very short summaries for modules and their functions. To change that you can put a custom template file into _templates/autosummary/module.rst (which will be parsed with Jinja2):

    {{ fullname }}
    {{ underline }}
    
    .. automodule:: {{ fullname }}
        :members:
   

In conclusion, there is no need to keep _autosummary directory under version control. Also, you may name it anything you want and place it anywhere in the source tree (putting it below _build will not work, though).

Sphinx AutoAPI does exactly this.

In each package, the __init__.py file can have .. automodule:: package.module components for each part of the package.

Then you can .. automodule:: package and it mostly does what you want.

Maybe what you're looking for is Epydoc and this Sphinx extension.