Make (install from source) python without running tests

PythonLinuxMakefileBuild

Python Problem Overview


I compiling python from source tar. All works good, but tests running 2 hours and two times. How to bypass these tests?

0:16:20 [178/405] test_inspect
0:16:26 [179/405] test_int
0:16:27 [180/405] test_int_literal
0:16:27 [181/405] test_io
0:18:18 [182/405] test_ioctl -- test_io passed in 1 min 51 sec
0:18:19 [183/405] test_ipaddress
0:18:22 [184/405] test_isinstance
0:18:23 [185/405] test_iter
0:18:24 [186/405] test_iterlen
0:18:25 [187/405] test_itertools
0:19:09 [188/405] test_json -- test_itertools passed in 44 sec
0:19:30 [189/405] test_keyword

As result

make 7724,86s user 188,63s system 101% cpu 2:10:18,93 total

I make its distribution like this

PYTHON_VERSION = 3.6.1
PYTHON_URL = https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tar.xz

wget -O dl/Python-${PYTHON_VERSION}.tar.xz ${PYTHON_URL}

cd dl 
tar xf Python-${PYTHON_VERSION}.tar.xz

mkdir -p dl/Python-${PYTHON_VERSION}-build/
cd Python-${PYTHON_VERSION} 
    ./configure --enable-optimizations --prefix=$$(pwd)-build --cache-file=$$(pwd)/cache-file

This commands runs tests twice:

make -C dl/Python-${PYTHON_VERSION} -j8 
make -C dl/Python-${PYTHON_VERSION} -j8 install

p.s. This is part of another make file.

Python Solutions


Solution 1 - Python

The configure option --enable-optimizations enables running test suites to generate data for profiling Python. The resulting python binary has better performance in executing python code. Improvements noted here

From configure help:
--enable-optimizations  Enable expensive optimizations (PGO, etc). Disabled by default.

From wikipedia

 profile-guided optimisation uses the results of profiling test runs of the instrumented program to optimize the final generated code.

In short, you should not skip tests when using --enable-optimizations as the data required for profiling is generated by running tests. You can run make -j8 build_all followed by make -j8 install to skip tests once(the tests would still run with install target), but that would defeat the purpose. You can instead drop the configure flag for better build times.

Solution 2 - Python

just build and install with

make -j8 build_all
make -j8 altinstall

Solution 3 - Python

I did some (quick) research on skipping the test runs when building Python by instructing either:

  • configure - passing some args (e.g. --without-tests, --disable-tests, --skip-tests)
  • make - specifying some variable (either via env vars or cmdline)

The former yielded no results. The latter (by looking in the Makefile template) revealed the fact that test execution is invoked by calling ${PYTHON_SRC_DIR}/Tools/scripts/run_tests.py (which sets some stuff and calls another script, which calls another one, ...).
Note that I found the file on Python 3.5(.4) and Python 3.6(.4) but not on Python 2.7(.14). A little bit more research revealed that it is possible to skip the (above) test run. What you need to do is:

make -C dl/Python-${PYTHON_VERSION} -j8 EXTRATESTOPTS=--list-tests install

Notes:

  • Googleing didn't reveal anything (relevant) on EXTRATESTOPTS, so I guess it's not officialy supported
  • You could also set EXTRATESTOPTS=--list-tests as an environment variable, before launching (inner) make
  • Needless to say that if some "minor" error happened during build (e.g. a non critical external module (like _ssl.so for example) failed to build), there will be no tests to fail, so you'll only find about it at runtime (which would be terribly nasty if it would happen in production)

@EDIT0:

After @amohr 's comment, I decided to play a little bit more, so I ran the whole process:

  1. configure (opts)
  2. make (opts)
  3. make install

on a Lnx (Ubtu 16) machine with 2 CPUs, where one (full) test run takes ~24 minutes. Here are my findings (Python 3.6):

  • It ran successfully on Python 3.5(.4)

  • The solution that I suggested earlier, operates at the 3rd step, so it only skips the 2nd test run: it operates on the (root) Makefile's test target (make test) which is invoked by install target

  • Regarding the 1st test run, by checking the Makefile, and make's output, here's what I discovered that happens at the 2nd (make) step:

    1. The C sources are built "normally"
    2. Tests are being run (I deducted that some profile data is stored somewhere)
    3. The C sources are rebuilt with different flags (e.g. in my case gcc's -fprofile-generate was replaced by -fprofile-use -fprofile-correction (check [GNU.GCC]: Options That Control Optimization for more details)) to make use of the profile info generated at previous (sub) step
  • Skipping the 1st test run would automatically imply no optimizations. Way(s) of achieving:

    1. make build_all (at 2nd step) - as suggested by other answers

      • Here's a snippet of the (root) Makefile generated by configure (with --enable-optimizations):

          all:            profile-opt
          build_all:      check-clean-src $(BUILDPYTHON) oldsharedmods sharedmods gdbhooks \
                          Programs/_testembed python-config
        
      • And here's one without it:

          all:            build_all
          build_all:      check-clean-src $(BUILDPYTHON) oldsharedmods sharedmods gdbhooks \
                          Programs/_testembed python-config
        
      • As seen, running:

        1. configure --enable-optimizations
        2. make build_all

        is identical to:

        1. configure
        2. make
    2. Manually modifying the (root) Makefile between 1st (configure --enable-optimizations) and 2nd (make) steps:

      • Find the macro definition PROFILE_TASK=-m test.regrtest --pgo (for me it was around line ~250)
      • Add --list-tests at the end
      • Substeps (#2.)#1. and (#2.)#3. are exactly the same, while for (#2.)#2., the tests are not being run. That can mean that either:
        • The 2nd sources build is identical to the 1st one (which would make it completely useless)
        • The 2nd does some optimizations (without having any information), which means that it could crash at runtime (I think / hope it's the former case)

Solution 4 - Python

the default build target for optimized builds includes running the tests. to skip them, try:

 make -C dl/Python-${PYTHON_VERSION} -j8 build_all

Solution 5 - Python

We faced the same problem with python 3.7.6 and based on reverse engineering the Makefile, found that following steps will build python quickly while running the tests too (so that we don't loose on --enable-optimization flag)

cd /home/ubuntu
PYTHON_VER=3.7.6
wget https://www.python.org/ftp/python/$PYTHON_VER/Python-$PYTHON_VER.tgz
tar xvf Python-$PYTHON_VER.tgz
cd Python-$PYTHON_VER/
./configure --enable-loadable-sqlite-extensions --enable-optimizations
make profile-gen-stamp; ./python -m test.regrtest --pgo -j8; make build_all_merge_profile; touch profile-run-stamp; make
make install

The key is run python tests in parallel by passing -j8 to it.

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestioneriView Question on Stackoverflow
Solution 1 - PythonnitView Answer on Stackoverflow
Solution 2 - PythonDavid DingView Answer on Stackoverflow
Solution 3 - PythonCristiFatiView Answer on Stackoverflow
Solution 4 - PythonumläuteView Answer on Stackoverflow
Solution 5 - PythonMonga ChingaView Answer on Stackoverflow