sys.path different in Jupyter and Python - how to import own modules in Jupyter?

In Jupyter my own little module is not loaded but in python/bpython is everything is fine. When typing

import sys
print(sys.path)

the path to my module will not in show in Jupyter but in python/bpython it is still there.

I am using:

1. PYTHONPATH in .bashrc to include my module,
2. Jupyter and bpython inside a virtualenv.

The most similar questions is this https://stackoverflow.com/questions/34389029/cannot-import-modules-in-jupyter-notebook-wrong-sys-path

How to configure Jupyter to load my modules automagically?

Here is what I do on my projects in jupyter notebook,

import sys
sys.path.append("../") # go to parent dir
from customFunctions import *

Then, to affect changes in customFunctions.py,

%load_ext autoreload
%autoreload 2

Jupyter is base on ipython, a permanent solution could be changing the ipython config options.

Create a config file

$ ipython profile create
$ ipython locate
/Users/username/.ipython

Edit the config file

$ cd /Users/username/.ipython
$ vi profile_default/ipython_config.py

The following lines allow you to add your module path to sys.path

c.InteractiveShellApp.exec_lines = [
    'import sys; sys.path.append("/path/to/your/module")'
]

At the jupyter startup the previous line will be executed

Here you can find more details about ipython config https://www.lucypark.kr/blog/2013/02/10/when-python-imports-and-ipython-does-not/

Jupyter has its own PATH variable, JUPYTER_PATH.

Adding this line to the .bashrc file worked for me:

export JUPYTER_PATH=<directory_for_your_module>:$JUPYTER_PATH

Suppose your project has the following structure and you want to do imports in the notebook.ipynb:

/app
  /mypackage
    mymodule.py
  /notebooks
    notebook.ipynb

If you are running Jupyter inside a docker container without any virtualenv it might be useful to create Jupyter (ipython) config in your project folder:

/app
  /profile_default
    ipython_config.py

Content of ipython_config.py:

c.InteractiveShellApp.exec_lines = [
    'import sys; sys.path.append("/app")'
]

Open the notebook and check it out:

print(sys.path)

> ['', '/usr/local/lib/python36.zip', '/usr/local/lib/python3.6', > '/usr/local/lib/python3.6/lib-dynload', > '/usr/local/lib/python3.6/site-packages', > '/usr/local/lib/python3.6/site-packages/IPython/extensions', > '/root/.ipython', '/app']

Now you can do imports in your notebook without any sys.path appending in the cells:

from mypackage.mymodule import myfunc

The verified solution doesn't work for me, since my notebook is not in my sys.path. This works however;

import os,sys
sys.path.insert(1, os.path.join(os.getcwd()  , '..'))

I don't like sys.path.append("../"). This actually adds string ../ to the path and not the absolute path.

• This causes path differences while loading data files (like csv). If we go down that route, we need to add relative paths to those file paths as well.
• This causes __file__ reference to have relative .. in them. If there are logics written around __file__ reference those may break.

The following approach adds absolute path to the sys.path and I didn't face any issue:

import sys
import os

sys.path.insert(0, os.path.abspath('..'))

Now everything works same way as it works from normal python & jupyter notebook.

We can check the sys path and validate with following :

# Testing
import sys
for p in sys.path:
    print(p)

We will still have issues with accessing file and their paths used within our modules. This is because file io like open uses current directory based paths.

Best solution would be to clean up the way we construct file paths to have relative path from that particular python module.

For example:

def get_transactions():    
    transaction_path = Path(__file__).parent.parent.parent / 'data_source/some_input.csv'
    print(transaction_path) 
    transactions_df = pd.read_parquet(path=transaction_path) if os.path.isfile(transaction_path) else None
    return transactions_df

You can use absolute imports:

/root
    /app
      /config
        newfile.py
      /source
        file.ipynb

# In the file.ipynb importing the newfile.py file
import os
os.chdir(ROOT_PATH)
from app.config import newfile