Reading file using relative path in python project

Say I have a python project that is structured as follows:

project
    /data
        test.csv
    /package
        __init__.py
        module.py
    main.py

__init__.py:

from .module import test

module.py:

import csv

with open("..data/test.csv") as f:
    test = [line for line in csv.reader(f)]

main.py:

import package

print(package.test)

When I run main.py I get the following error:

 C:\Users\Patrick\Desktop\project>python main.py
Traceback (most recent call last):
  File "main.py", line 1, in <module>
    import package
  File "C:\Users\Patrick\Desktop\project\package\__init__.py", line 1, in <module>
    from .module import test
  File "C:\Users\Patrick\Desktop\project\package\module.py", line 3, in <module>
    with open("../data/test.csv") as f:
FileNotFoundError: [Errno 2] No such file or directory: '../data/test.csv'

However, if I run module.py from the package directory I get no errors. So it seems that the relative path used in open(...) is only relative to where the originating file is being run from (i.e __name__ == "__main__")? I don't want to use absolute paths. What are some ways to deal with this?

Relative paths are relative to current working directory. If you do not your want your path to be, it must be absolute.

But there is an often used trick to build an absolute path from current script: use its __file__ special attribute:

from pathlib import Path

path = Path(__file__).parent / "../data/test.csv"
with path.open() as f:
    test = list(csv.reader(f))

This requires python 3.4+ (for the pathlib module).

If you still need to support older versions, you can get the same result with:

import csv
import os.path

my_path = os.path.abspath(os.path.dirname(__file__))
path = os.path.join(my_path, "../data/test.csv")
with open(path) as f:
    test = list(csv.reader(f))

[2020 edit: python3.4+ should now be the norm, so I moved the pathlib version inspired by jpyams' comment first]

For Python 3.4+:

import csv
from pathlib import Path

base_path = Path(__file__).parent
file_path = (base_path / "../data/test.csv").resolve()

with open(file_path) as f:
    test = [line for line in csv.reader(f)]

This worked for me.

with open('data/test.csv') as f:

 

My Python version is Python 3.5.2 and the solution proposed in the accepted answer didn't work for me. I've still were given an error

FileNotFoundError: [Errno 2] No such file or directory

when I was running my_script.py from the terminal. Although it worked fine when I run it through Run/Debug Configurations from PyCharm IDE (PyCharm 2018.3.2 (Community Edition)).

Solution:

instead of using:

my_path = os.path.abspath(os.path.dirname(__file__)) + some_rel_dir_path 

as suggested in the accepted answer, I used:

my_path = os.path.abspath(os.path.dirname(os.path.abspath(__file__))) + some_rel_dir_path

Explanation: Changing os.path.dirname(__file__) to os.path.dirname(os.path.abspath(__file__)) solves the following problem:

When we run our script like that: python3 my_script.py the __file__ variable has a just a string value of "my_script.py" without path leading to that particular script. That is why method dirname(__file__) returns an empty string "". That is also the reson why my_path = os.path.abspath(os.path.dirname(__file__)) + some_rel_dir_path is actually the same thing as my_path = some_rel_dir_path. Consequently FileNotFoundError: [Errno 2] No such file or directory is given when trying to use open method because there is no directory like "some_rel_dir_path".

Running script from PyCharm IDE Running/Debug Configurations worked because it runs a command python3 /full/path/to/my_script.py (where "/full/path/to" is specified by us in "Working directory" variable in Run/Debug Configurations) instead of justpython3 my_script.py like it is done when we run it from the terminal.

Hope that will be useful.

try

with open(f"{os.path.dirname(sys.argv[0])}/data/test.csv", newline='') as f:

I was thundered when the following code worked.

import os

for file in os.listdir("../FutureBookList"):
    if file.endswith(".adoc"):
        filename, file_extension = os.path.splitext(file)
        print(filename)
        print(file_extension)
        continue
    else:
        continue

So, I checked the documentation and it says:

> Changed in version 3.6: Accepts a path-like object.

path-like object:

> An object representing a file system path. A path-like object is > either a str or...

I did a little more digging and the following also works:

with open("../FutureBookList/file.txt") as file:
   data = file.read()