Each Answer to this Q is separated by one/two green lines.
I’ve tried reading through questions about sibling imports and even the
package documentation, but I’ve yet to find an answer.
With the following structure:
??? LICENSE.md ??? README.md ??? api ? ??? __init__.py ? ??? api.py ? ??? api_key.py ??? examples ? ??? __init__.py ? ??? example_one.py ? ??? example_two.py ??? tests ? ??? __init__.py ? ??? test_one.py
How can the scripts in the
tests directories import from the
api module and be run from the commandline?
Also, I’d like to avoid the ugly
sys.path.insert hack for every file. Surely
this can be done in Python, right?
Tired of sys.path hacks?
There are plenty of
sys.path.append -hacks available, but I found an alternative way of solving the problem in hand.
- Wrap the code into one folder (e.g.
setup.pyscript where you use setuptools.setup(). (see minimal
- Pip install the package in editable state with
pip install -e <myproject_folder>
- Import using
from packaged_stuff.modulename import function_name
The starting point is the file structure you have provided, wrapped in a folder called
. ??? myproject ??? api ? ??? api_key.py ? ??? api.py ? ??? __init__.py ??? examples ? ??? example_one.py ? ??? example_two.py ? ??? __init__.py ??? LICENCE.md ??? README.md ??? tests ??? __init__.py ??? test_one.py
I will call the
. the root folder, and in my example case it is located at
As a test case, let’s use the following ./api/api.py
def function_from_api(): return 'I am the return value from api.api!'
from api.api import function_from_api def test_function(): print(function_from_api()) if __name__ == '__main__': test_function()
Try to run test_one:
PS C:\tmp\test_imports> python .\myproject\tests\test_one.py Traceback (most recent call last): File ".\myproject\tests\test_one.py", line 1, in <module> from api.api import function_from_api ModuleNotFoundError: No module named 'api'
Also trying relative imports wont work:
from ..api.api import function_from_api would result into
PS C:\tmp\test_imports> python .\myproject\tests\test_one.py Traceback (most recent call last): File ".\tests\test_one.py", line 1, in <module> from ..api.api import function_from_api ValueError: attempted relative import beyond top-level package
- Make a setup.py file to the root level directory
The contents for the
setup.py would be*
from setuptools import setup, find_packages setup(name="myproject", version='1.0', packages=find_packages())
- Use a virtual environment
If you are familiar with virtual environments, activate one, and skip to the next step. Usage of virtual environments are not absolutely required, but they will really help you out in the long run (when you have more than 1 project ongoing..). The most basic steps are (run in the root folder)
- Create virtual env
python -m venv venv
- Activate virtual env
source ./venv/bin/activate(Linux, macOS) or
To learn more about this, just Google out “python virtual env tutorial” or similar. You probably never need any other commands than creating, activating and deactivating.
Once you have made and activated a virtual environment, your console should give the name of the virtual environment in parenthesis
PS C:\tmp\test_imports> python -m venv venv PS C:\tmp\test_imports> .\venv\Scripts\activate (venv) PS C:\tmp\test_imports>
and your folder tree should look like this**
. ??? myproject ? ??? api ? ? ??? api_key.py ? ? ??? api.py ? ? ??? __init__.py ? ??? examples ? ? ??? example_one.py ? ? ??? example_two.py ? ? ??? __init__.py ? ??? LICENCE.md ? ??? README.md ? ??? tests ? ??? __init__.py ? ??? test_one.py ??? setup.py ??? venv ??? Include ??? Lib ??? pyvenv.cfg ??? Scripts [87 entries exceeds filelimit, not opening dir]
- pip install your project in editable state
Install your top level package
pip. The trick is to use the
-e flag when doing the install. This way it is installed in an editable state, and all the edits made to the .py files will be automatically included in the installed package.
In the root directory, run
pip install -e . (note the dot, it stands for “current directory”)
You can also see that it is installed by using
(venv) PS C:\tmp\test_imports> pip install -e . Obtaining file:///C:/tmp/test_imports Installing collected packages: myproject Running setup.py develop for myproject Successfully installed myproject (venv) PS C:\tmp\test_imports> pip freeze myproject==1.0
myproject.into your imports
Note that you will have to add
myproject. only into imports that would not work otherwise. Imports that worked without the
pip install will work still work fine. See an example below.
Test the solution
Now, let’s test the solution using
api.py defined above, and
test_one.py defined below.
from myproject.api.api import function_from_api def test_function(): print(function_from_api()) if __name__ == '__main__': test_function()
running the test
(venv) PS C:\tmp\test_imports> python .\myproject\tests\test_one.py I am the return value from api.api!
* See the setuptools docs for more verbose setup.py examples.
** In reality, you could put your virtual environment anywhere on your hard disk.
Seven years after
Since I wrote the answer below, modifying
sys.path is still a quick-and-dirty trick that works well for private scripts, but there has been several improvements
- Installing the package (in a virtualenv or not) will give you what you want, though I would suggest using pip to do it rather than using setuptools directly (and using
setup.cfgto store the metadata)
- Using the
-mflag and running as a package works too (but will turn out a bit awkward if you want to convert your working directory into an installable package).
- For the tests, specifically, pytest is able to find the api package in this situation and takes care of the
sys.pathhacks for you
So it really depends on what you want to do. In your case, though, since it seems that your goal is to make a proper package at some point, installing through
pip -e is probably your best bet, even if it is not perfect yet.
As already stated elsewhere, the awful truth is that you have to do ugly hacks to allow imports from siblings modules or parents package from a
__main__ module. The issue is detailed in PEP 366. PEP 3122 attempted to handle imports in a more rational way but Guido has rejected it one the account of
The only use case seems to be running scripts that happen
to be living inside a module’s directory, which I’ve always seen as an
Though, I use this pattern on a regular basis with
# Ugly hack to allow absolute import from the root folder # whatever its name is. Please forgive the heresy. if __name__ == "__main__" and __package__ is None: from sys import path from os.path import dirname as dir path.append(dir(path)) __package__ = "examples" import api
path is your running script’s parent folder and
dir(path) your top level folder.
I have still not been able to use relative imports with this, though, but it does allow absolute imports from the top level (in your example
api‘s parent folder).
Here is another alternative that I insert at top of the Python files in
# Path hack. import sys, os sys.path.insert(0, os.path.abspath('..'))
You don’t need and shouldn’t hack
sys.path unless it is necessary and in this case it is not. Use:
import api.api_key # in tests, examples
Run from the project directory:
python -m tests.test_one.
You should probably move
tests (if they are api’s unittests) inside
api and run
python -m api.test to run all tests (assuming there is
python -m api.test.test_one to run
You could also remove
examples (it is not a Python package) and run the examples in a virtualenv where
api is installed e.g.,
pip install -e . in a virtualenv would install inplace
api package if you have proper
I don’t yet have the comprehension of Pythonology necessary to see the intended way of sharing code amongst unrelated projects without a sibling/relative import hack. Until that day, this is my solution. For
tests to import stuff from
..\api, it would look like:
import sys.path import os.path # Import from sibling directory ..\api sys.path.append(os.path.dirname(os.path.abspath(__file__)) + "/..") import api.api import api.api_key
For siblings package imports, you can use either the insert or the append method of the [sys.path] module:
if __name__ == '__main__' and if __package__ is None: import sys from os import path sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) ) import api
This will work if you are launching your scripts as follows:
python examples/example_one.py python tests/test_one.py
On the other hand, you can also use the relative import:
if __name__ == '__main__' and if __package__ is not None: import ..api.api
In this case you will have to launch your script with the ‘-m’ argument (note that, in this case, you must not give the ‘.py’ extension):
python -m packageName.examples.example_one python -m packageName.tests.test_one
Of course, you can mix the two approaches, so that your script will work no matter how it is called:
if __name__ == '__main__': if __package__ is None: import sys from os import path sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) ) import api else: import ..api.api
This method does not require setuptools, path hacks, additional command line arguments, or specifying the top level of the package in every single file of your project.
Just make a script in the parent directory of whatever your are calling to be your
__main__ and run everything from there. For further explanation continue reading.
This can be accomplished without hacking a new path together, extra command line args, or adding code to each of your programs to recognize its siblings.
The reason this fails as I believe was mentioned before is the programs being called have their
__name__ set as
__main__. When this occurs the script being called accepts itself to be on the top level of the package and refuses to recognize scripts in sibling directories.
However, everything under the top level of the directory will still recognize ANYTHING ELSE under the top level. This means the ONLY thing you have to do to get files in sibling directories to recognize/utilize each other is to call them from a script in their parent directory.
Proof of Concept
In a dir with the following structure:
. |__Main.py | |__Siblings | |___sib1 | | | |__call.py | |___sib2 | |__callsib.py
Main.py contains the following code:
import sib1.call as call def main(): call.Call() if __name__ == '__main__': main()
import sib2.callsib as callsib def Call(): callsib.CallSib() if __name__ == '__main__': Call()
and sib2/callsib.py contains:
def CallSib(): print("Got Called") if __name__ == '__main__': CallSib()
If you reproduce this example you will notice that calling
Main.py will result in “Got Called” being printed as is defined in
sib2/callsib.py even though
sib2/callsib.py got called through
sib1/call.py. However if one were to directly call
sib1/call.py (after making appropriate changes to the imports) it throws an exception. Even though it worked when called by the script in its parent directory, it will not work if it believes itself to be on the top level of the package.
For readers in 2021: If you’re not confident with
pip install -e :
Consider this hierarchy, as recommended by an answer from Relative imports in Python 3:
MyProject ??? src ? ??? bot ? ? ??? __init__.py ? ? ??? main.py ? ? ??? sib1.py ? ??? mod ? ??? __init__.py ? ??? module1.py ??? main.py
The content of
main.py, which is the starting point and we use absolute import (no leading dots) here:
from src.bot import main if __name__ == '__main__': main.magic_tricks()
The content of
bot/main.py, which takes advantage of explicit relative imports:
from .sib1 import my_drink # Both are explicit-relative-imports. from ..mod.module1 import relative_magic def magic_tricks(): # Using sub-magic relative_magic(in=["newbie", "pain"], advice="cheer_up") my_drink() # Do your work ...
Now here comes the reasoning:
- When executing
python MyProject/main.py, the
path/to/MyProjectis added into the
- The absolute import
import src.botwill read it.
from ..modpart means it will go up one level to
- Can we see it? YES, since
path/to/MyProjectis added into the
- Can we see it? YES, since
So the point is:
We should put the main script next to
MyProject/src, since that when doing relative-referencing, we won’t go out of the
src, and the absolute import
import src. provides the just-fit scope for us: the
You need to look to see how the import statements are written in the related code. If
examples/example_one.py uses the following import statement:
…then it expects the root directory of the project to be in the system path.
The easiest way to support this without any hacks (as you put it) would be to run the examples from the top level directory, like this:
PYTHONPATH=$PYTHONPATH:. python examples/example_one.py
Just in case someone using Pydev on Eclipse end up here: you can add the sibling’s parent path (and thus the calling module’s parent) as an external library folder using Project->Properties and setting External Libraries under the left menu Pydev-PYTHONPATH. Then you can import from your sibling, e. g.
from sibling import some_class.
If you’re using pytest then the pytest docs describe a method of how to reference source packages from a separate test package.
The suggested project directory structure is:
setup.py src/ mypkg/ __init__.py app.py view.py tests/ __init__.py foo/ __init__.py test_view.py bar/ __init__.py test_view.py
Contents of the
from setuptools import setup, find_packages setup(name="PACKAGENAME", packages=find_packages())
Install the packages in editable mode:
pip install -e .
The pytest article references this blog post by Ionel Cristian M?rie?.
I made a sample project to demonstrate how I handled this, which is indeed another sys.path hack as indicated above. Python Sibling Import Example, which relies on:
if __name__ == '__main__': import os import sys sys.path.append(os.getcwd())
This seems to be pretty effective so long as your working directory remains at the root of the Python project.
I wanted to comment on the solution provided by np8 but I don’t have enough reputation so I’ll just mention that you can create a setup.py file exactly as they suggested, and then do
pipenv install --dev -e . from the project root directory to turn it into an editable dependency. Then your absolute imports will work e.g.
from api.api import foo and you don’t have to mess around with system-wide installations.
in your main file add this:
import sys import os sys.path.append(os.path.abspath(os.path.join(__file__,mainScriptDepth)))
mainScriptDepth = the depth of the main file from the root of the project.
Here is your case
mainScriptDepth = "../../". Then you can import by specifying the path (
from api.api import * ) from the root of your project.
for the main question:
call sibling folder as module:
from .. import siblingfolder
call a_file.py from sibling folder as module:
from ..siblingfolder import a_file
call a_function inside a file in sibling folder as module:
from..siblingmodule.a_file import func_name_exists_in_a_file
The easiest way.
go to lib/site-packages folder.
if exists ‘easy_install.pth’ file, just edit it and add your directory that you have script that you want make it as module.
if not exists, just make it one…and put your folder that you want there
after you add it…, python will be automatically perceive that folder as similar like site-packages and you can call every script from that folder or subfolder as a module.
i wrote this by my phone, and hard to set it to make everyone comfortable to read.
First, you should avoid having files with the same name as the module itself. It may break other imports.
When you import a file, first the interpreter checks the current directory and then searchs global directories.
tests you can call:
from ..api import api
Now, if you want to access about.py module in the User package, from the info.py module in Tech package then you have to bring the cmd (in windows) path to Project i.e.
**C:\Users\Personal\Desktop\Project>**as per the above Package example. And from this path you have to enter, python -m Package_name.module_name
For example for the above Package we have to do,
C:\Users\Personal\Desktop\Project>python -m Tech.info
- Don’t use .py extension after info module i.e. python -m Tech.info.py
- Enter this, where the siblings packages are in the same level.
- -m is the flag, to check about it you can type from the cmd python –help