__all__ in Python

As we go deeper into packages and modules, we might encounter the variable __all__ set in different _init_.py files.

The __init__.py files are the files that make Python treat the directories as some containing packages. This file prevents directories with similar names, such as strings, from hiding valid modules that might occur later on a module search path.

In the simplest case, __init__.py might be an empty file, but it may also execute the initialization code for the package or set the __all__ variable.

Therefore the __init__.py can declare the __all__ variables for a package.

A list of public objects of that module is given in the __all__ variable. It is interpreted by the import *. This variable overrides the default of hiding everything that begins with an underscore from the given namespace.

For example,

__all__ = ["a", "b"]
c = 5
a = 10


def b():
    return "b"

Now we import this in the following code.

from sample import *

print(a)  # will work fine
print(b)  # will work fine
print(c)  # will generate an error

In the above example, we have used import * to import all the public objects from the file sample.py to this file. It means that this file will import and support all the public objects of the file sample.py.

Objects a and b will be imported, and the new code will work perfectly fine where these objects are used. The problem arises in using the third object, c. That object is never imported to the new file as it is not a public object, as not a part of the __all__ variable. So this part of the code will generate an error.

There is an alternative to this. By default, Python will take responsibility to export all names that do not start with an underscore _. And one could certainly rely on this mechanism. In the Python standard library, some packages do rely on this, but to do so, they alias their imports, for instance, os as _os, sys as _sys, etc.

Using the _ convention is handier as it removes the redundancy of naming the names repeatedly. But it does add the redundancy for imports (if you have many), and it is easier to forget to do this consistently.

A lot of packages in the standard library use __all__. It makes sense to use the _ prefix convention in the place of __all__ when still in early development mode and have no users and are constantly tweaking your API. Maybe there are some users, but one has unit tests covering the API and still actively updating and adding to the API and tweaking in the development.