Modular Programming and Modules
Modular programming is a software design technique, which is based on the general principal
of modular design. Modular design is an approach which has been proven as indispensable in engineering
even long before the first computers. Modular design means that a complex system is broken down
into smaller parts or components, i.e. modules. These components can be independently created and
tested. In many cases, they can be even used in other systems as well.
There is hardly any product nowadays, which doesn't heavily rely on modularisation, like cars, mobile phones. Computers belong to those products which are modularised to the utmost. So, what's a must for the hardware is an unavoidable necessity for the software running on the computers.
If you want to develop programs which are readable, reliable and maintainable without too much effort, you have to use some kind of modular software design. Especially if your application has a certain size. There exists a variety of concepts to design software in modular form. Modular programming is a software design technique to split your code into separate parts. These parts are called modules. The focus for this separation should be to have modules with no or just few dependencies upon other modules. In other words: Minimization of dependencies is the goal. When creating a modular system, several modules are built separately and more or less independently. The executable application will be created by putting them together.
Importing ModulesSo far we haven't explained what a Python module is. To put it in a nutshell: every file, which has the file extension .py and consists of proper Python code, can be seen or is a module! There is no special syntax required to make such a file a module. A module can contain arbitrary objects, for example files, classes or attributes. All those objects can be accessed after an import. There are different ways to import a modules. We demonstrate this with the math module:
import mathThe module math provides mathematical constants and functions, e.g. π (math.pi), the sine function (math.sin()) and the cosine function (math.cos()). Every attribute or function can only be accessed by putting "math." in front of the name:
>>> math.pi 3.141592653589793 >>> math.sin(math.pi/2) 1.0 >>> math.cos(math.pi/2) 6.123031769111886e-17 >>> math.cos(math.pi) -1.0
It's possible to import more than one module in one import statement. In this case the module names are separated by commas:
import math, randomimport statements can be positioned anywhere in the program, but it's good style to place them directly at the beginning of a program. If only certain objects of a module are needed, we can import only those:
from math import sin, piThe other objects, e.g. cos, are not available after this import. We are capable of accessing sin and pi directly, i.e. without prefixing them with "math."
Instead of explicitly importing certain objects from a module, it's also possible to import everything in the namespace of the importing module. This can be achieved by using an asterisk in the import:
>>> from math import * >>> sin(3.01) + tan(cos(2.1)) + e 2.2968833711382604 >>> e 2.718281828459045 >>>It's not recommended to use the asterisk notation in an import statement, except when working in the interactive Python shell. One reason is that the origin of a name can be quite obscure, because it can't be seen from which module it might have been imported. We will demonstrate another serious complication in the following example:
>>> from numpy import * >>> from math import * >>> print(sin(3)) 0.1411200080598672 >>> sin(3) 0.1411200080598672 >>>Let's slightly change the previous example by changing the order of the imports:
>>> from math import * >>> from numpy import * >>> print(sin(3)) 0.14112000806 >>> sin(3) 0.14112000805986721 >>>
People use the asterisk notation, because it is so convenient. It means avoiding a lot of tedious typing. Another way to shrink the typing effort consists in renaming a namespace. A good example for this is the numpy module. You will hardly find an example or a tutorial, in which they will import this module with the statement
import numpyIt's like an unwritten law to import it with
import numpy as npNow you can prefix all the objects of numpy with "np." instead of "numpy.":
>>> import numpy as np >>> np.diag([3, 11, 7, 9]) array([[ 3, 0, 0, 0], [ 0, 11, 0, 0], [ 0, 0, 7, 0], [ 0, 0, 0, 9]]) >>> np.e 2.718281828459045 >>>
Designing and Writing Modules
But how do we create modules in Python? A module in Python is just a file containing Python definitions and statements. The module name is moulded out of the file name by removing the suffix .py. For example, if the file name is fibonacci.py, the module name is fibonacci.
Let's turn our Fibonacci functions into a module. There is hardly anything to be done, we just save the following code in the file fibonacci.py:
def fib(n): if n == 0: return 0 elif n == 1: return 1 else: return fib(n-1) + fib(n-2) def ifib(n): a, b = 0, 1 for i in range(n): a, b = b, a + b return aThe newly created module "fibonacci" is ready for use now. We can import this module like any other module in a program or script. We will demonstrate this in the following interactive Python shell:
>>> import fibonacci >>> fibonacci.fib(7) 13 >>> fibonacci.fib(20) 6765 >>> fibonacci.ifib(42) 267914296 >>> fibonacci.ifib(1000) 43466557686937456435688527675040625802564660517371780402481729089536555417949051890403879840079255169295922593080322634775209689623239873322471161642996440906533187938298969649928516003704476137795166849228875 >>>
Don't try to call the recursive version of the Fibonacci function with large arguments like we did with the iterative version. A value like 42 is already too large. You will have to wait for a long time!
As you can easily imagine: It's a pain if you have to use those functions often in your program and you always have to type in the fully qualified name, i.e. fibonacci.fib(7). One solution consists in assigning a local name to a module function to get a shorter name:
>>> fib = fibonacci.ifib >>> fib(10) 55 >>>
But it's better, if you import the necessary functions directly into your module, as we will demonstrate further down in this chapter.
More on ModulesUsually, modules contain functions or classes, but there can be "plain" statements in them as well. These statements can be used to initialize the module. They are only executed when the module is imported.
Let's look at a module, which only consists of just one statement:
print("The module is imported now!")
We save with the name "one_time.py" and import it two times in an interactive session:
>>> import one_time The module is imported now! >>> import one_time >>>We can see that it was only imported once. Each module can only be imported once per interpreter session or in a program or script. If you change a module and if you want to reload it, you must restart the interpreter again. In Python 2.x, it was possible to reimport the module by using the built-in reload, i.e.reload(modulename):
$ python Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import one_time The module is imported now! >>> reload(one_time) The module is imported now!This is not possible anymore in Python 3.x.
You will cause the following error:
>>> import one_time The module is imported now! >>> reload(one_time) Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'reload' is not defined >>>
Since Python 3.0 the reload built-in function has been moved into the imp standard library module. So it's still possible to reload files as before, but the functionality has to be imported. You have to execute an "import imp" and use imp.reload(my_module). Alternatively, you can use "imp import reload" and use reload(my_module).
Example with reloading the Python3 way:
$ python3 Python 3.1.2 (r312:79147, Sep 27 2010, 09:57:50) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from imp import reload >>> import one_time The module is imported now! >>> reload(one_time) The module is imported now!Since version 3.4 you should use the "importlib" module, because imp.reload is marked as deprecated:
>>> from importlib import reload >>> import one_time The module is imported now! >>> reload(one_time) The module is imported now!A module has a private symbol table, which is used as the global symbol table by all functions defined in the module. This is a way to prevent that a global variable of a module accidentally clashes with a user's global variable with the same name. Global variables of a module can be accessed with the same notation as functions, i.e. modname.name
A module can import other modules. It is customary to place all import statements at the beginning of a module or a script.
Importing Names from a Module DirectlyNames from a module can directly be imported into the importing module's symbol table:
>>> from fibonacci import fib, ifib >>> ifib(500) 1 1 2 3 5 8 13 21 34 55 89 144 233 377This does not introduce the module name from which the imports are taken in the local symbol table. It's possible but not recommended to import all names defined in a module, except those beginning with an underscore "_":
>>> from fibonacci import * >>> fib(500) 1 1 2 3 5 8 13 21 34 55 89 144 233 377This shouldn't be done in scripts but it's possible to use it in interactive sessions to save typing.
Executing Modules as ScriptsEssentially a Python module is a script, so it can be run as a script:
python fibo.pyThe module which has been started as a script will be executed as if it had been imported, but with one exception: The system variable __name__ is set to "__main__". So it's possible to program different behaviour into a module for the two cases. With the following conditional statement the file can be used as a module or as a script, but only if it is run as a script the method fib will be started with a command line argument:
if __name__ == "__main__": import sys fib(int(sys.argv))If it is run as a script, we get the following output:
$ python fibo.py 50 1 1 2 3 5 8 13 21 34If it is imported, the code in the if block will not be executed:
>>> import fibo >>>
Renaming a Namespace
While importing a module, the name of the namespace can be changed:
>>> import math as mathematics >>> print(mathematics.cos(mathematics.pi)) -1.0After this import there exists a namespace mathematics but no namespace math.
It's possible to import just a few methods from a module:
>>> from math import pi,pow as power, sin as sinus >>> power(2,3) 8.0 >>> sinus(pi) 1.2246467991473532e-16
Kinds of Modules
There are different kind of modules:
- Those written in Python
They have the suffix: .py
- Dynamically linked C modules
Suffixes are: .dll, .pyd, .so, .sl, ...
- C-Modules linked with the Interpreter:
It's possible to get a complete list of these modules:
import sys print(sys.builtin_module_names)An error message is returned for Built-in-Modules.
Module Search Path
If you import a module, let's say "import abc", the interpreter searches for this module in the following locations and in the order given:
- The directory of the top-level file, i.e. the file being executed.
- The directories of PYTHONPATH, if this global variable is set.
- standard installation path Linux/Unix e.g. in /usr/lib/python3.4.
>>> import numpy >>> numpy.__file__ '/usr/lib/python3/dist-packages/numpy/__init__.py' >>> import random >>> random.__file__ '/usr/lib/python3.4/random.py' >>>The __file__ attribute doesn't always exists. This is the case with modules which are statically linked C libraries.
>>> import math >>> math.__file__ Traceback (most recent call last): File "<stdin>", line 1, in
AttributeError: 'module' object has no attribute '__file__' >>>
Content of a Module
With the built-in function dir() and the name of the module as an argument, you can list all valid attributes and methods for that module.
>>> import math >>> dir(math) ['__doc__', '__loader__', '__name__', '__package__', '__spec__', 'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'ceil', 'copysign', 'cos', 'cosh', 'degrees', 'e', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 'factorial', 'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'hypot', 'isfinite', 'isinf', 'isnan', 'ldexp', 'lgamma', 'log', 'log10', 'log1p', 'log2', 'modf', 'pi', 'pow', 'radians', 'sin', 'sinh', 'sqrt', 'tan', 'tanh', 'trunc'] >>>Calling dir() without an argument, a list with the names in the current local scope is returned:
>>> import math >>> cities = ["New York", "Toronto", "Berlin", "Washington"] >>> dir() ['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'cities', 'math'] >>>It's possible to get a list of the Built-in functions, exceptions, and other objects by importing the builtins module:
>>> import builtins >>> dir(builtins) ['ArithmeticError', 'AssertionError', 'AttributeError', 'BaseException', 'BlockingIOError', 'BrokenPipeError', 'BufferError', 'BytesWarning', 'ChildProcessError', 'ConnectionAbortedError', 'ConnectionError', 'ConnectionRefusedError', 'ConnectionResetError', 'DeprecationWarning', 'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'False', 'FileExistsError', 'FileNotFoundError', 'FloatingPointError', 'FutureWarning', 'GeneratorExit', 'IOError', 'ImportError', 'ImportWarning', 'IndentationError', 'IndexError', 'InterruptedError', 'IsADirectoryError', 'KeyError', 'KeyboardInterrupt', 'LookupError', 'MemoryError', 'NameError', 'None', 'NotADirectoryError', 'NotImplemented', 'NotImplementedError', 'OSError', 'OverflowError', 'PendingDeprecationWarning', 'PermissionError', 'ProcessLookupError', 'ReferenceError', 'ResourceWarning', 'RuntimeError', 'RuntimeWarning', 'StopIteration', 'SyntaxError', 'SyntaxWarning', 'SystemError', 'SystemExit', 'TabError', 'TimeoutError', 'True', 'TypeError', 'UnboundLocalError', 'UnicodeDecodeError', 'UnicodeEncodeError', 'UnicodeError', 'UnicodeTranslateError', 'UnicodeWarning', 'UserWarning', 'ValueError', 'Warning', 'ZeroDivisionError', '_', '__build_class__', '__debug__', '__doc__', '__import__', '__loader__', '__name__', '__package__', '__spec__', 'abs', 'all', 'any', 'ascii', 'bin', 'bool', 'bytearray', 'bytes', 'callable', 'chr', 'classmethod', 'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod', 'enumerate', 'eval', 'exec', 'exit', 'filter', 'float', 'format', 'frozenset', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'isinstance', 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'map', 'max', 'memoryview', 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property', 'quit', 'range', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'vars', 'zip'] >>>
PackagesIt's possible to put several modules into a Package. A package is basically a directory with Python files.
A package is imported like a "normal" module.
Each directory inside of the Python path, which should function as a package, needs to contain a file named __init__.py,
Otherwise the package can't be used as a package, i.e. it can't be imported! We will demonstrate with a very simple example how to create a package with some Python modules.
First of all, we need a directory. The name of this directory will be the name of the package, which we want to create. We will call our package "SimplePackage". This directory needs to contain a file with the name "__init__.py". This file can be empty, or it can contain valid Python code. This code will be executed when a package will be imported, so it can be used to initialize a package, e.g. to make sure that some other modules are imported or some values set. Now we can put into this directory all the Python files which will be the submodules of our module.
We create two simple files a.py and b.py just for the sake of filling the package with modules.
The content of a.py:
def bar(): print("Hello, function 'bar' from module 'a' calling")
The content of b.py:
def foo(): print("Hello, function 'foo' from module 'b' calling")We can import this module in the following way, if we want to use the modules a and b:
>>> from SimplePackage import a, b >>> a.bar() Hello, function 'bar' from module 'a' calling >>> b.foo() Hello, function 'foo' from module 'b' calling >>>It's not possible to access a and b, if you import only SimplePackage:
>>> import SimplePackage >>> SimplePackage.a.bar() Traceback (most recent call last): File "<stdin>", line 1, inWe can use the file __init__.py to automatically import a and b as well, when we import SimplePackage. We add the following lines to the empty file __init__.py:
AttributeError: 'module' object has no attribute 'a' >>> SimplePackage.a Traceback (most recent call last): File "<stdin>", line 1, in AttributeError: 'module' object has no attribute 'a' >>>
import SimplePackage.a import SimplePackage.bNow it works:
>>> import SimplePackage >>> SimplePackage.a.bar() Hello, function 'bar' from module 'a' calling