Python and the Shell

Shell

Python and Shell Shell is a term, which is often used and often misunderstood. Like the shell of an egg, either hen or Python snake, or a mussel, the shell in computer science is generally seen as a piece of software that provides an interface for a user to some other software or the operating system. So the shell can be an interface between the operating system and the services of the kernel of this operating system. But a web browser or a program functioning as an email client can be seen as shell as well.

Understanding this, it's obvious that a shell can be either But in most cases the term shell is used as a synonym for a command line interface (CLI). The best known and most often used shells under Linux and Unix are the Bourne-Shell, C-Shell or Bash shell. The Bourne shell (sh) was modelled after the Multics shell, and is the first Unix shell.
Most operating system shells can be used in both interactive and batch mode.

System programming

System programming (also known as systems programming) stands for the activity of programming system components or system software. System programming provides software or services to the computer hardware, while application programming produces software which provides tools or services for the user.

"System focused programming" as it is made possible with the aid of the sys and the os module, serves as an abstraction layer between the application, i.e. the Python script or program, and the operating system, e.g. Linux or Microsoft Windows. By means of the abstraction layer it is possible to implement platform independent applications in Python, even if they access operating specific functionalities.

Therefore Python is well suited for system programming, or even platform independent system programming. The general advantages of Python are valid in system focused programming as well:

The os Module

The os module is the most important module for interacting with the operating system. The os module allows platform independent programming by providing abstract methods. Nevertheless it is also possible by using the system() and the exec*() function families to include system independent program parts. (Remark: The exec*()-Functions are introduced in detail in our chapter "Forks and Forking in Python")
The os module provides various methods, e.g. the access to the file system.

Executing Shell scripts with os.system()

It's not possible in Python to read a character without having to type the return key as well. On the other hand this is very easy on the Bash shell. The Bash command "read -n 1 waits for a key (any key) to be typed. If you import os, it's easy to write a script providing getch() by using os.system() and the Bash shell. getch() waits just for one character to be typed without a return:
import os
def getch():
     os.system("bash -c \"read -n 1\"")
 
getch()
The script above works only under Linux. Under Windows you will have to import the module msvcrt. Pricipially we only have to import getch() from this module.
So this is the Windows solution of the problem:
from msvcrt import getch
The following script implements a platform independent solution to the problem:
import os, platform
if platform.system() == "Windows":
    import msvcrt
def getch():
    if platform.system() == "Linux":
        os.system("bash -c \"read -n 1\"")
    else:
        msvcrt.getch()

print("Type a key!")
getch()
print("Okay")
The previous script harbours a problem. You can't use the getch() function, if you are interested in the key which has been typed, because os.system() doesn't return the result of the called shell commands.
We show in the following script, how we can execute shell scripts and return the output of these scripts into python by using os.popen():
>>> import os
>>> dir = os.popen("ls").readlines()
>>> print dir
['curses.py\n', 'curses.pyc\n', 'errors.txt\n', 'getch.py\n', 'getch.pyc\n', 'more.py\n',
'numbers.txt\n', 'output.txt\n', 'redirecting_output.py\n', 'redirecting_stderr2.py\n', 
'redirecting_stderr.py\n', 'streams.py\n',  'test.txt\n']
>>> 
The output of the shell script can be read line by line, as can be seen in the following example:
import os

command = " "
while (command != "exit"):
    command = raw_input("Command: ")
    handle = os.popen(command)
    line = " "
    while line:
        line = handle.read()
        print line
    handle.close()

print "Ciao that's it!"

subprocess Module

The subprocess module is available since Python 2.4.
It's possible to create spawn processes with the module subprocess, connect to their input, output, and error pipes, and obtain their return codes.
The module subprocess was created to replace various other modules:

Working with the subprocess Module

Instead of using the system method of the os-Module
os.system('touch xyz')
we can use the the Popen() command of the subprocess Module. By using Popen() we are capable to get the output of the script:
>>> x = subprocess.Popen(['touch', 'xyz'])
>>> print x

>>> x.poll()
0
>>> x.returncode
0
The shell command cp -r xyz abc can be send to the shell from Python by using the Popen() method of the subprocess-Module in the following way:
p = subprocess.Popen(['cp','-r', "xyz", "abc"])
There is no need to escape the Shell metacharacters like $, > usw..
If you want to emulate the behaviour of os.system, the optional parameter shell has to be set to true, i.e.
shell=True
and we have to use a string instead of a list:
p=subprocess.Popen("cp -r xyz abc", shell=True)

As we have said above, it is also possible to catch the output from the shell command or shell script into Python. To do this, we have to set the optional parameter stdout of Popen() to subprocess.PIPE:
>>> process = subprocess.Popen(['ls','-l'], stdout=subprocess.PIPE)
>>> print process.stdout.read()
total 132
-rw-r--r-- 1 bernd bernd   0 2010-10-06 10:03 abc
-rw-r--r-- 1 bernd bernd   0 2010-10-06 10:04 abcd
-rw-r--r-- 1 bernd bernd 660 2010-09-30 21:34 curses.py


If a shell command or shell script has been started with Popen(), the Python script doesn't wait until the shell command or shell script is finished. To wait until it is finished, you have to use the wait() method:
>>> process = subprocess.Popen(['ls','-l'], stdout=subprocess.PIPE)
>>> process.wait()
0

Functions to manipulate paths, files and directories

Function Description
getcwd() returns a string with the path of the current working directory
chdir(path) Change the current working directory to path.
Example under Windows:
>>> os.chdir("c:\Windows")
>>> os.getcwd()
'c:\\Windows'
An similiar example under Linux:
>>> import os
>>> os.getcwd()
'/home/homer'
>>> os.chdir("/home/lisa")
>>> os.getcwd()
'/home/lisa'
>>> 
getcwdu() like getcwd() but unicode as output
listdir(path) A list with the content of the directory defined by "path", i.e. subdirectories and file names.
>>> os.listdir("/home/homer")
['.gnome2', '.pulse', '.gconf', '.gconfd', '.beagle', '.gnome2_private', '.gksu.lock', 'Public', '.ICEauthority', '.bash_history', '.compiz', '.gvfs', '.update-notifier', '.cache', 'Desktop', 'Videos', '.profile', '.config', '.esd_auth', '.viminfo', '.sudo_as_admin_successful', 'mbox', '.xsession-errors', '.bashrc', 'Music', '.dbus', '.local', '.gstreamer-0.10', 'Documents', '.gtk-bookmarks', 'Downloads', 'Pictures', '.pulse-cookie', '.nautilus', 'examples.desktop', 'Templates', '.bash_logout']
>>> 
mkdir(path[, mode=0755]) Create a directory named path with numeric mode "mode", if it doesn't already exist. The default mode is 0777 (octal). On some systems, mode is ignored. If it is used, the current umask value is first masked out. If the directory already exists, OSError is raised. Parent directories will not be created, if they don't exist.
makedirs(name[, mode=511]) Recursive directory creation function. Like mkdir(), but makes all intermediate-level directories needed to contain the leaf directory. Raises an error exception if the leaf directory already exists or cannot be created.
rename(old, new) The file or directory "old" is renamed to "new" If "new" is a directory, an error will be raised. On Unix and Linux, if "new" exists and is a file, it will be replaced silently if the user has permission to do so.
renames(old, new) Works like rename(), except that it creates recursively any intermediate directories needed to make the "new" pathname.
rmdir(path) remove (delete) the directory "path". rmdir() works only, if the direcotry "path" is empty, otherwise an error is raised. To remove whole directory trees, shutil.rmdtree() can be used.

Further function and methods working on files and directories can be found in the module shutil. Amongst other possibilities it provides the possibility to copy files and directories with shutil.copyfile(src,dst).