Python Course

Advanced Topics

---

Other Shells

There is hardly a word with so many completely different meanings than the expression shell: The most widely known meaning of course is the seashell of various marine animals or the eggshell, i.e. the outer covering of a hard-shelled egg.
A shotgun shell (shotshell) is a self-contained cartridge loaded with shot designed to be fired from a shotgun. In a similiar context the word shell is also used to describe a projectile which carries an explosive payload.
Shells are also known in theatres. Here they stand for a curved, hard surface designed to reflect sound towards an audience. This kind of shells are most commonly used for orchestras, bands and choirs. Often these shells are designed to be removable, either rolling away on wheels or flying into a flyspace.
Do you know, that a shell can also be used for a rowing boat? In watercraft, a racing shell is an extremely narrow, and often disproportionately long, rowing boat specifically designed for racing or exercise.
Shells in science: An electron shell may be thought of as an orbit followed by electrons around an atom nucleus.
And of course there is this large multinational energy company called Shell.
And believe it or not, there is a computer scientist called Shell: Donald L. Shell is a computer scientist born in 1941, who designed the Shell sort sorting algorithm.

---

This website is supported by:

Linux Courses and Seminars

---

This topic in German / Deutsche Version: Python und die Shell, Systemprogrammierung

Python Courses

If you want to learn Python fast and efficiently, you should consider one of the
Python Courses at Bodenseo.

Python Tricks

"Python tricks" is a tough one, cuz the language is so clean. E.g., C makes an art of confusing pointers with arrays and strings, which leads to lotsa neat pointer tricks; APL mistakes everything for an array, leading to neat one-liners; and Perl confuses everything period, making each line a joyous adventure . (Tim Peters, Pythoneer who formulated the "Zen of Python")

Python compared to Lisp

Greenspun's "Tenth Rule of Programming" states:
Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp. Sounds complicated? Lisp is complicated. Guido van Rossum compared it - or to be precise Common Lisp and Scheme - to Python with the following words: "These languages are close to Python in their dynamic semantics, but so different in their approach to syntax that a comparison becomes almost a religious argument: is Lisp's lack of syntax an advantage or a disadvantage? It should be noted that Python has introspective capabilities similar to those of Lisp, and Python programs can construct and execute program fragments on the fly. Usually, real-world properties are decisive: Common Lisp is big (in every sense), and the Scheme world is fragmented between many incompatible versions, where Python has a single, free, compact implementation."
(excerpt from "Comparing Python to Other Languages" by Guido van Rossum)

Wizards and Magic

"Things in Python are very clear, but are harder to find than the secrets of wizards. Things in Perl are easy to find, but look like arcane spells to invoke magic."

Python and the Shell

Shell

Shell is a term, which is often used and often misunderstood. Like the shell of an egg, either hen or Python snake, or a mussel, the shell in computer science is generally seen as a piece of software that provides an interface for a user to some other software or the operating system. So the shell can be an interface between the operating system and the services of the kernel of this operating system. But a web browser or a program functioning as an email client can be seen as shell as well.

Understanding this, it's obvious that a shell can be either

• a command-line interface (CLI)
  or
• a graphical user interface (GUI)

But in most cases the term shell is used as a synonym for a command line interface (CLI). The best known and most often used shells under Linux and Unix are the Bourne-Shell, C-Shell or Bash shell. The Bourne shell (sh) was modelled after the Multics shell, and is the first Unix shell.
Most operating system shells can be used in both interactive and batch mode.

System programming

System programming (also known as systems programming) stands for the activity of programming system components or system software. System programming provides software or services to the computer hardware, while application programming produces software which provides tools or services for the user.

"System focused programming" as it is made possible with the aid of the sys and the os module, serves as an abstraction layer between the application, i.e. the Python script or program, and the operating system, e.g. Linux or Microsoft Windows. By means of the abstraction layer it is possible to implement platform independent applications in Python, even if they access operating specific functionalities.

Therefore Python is well suited for system programming, or even platform independent system programming. The general advantages of Python are valid in system focused programming as well:

• simple and clear
• Well structured
• highly flexible

The os Module

The os module is the most important module for interacting with the operating system. The os module allows platform independent programming by providing abstract methods. Nevertheless it is also possible by using the system() and the exec*() function families to include system independent program parts. (Remark: The exec*()-Functions are introduced in detail in our chapter "Forks and Forking in Python")
The os module provides various methods, e.g. the access to the file system.

Executing Shell scripts with os.system()

It's not possible in Python to read a character without having to type the return key as well. On the other hand this is very easy on the Bash shell. The Bash command "read -n 1 waits for a key (any key) to be typed. If you import os, it's easy to write a script providing getch() by using os.system() and the Bash shell. getch() waits just for one character to be typed without a return:

import os
def getch():
     os.system("bash -c \"read -n 1\"")
 
getch()

The script above works only under Linux. Under Windows you will have to import the module msvcrt. Pricipially we only have to import getch() from this module.
So this is the Windows solution of the problem:

from msvcrt import getch

The following script implements a platform independant solution to the problem:

import os, platform
if platform.system() == "Windows":
    import msvcrt
def getch():
    if platform.system() == "Linux":
        os.system("bash -c \"read -n 1\"")
    else:
        msvcrt.getch()

print("Type a key!")
getch()
print("Okay")

The previous script harbours a problem. You can't use the getch() function, if you are interested in the key which has been typed, because os.system() doesn't return the result of the called shell commands.
We show in the following script, how we can execute shell scripts and return the output of these scripts into python by using os.popen():

>>> import os
>>> dir = os.popen("ls").readlines()
>>> print dir
['curses.py\n', 'curses.pyc\n', 'errors.txt\n', 'getch.py\n', 'getch.pyc\n', 'more.py\n',
'numbers.txt\n', 'output.txt\n', 'redirecting_output.py\n', 'redirecting_stderr2.py\n', 
'redirecting_stderr.py\n', 'streams.py\n',  'test.txt\n']
>>> 

The output of the shell script can be read line by line, as can be seen in the following example:

import os

command = " "
while (command != "exit"):
    command = raw_input("Command: ")
    handle = os.popen(command)
    line = " "
    while line:
        line = handle.read()
        print line
    handle.close()

print "Ciao, that's it!"

subprocess Module

The subprocess module is available since Python 2.4.
It's possible to create spawn processes with the module subprocess, connect to their input, output, and error pipes, and obtain their return codes.
The module subprocess was created to replace various other modules:

• os.system
• os.spawn*
• os.popen*
• popen2.*
• commands.*

Working with the subprocess Module

Instead of using the system method of the os-Module

os.system('touch xyz')

we can use the the Popen() command of the subprocess Module. By using Popen() we are capable to get the output of the script:

>>> x = subprocess.Popen(['touch', 'xyz'])
>>> print x

>>> x.poll()
0
>>> x.returncode
0

The shell command cp -r xyz abc can be send to the shell from Python by using the Popen() method of the subprocess-Module in the following way:

p = subprocess.Popen(['cp','-r', "xyz", "abc"])

There is no need to escape the Shell metacharacters like $, > usw..
If you want to emulate the behaviour of os.system, the optional parameter shell has to be set to true, i.e.

shell=True

and we have to use a string instead of a list:

p=subprocess.Popen("cp -r xyz abc", shell=True)

As we have said above, it is also possible to catch the output from the shell command or shell script into Python. To do this, we have to set the optional parameter stdout of Popen() to subprocess.PIPE:

>>> process = subprocess.Popen(['ls','-l'], stdout=subprocess.PIPE)
>>> print process.stdout.read()
total 132
-rw-r--r-- 1 bernd bernd   0 2010-10-06 10:03 abc
-rw-r--r-- 1 bernd bernd   0 2010-10-06 10:04 abcd
-rw-r--r-- 1 bernd bernd 660 2010-09-30 21:34 curses.py

If a shell command or shell script has been started with Popen(), the Python script doesn't wait until the shell command or shell script is finished. To wait until it is finished, you have to use the wait() method:

>>> process = subprocess.Popen(['ls','-l'], stdout=subprocess.PIPE)
>>> process.wait()
0

Functions to manipulate paths, files and directories

Function	Description
getcwd()	returns a string with the path of the current working directory
chdir(path)	Change the current working directory to path. Example under Windows: >>> os.chdir("c:\Windows") >>> os.getcwd() 'c:\\Windows' An similiar example under Linux: >>> import os >>> os.getcwd() '/home/homer' >>> os.chdir("/home/lisa") >>> os.getcwd() '/home/lisa' >>>
getcwdu()	like getcwd() but unicode as output
listdir(path)	A list with the content of the directory defined by "path", i.e. subdirectories and file names. >>> os.listdir("/home/homer") ['.gnome2', '.pulse', '.gconf', '.gconfd', '.beagle', '.gnome2_private', '.gksu.lock', 'Public', '.ICEauthority', '.bash_history', '.compiz', '.gvfs', '.update-notifier', '.cache', 'Desktop', 'Videos', '.profile', '.config', '.esd_auth', '.viminfo', '.sudo_as_admin_successful', 'mbox', '.xsession-errors', '.bashrc', 'Music', '.dbus', '.local', '.gstreamer-0.10', 'Documents', '.gtk-bookmarks', 'Downloads', 'Pictures', '.pulse-cookie', '.nautilus', 'examples.desktop', 'Templates', '.bash_logout'] >>>
mkdir(path[, mode=0755])	Create a directory named path with numeric mode "mode", if it doesn't already exist. The default mode is 0777 (octal). On some systems, mode is ignored. If it is used, the current umask value is first masked out. If the directory already exists, OSError is raised. Parent directories will not be created, if they don't exist.
makedirs(name[, mode=511])	Recursive directory creation function. Like mkdir(), but makes all intermediate-level directories needed to contain the leaf directory. Raises an error exception if the leaf directory already exists or cannot be created.
rename(old, new)	The file or directory "old" is renamed to "new" If "new" is a directory, an error will be raised. On Unix and Linux, if "new" exists and is a file, it will be replaced silently if the user has permission to do so.
renames(old, new)	Works like rename(), except that it creates recursively any intermediate directories needed to make the "new" pathname.
rmdir(path)	remove (delete) the directory "path". rmdir() works only, if the direcotry "path" is empty, otherwise an error is raised. To remove whole directory trees, shutil.rmdtree() can be used.

Further function and methods working on files and directories can be found in the module shutil. Amongst other possibilites it provides the possibility to copy files and directories with shutil.copyfile(src,dst).

© 2011 Bernd Klein, Bodenseo; Design by Denise Mitchinson adapted for python-course.eu by Bernd Klein