Python Threading - Basics

[Zecho]

If you want your application to perform several tasks at once, you can use threads. Python can handle threads, but many developers find thread programming to be very tricky. Threads let you set up a series of processes (or sub-processes) each of which can be run independently, but which can be brought back together later and/or co-ordinated as they run. For many applications, threads are overkill but on some occasions they can be useful.

Before we begin to look at the code that makes threading work, the most important feature we must look at is Python's global interpreter lock. If two or more threads were to attempt to manipulate the same object at the same time, problems would inevitably pop up. The global interpreter lock fixes this. Only one thread can perform an action at any given time. Python automatically switches between threads when it is needed.

In order to support multi threaded Python programs the interpreter regularly releases and reacquires the lock, by default every 10 bytecode instructions. This can however be changed using the sys.setcheckinterval() function. The lock is also released and reacquired around potentially blocking I/O operations like reading or writing a file, so that other threads can run while the thread that requests the I/O is waiting for the I/O operation to complete.

The Python Interpreter keeps some book keeping info per thread, for which it uses a data structure called PyThreadState. Earlier the state was stored in global variables and switching threads could cause problems. In particular, exception handling is now thread safe when the application uses sys.exc_info() to access the exception last raised in the current thread. There's one global variable left, however: the pointer to the current PyThreadState structure. While most thread packages have a way to store ``per-thread global data,'' Python's internal platform independent thread abstraction doesn't support this yet. Therefore, the current thread state must be manipulated explicitly.

The global interpreter lock is used to protect the pointer to the current thread state. When releasing the lock and saving the thread state, the current thread state pointer must be retrieved before the lock is released (since another thread could immediately acquire the lock and store its own thread state in the global variable). Conversely, when acquiring the lock and restoring the thread state, the lock must be acquired before storing the thread state pointer

[Reegan]

Python manages to get a lot done using so little. The Threading module uses the built in thread package to provide some very interesting features that would make your programming a whole lot easier. There are in built mechanisms which provide critical section locks, wait/notify locks etc. In particular we shall look at:

• Using the Thread object
• Profiling the Threaded code
• Using the Condition, Event, and Queue object.

Executing the thread is also simple. All we have to do is create an instance of our thread class and then call its start method:

import threading

class MyThread ( threading.thread ):

def run ( self ):

print 'You called my start method, yeah.'
print 'Were you expecting something amazing?'

MyThread().start()

Of course, it's no fun having just one thread. Just like us humans, threads get lonely after a while. Let's create a group of threads:

import threading

theVar = 1

class MyThread ( threading.Thread ):

def run ( self ):

global theVar
print 'This is thread ' + str ( theVar ) + ' speaking.'
print 'Hello and good bye.'
theVar = theVar + 1

for x in xrange ( 20 ):
MyThread().start()

[Modifier]

If you're going to be writing a threaded application, there are (broadly) two main approaches you can take. The example that I've shown above uses a separate thread to take each request through from beginning to end and all the threads have the same structure. An alternative strategy is to write a series of "worker" threads each of which performs a step in a multistep process, and have them passing data on to one another. It's the difference between having an employee to walk each order through your factory and having employees at a production line passing each job on.

Where you're running threads, you have to be very much aware of the effect they can have on one another. As soon as you have two workers, their work may interfere with one another and you get involved in synchronisation (locking) of objects to ensure that this doesn't happen.

Locking / synchronisation brings its own further complications in that you have to involve "deadlocks" where two threads both require two resources ... each grabs the first, then waits for the second which (because the other has it) never becomes available.

FOOTNOTES

We've used the operating system's ping process in this example program. Ping responses vary between operating systems and you may need to alter the ping command and regular expression to match the response. The example above has been tested on Fedora Core Linux.

Threading makes heavy use of Operating System capabilities and is NOT as portable (no matter what language you're programming in) than most code

[labman]

Conditions are a way of synchronizing access between multiple threads, which wait for a particular condition to be true to start any major processing. Condition Objects are a very elegant mechanism by which it is possible to implent the Producer Consumer Problem. Indeed this is true for anything in Python. Conditions take a lock object or if none is supplied creates its own RLock object.

A Thread waits for a particular condition to be true by using the wait() function, while it signals another thread by using the notify() or notifyAll() method.