1. Answers to frequently asked questions for comp.programming.threads: Part 1 of 1

Last modified: Fri Oct 4 14:42:35 1996
$Revision: 1.7 $

0. Table of contents

2. Introduction

The FAQ is posted monthly to comp.programming.threads, in multiple parts. It is also available on the World-Wide Web, at <URL: http://www.serpentine.com/~bos/threads-faq>. You may prefer to browse the FAQ on the Web rather than on Usenet, as it contains many useful hyperlinks (and tables are readable, which is unfortunately not the case for the text version).

2.1. Reader contributions and comments

While I am more than happy to include submissions of material for the FAQ if they seem appropriate, it would make my life a lot easier if such text were proof-read in advance, and kept concise. I don't have as much time as I would like to digest 15K text files and summarise them in three paragraphs for inclusion here. If you are interested in contributing material, please see the to-do list at the end of part 3 of the FAQ.

2.2. How to read this FAQ

Wherever it may not otherwise be obvious that a particular section refers only to some families or implementations, you will find one or more of the following key words to help you.

Key	Implementation
DCE	OSF/DCE threads (POSIX draft 4)
OS/2	IBM OS/2 threads
POSIX	POSIX 1003.1c-1995 standard threads
UI	Unix International threads
Unix	Of general relevance to Unix users
WIN32	Microsoft Win32 API threads

2.3. Acknowledgments and caveats

3. What are threads?

Why are some phrases above in quotes? In a typical process in which multiple threads exist, zero or more threads may actually be running at any one time. This depends on the number of CPUs the computer on which the process is running, and also on how the threads system is implemented. A machine with n CPUs can, intuitively enough, run no more than n threads in parallel, but it may give the appearance of running many more than n "simultaneously", by sharing the CPUs among threads.

3.1. Why are threads interesting?

3.2. A little history

4. What are the main families of threads?

• POSIX-style threads, which generally run on Unix systems.
• Microsoft-style threads, which generally run on PCs.

4.1. POSIX-style threads

• "Real" POSIX threads, based on the IEEE POSIX 1003.1c-1995 (also known as the ISO/IEC 9945-1:1996) standard, part of the ANSI/IEEE 1003.1, 1996 edition, standard. POSIX implementations are, not surprisingly, the emerging standard on Unix systems.
  • POSIX threads are usually referred to as Pthreads.
  • You will often see POSIX threads referred to as POSIX.1c threads, since 1003.1c is the section of the POSIX standard that deals with threads.
  • You may also see references to draft 10 of POSIX.1c, which became the standard.
• DCE threads are based on draft 4 (an early draft) of the POSIX threads standard (which was originally named 1003.4a, and became 1003.1c upon standardisation). You may find these on some Unix implementations.
• Unix International (UI) threads, also known as Solaris threads, are based on the Unix International threads standard (a close relative of the POSIX standard). The only major Unix variants that support UI threads are Solaris 2, from Sun, and UnixWare 2, from SCO.

Those few tardy Unix vendors who do not yet ship POSIX threads implementations are expected to do so "real soon now". If you are developing multithreaded applications from scratch on Unix, you would do well to use POSIX threads.

4.2. Microsoft-style threads

• WIN32 threads are the standard threads on Microsoft Windows 95 and Windows NT.
• OS/2 threads are the standard threads on OS/2, from IBM.

4.3. Others

5. Some terminology

5.1. (DCE, POSIX, UI) Async safety

5.2. Asynchronous and blocking system calls

Asynchronous system calls are generally much harder for the programmer to deal with than blocking calls.

5.3. Context switch

5.4. Critical section

If other threads can access these data during a critical section, your program may not behave correctly. This may cause it to crash, lock up, produce incorrect results, or do just about any other unpleasant thing you care to imagine.

Other threads are generally denied access to inconsistent data during a critical section (usually through use of locks). If some of your critical sections are too long, however, it may result in your code performing poorly.

5.5. Lightweight process

Several operating systems allow lightweight processes to be "bound" to particular CPUs; this guarantees that those threads will only execute on the specified CPUs.

5.6. MT safety

Wrapping a global lock around MT-unsafe code will generally let you call it from within a multithreaded program, but since this does not permit concurrent access to that code, it is not considered to make it MT-safe.

If you are trying to write MT-safe code using POSIX threads, you need to worry about a few issues such as dealing correctly with locks across calls to fork(2) (if you are wondering what to do, read about the pthread_atfork(3) library call).

5.7. Protection boundary

The canonical example of a protection boundary on most modern systems is that between processes and the kernel. The kernel is protected from processes, so that they can only examine or change its internal state in certain strictly-defined ways.

Protection boundaries also exist between individual processes on most modern systems. This prevents one buggy or malicious process from wreaking havoc on others.

Why are protection boundaries interesting? Because transferring control across them is expensive; it takes a lot of time and work.

5.8. Scheduling

6. What are the different kinds of threads?

• User-space threads, and
• Kernel-supported threads.

6.1. Architectural differences

Kernel-supported threads fall into two classes.

• In a "pure" kernel-supported system, the kernel is responsible for scheduling all threads.
• Systems in which the kernel cooperates with a user-level library to do scheduling are known as two-level, or hybrid, systems. Typically, the kernel schedules LWPs, and the user-level library schedules threads onto LWPs.

6.2. Performance differences

6.3. Potential problems with functionality

This problem also exists with two-level kernel-supported threads, though it is not as acute as for user-level threads. What usually happens here is that system calls block entire LWPs. This means that if more threads exist than do LWPs and all of the LWPs are blocked in system calls, then other threads that could potentially make forward progress are prevented from doing so.

The Solaris threads library provides a reasonable solution to this problem. If the kernel notices that all LWPs in a process are blocked, it sends a signal to the process. This signal is caught by the user-level threads library, which can create another LWP so that the process will continue to make progress.

7. Where can I find books on threads?

7.1. POSIX-style threads

Bil Lewis and Daniel J. Berg, Threads Primer. SunSoft Press, ISBN 0-13-443698-9. <URL: http://www.sun.com/smi/ssoftpress/books/Lewis/Lewis.html>

This is a good introduction to programming with threads for programmers and managers. It concentrates on UI and POSIX threads, but also covers use of OS/2 and WIN32 threads.

Steve Kleiman, Devang Shah and Bart Smaalders, Programming With Threads. SunSoft Press, ISBN 0-13-172389-8. <URL: http://www.sun.com/smi/ssoftpress/books/Kleiman/Kleiman.html>

This book goes into considerably greater depth than the other SunSoft Press offering, and is recommended for the working programmer who expects to deal with threads on a day-to-day basis. It includes many detailed examples.

Charles J. Northrup, Programming With Unix Threads. John Wiley & Sons, ISBN 0-471-13751-0. <URL: http://www.wiley.com/compbooks/catalog/14/13751-0.html>

This book details the UI threads interface, focusing mostly on the Unixware implementation. This is an introductory book.

7.2. Microsoft-style threads

Len Dorfman, Marc J. Neuberger, Effective Multithreading with OS/2. Publisher and ISBN unknown.

This book covers the OS/2 threads API and contains many examples, but doesn't have much by way of concepts.

Thuan Q. Pham, Pankaj K. Garg, Multithreaded Programming with Windows NT. Prentice Hall, ISBN 0-131-20643-5. <URL: http://www.prenhall.com/013/120642/12064-2.html>

Not surprisingly, this book focuses on WIN32 threads, but it also mentions other libraries in passing. It also deals with some relatively advanced topics, and has a thorough bibliography.

7.3. Books on implementations

Curt Schimmel, Unix Systems for Modern Architectures. Addison-Wesley, ISBN 0-201-63338-8. <URL: http://www.aw.com/cp/schimmel.html>

This book gives a lucid account of the work needed to get Unix (or, for that matter, more or less anything else) working on a modern system that incorporates multiple processors, each with its own cache. While it has some overlap with the Vahalia book (see below), it has a smaller scope, and thus deals with shared topics in more detail.

Uresh Vahalia, Unix Internals: the New Frontiers. Prentice Hall, ISBN 0-13-101908-2. <URL: http://www.prenhall.com/013/101907/10190-7.html>

This is the best kernel internals book currently available. It deals extensively with building multithreaded kernels, implementing LWPs, and scheduling on multiprocessors. Given a choice, I would buy both this and the Schimmel book.

Ben Catanzaro, Multiprocessor System Architectures. SunSoft Press, ISBN 0-13-089137-1. <URL: http://www.sun.com/smi/ssoftpress/books/Catanzaro/Catanzaro.html>

I don't know much about this book, but it deals with both the hardware and software (kernel and user) architectures used to put together modern multiprocessor systems.

7.4. The POSIX threads standard

Unless you are implementing a POSIX threads package, you should not ever need to look at the POSIX threads standard. It is the last place you should look if you wish to learn about threads!

Neither IEEE nor ISO makes standards available for free; please do not ask whether the POSIX threads standard is available on the Web. It isn't.

8. Where can I obtain training on using threads?

9. (Unix) Are there any freely-available threads packages?

• Xavier Leroy <xleroy@inria.fr> has written a POSIX threads implementation for Linux 2.x that uses pure kernel-supported threads. While the package is currently in alpha testing, it is allegedly very stable. For more information, see <URL: http://pauillac.inria.fr/~xleroy/linuxthreads>.
• Michael T. Peterson <mtp@big.aa.net> has written a user-space POSIX and DCE threads package for Intel-based Linux systems; it is called PCthreads. See <URL: http://www.aa.net/~mtp/PCthreads.html> for more information.
• Christopher Provenzano <proven@mit.edu> has written a fairly portable implementation of draft 8 of the POSIX threads standard. See <URL: http://www.mit.edu:8001/people/proven/pthreads.html> for further details. Note: as far as I can see, development of this library has halted (at least temporarily), and it still contains many serious bugs.
• Georgia Tech's OS group has a fairly portable user-level threads implementation of the Mach C threads package. It is called Cthreads, and can be found at <URL: ftp://ftp.cc.gatech.edu/pub/groups/systems/Falcon/cthreads_distribution.tar.gz>.
• Frank Müller, of the POSIX / Ada-Runtime Project (PART) has made available an implementation of draft 6 of the POSIX 1003.4a Pthreads specification, which runs under SunOS 4, Solaris 2.x, SCO Unix, FreeBSD and Linux. For more information, see <URL: file://ftp.cs.fsu.edu/pub/PART/PTHREADS/pthreads_ANNOUNCE>.
• Elan Feingold has written a threads package called ethreads; I don't know anything about it, other than that it is available from <URL: ftp://frmap711.mathp7.jussieu.fr/pub/scratch/rideau/misc/threads/ethreads/ethreads.tgz>.
• QuickThreads is a toolkit for building threads packages, written by David Keppel <pardo@cs.washington.edu>. It is available from <URL: ftp://ftp.cs.washington.edu/pub/qt-001.tar.Z>, with an accompanying tech report at <URL: ftp://ftp.cs.washington.edu/tr/1993/05/UW-CSE-93-05-06.PS.Z>. The code as distributed includes ports for the Alpha, x86, 88000, MIPS, SPARC, VAX, and KSR1.

10. (DCE, POSIX, UI) Why does my threaded program not handle signals sensibly?

When it comes to dealing with signals, the best thing you can do is create a thread whose sole purpose is to handle signals for the entire process. This thread should loop calling sigwait(2); this allows it to deal with signals synchronously. You should also make sure that all threads (including the one that calls sigwait) have the signals you are interested in handling blocked. Handling signals synchronously in this way greatly simplifies things.

Note, also, that sending signals to other threads within your own process is not a friendly thing to do, unless you are careful with signal masks. For an explanation, see the section on asynchronous cancellation.

Finally, using sigwait and installing signals handlers for the signals you are sigwaiting for is a bad idea.

11. (DCE?, POSIX) Why does everyone tell me to avoid asynchronous cancellation?

Code that can deal sensibly with asynchronous cancellation requests is not referred to as async-safe; that means something else (see the terminology section of the FAQ). You won't see much code around that handles asynchronous cancellation requests properly, and you shouldn't try write any of your own unless you have compelling reasons to do so. Deferred cancellation is your friend.

12. Why are reentrant library and system call interfaces good?

While this approach mostly works, it provides terrible performance. For functions that maintain state across multiple invocations (e.g. strtok() and friends), this approach simply doesn't work at all, hence the existence of "_r" interfaces on many Unix systems (see below).

A better solution is to ensure that library calls can safely be performed by multiple threads at once.

12.1. (DCE, POSIX, UI) When should I use thread-safe "_r" library calls?

Unfortunately, the supersets that different vendors support do not necessarily overlap, so you can only safely use the standard POSIX-mandated functions. The thread-safe routines are conceptually "cleaner" than their stateful counterparts, though, so it is good practice to use them wherever and whenever you can.

13. (POSIX) How can I perform a join on any thread?

Unix programmers are used to being able to call wait() in such a way that it will return when "any" process exits, but expecting this to work for threads can cause confusion for programmers trying to use threads. The important thing to note here is that Unix processes are based around a notion of parent and child; this is a notion that is not present in most threads systems. Since threads don't contain this notion, joining on "any" thread could have the undesirable effect of having the join return once a completely unrelated thread happened to exit.

In many (perhaps even most) threaded applications, you do not want to be able to join with any thread in your process. Consider, for example, a library call that one of your threads might make, which in its turn might start a few threads and try to join on them. If another of your threads, joining on "any" thread, happened to join on one of the library call's threads, that would lead to incorrect program behaviour.

If you want to be able to join on any thread so that, for example, you can keep track of the number of running threads, you can achieve the same functionality by starting detached threads and having them decrememnt a (suitably locked, of course) counter as they exit.

14. (DCE, UI, POSIX) After I create a certain number of threads, my program crashes

15. Where can I find POSIX thread benchmarks?

16. Does any DBMS vendor provide a thread-safe interface?

If you are still using an older release of any of these products, it is possible to hack together a set of intermediate thread-safe interfaces to your database if you really need it, but this requires a moderately large amount of work.

17. Why is my threaded program running into performance problems?

• You can probably discount the performance of the threads package you are using almost straight away, unless it is a user-level package. If so, you may want to try to find out whether your whole process is blocking when you execute certain system calls. Otherwise, you should look at your own code unless you have a very strong reason for believing that there may be a problem with your threads package.
• Look at the granularity of your locks. If a single lock protects too much data for which there is contention, you will needlessly serialise your code. On the other hand, if your locks protect very small amounts of data, you will spend too much time obtaining and releasing locks. If your vendor is worth their salt, they will have a set of tools available that allow you to profile your program's behaviour and look for areas of high contention for locks. Tools of this kind are invaluable.

18. What tools will help me to program with threads?

• The TNF Tools are a set of extensible tools that allow users to gather and analyse trace information from the Solaris kernel and from user processes and threads. They can be used to correlate user and kernel thread activity, and also to determine such things as lock hold times. They are available for free from Sun; for more information, see <URL: http://opcom.sun.ca/toolpages/tnftools.html>.
• The debugger that comes with the DevPro compiler set from Sun understands threads.
• GDB nominally understands threads, but only supports them (and in a flaky way) under some versions of Irix and a few other systems (mostly embedded machines).

19. What operating systems provide threads?

1. OpenVMS 7.0 for Alpha shipped with kernel threads support disabled by default. The "mixed" threads model can be turned on by setting the MULTITHREAD sysgen parameter. With patch kit, the "mixed" or "user" model is determined by the main program binary (i.e. via the linker or the threadcp qualifier) in addition to MULTITHREAD.
2. SGI ships the POSIX 1003.1c API as a patch for Irix 6.2, but it will be bundled with future revisions of the OS.

20. What about other threads-related software?

• Douglas C. Schmidt <schmidt@cs.wustl.edu> has written a package called ACE (Adaptive Communication Environment). ACE supports multithreading on Unix and WIN32 platforms, and integrates popular IPC mechanisms (sockets, RPC, System V IPC) and a host of other features that C++ programmers will find useful. For details, see <URL: http://www.cs.wustl.edu/~schmidt/ACE.html>.

21. Where can I find other information on threads?

• The most comprehensive collection of threads-related information on the Web is at Sun's threads page, at <URL: http://www.sun.com/sunsoft/Products/Developer-products/sig/threads>.
• IBM has a thorough treatment of AIX 4.1 threads (based on POSIX draft 7) at <URL: http://developer.austin.ibm.com/sdp/library/ref/about4.1/df4threa.html>.
• Digital has a brief overview of threads in Digital UNIX at <URL: http://www.unix.digital.com/unix/smp>.
• A bibliography on threads is available at <URL: http://liinwww.ira.uka.de/bibliography/Os/threads.html>.
• Tom Wagner <wagner@cs.umass.edu> and Don Towsley have written an introductory tutorial on programming with POSIX threads at <URL: http://centaurus.cs.umass.edu/~wagner/threads_html/tutorial.html>

21.1. Articles appearing in periodicals

• An introduction to programming with threads, at <URL: http://www.sun.com/sunworldonline/swol-02-1996/swol-02-threads.html>, from SunWorld Online's February 1996 issue
• An introduction to programming with threads, at <URL: http://developer.austin.ibm.com/sdp/library/aixpert/nov94/aixpert_nov94_intrmult.html>, from AIXpert Magazine's November 1994 issue
• Porting DCE threads to AIX 4.1 (POSIX draft 7), at <URL: http://www.developer.ibm.com/sdp/library/aixpert/aug94/aixpert_aug94_PTHREADS.html>, from AIXpert Magazine's August 1994 issue
• A less thorough introduction to programming with threads, at <URL: http://developer.austin.ibm.com/sdp/library/aixpert/aug95/aixpert_aug95_thread.html>, from AIXpert Magazine's August 1995 issue
• Using signals with POSIX threads, at <URL: http://developer.austin.ibm.com/sdp/library/aixpert/aug95/aixpert_aug95_signal.html>, from AIXpert Magazine's August 1995 issue

22. Notice of copyright and permissions

These Documents are provided "as is". The information in them is not warranted to be correct; you use it at your own risk. The Author makes no representation or warranty as to the suitability of the information in These Documents, either express or implied, including but not limited to the implied warranties of fitness for a particular purpose or non-infringement. The Author shall not be liable for any damages suffered as a result of using information distributed in These Documents or any derivatives.

These Documents may be reproduced and distributed in whole or in part, subject to the following conditions:

• This copyright and permission notice must be retained on all complete or partial copies of These Articles.
• These Articles may be copied or distributed in part or in full for personal or educational use. Any translation, derivative work, or copies made for other purposes must be approved by the copyright holder before distribution, unless otherwise stated.
• If you distribute These Articles, instructions for obtaining the complete current versions of them free or at cost price must be included. Redistributors must make reasonable efforts to maintain current copies of These Articles.