Mutex vs.

Semaphores Part 1: Semaphores

It never ceases to amaze me how often I see postings in newsgroups, etc. asking the difference between a semaphore and a mutex. Probably what baffles me more is that over 90% of the time the responses given are either incorrect or missing the key differences. The most often quoted response is that of the The Toilet Example (c) Copyright 2005, Niclas Winquist . This summarises the differences as:

A mutex is really a semaphore with value 1

No, no and no again. Unfortunately this kind of talk leads to all sorts of confusion and misunderstanding (not to mention companies like Wind River Systems redefining a mutex as a Mutual-Exclusion Semaphore now where is that wall to bang my head against?). Firstly we need to clarify some terms and this is best done by revisiting the roots of the semaphore. Back in 1965, Edsger Dijkstra, a Dutch computer scientist, introduced the concept of a binary semaphore into modern programming to address possible race conditions in concurrent programs. His very simple idea was to use a pair of function calls to the operating system to indicate entering and leaving a critical region. This was achieved through the acquisition and release of an operating system resource called a semaphore. In his original work, Dijkstra used the notation of P & V, from the Dutch words Prolagen (P), a neologism coming from To try and lower, and Verhogen (V) To raise, To increase.

With this model the first task arriving at the P(S) [where S is the semaphore] call gains access to the critical region. If a context switch happens while that task is in the critical region, and another task also calls on P(S), then that second task (and any subsequent tasks) will be blocked from entering the critical region by being put in a waiting state by the operating system. At a later point the first task is rescheduled and calls V(S) to indicate it has left the critical region. The second task will now be allowed access to the critical region. A variant of Dijkstras semaphore was put forward by another Dutchman, Dr. Carel S. Scholten. In his proposal the semaphore can have an initial value (or count) greater than one. This enables building programs where more than one resource is being managed in a given critical region. For example, a counting semaphore could be used to manage the parking spaces in a robotic parking system. The initial count would be set to the initial free parking places. Each time a place is used the count is

decremented. If the count reaches zero then the next task trying to acquire the semaphore would be blocked (i.e. it must wait until a parking space is available). Upon releasing the semaphore (A car leaving the parking system) the count is incremented by one. Scholtens semaphore is referred to as the General or Counting Semaphore, Dijkstras being known as the Binary Semaphore. Pretty much all modern Real-Time Operating Systems (RTOS) support the semaphore. For the majority, the actual implementation is based around the counting semaphore concept. Programmers using these RTOSs may use an initial count of 1 (one) to approximate to the binary semaphore. One of the most notable exceptions is probably the leading commercial RTOS VxWorks from Wind River Systems. This has two separate APIs for semaphore creation, one for the Binary semaphore (semBCreate) and another for the Counting semaphore (semCCreate). Hopefully we now have a clear understanding of the difference between the binary semaphore and the counting semaphore. Before moving onto the mutex we need to understand the inherent dangers associated with using the semaphore. These include:

Accidental release Recursive deadlock Task-Death deadlock Priority inversion Semaphore as a signal

All these problems occur at run-time and can be very difficult to reproduce; making technical support very difficult. Accidental release This problem arises mainly due to a bug fix, product enhancement or cut-and-paste mistake. In this case, through a simple programming mistake, a semaphore isnt correctly acquired but is then released.

When the counting semaphore is being used as a binary semaphore (initial count of 1 the most common case) this then allows two tasks into the critical region. Each time the buggy code is executed

the count is increment and yet another task can enter. This is an inherent weakness of using the counting semaphore as a binary semaphore. Deadlock Deadlock occurs when tasks are blocked waiting on some condition that can never become true, e.g. waiting to acquire a semaphore that never becomes free. There are three possible deadlock situations associated with the semaphore:

Recursive Deadlock Deadlock through Death Cyclic Deadlock (Deadly Embrace)

Here we shall address the first two, but shall return to the cyclic deadlock in a later posting. Recursive Deadlock Recursive deadlock can occur if a task tries to lock a semaphore it has already locked. This can typically occur in libraries or recursive functions; for example, the simple locking of malloc being called twice within the framework of a library. An example of this appeared in the MySQL database bug reporting system: Bug #24745 InnoDB semaphore wait timeout/crash deadlock waiting for itself Deadlock through Task Death What if a task that is holding a semaphore dies or is terminated? If you cant detect this condition then all tasks waiting (or may wait in the future) will never acquire the semaphore and deadlock. To partially address this, it is common for the function call that acquires the semaphore to specify an optional timeout value. Priority Inversion The majority of RTOSs use a priority-driven pre-emptive scheduling scheme. In this scheme each task has its own assigned priority. The pre-emptive scheme ensures that a higher priority task will force a lower priority task to release the processor so it can run. This is a core concept to building real-time systems using an RTOS. Priority inversion is the case where a high priority task becomes blocked for an indefinite period by a low priority task. As an example:

An embedded system contains an information bus Sequential access to the bus is protected with a semaphore.

A bus management task runs frequently with a high priority to move certain kinds of data in and out of the information bus. A meteorological data gathering task runs as an infrequent, low priority task, using the information bus to publish its data. When publishing its data, it acquires the semaphore, writes to the bus, and release the semaphore.

The system also contains a communications task which runs with medium priority. Very infrequently it is possible for an interrupt to occur that causes the (medium priority) communications task to be scheduled while the (high priority) information bus task is blocked waiting for the (low priority) meteorological data task.

In this case, the long-running communications task, having higher priority than the meteorological task, prevents it from running, consequently preventing the blocked information bus task from running.

After some time has passed, a watchdog timer goes off, notices that the data bus task has not been executed for some time, concludes that something has gone drastically wrong, and initiates a total system reset.

This well reported event actual sequence of events happened on NASA JPLs Mars Pathfinder spacecraft. Semaphore as a Signal Unfortunately, the term synchronization is often misused in the context of mutual exclusion. Synchronization is, by definition To occur at the same time; be simultaneous. Synchronization between tasks is where, typically, one task waits to be notified by another task before it can continue execution (unilateral rendezvous). A variant of this is either task may wait, called the bidirectional rendezvous. This is quite different to mutual exclusion, which is a protection mechanism. However, this misuse has arisen as the counting semaphore can be used for unidirectional synchronization. For this to work, the semaphore is created with a count of 0 (zero).

Note that the P and V calls are not used as a pair in the same task. In the example, assuming Task1 calls the P(S) it will block. When Task 2 later calls the V(S) then the unilateral synchronization takes place and both task are ready to run (with the higher priority task actually running). Unfortunately misusing the semaphore as synchronization primitive can be problematic in that it makes debugging harder and increase the potential to miss accidental release type problems, as a V(S) on its own (i.e. not paired with a P(S)) is now considered legal code.

In the next posting I shall look at how the mutex address most of the weaknesses of the semaphore.

14 Comments a Mutex vs. Semaphores Part 1: Semaphores

1.

Michael Barr says:

September 8th, 2009 at 6:38 pm

Niall, Welcome to the blogosphere! As I would expect from you, you've done a nice job above explaining some rather complex (and often misunderstood) issues. However, I have a problem with one of the examples. The robotic parking garage implementation suffers from implementation by a counting semaphore. Only a mutex is truly needed. Here's why Each of the parking spaces is an individually identifiable object. In a computer, it is analogous to a fixed-sized memory buffer. If the buffer contains data, some part of the application code has a pointer to it. If the buffer is empty, it must be tracked in a "free list" data structure. One suitable data structure is a linked list maintained within the empty buffers plus a head pointer. (But the important general point is there is always at least one piece of metadata.) The free list data structure must be protected via a mutual exclusion primitive (preferably a mutex). Gaining access to that data structure will tell the caller if there are any free parking spots. The suggested use of a counting semaphore neither (a) gets you deep enough into the implementation to pick a specific parking spot nor (b) is an alternative to the mutex, which must always be used. Thus the counting semaphore is a waste of space in the solution to that problem. Generalizing, I recommend avoiding use of the term "counting semaphore" altogether. Only mutexes and semaphores are of practical use, for data protection and signaling respectively. I've blogged a bunch on mutexes and semaphores at http://www.embeddedgurus.net/barrcode and, on the points you're discussing, highly recommend the article Mutexes and Semaphores Demystified at http://www.netrino.com/Embedded-Systems/How-To/RTOS-Mutex-Semaphore Cheers, Michael

2.

Rennie Allen says:

September 8th, 2009 at 9:39 pm

While I think I agree on where you are going with this, the argument is difficult to grasp because you seem to be comparing an implementation with an abstract functional concept.

From the functional concept perspective, a conceptual mutex (rather than a specific implementation such as pthreads) could be considered to be a semaphore with a count of one. Because the generic concept of the function of a mutex is ill-defined, I am convinced that I could write a set of cover functions for a mutex, implemented in terms of a semaphore with a count of one; and that you'd be hard pressed to show me why that doesn't provide all the attributes of a conceptual mutex function (i.e. conceptually, they are the same). Certainly, if we consider this question within the constraints of the SysV behavioral model of the semaphore and the pthreads behavioral model of a mutex, then I think we can agree that there are many behavioral details that make a SysV semaphore with a count of 1, and a pthread mutex significantly different. I think it is important to make clear though, that you are talking about SysV semaphores and pthread mutexes (you are, aren't you?) rather than the conceptual model of semaphores and mutexes.

3.

Dan says:
September 8th, 2009 at 11:01 pm

Michael Barr (former editor of Embedded Systems Programming, now president of Netrino) has a good article about the differences between mutexes & semaphores at the following location: http://www.netrino.com/node/202 Also another article discusses the "Perils of Preemption" (including issues you covered such as deadlock & priority inversion) at this link: http://www.netrino.com/Embedded-Systems/How-To/Preemption-Perils Keep up the good work. More people need to learn about such topics.

4.

Bill Dittmann says:

September 9th, 2009 at 3:16 pm

Niall, Good article. In the RTXC Quadros RTOS mutex implementation, nesting locks on the same mutex are allowed, supported, and safe. We do 2 things to make it safe. First, we remember who locked (owns) the mutex and keep a counter as it is locked and locked. The counter scheme properly tracks the scope of the mutex lock. We also make sure that only the owner can release (unlock) the mutex.

In my experiences, during development phases, it is not uncommon for a task to inadvertently release when it is doesnt even own the lock. RTXC Quadros detects this lock underflow or release by non-owner condition, e.g., a program design flaw, during runtime.

5.

Solti says:
February 1st, 2010 at 3:11 am

Hi Niall, Can I say when we are dealing with mutual exclusion problems; a binary semaphore is the same to a mutex? Suppose we use them correctly, and there is a resource which allows only one user at every time instant. There must be a get/return pair in 'user' if we use mutex, and there must also be a wait/signal pair if we use binary semaphore. So in this case, can I say they are totally the same? Thank you very much, and I really appreciate your articles. Solti

6.

Shreshtha says:
May 20th, 2010 at 7:18 am

Hi All, With such a great quality of information and discussion from industrys eminent people, this page should come in top every time mutex semaphore is searched. Wonderful discussion! From Linux perspective here is my understanding about difference of Mutex and Sem (please correct me if I am wrong anywhere) Note I find its very important that we are clear about the part of Linux we are discussing this topic Kernel Space or User Space. i) Scope The scope of mutex is within a process address space which has created it and is used for synchronization of common resource access. Whereas semaphore can be used across processes space and hence it can be used for interprocess synchronization/signaling. Hence mutex must be released by same thread which is taking it. ii) Mutex is lightweight and faster than semaphore (only valid for userspace) iii) Mutex can be acquired by same thread successfully multiple times with condition that it should

release it same number of times. Other thread trying to acquire will block. Whereas in case of semaphore if same process tries to acquire it again it blocks as it can be acquired only once. I found this tread very informative in this context http://alinux.tv/Kernel-2.6.29/mutex-design.txt Cheers, @Shreshtha19

7.

Shreshtha says:
July 12th, 2010 at 8:06 am

above link is not working. here is working one http://www.kernel.org/doc/Documentation/mutex-design.txt

Mutex vs. Semaphores Part 2: The Mutex

In Part 1 of this series we looked at the history of the binary and counting semaphore, and then went on to discuss some of the associated problem areas. In this posting I aim to show how a different RTOS construct, the mutex, may overcome some, if not all, of these weaknesses. To address the problems associated with semaphore, a new concept was developed during the late 1980s. I have struggled to find its first clear definition, but the major use of the term mutex (another neologism based around MUTual EXclusion) appears to have been driven through the development of the common programming specification for UNIX based systems. In 1990 this was formalised by the IEEE as standard IEEE Std 1003.1 commonly known as POSIX. The mutex is similar to the principles of the binary semaphore with one significant difference: the principle of ownership. Ownership is the simple concept that when a task locks (acquires) a mutex only it can unlock (release) it. If a task tries to unlock a mutex it hasnt locked (thus doesnt own) then an error condition is encountered and, most importantly, the mutex is not unlocked. If the mutual exclusion object doesnt have ownership then, irrelevant of what it is called, it is not a mutex.

The concept of ownership enables mutex implementations to address the problems discussed in part 1: 1. Accidental release 2. Recursive deadlock 3. Task-Death deadlock 4. Priority inversion 5. Semaphore as a signal

Accidental Release As already stated, ownership stops accidental release of a mutex as a check is made on the release and an error is raised if current task is not owner. Recursive Deadlock Due to ownership, a mutex can support relocking of the same mutex by the owning task as long as it is released the same number of times. Priority Inversion With ownership this problem can be addressed using one of the following priority inheritance protocols:

[Basic] Priority Inheritance Protocol Priority Ceiling Protocol

The Basic Priority Inheritance Protocol enables a low-priority task to inherit a higher-priorities tasks priority if this higher-priority task becomes blocked waiting on a mutex currently owned by the lowpriority task. The low priority task can now run and unlock the mutex at this point it is returned back to its original priority. The details of the Priority Inheritance Protocol and Priority Ceiling Protocol (a slight variant) will be covered in part 3 of this series. Death Detection If a task terminates for any reason; the RTOS can detect if that task current owns a mutex and signal waiting tasks of this condition. In terms of what happens to the waiting tasks, there are various models, but two dominate:

All tasks readied with error condition; Only one task readied; this task is responsible for ensuring integrity of critical region.

When all tasks are readied, these tasks must then assume critical region is in an undefined state. In this model no task currently has ownership of the mutex. The mutex is in an undefined state (and cannot be locked) and must be reinitialized. When only one task is readied, ownership of the mutex is passed from the terminated task to the readied task. This task is now responsible for ensuring integrity of critical region, and can unlock the mutex as normal. Mutual Exclusion / Synchronization Due to ownership a mutex cannot be used for synchronization due to lock/unlock pairing. This makes the code cleaner by not confusing the issues of mutual exclusion with synchronization. Caveat A specific Operating Systems mutex implementation may or may not support the following:

Recursion Priority Inheritance Death Detection

Review of some APIs It should be noted that many Real-Time Operating Systems (or more correctly Real-Time Kernels) do not support the concept of the mutex, only supporting the Counting Semaphore (e.g. MicroC/OS-II). [CORRECTION: The later versions of uC/OS-II do support the mutex, only the original version did not]. In this section we shall briefly examine three different implementations. I have chosen these as they represent the broad spectrum of APIs offered (Footnote 1):

VxWorks Version 5.4 POSIX Threads (pThreads) IEEE Std 1003.1, 2004 Edition Microsoft Windows Win32 Not .NET

VxWorks from Wind River Systems is among the leading commercial Real-Time Operating System used in embedded systems today. POSIX Threads is a widely supported standard, but has become more widely used due to the growth of the use of Embedded Linux. Finally Microsoft Windows common programming API, Win32 is examined. Windows CE, targeted at embedded development, supports this API. However, before addressing the APIs in detail we need to introduce the concept of a Release Order Policy. In Dijkstras original work the concept of task priorities was not part of the problem domain.

Therefore it was assumed that if more than one task was waiting on a held semaphore, when released the next task to acquire the semaphore would be chosen on a First-Come-First-Serve (First-In-FirstOut; FIFO) policy. However once tasks have priorities, the policy may be:

FIFO Priority Undefined

waiting tasks ordered by arrival time waiting tasks ordered by priority - implementation doesnt specify

VxWorks v5.4 VxWorks supports the Binary Semaphore, the Counting Semaphore and the Mutex (called the MutualExclusion Semaphore in VxWorks terminology). They all support a common API for acquiring (semTake) and releasing (semGive) the particular semaphore. For all semaphore types, waiting tasks can be queued by priority or FIFO and can have a timeout specified. The Binary Semaphore has, as expected, no support for recursion or inheritance and the taker and giver do not have to be same task. Some additional points of interest are that there is no effect of releasing the semaphore again; It can be used as a signal (thus can be created empty); and supports the idea of a broadcast release (wake up all waiting tasks rather than just the first). The Counting Semaphore, as expected, is the same as the Binary Semaphore with ability to define an initial count. The Mutual-Exclusion Semaphore is the VxWorks mutex. Only the owning task may successfully call semGive. The VxWorks mutex also has the ability to support both priority inheritance (basic priority inheritance protocol) and deletion safety. POSIX POSIX is an acronym for Portable Operating System Interface (the X has no meaning). The current POSIX standard is formally defined by IEEE Std 1003.1, 2004 Edition. The mutex is part of the core POSIX Threads (pThreads) specification (historically referred to as IEEE Std 1003.1c-1995). POSIX also supports both semaphores and priority-inheritance mutexes as part of what are called Feature Groups. Support for these Feature Groups is optional, but when an implementation claims that a feature is provided, all of its constituent parts must be provided and must comply with this specification. There are two main Feature Groups of interest, the Realtime Group and Realtime Threads Groups. The semaphore is not part of the core standard but is supported as part of the Realtime Feature Group. The Realtime Semaphore is an implementation of the Counting semaphore. The default POSIX mutex is non-recursive , has no priority inheritance support or death detection. However, the Pthreads standard allows for non-portable extensions (as long as they are tagged with -

np). A high proportion of programmers using POSIX threads are programming for Linux. Linux supports four different mutex types through non-portable extensions:

Fast mutex Recursive mutex Adaptive mutex

non-recursive and will deadlock [default] as the name implies - extra fast for mutli-processor systems

Error checking mutex non-recursive but will report error

These are extremely well covered by Chris Simmonds in his posting Mutex mutandis: understanding mutex types and attributes. Finally the Realtime Threads Feature Group adds mutex support for both priority inheritance and priority ceiling protocols. Win32 API Microsoft Windows common API is referred to as Win32. This API supports three different primitives:

Semaphore Critical Section Mutex

The counting semaphore - Mutex between threads in the same process; Recursive, no timeout, As per critical sections, but can be used by threads in different

queuing order undefined processes; Recursive, timeout, queuing order undefined The XP/Win32 mutex API does not support priority inheritance in application code, however the WinCE/Win32 API does! Win32 mutexes do have built-in death detection; if a thread terminates when holding a mutex, then that mutex is said to be abandoned. The mutex is released (with WAIT_ABANDONED error code) and a waiting thread will take ownership. Note that Critical sections do not have any form of death detection. Critical Sections have no timeout ability, whereas mutexes do. However Critical Sections support a separate function call TryEnterCriticalSection. A major weakness of the Win32 API is that the queuing model is undefined (i.e. neither Priority nor FIFO). According to Microsoft this is done to improve performance. So, what can we gather from this? First and foremost the term mutex is less well defined than the semaphore. Secondly,the actual implementations from RTOS to RTOS vary massively. I urge you to go back and look at your faviourite RTOS and work out what support, if any, you have for the mutex. Id

love to hear from people regarding mutual exclusion support (both semaphores and mutexes) for their RTOS of choice. If youd like to contact me do so at nsc(at)acm.org. Finally, Part 3 will look at a couple of problems the mutex doesnt solve, and how these can be overcome. As part of that it will review the Basic Priority Inheritance Protcol and the Prority Ceiling Protocol. At a later date I will also address the use of, and problems associted with, the semaphore being used for task synchronisation. ENDNOTES 1. Please I do not want to get into the thats not a real-time OS debate here lets save that for another day! 2. A number of people pointed out that Michael Barr (former editor of Embedded Systems Programming, now president of Netrino) has a good article about the differences between mutexes & semaphores at the following location: http://www.netrino.com/node/202. I urge you to read his posting as well. 3. Apologies about not having the atom feed sorted this should all be working now

6 Comments a Mutex vs. Semaphores Part 2: The Mutex

1.

Randell J says:
September 19th, 2009 at 6:54 am

Back In The Day Semaphores in the Amiga Exec (which was modeled on Xinu) were by your/modern terminology Mutexes with support for recursion. You can think of them (and recursive Mutexes in general) as Counting Semaphores with the counts going in the opposite direction: acquiring a "Semaphore" increases the count (if unowned (count == 0) or if you already own it). On top of that, we implemented "Read/Write Semaphores", where any number of people could acquire the "semaphore" for reading protected data, but only one could for writing to it. This was effectively the equivalent of a "read-permission" Mutex and a Write-permission mutex (though higher performance than a decomposed implementation). Roughly (pseudo-code): acquire(read): -if (I own a lock) // if recursion is allowed count++ // note: others may also own it -else if (writer_count != 0)

add self to queue -else count++ //increase count of (read) owners acquire(write): -if (I own a lock) count++ writer_count++ -else if (count != 0) add self to queue writer_count++ -else count++ writer_count++ // makes sure readers won't read while we're writing The operations for release are obvious (make sure writer_count is dealt with correctly). It can be tricky to implement recursion with this; you can't just store a handle of the 'owner' since you have multiple owners. You can extend this with queuing mods, like "a write request jumps ahead of all read requests" (note that you can only have a queue if a writer either owns the resource or is in the queue also, waiting for all the readers to release). I think we had that as an option (and I think I used an internal implementation of that in the filesystem code, which was all coded as coroutines now there's a paradigm that virtually disappeared, at least formally).

2.

Sergey Oboguev says:

November 9th, 2011 at 10:05 pm

The term was not used at user level until PPL (see below), but mutex primitive is used very extensively in VMS kernel. See VMS Internals and Data Structures, any edition (still easily findable at amazon) or VMS documentation on driver writing (there are PDFs easily findable online) and/or System Dump Analyzer documentation or, better yet, kernel sources/listings (a little harder to get, but obtainable for motivated people ).

The relevant functions are SCH$LOCKR, SCH$LOCKW, SCH$UNLOCK, SCH$LOCKRNOWAIT, SCH$LOCKWNOWAIT, SCH$LOCKWEXEC, SCH$LOCKREXEC located in module [SYS.SRC]MUTEX.MAR. Many mutex objects throughout the kernel also bear a word mutex in their name, for example I/O database mutex IOC$GL_MUTEX and so on, there are dozens of various mutexes in the system. In terms of functionality VMS kernel mutexes are actually closer to modern RWLocks since they allow both exclusive locking (writer lock) and shared locking (reader lock). They also elevate process IPL to ASTDEL (2), to prevent process deletion while a mutex is held the process can still be preempted, but control flow cannot be interrupted by ASTs and, in particular, cannot be deleted while any kernel mutex is held.

At user level, principal VMS mechanism for exclusion were ENQ/DEQ locks, which had a richer semantics than most existing locking primitives, and in addition were cluster-wide, but of course had greater overhead than grandmas futex. Besides VMS kernel mutexes, mutexes (of different design than kernel ones) were also used in user space, internally in run-time libraries for ADA and at some point CRTL, and eventually were released for developers use as a part of Parallel Processing Library which, according to the headers in the sources, dates back to 1986.

3.

Sergey Oboguev says:

November 17th, 2011 at 5:49 am

On a somewhat related curious matter, one of the big discoveries in the field of synchronization was 1990 discovery that test-and-set hogs the interconnect bus with intense traffic and that testand-test-and-set performs much better, since it spins on the cached value and does not issue interlocked instruction until the cache coherence indicates the lock may be available. The most quoted article on the issue nowadays is Andersons 1990 article The Performance of Spin Lock Alternatives for SharedMemory Multiprocessors, though it was actually preceded by other articles on the matter published in 1990. I am sure there are thousands of references to Andersons article, indeed google alone finds 16600 references on the web and 800 references in google books. Web also contains 236.000 references to test and test and set idea allegedly pioneered by the Andersons article. Yet, VMS SMP spinlock code (module [SYS.SRC]SPINLOCK.MAR) written 5 years earlier, in 1985, does just that spins on local cached value, and performs interlocked operation only after the cache is updated, with the code section prepended by the comment: The busy wait loop below assumes that a cache is present and that cache coherency is maintained on all available processors. Note that if the cache is not present and working, the busy wait is likely to impact the system performance overall by making many references to memory. I would expect many other examples like this could be found, and thus invention and use of mutex in the industry many years before it made a splash in academic publications is by no means anything too unique.

4.

Sergey Oboguev says:

November 17th, 2011 at 6:14 am

Back to the origins of the term mutex, lookup in google books suggests that the term originated in 1967, originally as a designation for binary semaphore (indeed, the very first use findable in google books appears to be as a name of semaphore variable) and then gradually acquiring a separate meaning.

The word was introduced apparently not without some resistance to the readers ears. One reviewer of J.L. Jolleys Data Study (published in 1968) writing in New Scientist (1968) complains: The basic idea is a simple one and many will be irritated by the authors use of such words as mutex, ambisubterm, homeostasis and idempotency to provide names for many quite ordinary phenomena.

Mutex vs. Semaphores Part 3 (final part): Mutual Exclusion Problems

As hopefully you can see from the previous posting, the mutex is a significantly safer mechanism to use for implementing mutual exclusion around shared resources. Nevertheless, there are still a couple of problems that use of the mutex (in preference to the semaphore) will not solve. These are:

Circular deadlock Non-cooperation

Circular Deadlock Circular deadlock, often referred to as the deadly embrace problem is a condition where two or more

tasks develop a circular dependency of mutual exclusion. Simply put, one task is blocked waiting on a mutex owned by another task. That other task is also block waiting on a mutex held by the first task. So how can this happen? Take as an example a small control system. The system is made up of three tasks, a low priority Control task, a medium priority System Identification (SI) task and a high priority Alarm task. There is an analogue input shared by the Control and the SI tasks, which is protected by a mutex. There is also an analogue output protected by a different mutex.

The Control task waits for mutexes ADC and DAC: mutex_lock (ADC); mutex_lock (DAC); /* critical section */ mutex_unlock (ADC); mutex_unlock (DAC); The SI Task waits for mutexes DAC and ADC: mutex_lock (DAC); mutex_lock (ADC); /* critical section */ mutex_unlock (DAC); mutex_unlock (ADC); Unfortunately, under certain timing conditions, this can lead to deadlock. In this example the Control task has locked the ADC, but before locking the DAC has been pre-empted by the higher priory SI task. The SI task then locks the DAC and tries to lock the ADC. The SI task is now blocked as the ADC

is already owned by the Control task. The Control task now runs and tries to lock the DAC. It is blocked as the DAC is held by the SI task. Neither task can continue until the mutex is unlocked and neither mutex can be unlocked until either task runs classic deadlock.

For circular deadlock to occur the following conditions must all be true:

A thread has exclusive use of resources (Mutual exclusion) A thread can hold on to a resource(s) whilst waiting for another resource (Hold and wait) A circular dependency of thread and resources is set up (Circular waiting) A thread never releases a resource until it is completely finished with it (No resource preemption)

These conditions can be addressed in a number of ways. For example, a design policy may stipulate that if a task needs to lock more than one mutex it must either lock all or none. Priority Ceiling Protocol With the Priority Ceiling Protocol (PCP) method each mutex has a defined priority ceiling, set to that of the highest priority task which uses the mutex. Any task using a mutex executes at its own priority until a second task attempts to acquire the mutex. At this point it has its priority raised to the ceiling value, preventing suspension and thus eliminating the hold and wait condition. In the deadlock example shown before, the significant point is when the SI task tries to lock the DAC. Before that succeeded and lead to cyclic deadlock. However with a PCP mutex, both the ADC and DAC mutex will have a ceiling priority equal to the SIs task priority. When the SI task tries to lock the DAC, then the run-time system will detect that the SIs task priority is not higher than the priority of the locked mutex ADC. The run-time system suspends the SI task without locking the DAC mutex. The control task now inherits the priority of the SI task and resumes execution.

Non-cooperation The last, but most important aspect of mutual exclusion covered in these ramblings relies on one founding principle: we have to rely on all tasks to access critical regions using mutual exclusion primitives. Unfortunately this is dependent on the design of the software and cannot be detected by the run-time system. This final problem was addressed by Tony Hoare, called the Monitor. The Monitor The monitor is a mechanism not typically supplied by the RTOS, but something the programmer tends to build (a notable exception is Ada95s protected object mechanism). A monitor simply encapsulates the shared resource and the locking mechanism into a single construct (e.g. a C++ Object that encapsulates the mutex mechanism). Access to the shared resource, then, is through a controlled interface which cannot be bypassed (i.e. the application never explicitly calls the mutex, but calls upon access functions). Finishing Off This goal of these initial postings is to demonstrate that common terms used in the real-time programming community are open to ambiguity and interpretation. Hopefully you should now be clear about the core differences between the Binary Semaphore, General (counting) Semaphore and the Mutex. The underlying difference between the Semaphores and the Mutex is the Principle of Ownership. Given the principle of ownership a particular implementation of a mutex may support Recursion, Priority inheritanceand Death Detection. ENDNOTE An aspect of the mutex I havent covered here is that many operating systems support the concept of acondition variable. A condition variable allows a task to wait on a synchronization primitive within a

critical region. The whole aspect Synchronization Patterns (e.g. semaphore as a signal) within the context of RTOSs will be the subject of my next posting.
Posted on October 5th, 2009 Feed to this thread Trackback

1 Comments a Mutex vs. Semaphores Part 3 (final part): Mutual Exclusion Problems

1.

Pete says:
November 11th, 2009 at 3:51 pm

Your description of the Priority Ceiling Protocol is incorrect. What you describe is more similar to PCP *Emulation* (also called the Priority Protect Protocol in POSIX). Real PCP doesn't raise the task's priority to the blocked task's priority (not the ceiling). It uses the ceiling to block a task on a wait/lock operation of a mutex if the locking task has a lower priority than the current priority ceiling (highest ceiling of any currently locked mutex), even if the mutex was available. It's complex, which is why the version you state is more commonly seen.

Mutex mutandis: understanding mutex types and attributes

The mutex is a simple little thing with one job to do: ensure mutual exclusion between threads. Yet Linux has up to 16 variations of the mutex, depending on which versions of the kernel and C library you have (see the table at the end). In this article I will try to explain why there are so many and how they affect your programs.

Basics
A mutex has two states: locked and unlocked. Once it has been locked by one thread any others are forced to wait until the first unlocks it again. Only the thread that locked the mutex may unlock it. With these three facts you can use a mutex to ensure exclusive access to a shared resource, such as a data structure. In the example below I have included the mutex in the structure being protected, which is good design practice

struct some_data { pthread_mutex_t data_lock; // other data items } my_data; ... pthread_mutex_lock (&my_data.data_lock); // write some values pthread_mutex_unlock (&my_data.data_lock);
Only one thread can hold data_lock at a time, ensuring that the data values are always consistent.

Complications
Mutexes are so important to the correct behavior of an application that small details about their implementation can make a big difference overall. Here are some things to consider

what is most important: speed or correct behavior? what happens if you try to lock the same mutex twice? if several threads are waiting for a mutex to be unlocked, which one should get the mutex next? is it acceptable for a high priority thread to be blocked indefinitely by a lower priority thread (leading to priority inversion)? what happens if the thread that has locked the mutex terminates without unlocking? Your response to these questions will determine the sort of mutex you need.

Types of mutex: fast, error checking, recursive and adaptive

Linux has four types of mutex. The code snippets below show how to declare and initialise the default type, which is fast. Sounds good, but what does that mean? It means that speed is preferred over correctness: there is no check that you are the owner in pthread_mutex_unlock() so any thread can unlock a fast mutex. Also it doesn't check if you have already locked the mutex, so you can deadlock yourself, and there are no checks anywhere that the mutex has been intialised correctly. You declare a mutex of default (fast) type at run-time like this

pthread_mutex_t mutex; ... pthread_mutex_init (&mutex, NULL);

or statically at compile-time like this

pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;

If you prefer correctness over speed, you need to set the type to be error checking. Error checking mutexes return EDEADLK if you try to lock the same one twice and EPERM if you unlock a mutex that isn't yours. To create such a mutex you need to initialise a muter_attr and pass it to pthread_mutex_init() like so:

pthread_mutex_t mutex; pthread_mutexattr_t attr; pthread_mutexattr_init (&attr); pthread_mutexattr_settype (&attr, PTHREAD_MUTEX_ERRORCHECK_NP); pthread_mutex_init (&mutex, &attr);
or, you can do it statically at compile-time in one line like this:

pthread_mutex_t mutex = PTHREAD_ERRORCHECK_MUTEX_INITIALIZER_NP;

Note: the _NP suffix indicates a non-portable extension to the POSIX specification. In fact the latest specification, 1003.1-2008 [2], includes most of the Linux additions so you can leave the _NP ending off. I have chosen not to because some older versions of the header files only have the _NP variants. Next is the recursive mutex, which does everything that the error checking mutex does except that you can lock the same mutex multiple times. It keeps a count of the number of times it has been locked and you must unlock it the same number of times before it becomes truly unlocked. As with the other types, you can declare and initialise one like this:

pthread_mutex_t mutex; pthread_mutexattr_t attr; pthread_mutexattr_init (&attr); pthread_mutexattr_settype (&attr, PTHREAD_MUTEX_RECURSIVE_NP); pthread_mutex_init (&mutex, &attr);
or like this:

pthread_mutex_t mutex = PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP;

The last type is adaptive which is an extra fast mutex for multi processor systems. It combines a spinlock with an ordinary mutex: instead of blocking straight away on a locked mutex it spins for a short while re-trying the lock and then blocks in the normal way. On a single processor it doesn't spin and so is identical to a fast mutex.

Handling errors
There is not much point using error checking mutexes if you ignore the return value! Except that in reality the main reason that you will get errors from pthread_mutex_lock() and pthread_mutex_unlock() is because of logic errors in your code. I use this macro to detect errors during development and then compile it out for production.

#ifdef DEBUG #define pthread_mutex_lock_check(mutex) ({ if (__ret != 0) %s\n", \ __FILE__, __LINE__, __ret, strerror (__ret)); \ __ret; }) #else #define pthread_mutex_lock_check pthread_mutex_lock #endif \ \ int __ret = pthread_mutex_lock (mutex); \ \ \

printf ("pthread_mutex_lock_check in %s line %u: error %d -

Wake-up order
when a mutex is unlocked and there are several threads blocked waiting for it the system has to decide which gets the mutex next. Until recently the choice was simply the thread that had been waiting longest, but with Linux kernels from 2.6.22 onwards the thread chosen will be the highest priority real-time thread. If there are no real-time threads then it will be the longest waiter as before.

Sharing a mutex between processes

Most often the shared resource being protected by a mutex is a global variable in a process address space and so the threads using the mutex are all local to that process. Sometimes you will have data in a shared memory segment, for example using the POSIX or SYSV IPC shared memory functions, and so the mutex needs to be locked and unlocked by threads from different processes. In such a case, the mutex must be initialsed with the shared attribute

pthread_mutex_t mutex; pthread_mutexattr_t attr; pthread_mutexattr_init (&attr); pthread_mutexattr_setpshared (&attr, PTHREAD_PROCESS_SHARED); pthread_mutex_init (&mutex, &attr);
Apart from being shared the behavior of the mutex is the same as local ones.

Problems with real-time threads: priority inversion

If (and only if) you have threads with real-time scheduling policies SCHED_FIFO or SCHED_RR you may experience priority inversion[3] which results in a high priority thread that is waiting to lock a mutex being blocked by a lower priority thread. One way to resolve the problem is to set the priority protocol of the mutex to priority inheritance, with the result that the thread holding the mutex inherits the priority of the highest priority thread waiting for the mutex and so it cannot be preempted by intermediate priority threads. Here is how to do it

#define __USE_UNIX98 /* Needed for PTHREAD_PRIO_INHERIT */ #include <pthread.h> pthread_mutex_t mutex; pthread_mutexattr_t attr; pthread_mutexattr_init (&attr); pthread_mutexattr_setprotocol (&attr, PTHREAD_PRIO_INHERIT); pthread_mutex_init (&mutex, &attr);
Priority inheritance can be combined with any of the four types. However, it adds a large overhead to the implementation and so it does not make sense to combine it with the fast or adaptive types.

Unexpected termination: the robust mutex

Supposing a thread has locked a mutex and then terminates, what then? In the normal run of things the mutex will remain locked for ever (OK, until the next reboot), causing any threads trying to lock it

to deadlock. This is particularly a problem if you are sharing a mutex between processes and one of them segfaults or is killed. This is where the robust attribute comes in. The first stage is to initialise the mutex with the robust option. It can be combined with any of the four types and with the priority inheritance attribute. Here we go:

#define __USE_GNU /* Needed for PTHREAD_MUTEX_ROBUST_NP */ #include <pthread.h> pthread_mutex_t mutex; pthread_mutexattr_t attr; pthread_mutexattr_init (&attr); pthread_mutexattr_setrobust_np (&attr, PTHREAD_MUTEX_ROBUST_NP); pthread_mutex_init (&mutex, &attr);
Now, if the thread owning the mutex terminates with it locked, any other thread that is trying to lock it will unblock with error code EOWNERDEAD. In other words, this mutex no longer functions as a mutex. If you want to repair the situation you must validate the data that the mutex was protecting, maybe remove some inconsistent state, and then call pthread_mutex_consistent_np(). Then you must lock it, in the same thread that marked it as consistent. Now it is a fully functional mutex again. Finally, if the mutex is unlocked without being made consistent, it will be in a permanently unusable state and all attempts to lockit will fail with the error ENOTRECOVERABLE. You have blown it: the only thing you can do with such a mutex is to destroy it. All the above adds quite a lot of complexity to the implementation of a mutex, so robust mutexes are NOT going to be fast.

Summary
We have four types of mutex each of which may be robust and may have the priority inheritance protocol, which gives us 4 x 2 x 2 = 16 different possibilities. Here are my suggestions on which to use.

During development, use error checking - the extra overhead is very small Use recursive mutexes if you have library code where you cannot be sure that a mutex has already been locked elsewhere In high performance production code, use fast or (if you have more than one CPU) adaptive types If you have real time threads, look carefully at the dependencies between the threads and use priority inheritance where necessary

If you share mutexes between processes (or if your threads terminate in odd places) use robust mutexes

Features vs libc version

Some features are dependent on the version of the 'C' library you use. Here I am considering the two most often used in embedded devices: GNU libc (glibc) and the microcontroller C library, uClibc [4].

Fast, error checking and recursive mutex types are present in all versions of glibc and uClibc The adaptive type and the robust and priority inheritance protocol are ONLY implemented in glibc 2.5 onwards Main-line uClibc DOES NOT implement adaptive, robust and priority inheritance (*) (*) Because uClibc uses the older LinuxThreads library, not the Native POSIX Threads (nptl) library as glibc does. There is an on-going project to port nptl to uClibc but it is not yet functional on all architectures.

Features vs kernel version

If you are using glibc (or uClibc with nptl) you also need to keep a watch on the kernel version. Here are the minimum versions for the various features

Normal

Robust

Priority Inheritance

Priority wake-up queue

2.6.0

2.6.17

2.6.18

2.6.22