Multithreaded programming is a fundamental skill for every programmer, yet also one of the trickiest areas in development. This post covers thread creation and the three common thread synchronization mechanisms, followed by a summary and reflections on multithreaded programming with code examples.

1. Creating Threads

The first step in multithreaded programming is creating threads. Creating a thread essentially adds a new flow of control, allowing multiple flows to execute concurrently or in parallel within the same process.

The thread creation function (other functions are omitted here; refer to pthread.h):

#include<pthread.h>

int pthread_create(
    pthread_t *restrict thread,  /* thread id */
	const pthread_attr_t *restrict attr,    /* thread attributes; pass NULL for defaults */
	void *(*start_routine)(void*),  /* thread entry function */
	void *restrict arg  /* argument to the thread entry function */
	);

Example:

#include<stdio.h>
#include<string.h>
#include<stdlib.h>
#include<unistd.h>
#include<pthread.h>

char* thread_func1(void* arg) {
    pid_t pid = getpid();
    pthread_t tid = pthread_self();
    printf("%s pid: %u, tid: %u (0x%x)\n", (char*)arg, (unsigned int)pid, (unsigned int)tid, (unsigned int)tid);

    char* msg = "thread_func1";
    return msg;
}

void* thread_func2(void* arg) {
    pid_t pid = getpid();
    pthread_t tid = pthread_self();
    printf("%s pid: %u, tid: %u (0x%x)\n", (char*)arg, (unsigned int)pid, (unsigned int)tid, (unsigned int)tid);
    char* msg = "thread_func2 ";
    while(1) {
        printf("%s running\n", msg);
        sleep(1);
    }
    return NULL;
}

int main() {
    pthread_t tid1, tid2;
    if (pthread_create(&tid1, NULL, (void*)thread_func1, "new thread:") != 0) {
        printf("pthread_create error.");
        exit(EXIT_FAILURE);
    }

    if (pthread_create(&tid2, NULL, (void*)thread_func2, "new thread:") != 0) {
        printf("pthread_create error.");
        exit(EXIT_FAILURE);
    }
    pthread_detach(tid2);

    char* rev = NULL;
    pthread_join(tid1, (void *)&rev);
    printf("%s return.\n", rev);
    pthread_cancel(tid2);

    printf("main thread end.\n");
    return 0;
}

2. Thread Synchronization

Sometimes we need multiple threads to cooperate, which requires synchronization. Common synchronization mechanisms include:

  • Mutex
  • Semaphore
  • Condition Variable

First, let’s look at an example without synchronization:

#include<stdio.h>
#include<string.h>
#include<stdlib.h>
#include<unistd.h>
#include<pthread.h>

#define LEN 100000
int num = 0;

void* thread_func(void* arg) {
    for (int i = 0; i< LEN; ++i) {
        num += 1;
    }
    
    return NULL;
}

int main() {
    pthread_t tid1, tid2;
    pthread_create(&tid1, NULL, (void*)thread_func, NULL);
    pthread_create(&tid2, NULL, (void*)thread_func, NULL);

    char* rev = NULL;
    pthread_join(tid1, (void *)&rev);
    pthread_join(tid2, (void *)&rev);

    printf("correct result=%d, wrong result=%d.\n", 2*LEN, num);
    return 0;
}

Output: correct result=200000, wrong result=106860..

[1] Mutex

This is the easiest to understand: when accessing a critical resource, mutual exclusion ensures that at most one thread can acquire the critical resource at any given time.

The mutex logic works as follows: when a thread reaches the critical resource and finds no other thread has locked it, it locks it and accesses the resource. If another thread arrives at the mutex and finds it already locked, that thread suspends and waits for the lock to be released. After the current thread finishes with the critical resource, it unlocks and wakes up other threads suspended on that mutex, which then wait to be scheduled again.

How are “suspend and wait” and “wake up waiting threads” implemented? Each mutex has a wait queue. When a thread needs to suspend on a mutex, it first adds itself to the wait queue, then sets its thread state to sleeping, and calls the scheduler function to switch to another thread. To wake up waiting threads, one simply removes an entry from the wait queue, changes its state from sleeping to ready, and adds it to the ready queue — then the next time the scheduler runs, the awakened thread may be selected.

Key functions:

#include <pthread.h>

int pthread_mutex_init(pthread_mutex_t *restrict mutex,     
       const pthread_mutexattr_t *restrict attr);       /* initialize mutex */
int pthread_mutex_destroy(pthread_mutex_t *mutex);      /* destroy mutex */
int pthread_mutex_lock(pthread_mutex_t *mutex);
int pthread_mutex_trylock(pthread_mutex_t *mutex);
int pthread_mutex_unlock(pthread_mutex_t *mutex);

Using a mutex to fix the incorrect result problem above:

#include<stdio.h>
#include<string.h>
#include<stdlib.h>
#include<unistd.h>
#include<pthread.h>

#define LEN 100000
int num = 0;

void* thread_func(void* arg) {
    pthread_mutex_t* p_mutex = (pthread_mutex_t*)arg;
    for (int i = 0; i< LEN; ++i) {
        pthread_mutex_lock(p_mutex);
        num += 1;
        pthread_mutex_unlock(p_mutex);
    }
    
    return NULL;
}

int main() {
    pthread_mutex_t m_mutex;
    pthread_mutex_init(&m_mutex, NULL);

    pthread_t tid1, tid2;
    pthread_create(&tid1, NULL, (void*)thread_func, (void*)&m_mutex);
    pthread_create(&tid2, NULL, (void*)thread_func, (void*)&m_mutex);

    pthread_join(tid1, NULL);
    pthread_join(tid2, NULL);

    pthread_mutex_destroy(&m_mutex);

    printf("correct result=%d, result=%d.\n", 2*LEN, num);
    return 0;
}

Output: correct result=200000, result=200000.

If mutex sections are nested with other mutex code, watch out for deadlocks.

Two common deadlock scenarios:

  • Scenario 1: If the same thread calls lock twice in succession, on the second call the lock is already held by itself. The thread suspends waiting for another thread to release the lock, but the lock is held by itself — so it suspends forever with no chance to release it, causing a deadlock.
  • Scenario 2: Thread A acquires lock 1, Thread B acquires lock 2. Then Thread A calls lock on lock 2 and must suspend waiting for Thread B to release lock 2. Meanwhile Thread B calls lock on lock 1 and must suspend waiting for Thread A to release lock 1. Both threads are now permanently suspended.

How to avoid deadlocks:

  1. Don’t use mutexes (often impractical).
  2. Try to avoid holding multiple locks simultaneously.
  3. If you must hold multiple locks, follow this principle: if all threads acquire multiple locks in the same order (commonly by mutex variable address order), deadlocks won’t occur. (For example, if a program uses lock 1, lock 2, and lock 3 with addresses lock1 < lock2 < lock3, then every thread that needs two or three locks must acquire them in the order lock1, lock2, lock3. If establishing a total order for all locks is difficult, prefer pthread_mutex_trylock over pthread_mutex_lock to avoid deadlocks.)

Approaches to solving deadlocks

  1. One approach is to acquire all required critical resources atomically; if you can’t acquire them all at once, block and wait. The downside is lower efficiency.
  2. Have the program predict whether acquiring a resource could cause a deadlock — if so, give up this attempt and retry on the next scheduling. How do you predict deadlock potential? One viable algorithm is the Banker’s algorithm. In real-world systems, however, the Banker’s algorithm still suffers from efficiency issues.
[2] Condition Variable

In a nutshell: a thread needs a certain condition to be true (a condition determined by other threads) before it can proceed. If the condition is false, the thread blocks and waits; when another thread makes the condition true during execution, it wakes the waiting thread to continue.

Relevant functions:

#include <pthread.h>

int pthread_cond_destroy(pthread_cond_t *cond);
int pthread_cond_init(pthread_cond_t *restrict cond,
       const pthread_condattr_t *restrict attr);
int pthread_cond_timedwait(pthread_cond_t *restrict cond,
       pthread_mutex_t *restrict mutex,
       const struct timespec *restrict abstime);
int pthread_cond_wait(pthread_cond_t *restrict cond,
       pthread_mutex_t *restrict mutex);
int pthread_cond_broadcast(pthread_cond_t *cond);
int pthread_cond_signal(pthread_cond_t *cond);

The classic example for condition variables is the producer-consumer pattern: the producer thread sends data to a queue, and the consumer thread takes data from the queue. When the consumer processes faster than the producer, the queue may become empty. One approach is to wait and poll periodically, but this is suboptimal — you don’t know how long to wait. Condition variables solve this elegantly. Here’s the code:

#include<sys/types.h>
#include<unistd.h>
#include<stdlib.h>
#include<stdio.h>
#include<pthread.h>
#include<errno.h>
#include<string.h>

#define LIMIT 1000

struct data {
    int n;
    struct data* next;
};

pthread_cond_t condv = PTHREAD_COND_INITIALIZER;
pthread_mutex_t mlock = PTHREAD_MUTEX_INITIALIZER; 
struct data* phead = NULL;

void producer(void* arg) {
    printf("producer thread running.\n");
    int count = 0;
    for (;;) {
        int n = rand() % 100;
        struct data* nd = (struct data*)malloc(sizeof(struct data));
        nd->n = n;

        pthread_mutex_lock(&mlock);
        struct data* tmp = phead;
        phead = nd;
        nd->next = tmp;
        pthread_mutex_unlock(&mlock);
        pthread_cond_signal(&condv);

        count += n;

        if(count > LIMIT) {
            break;
        }
        sleep(rand()%5);
    }
    printf("producer count=%d\n", count);
}

void consumer(void* arg) {
    printf("consumer thread running.\n");
    int count = 0;
    for(;;) {
        pthread_mutex_lock(&mlock);
        if (NULL == phead) {
            pthread_cond_wait(&condv, &mlock);
        } else {
            while(phead != NULL) {
                count += phead->n;
                struct data* tmp = phead;
                phead = phead->next;
                free(tmp);
            }
        }
        pthread_mutex_unlock(&mlock);
        if (count > LIMIT)
            break;
    }
    printf("consumer count=%d\n", count);
}

int main() {
    pthread_t tid1, tid2;
    pthread_create(&tid1, NULL, (void*)producer, NULL);
    pthread_create(&tid2, NULL, (void*)consumer, NULL);
    
    pthread_join(tid1, NULL);
    pthread_join(tid2, NULL);

    return 0;
}

Execution logic of condition variables:

The key is understanding what happens inside int pthread_cond_wait(pthread_cond_t *restrict cond, pthread_mutex_t *restrict mutex) — the rest is relatively straightforward. Before calling this function, you must first acquire the mutex, then check the condition. If the condition is met, continue executing and then release the lock. If the condition is not met, release the lock, the thread blocks here, and waits until another thread signals that the condition is now true — at which point the thread is woken up, re-acquires the lock, continues execution, and releases the lock. (In short: release lock → block and wait → wake up, re-acquire lock, and return.)

See the source for implementation details: pthread_cond_wait.c and pthread_cond_signal.c

The example above may be somewhat verbose; here is a more concise one:

#include<sys/types.h>
#include<unistd.h>
#include<stdlib.h>
#include<stdio.h>
#include<pthread.h>
#include<errno.h>
#include<string.h>

#define NUM 3
pthread_cond_t condv = PTHREAD_COND_INITIALIZER;
pthread_mutex_t mlock = PTHREAD_MUTEX_INITIALIZER; 

void producer(void* arg) {
    int n = NUM;
    while(n--) {
        sleep(1);
        pthread_cond_signal(&condv);
        printf("producer thread send notify signal. %d\t", NUM-n);
    }
}

void consumer(void* arg) {
    int n = 0;
    while (1) {
        pthread_cond_wait(&condv, &mlock);
        printf("recv producer thread notify signal. %d\n", ++n);
        if (NUM == n) {
            break;
        }
    }
}

int main() {
    pthread_t tid1, tid2;
    pthread_create(&tid1, NULL, (void*)producer, NULL);
    pthread_create(&tid2, NULL, (void*)consumer, NULL);
    
    pthread_join(tid1, NULL);
    pthread_join(tid2, NULL);

    return 0;
}

Output:

producer thread send notify signal. 1   recv producer thread notify signal. 1
producer thread send notify signal. 2   recv producer thread notify signal. 2
producer thread send notify signal. 3   recv producer thread notify signal. 3
[3] Semaphore

Semaphores are suited for controlling a shared resource that supports only a limited number of users. They maintain a counter between 0 and a specified maximum value. When a thread completes a wait on the semaphore, the counter is decremented by 1; when a thread completes a post (release) on the semaphore, the counter is incremented by 1. When the counter is 0, threads suspend and wait until the counter exceeds 0.

Key functions:

#include <semaphore.h>

int sem_init(sem_t *sem, int pshared, unsigned int value);
int sem_wait(sem_t *sem);
int sem_trywait(sem_t *sem);
int sem_post(sem_t * sem);
int sem_destroy(sem_t * sem);

Example:

#include<sys/types.h>
#include<unistd.h>
#include<stdlib.h>
#include<stdio.h>
#include<pthread.h>
#include<errno.h>
#include<string.h>
#include<semaphore.h>

#define NUM 5

int queue[NUM];
sem_t psem, csem; 

void producer(void* arg) {
    int pos = 0;
    int num, count = 0;
    for (int i=0; i<12; ++i) {
        num = rand() % 100;
        count += num;
        sem_wait(&psem);
        queue[pos] = num;
        sem_post(&csem);
        printf("producer: %d\n", num); 
        pos = (pos+1) % NUM;
        sleep(rand()%2);
    }
    printf("producer count=%d\n", count);
}

void consumer(void* arg){
    int pos = 0;
    int num, count = 0;
    for (int i=0; i<12; ++i) {
        sem_wait(&csem);
        num = queue[pos];
        sem_post(&psem);
        printf("consumer: %d\n", num);
        count += num;
        pos = (pos+1) % NUM;
        sleep(rand()%3);
    }
    printf("consumer count=%d\n", count);    
} 

int main() {
    sem_init(&psem, 0, NUM);
    sem_init(&csem, 0, 0);

    pthread_t tid[2];
    pthread_create(&tid[0], NULL, (void*)producer, NULL);
    pthread_create(&tid[1], NULL, (void*)consumer, NULL);
    pthread_join(tid[0], NULL);
    pthread_join(tid[1], NULL);
    sem_destroy(&psem);
    sem_destroy(&csem);

    return 0;
}

Execution logic of semaphores:

When a thread needs access to the shared resource, it first checks the semaphore. If the value is greater than 0, it decrements it by 1, accesses the shared resource, and after finishing, increments it by 1 — waking one of the threads suspended on that semaphore if any exist. If the semaphore value is 0, the thread suspends and waits.

See sem_post.c for source reference.

3. Summary and Reflections on Multithreaded Programming

To conclude, here are some reflections on multithreaded programming:

  • First: always consider synchronization when writing multithreaded code. In most cases, we create multiple threads so they can cooperate; without synchronization, problems will arise.
  • Second: beware of deadlocks. When multiple threads need to access multiple critical resources, improper handling leads to deadlocks. If the program compiles but hangs at runtime, suspect a deadlock. Think about which threads access multiple critical resources — this narrows down the search.
  • Third: critical resource handling. Most multithreading issues stem from multiple threads accessing critical resources simultaneously. One approach is to centralize all access and processing of critical resources into a single thread, which services requests from other threads — having only one thread touch critical resources solves many problems.
  • Fourth: thread pools. When handling many short-lived tasks, create a thread pool upfront. Threads in the pool continuously pull tasks from a task queue, avoiding the overhead of repeatedly creating and destroying threads. Thread pools are not covered in detail here.