Multi-threading & Multi-Processing in Python: A Complete Beginner-to-Intermediate Guide

Introduction

Hey Pythonistas! Ever wondered how to make your programs run faster by doing multiple things at once? That’s where multi-threading and multi-processing come in. Python’s Global Interpreter Lock (GIL) means threads don’t give true parallelism for CPU-heavy tasks, but they shine for I/O-bound work (like waiting for network responses or file reads). Multi-processing bypasses the GIL by using separate processes — perfect for CPU-bound jobs.

In this hands-on guide, we’ll explore threading deeply with real examples you can copy-paste, then touch on multiprocessing as the powerful sibling. By the end, you’ll be ready to speed up your own projects!

Multitasking vs Multi-threading

Multitasking is what your operating system does — running many programs (processes) simultaneously (e.g., browser + music player + code editor).

Multi-threading happens inside one program. Threads are lightweight “mini-processes” that share the same memory. They’re great for concurrency but not always for speed on multi-core CPUs because of the GIL.

Quick analogy: Multitasking = multiple chefs in different kitchens. Multi-threading = multiple chefs sharing one kitchen (they take turns using the stove).

The Threading Module

Python’s built-in threading module is all you need. Import it like this:

import threading
import time

Creating Threads – Two Popular Ways

1. Inheriting from the Thread Class

This is clean when you want to add more methods later.

class WorkerThread(threading.Thread):
    def __init__(self, name):
        super().__init__()
        self.name = name
    
    def run(self):
        print(f"Thread {self.name} started")
        time.sleep(2)
        print(f"Thread {self.name} finished")

# Create and start
t1 = WorkerThread("A")
t1.start()

2. Using a Callable Object (target)

Simpler for quick tasks — no subclass needed.

def worker(name):
    print(f"Worker {name} started")
    time.sleep(1.5)
    print(f"Worker {name} finished")

t2 = threading.Thread(target=worker, args=("B",))
t2.start()

Life Cycle of a Thread

Every thread goes through these states:

• New – just created (Thread object exists)
• Runnable – ready to run after start()
• Running – executing run()
• Blocked/Waiting – sleeping, waiting for lock, or join()
• Terminated – finished or crashed

Single-Threaded Application Example

Let’s simulate downloading 3 files sequentially.

def download_file(file_id):
    print(f"Downloading file {file_id}...")
    time.sleep(2)  # simulate network delay
    print(f"File {file_id} downloaded!")

start = time.time()
for i in range(3):
    download_file(i)
print(f"Total time: {time.time() - start:.2f} seconds")

Output: Takes ~6 seconds (3 × 2s).

Multi-Threaded Application Example

Same task, but now running in parallel!

start = time.time()
threads = []

for i in range(3):
    t = threading.Thread(target=download_file, args=(i,))
    t.start()
    threads.append(t)

for t in threads:
    t.join()  # wait for all to finish

print(f"Total time: {time.time() - start:.2f} seconds")

Output: Takes only ~2 seconds! Threads run concurrently.

Can We Call run() Directly?

Yes, but it runs in the main thread — no new thread is created.

t = threading.Thread(target=worker, args=("Test",))
t.run()  # Runs immediately in current thread

That defeats the purpose. Always use start()!

Why Do We Need the start() Method?

start() tells the OS to create a new thread and then calls run() inside it. Without it, you’re just calling a normal function.

Thread.sleep() – Adding Pauses

Use time.sleep(seconds) to simulate work or prevent busy-waiting.

Tip: Never use sleep() in production to “fix” race conditions — use synchronization instead!

The join() Method – Waiting for Threads

thread.join() blocks the calling thread until the target thread finishes. We used it above to make sure the program doesn’t exit before downloads complete.

You can even add a timeout: t.join(timeout=5).

Synchronization – Using Lock to Prevent Race Conditions

When multiple threads access the same variable, chaos happens. Let’s see a classic counter example.

Without Lock (Race Condition)

counter = 0

def increment():
    global counter
    for _ in range(100_000):
        counter += 1

threads = [threading.Thread(target=increment) for _ in range(5)]
for t in threads: t.start()
for t in threads: t.join()

print("Final counter:", counter)  # Expected 500_000, but you’ll get random lower number!

With Lock – Fixed!

counter = 0
lock = threading.Lock()

def safe_increment():
    global counter
    for _ in range(100_000):
        lock.acquire()   # or use 'with lock:'
        counter += 1
        lock.release()

# Same thread creation as above...
print("Final counter:", counter)  # Now always 500_000 🎉

Even better — use context manager:

with lock:
    counter += 1

Case Studies – Real-World Applications

Case Study 1: Parallel Web Scraping

Scrape 10 product pages from an e-commerce site. Each thread handles one page → 10× faster than sequential!

(Use requests + BeautifulSoup inside the worker function. Just remember to use a lock if writing to a shared list.)

Case Study 2: Batch Image Processing

Resize 50 images. Even with GIL, threading helps if you’re waiting for disk I/O. For true speed, switch to multiprocessing (see below).

Case Study 3: Producer-Consumer Pattern

A thread produces data (e.g., reading sensor values), another consumes it. Use queue.Queue + Lock for safe communication.

Bonus: Quick Look at Multi-Processing

When you need real parallelism (CPU-heavy tasks like data crunching or machine learning), use multiprocessing.

from multiprocessing import Process, Pool

def cpu_heavy(n):
    return n * n

if __name__ == "__main__":
    with Pool(4) as pool:  # 4 processes
        results = pool.map(cpu_heavy, range(10000))
    print("Done!")

Rule of thumb:
• I/O bound (network, files) → Threading
• CPU bound (math, image processing) → Multiprocessing

Final Tips for Python Developers

• Always use if __name__ == "__main__": when using multiprocessing.
• Threading is simpler but remember the GIL.
• Start small — test with print statements before adding locks.
• Tools like concurrent.futures.ThreadPoolExecutor make threading even easier (highly recommended!).

That’s it! You now have a solid foundation in Python concurrency. Grab the code snippets, experiment in your IDE, and watch your programs fly 🚀

Happy coding 🎉