What is parallelism?

Parallelism is the ability of a system to perform multiple tasks or processes simultaneously. Unlike concurrency, where tasks may switch, creating an illusion of simultaneity (for example, on a single-core processor), parallelism requires multiple physical resources (such as processor cores).

Main types of parallelism:

• Bit-level: Increasing the size of the machine word (for example, moving from 32-bit to 64-bit processors).
• Instruction-level: Executing multiple instructions in one cycle (for example, pipelining, superscalar architecture).
• Data-level: Applying one operation to many data elements simultaneously (for example, SIMD instructions).
• Task (or thread/process) level: Executing independent blocks of code (tasks) on multiple cores or processors.

In Python, parallelism at the task level is mainly implemented in two ways:

1. Multiprocessing (multiprocessing): Creating independent processes, each with its own address space and Python interpreter. Data exchange between processes requires explicit mechanisms (queues, channels, shared memory). This allows bypassing the Global Interpreter Lock (GIL) limitation and utilizing multiple CPU cores for resource-intensive tasks.

   // Example of using multiprocessing
   import multiprocessing
   
   def worker(num):
       """Function that performs work in a separate process"""
       print(f'Worker {num} starting')
       import time
       time.sleep(1)
       print(f'Worker {num} finishing')
   
   if __name__ == '__main__':
       processes = []
       for i in range(3):
           p = multiprocessing.Process(target=worker, args=(i,))
           processes.append(p)
           p.start()
   
       for p in processes:
           p.join() // Waiting for process to finish
   
       print('All processes finished')
   

2. Multithreading (threading): Creating threads within a single process. Threads share the same address space, which simplifies data exchange, but are limited by the GIL in cPython, which prevents multiple threads from executing Python code simultaneously on different cores. Multithreading is useful for I/O-bound tasks (network, disk operations), where threads can switch while one is waiting.

   // Example of using threading
   import threading
   
   def worker(num):
       """Function that performs work in a separate thread"""
       print(f'Thread {num} starting')
       import time
       time.sleep(1)
       print(f'Thread {num} finishing')
   
   threads = []
   for i in range(3):
       t = threading.Thread(target=worker, args=(i,))
       threads.append(t)
       t.start()
   
   for t in threads:
       t.join() // Waiting for thread to finish
   
   print('All threads finished')
   

The choice between multiprocessing and multithreading depends on the task type: multiprocessing is preferred for CPU-bound tasks, multithreading for I/O-bound tasks.

Differences:

Attribute	Multiprocessing (multiprocessing)	Multithreading (threading)
Resource	Separate process	Thread within a process
Address space	Own	Shared
Data exchange	Complex (requires explicit mechanisms)	Simple (shared space)
GIL	Bypasses (each process has its own GIL)	Limits (shared GIL on process)
CPU utilization	Effective for CPU-bound tasks	Ineffective for CPU-bound tasks
Creation	Expensive (fork process)	Cheap
Termination	OS terminates	Explicit termination or exit from function
Errors	Do not affect other processes	Can affect the entire process