What happens inside the Linux system when memory is allocated and runs out?

In Linux, memory allocation for a process involves several levels of management:

1. Virtual Memory Allocation:

   • The process requests memory through system calls (e.g., brk/sbrk for the heap or mmap for file mapping and anonymous memory).
   • The kernel allocates virtual addresses, but physical memory may not be allocated immediately (lazy allocation).

2. If memory runs out:

   • If there is not enough free physical memory, the kernel tries to free it by invoking swapping — moving inactive pages to a swap file or swap partition on disk.
   • If both swap and physical memory are full, memory allocation fails (e.g., malloc returns NULL).

3. OOM Killer:

   • In case of critical memory shortage, Linux activates the Out-Of-Memory Killer, which terminates processes to free up memory.

Thus, the system first attempts to allocate virtual memory, then manages physical memory and swap, and when resources are exhausted, kills processes to prevent system crash.