Operating Systems — Cheat Sheet

A process is a running program with its own address space. A thread is a unit of execution inside a process, sharing that address space with its siblings. Each process gets its own virtual memory, file descriptor table, and permissions. Threads within a process share the heap, globals, and open files, but each gets its own stack, program counter, and registers. That sharing is the whole trade. Threads communicate by reading the same memory, which is fast but means every shared mutable variable is a potential race. Processes are isolated, so a crash or memory corruption in one cannot touch another — but they must communicate through pipes, sockets, or shared memory segments, which costs more. Context switching between threads is cheaper because the memory mappings do not change, so the TLB and much of the cache stay warm. Switching processes flushes far more state. The practical rule: threads for concurrency within one workload, processes for isolation and fault containment.

The PCB is the kernel's record of everything it needs to suspend a process and later resume it as if nothing happened. It holds the process ID and parent ID, the current state, the saved CPU registers and program counter, the stack pointer, scheduling information such as priority and accumulated CPU time, memory management data like page table pointers or segment bases, the file descriptor table, and accounting and signal handling information. The register snapshot is the part that makes context switching possible. On a switch the kernel dumps the live registers into the outgoing PCB and loads the incoming one — that copy is essentially what a context switch is. PCBs live in kernel memory and are never directly accessible to user code, which is why reading process information requires a system call or the /proc filesystem on Linux. The practical consequence: PCBs are why processes are heavier than threads. Threads in the same process share most of this structure.

The kernel saves the state of the running process, picks another, and restores its state. Concretely: a trigger occurs — a timer interrupt, a blocking system call, a higher-priority task becoming runnable. The CPU traps into kernel mode. The current registers, program counter and stack pointer are written into the outgoing PCB. The scheduler selects the next process. Its registers are loaded, its page table base register is updated, and execution resumes in user mode. The direct cost is a few microseconds of register copying. The indirect cost is usually larger: changing the address space invalidates TLB entries, and the incoming process finds a cache full of someone else's data. Those misses are what actually hurt. That is why thread switches within a process are cheaper — the address space is unchanged, so the TLB survives. It is also why thrashing under heavy context switching is so damaging: the machine spends its time switching rather than computing, and every switch starts cold.

New, Ready, Running, Waiting (or Blocked), and Terminated. New means the process is being created and its PCB set up. Ready means it is runnable and waiting only for a CPU. Running means it is currently executing on a core. Waiting means it is blocked on something external — I/O, a lock, a signal — and cannot be scheduled even if a CPU is free. Terminated means it has finished but its exit status may not yet have been collected. The transitions matter more than the names. Ready to Running is dispatch. Running to Ready is preemption, typically a timer interrupt. Running to Waiting is a blocking call. Waiting to Ready is the completion of whatever it waited on — note that it goes to Ready, not straight to Running, because it must still be scheduled. The common misconception is that a blocked process consumes CPU. It does not. A machine with a thousand blocked processes can sit at zero percent utilisation, which is why load average and CPU usage tell you different things.

A zombie has finished executing but its parent has not yet collected its exit status, so the kernel keeps a minimal entry alive holding that status. It consumes no memory or CPU — only a process table slot. That sounds harmless until a buggy parent leaks thousands of them and exhausts the PID table, at which point nothing on the machine can fork. The fix is for the parent to call wait() or waitpid(), or to explicitly ignore SIGCHLD so the kernel reaps children automatically. An orphan is the opposite situation: the parent died first while the child is still running. Orphans are re-parented to init (PID 1), which reaps them properly, so orphans are not a leak. This matters in containers. A container's PID 1 is often your application, not a real init, and most applications do not reap adopted children. That is exactly why you run with --init or tini in Docker — otherwise zombies accumulate inside the container.

fork() creates a near-identical copy of the calling process. exec() replaces the current process image with a new program, keeping the same PID. fork() returns twice — zero in the child, the child's PID in the parent — which is how each copy knows which one it is. exec() does not return at all on success, because the code that called it no longer exists. The separation looks odd but is deliberate and useful. Between the fork and the exec, the child is still running your code and can adjust the environment it is about to hand to the new program: redirect stdin and stdout, close file descriptors, change the working directory, drop privileges. That is exactly how a shell implements pipes and redirection. A combined spawn call would have to expose every one of those adjustments as a parameter. The cost concern is answered by copy-on-write: fork does not physically copy memory, it marks pages shared and read-only, copying only on the first write. Since exec discards the address space immediately, almost nothing is ever copied.

Rather than duplicating memory eagerly, the kernel maps the same physical pages into both processes and marks them read-only. The copy happens only when one side actually writes, triggering a page fault that allocates a private copy of just that page. It makes fork() cheap. Copying a process with a 4 GB heap would be absurd if done eagerly; with copy-on-write, forking costs page table setup and nothing more. Since most forks are immediately followed by exec, the pages are usually discarded before any copy occurs. It shows up elsewhere too. Redis uses it for background persistence — it forks, and the child writes a consistent snapshot while the parent keeps serving. The catch is that heavy writes during the snapshot cause widespread page copying and memory usage can spike toward double. That spike is the practical gotcha worth naming: a machine sized for one copy of the dataset can be OOM-killed during a fork-based save, even though "nothing was copied".

Kernel-level threads are known to and scheduled by the operating system. User-level threads are managed entirely by a library in user space, invisible to the kernel. User threads are extremely cheap to create and switch, since switching never enters the kernel. But because the kernel sees only one thread, a single blocking system call blocks every user thread in that process, and they cannot run on multiple cores in parallel. Kernel threads solve both problems — true parallelism and independent blocking — at the cost of a system call per switch and real kernel memory per thread. Java threads have been one-to-one with OS threads since the green-threads era ended, which is why creating tens of thousands of them is expensive. Virtual threads in Java 21 reintroduce the user-level model on top: many virtual threads multiplex onto few carrier threads, and the runtime unmounts a virtual thread when it blocks so the carrier stays busy. Go's goroutines are the same M:N idea, which is why both handle high-concurrency I/O well.

Concurrency is about dealing with many things at once; parallelism is about doing many things at once. A concurrent program is structured as independent tasks that can make progress interleaved. On a single core they take turns — only one truly executes at any instant, but all are in flight. A parallel program actually runs tasks simultaneously, which requires multiple cores. So concurrency is a design property and parallelism is an execution property. You can have concurrency without parallelism (a single-core machine running a web server), and parallelism without meaningful concurrency (a data-parallel loop across cores). The practical consequence is which one your bottleneck needs. I/O-bound work benefits from concurrency: threads are blocked most of the time, so overlapping the waiting is the win, and you can serve thousands of connections on a few cores. CPU-bound work needs parallelism, and adding threads beyond the core count only adds context switching. Misdiagnosing this is why people add threads to a CPU-saturated service and watch it get slower.

It depends entirely on whether the work is CPU-bound or I/O-bound. For CPU-bound work the answer is roughly the number of cores. More threads than cores adds context switching without adding throughput, since every thread wants a core continuously. For I/O-bound work threads spend most of their life blocked, so you want many more. The standard formula is cores × (1 + wait time / service time). A task that waits 90 ms on a database for every 10 ms of computation has a ratio of 9, suggesting roughly ten times the core count. In practice you measure rather than compute, because the formula assumes uniform tasks. Two things matter more than the exact number. Separate pools for different workload types, so a slow downstream call cannot starve fast work — this is bulkheading. And bound the queue: an unbounded queue converts a throughput problem into an out-of-memory crash, which is what makes Executors.newFixedThreadPool dangerous under sustained overload.

A system call is the mechanism by which user code requests a service only the kernel can perform — file I/O, network access, memory mapping, process creation. It is not an ordinary function call. It triggers a controlled transition into kernel mode via a dedicated instruction, the kernel validates the arguments, performs the work, and returns to user mode. The privilege boundary is the point: user code cannot be trusted to touch hardware or another process's memory directly. The cost is the mode switch plus argument validation — historically a few microseconds, now closer to hundreds of nanoseconds, though Spectre and Meltdown mitigations made it worse again by requiring page table isolation. That cost is why buffering exists. Writing a file byte-by-byte means one syscall per byte; a BufferedWriter accumulates and issues one syscall per few kilobytes. The same reasoning drives batching in network code and why readv/writev exist. Strace or perf will show you the syscall count, which is often the first thing to look at in an unexpectedly slow I/O path.

They are hardware-enforced privilege levels. Kernel mode can execute any instruction and access any memory; user mode is restricted to a safe subset and its own address space. The CPU tracks the current mode in a register. Privileged instructions — modifying page tables, disabling interrupts, direct device I/O — fault if attempted in user mode. Memory outside the process's mappings is simply not addressable. This is what makes an operating system able to enforce anything. Without it, any program could read another's memory, scribble on the page tables, or halt the machine. Transitions happen three ways: a deliberate system call, a hardware interrupt, or a fault such as a page fault or divide-by-zero. The practical relevance is performance analysis. Tools separate user time from system time, and a process burning most of its cycles in system time is usually doing too much I/O or too many syscalls — a very different problem from one burning user time in application code.

First Come First Served is simple and non-preemptive, but suffers the convoy effect — one long job blocks everything behind it, wrecking average wait time. Shortest Job First is provably optimal for average waiting time, but requires knowing burst lengths in advance, which you generally do not. Its preemptive form is Shortest Remaining Time First. Both can starve long jobs indefinitely. Round Robin gives each process a fixed time quantum and cycles. Fair and responsive, which is what interactive systems need. Quantum choice is the tuning knob: too large and it degenerates into FCFS, too small and context switching dominates. Priority scheduling runs the highest priority first, and starves low-priority work unless you add ageing, which gradually raises the priority of waiting processes. Multilevel Feedback Queue is what real systems approximate. Processes start high-priority with short quanta; those that use their full quantum are demoted, which naturally separates interactive work from batch work without needing to be told which is which.

Non-preemptive scheduling lets a running process keep the CPU until it blocks or exits. Preemptive scheduling can forcibly take the CPU away, usually on a timer interrupt or when a higher-priority process becomes ready. Non-preemptive is simpler and has less overhead, and it avoids some race conditions because a process cannot be interrupted mid-update. But one misbehaving process — an infinite loop — hangs the entire machine, and response time for interactive work is unbounded. Preemptive guarantees responsiveness and is what every general-purpose OS uses. The cost is context switch overhead and the fact that shared data can be interrupted mid-modification, which is precisely why synchronisation primitives exist. The historical illustration is instructive: cooperative multitasking in early Windows and Mac OS meant one frozen application froze the whole system. Preemption is why that no longer happens. Real-time systems complicate this further, since preemption is required for deadline guarantees but must itself be bounded in duration.

Starvation is a process that is ready to run but never gets scheduled, because something is always preferred over it. It is a consequence of any strictly priority-based policy. Under priority scheduling, a steady stream of high-priority arrivals means a low-priority process waits forever. Under Shortest Job First, a continuous supply of short jobs starves the long one. Ageing fixes it by making waiting itself increase priority. A process that has been ready for a long time is gradually promoted until it eventually outranks new arrivals and runs. It bounds the wait without abandoning priorities. The distinction from deadlock matters and interviewers probe it. A deadlocked process can never proceed regardless of what the scheduler does, because it waits on a resource that will never be released. A starved process could run at any moment — the scheduler simply keeps choosing others. Starvation is a scheduling fairness problem; deadlock is a resource dependency problem. The same idea appears in application code: a fair ReentrantLock hands the lock to the longest waiter to prevent barging threads from starving others.

Turnaround time is completion time minus arrival time — the total elapsed time from submission to finish, including all waiting. Waiting time is turnaround time minus burst time — how long the process sat ready without running. Equivalently, the time it spent wanting the CPU and not getting it. Response time is a third metric: first time on the CPU minus arrival time. It matters for interactive systems, where users care about when something starts reacting more than when it completes. The three can point in different directions, which is the substance of the question. Round Robin has excellent response time and mediocre turnaround, because everything starts quickly but nothing finishes quickly. Shortest Job First has optimal average turnaround but terrible response time for long jobs. Averaging also hides distribution. An algorithm with a good average can have a dreadful tail, which is why production systems track p99 latency rather than the mean. Interviewers like hearing that connection between OS theory and service-level objectives.

A high-priority task is blocked waiting on a lock held by a low-priority task, and a medium-priority task preempts the low-priority one — so the medium task effectively outranks the high one. The high-priority task cannot proceed until the low-priority task releases the lock, but the low-priority task cannot run because the medium-priority task keeps taking the CPU. Priorities are inverted in practice. The famous case is the Mars Pathfinder, which kept resetting on the surface because of exactly this, and was fixed by enabling priority inheritance remotely. Priority inheritance is the standard solution: while a low-priority task holds a lock a high-priority task wants, it temporarily inherits the high priority, so it cannot be preempted by medium tasks and releases the lock quickly. The alternative is the priority ceiling protocol, where a lock carries the highest priority of any task that can acquire it, and a holder is immediately raised to that level. It is mainly a real-time concern, but it explains why lock hold times matter in any latency-sensitive service.

A long-running process holds a resource — typically the CPU under FCFS — while many short processes queue behind it, so average waiting time collapses. The name comes from the traffic analogy: one slow lorry on a single-lane road, with a convoy of fast cars stuck behind it. The cars could each pass in seconds, but they all wait for the lorry. Concretely: one CPU-bound job of 100 ms followed by ten jobs of 1 ms each. The short jobs wait an average of over 100 ms for 1 ms of work, so utilisation of I/O devices collapses while everything waits on the CPU. Preemption is the fix. Round Robin interleaves so short jobs finish promptly; Shortest Job First reorders so they go first. The same pattern appears well beyond CPU scheduling. A single slow query holding a connection from a small pool convoys every other request. Head-of-line blocking in HTTP/1.1 is the same effect on a network connection, and it is exactly what HTTP/2 multiplexing was designed to remove.

CFS models an ideal processor that runs every runnable task simultaneously at equal speed, then approximates it by always running whichever task is furthest behind that ideal. It tracks virtual runtime per task — the CPU time consumed, weighted by priority. The scheduler picks the task with the smallest vruntime, runs it, and its vruntime advances. Tasks are kept in a red-black tree keyed by vruntime, so selecting the next one is O(log n). There is no fixed time quantum. The slice is derived from a target latency divided among runnable tasks, with a floor so slices do not become uselessly small under high load. Nice values act as weights on how fast vruntime accumulates, so a low-priority task accrues virtual time faster and is chosen less often. The result is that I/O-bound tasks get good latency without special-casing: they consume little CPU, so their vruntime stays low, and they are picked promptly when they wake. Newer kernels have moved to EEVDF, which adds explicit latency guarantees on top of the same fairness idea.

CPU utilisation is the percentage of time the CPU is executing work. Load average is the number of processes runnable or, on Linux, uninterruptibly blocked, averaged over one, five and fifteen minutes. They diverge in informative ways. A load average of 8 on an 8-core machine means fully loaded but not queueing. The same load on a 2-core machine means heavy queueing and processes waiting. Load must always be read relative to core count. The Linux-specific twist is that load includes processes in uninterruptible sleep — usually blocked on disk I/O. So a machine with a failing disk can show a load average of 50 at near-zero CPU utilisation. That combination is a strong signal to look at I/O rather than compute. The three windows show trend. A one-minute load far above the fifteen-minute one means a spike is developing; the reverse means it is subsiding. For diagnosis, pair them: high load and high CPU means genuinely CPU-bound; high load and low CPU means blocked on something.

CPU affinity binds a process or thread to a specific core or set of cores, preventing the scheduler from migrating it. The motivation is cache warmth. A thread that has been running on one core has its working set in that core's L1 and L2 cache. Migrating it to another core abandons all of that, and the thread runs cold until the cache refills. On NUMA systems it is worse — memory allocated near the old core is now remote, adding latency to every access. So affinity helps latency-sensitive work: trading systems, packet processing, and real-time audio commonly pin threads. The cost is lost flexibility. The scheduler can no longer balance load, so a pinned thread can sit idle while another core is oversubscribed. Pin badly and you make things worse. The scheduler already prefers to keep threads on the same core — soft affinity — so explicit pinning is only worth it when you have measured a migration problem. In containers, CPU limits and cpuset constraints interact with this, which is a common source of surprising performance in Kubernetes.

A CPU-bound process spends most of its time computing, with long CPU bursts. An I/O-bound process computes briefly then blocks on a device, giving short frequent bursts. The scheduler cares because favouring I/O-bound processes improves overall throughput. An I/O-bound process that gets the CPU promptly issues its next request quickly, keeping the device busy. Making it wait leaves expensive hardware idle. This is why multilevel feedback queues demote processes that consume their whole quantum: consuming the full slice is evidence of CPU-bound behaviour, and yielding early is evidence of I/O-bound behaviour. The scheduler infers the class from observed behaviour rather than being told. A good mix is what you want on a machine — CPU-bound work uses the cycles that I/O-bound work leaves idle while it waits. The application-level parallel is thread pool sizing, and it is the same distinction: core-count pools for computation, much larger pools for blocking I/O, and separate pools so the two do not interfere.

A race condition is when the result depends on the unpredictable timing of concurrent operations on shared state. The canonical example is an unsynchronised counter increment. count++ looks atomic but compiles to three steps: read, add, write. Two threads can both read 5, both compute 6, and both write 6 — one increment is lost. Three conditions must all hold: shared mutable state, concurrent access, and at least one writer. Remove any one and the race disappears, which is why immutable objects and thread confinement are such effective defences — they eliminate the problem rather than managing it. Races are nasty because they are timing-dependent. They pass tests, pass code review, and fail in production under load, then refuse to reproduce. The fixes in order of preference: do not share (thread-local or immutable), share something the platform makes atomic (AtomicInteger, a concurrent collection), or synchronise explicitly with a lock. Reaching for a lock first is common and usually more expensive than necessary.

A critical section is the region of code that accesses shared resources and must not be executed by more than one thread at a time. A correct solution must provide three things. Mutual exclusion: at most one process inside at a time. Progress: if no one is inside and someone wants in, the decision of who enters cannot be postponed indefinitely by processes not trying to enter. Bounded waiting: there is a limit on how many times others can enter before a waiting process gets its turn, which is what rules out starvation. Mutual exclusion alone is easy — a solution that lets nobody in satisfies it. The other two are what make a solution useful. The engineering advice that follows: keep critical sections as short as possible. Every instruction inside is serialised, so a long one destroys scalability regardless of core count — that is Amdahl's law in practice. And never perform I/O or call unknown code inside one. A network call under a lock turns a millisecond of contention into seconds, and calling a callback risks deadlock or reentrancy you did not anticipate.

A mutex enforces mutual exclusion for a single resource and has ownership: the thread that locks it must be the one to unlock it. A semaphore is a counter permitting up to N concurrent holders, with no ownership — any thread may signal it, including one that never waited on it. So a binary semaphore and a mutex look similar but differ in intent and semantics. Ownership lets a mutex support recursion and priority inheritance; a semaphore cannot, because it does not know who holds it. The usage difference follows from that. A mutex protects a critical section. A counting semaphore limits concurrency — at most ten simultaneous connections — or signals between threads, as in producer-consumer where one thread signals availability and another waits. In Java, synchronized and ReentrantLock are mutexes; Semaphore is the counting primitive. The classic misuse is using a semaphore for mutual exclusion and having a thread accidentally release one it never acquired, silently raising the permit count and letting two threads into the critical section.

A spinlock waits by looping — repeatedly testing whether the lock is free — rather than sleeping. That wastes CPU while waiting, but avoids the cost of blocking: no context switch out, no scheduler involvement, no switch back on wake. A context switch costs a few microseconds, so if the lock will be held for less than that, spinning is genuinely cheaper. So the rule is: spin when hold times are very short and you have a spare core to spin on. Kernel code protecting brief data structure updates is the classic case. Spinning is a bad idea on a single core — the spinner burns the entire quantum waiting for a holder that cannot run until the spinner yields. It is also bad for long hold times, where you burn a core doing nothing. Real implementations hedge with adaptive locking: spin briefly, then block if the lock has not been released. The JVM does this, and biased and thin locks are further optimisations for the uncontended case. This is why hand-rolling locks rarely beats the platform.

Producers add items to a shared bounded buffer and consumers remove them. The buffer must never overflow or underflow, and access must be mutually exclusive. The classic semaphore solution uses three primitives: a mutex protecting the buffer, an "empty" semaphore counting free slots, and a "full" semaphore counting available items. A producer waits on empty, acquires the mutex, inserts, releases the mutex, and signals full. The consumer mirrors it. The order matters critically. Acquiring the mutex before waiting on the counting semaphore deadlocks: a producer holding the mutex on a full buffer waits for a consumer that cannot acquire the mutex to drain it. With monitors it is a lock plus two condition variables, and the wait must be in a while loop, not an if, because of spurious wakeups and because another thread may have consumed the item between the signal and your reacquiring the lock. In practice you use a BlockingQueue, which packages all of this. Understanding the primitives still matters for tuning capacity and choosing between blocking, dropping, and back-pressure when full.

A condition variable lets a thread sleep until some predicate becomes true, releasing the associated lock while it waits and reacquiring it on wake. The loop is mandatory for two reasons. Spurious wakeups: a thread can wake without any signal at all, permitted by the specification because it makes implementations simpler and faster. And stolen wakeups: between the signal and the waiter reacquiring the lock, a third thread can run and invalidate the condition — the item it was signalled about has already been taken. So the pattern is always: acquire the lock, while the predicate is false wait, then act. Using if instead of while produces a bug that appears rarely and under load, which is the worst kind. notify versus notifyAll is the related trap. notify wakes one arbitrary waiter, and if waiters are waiting on different predicates it can wake the wrong one, which goes back to sleep while a thread that could have proceeded is never woken. Use notifyAll unless all waiters are provably interchangeable. Java's Condition with ReentrantLock gives multiple condition queues per lock, which solves that properly.

Visibility and ordering — not atomicity. Visibility: a write to a volatile variable is immediately visible to other threads. Without it, a thread may cache the value in a register or core-local cache and never observe another thread's update, which is how a plain boolean stop flag can loop forever. Ordering: volatile establishes a happens-before relationship. Everything written before a volatile write is visible to any thread that reads that volatile afterwards. This is what makes the double-checked locking idiom correct, and why it was broken before Java 5. What it does not give you is atomicity of compound operations. A volatile counter++ still races, because the read-modify-write is three steps and volatile makes each step visible without making the sequence indivisible. So: volatile for a flag written by one thread and read by others. AtomicInteger or a lock for anything read-modify-write. The practical tell is that if you find yourself wanting volatile on something you also increment, you want an atomic type instead.

A memory barrier is an instruction that constrains how memory operations may be reordered around it. Both compilers and CPUs reorder aggressively for performance. Within a single thread that reordering is invisible, because the hardware preserves the appearance of sequential execution for that thread. Across threads it is very visible: another thread can observe writes in an order the source code never expresses. A barrier forbids specific reorderings. A store barrier ensures earlier writes are visible before later ones; a load barrier ensures earlier reads complete before later ones; a full barrier does both. You rarely write barriers directly in application code. Volatile reads and writes, lock acquisition and release, and atomic operations all emit the appropriate barriers on your behalf — that is a large part of what they are actually doing under the hood. The reason to understand them is diagnosis. Code that works on x86 and fails on ARM is almost always a missing barrier, because x86 has a strong memory model that hides many mistakes while ARM's weaker model exposes them.

Multiple readers can safely access shared data concurrently, but a writer needs exclusive access. The problem is coordinating that without starving either side. A reader-preferring solution lets readers in whenever no writer is active. It maximises read throughput but starves writers — a continuous stream of readers means a waiting writer never gets in. A writer-preferring solution blocks new readers once a writer is waiting. Writers make progress, but a heavy write load starves readers. A fair solution queues both in arrival order, sacrificing some throughput for bounded waiting on both sides. Java provides ReentrantReadWriteLock, which can be constructed fair or unfair, and StampedLock, which adds an optimistic read mode: read without locking, then validate that no write intervened, retrying if it did. That is dramatically faster when writes are rare. The judgement call is whether it is worth it at all. A read-write lock has higher overhead than a plain mutex, so it only pays off when reads genuinely dominate and critical sections are long enough for the concurrency to matter.

An atomic operation completes entirely or not at all, with no observable intermediate state and no possibility of interleaving. Hardware provides the foundation with instructions such as compare-and-swap: atomically check that a memory location holds an expected value and, if so, replace it. The CPU guarantees indivisibility, typically by locking the cache line for the duration. Lock-free algorithms are built on this. AtomicInteger.incrementAndGet loops: read the current value, compute the new one, attempt a compare-and-swap, and retry if another thread intervened. No thread ever blocks — a thread that fails simply tries again. That is why atomics outperform locks under moderate contention: no context switching, no scheduler involvement. Under very high contention the retry loop can waste more CPU than blocking would, and LongAdder exists for exactly that case, spreading updates across cells and summing on read. The hazard worth naming is the ABA problem: a value changing from A to B and back to A passes the compare-and-swap even though the state was modified. AtomicStampedReference solves it by pairing the value with a version counter.

Both give mutual exclusion and reentrancy. ReentrantLock adds capabilities synchronized cannot express. synchronized is a language construct, so the lock is released automatically when the block exits, including on exception. That safety is its main virtue — you cannot forget to unlock. ReentrantLock is a class requiring an explicit unlock in a finally block, and forgetting it leaks the lock permanently. In exchange you get tryLock, so you can attempt acquisition and do something else on failure; timed acquisition, so you can give up after a bound; interruptible acquisition, so a blocked thread can be cancelled; optional fairness, granting the lock to the longest waiter; and multiple condition variables per lock. Performance is no longer a differentiator — modern JVMs optimise synchronized heavily with biased and thin locking, and the two are comparable under most contention. So the rule is: use synchronized by default for its safety, and reach for ReentrantLock only when you need one of those specific capabilities. Needing a timeout or multiple conditions is the usual trigger.

Thread confinement means data is only ever accessed by one thread, so no synchronisation is needed at all. It is the best strategy because it removes the problem rather than managing it. There is no lock to contend on, no deadlock risk, no memory visibility question, and no performance cost. It takes several forms. Stack confinement: local variables in a method are inherently per-thread. ThreadLocal: explicitly per-thread storage, used for things like SimpleDateFormat instances, which are notoriously not thread-safe. And ownership confinement, where a design convention hands an object to exactly one thread — the Swing and Android UI thread rules work this way. The risk is accidental publication. Confinement is a convention the compiler cannot check, so returning a reference or storing it in a shared field silently breaks it. The ThreadLocal caveat worth raising: with pooled threads, values persist across unrelated tasks, so failing to remove them leaks memory and can leak data between requests. In a servlet container that is a genuine security concern, not just a leak.

Two threads modify different variables that happen to sit on the same cache line, so the hardware treats them as contended even though the program logically has no sharing at all. Caches operate on lines, typically 64 bytes, not individual variables. When one core writes to a line, that line is invalidated in every other core's cache. If two threads are updating adjacent counters, each write forces the other core to refetch, and the line ping-pongs between them. The result is that a perfectly parallel workload scales badly or gets slower with more threads, with no lock and no shared variable in sight — which makes it maddening to diagnose. The fix is padding: separate the variables so they occupy different lines. Java has @Contended for this, though it needs a JVM flag to enable. Alternatively, restructure so each thread accumulates locally and combines at the end. This is precisely why LongAdder outperforms AtomicLong under contention — it spreads updates across padded cells rather than hammering one location.

A monitor is a synchronisation construct bundling mutual exclusion with the ability to wait for a condition — a lock plus one or more condition variables, attached to an object. Only one thread may be active inside the monitor at a time. A thread that cannot proceed calls wait, which atomically releases the lock and suspends it; another thread later signals, and the waiter reacquires the lock before continuing. That atomic release-and-suspend is the essential detail. If releasing the lock and sleeping were separate steps, a signal arriving in between would be missed and the thread would sleep forever — the lost wakeup problem. Java builds a monitor into every object: synchronized acquires it, and wait, notify and notifyAll operate on its condition queue. That is why those methods must be called while holding the lock, and throw IllegalMonitorStateException otherwise. The limitation of Java's built-in monitor is a single condition queue per object, so all waiters share one queue regardless of what they are waiting for. ReentrantLock with multiple Conditions is the way to get separate queues.

Mutual exclusion: at least one resource is held exclusively. Hold and wait: a process holds one resource while waiting for another. No preemption: resources cannot be forcibly taken, only voluntarily released. Circular wait: a cycle of processes each waiting on a resource held by the next. All four must hold simultaneously. That is the useful part, because breaking any one prevents deadlock entirely. Mutual exclusion is usually not negotiable — some resources genuinely cannot be shared. Hold and wait can be broken by requiring all resources be acquired at once, at the cost of poor utilisation. No preemption can be broken by allowing rollback, which is what databases do when they abort a transaction. Circular wait is broken by imposing a global ordering on resource acquisition. In practice, ordering is the technique that gets used, because it is cheap and requires no runtime machinery: agree that lock A is always taken before lock B, everywhere, and a cycle becomes impossible. Being able to name which condition your fix breaks is what separates a memorised answer from an understood one.

Lock ordering is the primary technique. Establish a global order for acquiring locks and follow it everywhere. If every thread takes A before B, no cycle can form. When locks are dynamic — transferring money between two accounts — order by something stable such as the account ID or the identity hash code. Lock timeouts are the second line. tryLock with a timeout means a thread that cannot acquire everything backs off, releases what it holds, and retries. That breaks hold-and-wait, though you must add jitter or two threads can livelock retrying in lockstep. Beyond that: reduce lock scope so windows are short, avoid calling unknown code while holding a lock since a callback may acquire locks you cannot see, and prefer higher-level constructs — concurrent collections and immutable objects remove the question entirely. The pragmatic answer many systems settle on is detection rather than prevention. Databases let deadlocks happen, detect the cycle, and abort the cheapest transaction, because prevention would cost more than the occasional retry.

A deadlock avoidance algorithm. Before granting a resource request, it checks whether the resulting state is safe — meaning there exists some ordering in which all processes can complete. It requires each process to declare its maximum resource needs upfront. The system tracks allocated, maximum, and available resources, and computes need as maximum minus allocated. A state is safe if you can find a process whose remaining need is satisfiable from what is available; assume it runs, completes, and returns everything, then repeat. If all processes can be sequenced this way, the state is safe. A request is granted only if the hypothetical resulting state is still safe; otherwise the process waits, even though the resources are currently free. It is a classic exam topic and essentially unused in practice. Declaring maximum needs in advance is unrealistic, the safety check is O(n²m) on every request, and it assumes a fixed process count. Real systems use prevention through lock ordering, or detection and recovery. Knowing why it is impractical is the more useful half of the answer.

Take a thread dump — jstack against the PID, or kill -3, or jcmd Thread.print. The JVM detects monitor deadlocks itself and prints an explicit "Found one Java-level deadlock" section naming the threads and the locks each holds and wants. That section is the fastest path. Without it, look for threads in BLOCKED state and follow the "waiting to lock" and "locked" annotations to find a cycle. Programmatically, ThreadMXBean.findDeadlockedThreads() returns the involved thread IDs, which is useful for a health check that fails a pod rather than leaving it hung. The caveat worth stating: the JVM only detects deadlocks on intrinsic monitors and ReentrantLocks. A deadlock formed with semaphores, or across a database and application lock, or on a bounded thread pool where tasks wait on other tasks in the same pool, will not be reported — you will just see stuck threads. That last case, pool exhaustion by inter-dependent tasks, is far more common in practice than classic lock-ordering deadlock, and it looks identical from the outside.

Deadlock: processes are blocked forever, each waiting on a resource another holds. No one makes progress and no one is executing. Livelock: processes are actively executing and changing state, but making no useful progress. The corridor analogy — two people repeatedly stepping aside in the same direction. It often arises from naive deadlock recovery: both threads detect contention, both release and retry in lockstep, and they collide again indefinitely. Random backoff is the standard fix. Starvation: a process could run but never gets the chance, because the scheduler or lock always favours others. Unlike deadlock, the resource does become available — this particular process is just never chosen. The distinctions matter for diagnosis. Deadlock shows threads blocked with zero CPU. Livelock shows high CPU with no throughput, which is easy to mistake for being busy. Starvation shows overall progress with one component making none. So: no CPU and no progress means deadlock, high CPU and no progress means livelock, progress overall but not for you means starvation.

Five philosophers sit around a table with one fork between each pair. Each needs both neighbouring forks to eat. If every philosopher picks up their left fork simultaneously, all five hold one fork and wait forever for the right. It is a compact demonstration of all four deadlock conditions at once, which is why it survives as a teaching problem. The solutions map onto the general prevention techniques. Resource ordering: number the forks and always pick up the lower-numbered one first, so the last philosopher reaches right-then-left and the cycle breaks. Limiting concurrency: allow at most four philosophers to attempt eating, guaranteeing one can always complete. Asymmetry: odd philosophers take left first, even take right first. Or an arbitrator that grants both forks atomically. The ordering solution is the one that generalises to real code, and it is worth saying so — this is the same technique as consistent lock ordering in a service. The follow-up is usually fairness: several solutions prevent deadlock but allow one philosopher to starve, which is a separate property to check.

Once a deadlock exists, something must be broken. There are two levers: terminate processes, or preempt resources. Termination can abort every deadlocked process, which is simple and definitely works but discards a lot of work. Or abort them one at a time, re-running detection after each, until the cycle breaks. Choosing the victim uses cost heuristics: how long it has run, how much work remains, how many resources it holds, and whether it can be restarted safely. Preemption takes a resource from one process and gives it to another. That requires the ability to roll back to a safe state, since a process interrupted mid-update leaves inconsistent data. Databases can do this because transactions have rollback built in; general applications usually cannot. The practical concern is starvation during recovery: if victim selection always picks the same process, it never completes. Including the number of prior rollbacks in the cost function prevents that. This is exactly what a database does on a deadlock — detect the cycle, pick the cheapest transaction, roll it back, and return a deadlock error for the client to retry.

Pool-induced deadlock happens when tasks in a pool wait on the results of other tasks submitted to the same pool. If all threads are occupied by waiting tasks, the tasks they are waiting for can never be scheduled. It requires no locks at all, which is why it is missed. A pool of ten threads running ten tasks that each submit a subtask and block on its result will hang permanently, and a thread dump shows ten threads in WAITING with no deadlock reported. The fixes: never submit and block within the same pool — use separate pools for dependent stages so the dependency cannot consume its own capacity. Or make the dependency non-blocking, composing with CompletableFuture instead of calling get. Or use a work-stealing pool such as ForkJoinPool, where a blocked worker can execute pending tasks rather than idling, which is what ManagedBlocker exists to support. The general principle is the same as ordinary deadlock: a cycle in the wait-for graph. Here the resource is a pool thread rather than a lock, which makes the cycle harder to see.

Virtual memory gives every process its own private address space, translated to physical memory by the hardware MMU using page tables maintained by the kernel. It solves several problems at once. Isolation: a process cannot address another's memory, because the translation simply does not exist. Simplicity: every program can be compiled assuming it starts at the same address, with no relocation. Overcommit: the total virtual memory across processes can exceed physical RAM, with unused pages paged out to disk. And fragmentation: physically scattered frames can appear contiguous in the virtual address space. The cost is a translation on every memory access, which is why the TLB exists to cache recent translations. The practical consequence backend engineers meet is the gap between virtual and resident memory. A JVM with -Xmx8g reserves 8 GB of virtual address space immediately, but resident memory grows only as pages are touched. Alarming at virtual size is a common monitoring mistake — RSS is the number that reflects real usage.

Paging divides the virtual address space into fixed-size pages and physical memory into equally sized frames, then maps pages to frames arbitrarily. A virtual address splits into a page number and an offset. The page number indexes the page table to find the frame number; the offset is carried through unchanged, since pages and frames are the same size. Concatenating the frame number with the offset gives the physical address. With 4 KB pages the offset is the low 12 bits and everything above is the page number. The problem is that a single-level page table for a 64-bit address space would be astronomically large. Real systems use multi-level tables — typically four levels on x86-64 — so only the portions actually mapped need to exist. Sparse address spaces then cost almost nothing. The downside is that translation now requires several memory accesses to walk the levels, which would be crippling. The TLB caches completed translations so the walk is skipped on a hit, and hit rates above 99% are normal.

The Translation Lookaside Buffer is a small, very fast cache of recent virtual-to-physical page translations, sitting inside the MMU. Without it, every memory access would require walking the multi-level page table — four extra memory accesses on x86-64 just to find out where the data lives. The TLB collapses that to a single lookup on a hit. It is small, typically a few hundred to a couple of thousand entries, so it covers only a limited working set. With 4 KB pages, 1,500 entries covers about 6 MB. An application whose hot data exceeds that thrashes the TLB and pays the walk repeatedly. That is what huge pages address. A 2 MB page means one entry covers 512 times more memory, so the same TLB covers gigabytes. Databases and JVMs with large heaps benefit measurably — -XX:+UseLargePages exists precisely for this. Context switches flush the TLB unless the hardware supports address space identifiers, which is a significant part of why process switches cost more than thread switches.

Internal fragmentation is wasted space inside an allocated block. Request 100 bytes from a 4 KB page and the remaining bytes are unusable by anyone else, even though they are technically free. External fragmentation is wasted space between allocations. Enough total memory is free to satisfy a request, but no single contiguous run is large enough. Paging eliminates external fragmentation entirely, because any free frame can back any page — contiguity in physical memory is no longer required. It introduces internal fragmentation instead, bounded by half a page per allocation on average. That is a good trade, and it is the main reason paging beat segmentation. Segmentation has the opposite profile: variable-size segments mean no internal waste but severe external fragmentation, requiring compaction to fix. The same tension appears in application allocators. Slab allocators and size-class allocators like jemalloc accept internal fragmentation to avoid external fragmentation and to make allocation O(1), which is why long-running services with varied allocation sizes usually perform better on them than on a naive free-list.

A page fault is a trap raised when a process accesses a virtual page that is not currently mapped to a physical frame. A minor fault means the page is in memory but not mapped into this process's page table — it is already in the page cache, or shared with another process, or being mapped lazily on first touch. Resolution is fast: update the page table and continue. No disk involved. A major fault means the page must be read from disk, either from swap or from a memory-mapped file. That costs milliseconds on spinning disks, tens of microseconds on NVMe — orders of magnitude slower than a minor fault. An invalid fault means the access is genuinely illegal, and the process gets a segmentation fault. For diagnosis this distinction is the whole point. A high minor fault rate is normal and largely harmless. A high major fault rate means you are swapping, and performance will be dreadful. On Linux, ps and /proc/PID/stat expose both counts, and vmstat's si/so columns show swap activity directly.

FIFO evicts the oldest page. Simple, but it ignores usage, so a heavily used page gets evicted purely for being old. It also suffers Belady's anomaly, where adding more frames can increase faults. Optimal (OPT) evicts the page that will not be used for the longest time. It is provably best but requires knowing the future, so it exists only as a benchmark. LRU evicts the least recently used page, approximating OPT by assuming recent use predicts future use. It performs well but true LRU requires updating a timestamp or list on every access, which is far too expensive in hardware. So real systems approximate it. The clock (second chance) algorithm keeps a reference bit per page, sweeps a pointer round, and evicts the first page whose bit is clear, clearing bits as it goes. Cheap and close enough. LFU evicts the least frequently used, which handles some patterns better but adapts badly when access patterns change. Linux uses a two-list variant separating active and inactive pages, which resists the classic failure of a single large sequential scan evicting the entire working set.

Thrashing is when a system spends more time paging than executing. The working sets of the running processes exceed physical memory, so every process's pages are evicted by another's, and each then faults them straight back in. The signature is distinctive: CPU utilisation collapses while disk I/O saturates. That combination is counterintuitive and is exactly what identifies it — the CPU is idle because every process is blocked on a page fault. The pathological feedback loop is what makes it so damaging. A scheduler that responds to low CPU utilisation by admitting more processes makes it strictly worse. The fixes are to reduce the memory demand — kill or suspend processes, which is what the OOM killer does — or add physical memory. Tuning the page replacement algorithm does not help, because the problem is capacity, not policy. On Linux, vmstat showing sustained non-zero si and so columns alongside low CPU is the confirmation. In containers this often appears instead as an OOM kill, since swap is typically disabled and the kernel has no option but to kill.

The working set is the set of pages a process has referenced in the most recent window of time. It approximates the memory the process actually needs right now. The insight is locality: programs do not access memory uniformly. Over any short interval they touch a small subset — a loop and its data — and that subset shifts gradually as execution moves between phases. The practical use is admission control. If the sum of all working sets exceeds physical memory, thrashing is inevitable, so the system should suspend a process rather than let everything degrade. That is far better than reacting after thrashing has started. The window size is the tuning parameter, and it is awkward: too small and you miss pages the process genuinely needs, too large and you include pages from a previous phase it has finished with. The application-level parallel is sizing a cache. A cache smaller than the working set has a poor hit rate no matter how good the eviction policy is, which is why measuring the working set matters more than choosing between LRU and LFU.

The stack holds function call frames — parameters, locals, return addresses — and grows and shrinks automatically as calls are made and return. Allocation is a pointer bump, so it is essentially free, and memory is reclaimed deterministically on return. The heap holds dynamically allocated objects whose lifetime is not tied to a call frame. Allocation involves finding a suitable block, so it is slower, and reclamation requires explicit freeing or garbage collection. The stack is small and fixed — typically 1 MB per thread in Java — which is why deep recursion overflows it and why large arrays go on the heap. It is also per-thread, so stack data is inherently thread-confined. The heap is large and shared, which is exactly why it needs synchronisation and why escape analysis matters: the JVM can allocate an object on the stack instead if it proves the object never escapes the method, avoiding GC pressure entirely. Cache behaviour differs too. The stack is hot and contiguous; the heap is scattered, so heap-heavy code has worse locality.

mmap maps a file directly into a process's virtual address space, so reading the file is just reading memory and the kernel pages content in on demand. The advantages are real. It eliminates a copy: conventional read() copies from the page cache into your buffer, while mmap lets you access the page cache directly. It gives lazy loading, since only pages actually touched are read. And multiple processes mapping the same file share physical pages, so the memory cost is paid once. That is why databases and message brokers use it heavily — Kafka's performance rests substantially on mmap plus sendfile. The drawbacks matter too. Page faults are invisible in the code, so a simple array access can block on disk for milliseconds with nothing in the source suggesting I/O. Error handling is worse, since an I/O error surfaces as a SIGBUS rather than a return code. And on 32-bit systems address space limits the file size. For sequential streaming of a large file read once, ordinary buffered I/O is usually simpler and no slower.

They move data in opposite directions and mean different things. The page cache holds file contents in memory — pages read from or written to disk files. It is a pure optimisation: dropping a clean page loses nothing, since it can be re-read from the file. This is why Linux appears to use all available RAM, and why "free" memory being low is not a problem. Swap holds anonymous memory — heap and stack pages that have no file backing — written out to a swap device when memory is scarce. Dropping them is impossible, because the data exists nowhere else. So page cache eviction is cheap and swap is expensive. Swapping means the system has run out of options and is trading enormous latency for capacity. The practical implication is that free memory is the wrong metric — available memory, which counts reclaimable page cache, is the one to alarm on. And sustained swap activity is always a problem, whereas a full page cache is healthy. Most container deployments disable swap entirely, preferring a fast OOM kill to unpredictable latency.

When Linux cannot satisfy an allocation and cannot reclaim enough memory, the out-of-memory killer terminates a process to free memory rather than letting the whole system fail. It scores each process, primarily by memory consumption relative to the total, adjusted by oom_score_adj which ranges from -1000 (never kill) to +1000 (kill first). The highest score is killed. That heuristic means the biggest consumer usually dies, which is often your main application rather than whatever actually caused the pressure — a frequent source of confusion when the JVM is killed by a memory spike in a sidecar. It exists because Linux overcommits: it grants more virtual memory than it has physical memory plus swap, betting that not everything will be touched. Usually that is right, and OOM is what happens when the bet fails. In containers, the memory cgroup limit triggers a cgroup-scoped OOM kill, which is what exit code 137 means in Kubernetes. Diagnosing that requires checking dmesg or kernel logs — the application itself gets no chance to log anything, because SIGKILL cannot be handled.

Paging divides memory into fixed-size blocks with no relationship to program structure. Segmentation divides it into variable-size units that correspond to logical divisions — code, data, stack, a particular module. Segmentation is closer to how a programmer thinks, and it allows protection and sharing at a meaningful granularity: mark the code segment read-only and executable, share a library segment between processes. But variable sizes cause external fragmentation. After enough allocation and freeing, memory is a patchwork of gaps too small to use, requiring expensive compaction. Paging avoids that entirely, at the cost of losing the logical correspondence — a page boundary can fall in the middle of anything. Modern x86 supports both, but practically everything uses paging with a flat segmentation model where the segments span the whole address space and effectively do nothing. Combined segmentation-with-paging was used historically but the complexity was not worth it. The residue you still meet is terminology — "segmentation fault" is named for a mechanism the system barely uses any more.

The JVM reserves its maximum heap as virtual address space at startup, then commits physical pages as the heap grows. So -Xmx8g reserves 8 GB of address space immediately — visible as virtual size — but resident memory starts small and grows as objects are allocated and pages are touched. Alarming on virtual size produces constant false positives. Crucially, the heap is not the whole footprint. A JVM process also consumes metaspace for class metadata, thread stacks at roughly 1 MB each, the code cache for JIT-compiled methods, GC data structures, direct byte buffers, and native memory from libraries. Total RSS routinely runs 25 to 50 percent above the heap size. That gap is what kills containers. Setting -Xmx equal to the container memory limit guarantees an eventual OOM kill, because the non-heap portion has nowhere to live. Modern JVMs are container-aware and read cgroup limits, so -XX:MaxRAMPercentage is the better control than a fixed -Xmx. Native Memory Tracking is the tool for attributing the non-heap portion when RSS is unexpectedly high.

Pages are loaded into physical memory only when actually accessed, rather than when the process starts. Starting a process maps its address space but commits almost nothing. The first access to each page triggers a fault, and the kernel loads that page from the executable or allocates a zero-filled frame. The benefits are substantial. Process startup is fast, because you are not waiting to load a whole binary. Memory holding code paths that are never executed is never consumed — error handling, rarely used features. And you can run programs larger than physical memory. The cost is that the first touch of each page is slow, so startup latency is spread out rather than eliminated. That is part of why JVM applications have a warm-up period. Prepaging is the counter-optimisation: when faulting on a page, load some neighbours too, betting on spatial locality. Reading 16 pages costs barely more than reading one, since the seek dominates. Linux does this with readahead, and it is why sequential file access is so much faster than random.

The counterintuitive result that adding more page frames can increase the number of page faults, under certain replacement algorithms and reference strings. It occurs with FIFO. The standard demonstration is the reference string 1,2,3,4,1,2,5,1,2,3,4,5: with three frames it produces nine faults, with four frames it produces ten. The reason is that FIFO ignores usage. Adding a frame changes which pages happen to be oldest at each point, and can cause a page that is about to be used to be evicted where previously it survived. There is no guarantee that the set of pages resident with n+1 frames includes those resident with n. That property has a name: stack algorithms are those where the resident set with n frames is always a subset of that with n+1 frames. LRU and OPT are stack algorithms, so they cannot exhibit the anomaly. FIFO is not. The practical lesson is not about FIFO specifically but about the danger of assuming more resources always help. The same reasoning applies to cache sizing and connection pool sizing.

A logical (virtual) address is what a program generates and sees. A physical address is an actual location in RAM. The program only ever works with logical addresses. The MMU translates each one to a physical address at access time using the page tables — hardware doing the work, not software, because it happens on every single memory access. The separation is what makes isolation and relocation possible. Two processes can both use address 0x400000 and reference completely different physical memory. A process can be swapped out and back into different physical frames, and it never notices, because its logical addresses are unchanged. Compile-time and load-time binding produce addresses fixed before execution and cannot relocate a running process. Execution-time binding, which virtual memory provides, defers translation to every access and is what allows swapping and dynamic relocation. The practical relevance is debugging: a pointer value in a core dump is a virtual address, meaningful only within that process's address space, which is why the same address in two processes tells you nothing about sharing.

They avoid allocating table entries for regions of the address space that are not mapped. A single-level table must have an entry for every possible page. For a 48-bit address space with 4 KB pages that is 2^36 entries — hundreds of gigabytes per process, which is absurd given most of the space is unused. A multi-level table splits the page number into several indexes. The top-level table is small and always present, but its entries point to second-level tables that are allocated only if that region is actually used. A process using a few megabytes needs a handful of tables rather than a complete map of the address space. x86-64 uses four levels, with a fifth optional for very large address spaces. The cost is that a translation now requires walking every level — four memory accesses before touching the data. That is why the TLB is essential rather than merely helpful, and why TLB misses are so much more expensive than they first appear. Inverted page tables are the alternative, indexing by frame rather than page, which bounds size by physical memory but makes lookup harder.

Huge pages are larger page sizes — typically 2 MB or 1 GB on x86-64 instead of 4 KB. The benefit is TLB coverage. Each TLB entry maps one page, so with 4 KB pages a 1,500-entry TLB covers about 6 MB. With 2 MB pages the same TLB covers 3 GB. An application with a large working set stops thrashing the TLB, and page table walks drop dramatically. They also shrink the page tables themselves, since fewer entries are needed. The wins are largest for applications with big, randomly accessed memory: databases, large JVM heaps, in-memory caches. Improvements of several percent to tens of percent are reported. The costs are internal fragmentation — a 2 MB page allocated for a small mapping wastes most of it — and allocation difficulty, since the kernel needs 2 MB of contiguous physical memory, which is hard on a fragmented system. Transparent Huge Pages automate this, but their background defragmentation can cause latency spikes, which is why databases such as MongoDB and Redis explicitly recommend disabling THP while still supporting explicit huge pages.

A dirty page is one that has been modified in memory but not yet written back to its backing store. The hardware tracks this with a dirty bit set on the first write. It matters for eviction cost: a clean page can simply be discarded, since an identical copy exists on disk, while a dirty page must be written out first. That makes evicting dirty pages substantially more expensive, and replacement algorithms prefer clean pages when the choice is close. For file data, dirty pages accumulate in the page cache and are flushed by kernel writeback threads, controlled by vm.dirty_ratio and vm.dirty_background_ratio. Buffering writes this way is what makes file I/O fast — you return as soon as the page cache is updated. The risk is data loss on a crash, since acknowledged writes may still be only in memory. That is exactly why fsync exists and why databases call it before acknowledging a commit. A large dirty page backlog also causes latency spikes when writeback finally triggers, which is why tuning those ratios matters on write-heavy systems.

Overcommit means the kernel grants more virtual memory than it has physical memory plus swap to back. It does this because processes routinely reserve far more than they touch. fork() duplicates an address space that is usually immediately discarded by exec. Programs allocate generous buffers they never fill. A JVM reserves its whole maximum heap upfront. Refusing these allocations would waste enormous capacity. So the kernel bets that not everything will be used simultaneously. Usually it is right. When it is wrong, there is no graceful option — the memory was already promised and the process is writing to it — so the OOM killer terminates something. The behaviour is tunable via vm.overcommit_memory: heuristic by default, always allow, or strict accounting which refuses allocations exceeding a computed limit. Strict mode makes malloc fail rather than risking an OOM kill later, which some databases prefer because a failed allocation can be handled while SIGKILL cannot. The practical takeaway is that a successful allocation on Linux is not a guarantee the memory exists — only that the kernel expects it will.

Non-Uniform Memory Access describes multi-socket systems where each CPU has its own local memory. Accessing local memory is fast; accessing another socket's memory goes over an interconnect and costs noticeably more — often 1.5 to 2 times the latency. It matters because the default allocation policy assigns memory near whichever CPU first touches it. A thread that allocates on one socket and is then migrated to another does every subsequent access remotely. The symptom is confusing: an application performs well on a single-socket machine and worse on a bigger dual-socket one, with no obvious bottleneck. The mitigations are to pin threads and their memory to the same node with numactl, to use first-touch allocation deliberately so each thread initialises the memory it will use, or to interleave allocation across nodes when access is genuinely uniform. For the JVM, -XX:+UseNUMA makes the collector NUMA-aware, allocating in node-local regions. Most cloud instances are single-socket or expose a single NUMA node, so this often does not arise — but on large bare-metal database servers it is a first-order effect.

VSZ is virtual size — the total address space the process has reserved, including memory never touched, files mapped but not read, and shared libraries. It is almost always much larger than real usage and is a poor metric to alarm on. RSS is resident set size — physical memory currently in use by the process. Closer to reality, but it counts shared pages in full for every process sharing them. Twenty processes sharing one 50 MB library each report the full 50 MB, so summing RSS across processes massively overcounts. PSS is proportional set size, which divides each shared page's cost among the processes sharing it. That 50 MB library contributes 2.5 MB to each of twenty processes, so PSS actually sums correctly to the true total. For a single process RSS is the usual answer. For total system usage across many processes PSS is the honest one, available in /proc/PID/smaps. In containers the number that triggers OOM kills is the cgroup memory usage, which includes page cache attributed to the cgroup — another reason RSS alone can mislead.

An unintentional retention: objects that are no longer needed but remain reachable, so the collector cannot free them. Garbage collection eliminates the classic C leak of forgetting to free, but it cannot know intent. If a reference exists, the object stays. The common causes are all forms of accidental retention. A static collection that only ever grows. Listeners or callbacks registered and never removed, keeping their enclosing objects alive. ThreadLocal values in a pooled thread, which outlive the request. Caches with no eviction policy. Inner classes holding an implicit reference to their outer instance. And unclosed resources holding native memory. The symptom is a heap that grows steadily across GC cycles, with full collections reclaiming progressively less — eventually GC thrashing, where the collector runs constantly and reclaims almost nothing, before OutOfMemoryError. Diagnosis is a heap dump plus a dominator tree in a tool such as Eclipse MAT, which shows what is retaining the memory. The question is always "what still references this?", and the path to GC roots answers it.

Badly, and it is a genuine production trap. After fork, parent and child share physical pages marked read-only, and pages are copied only on write. The expectation is that a mostly-read child costs almost no additional memory. A tracing garbage collector breaks that assumption. Marking typically sets a bit in the object header, which is a write. Compacting collectors move objects, writing even more. So a GC cycle in either process touches a large fraction of the heap and forces copying of pages that were logically never modified. The result is that memory usage can approach double the heap shortly after a fork, on a workload that appeared read-only. This is why Redis, which forks for background saves, warns that memory can spike and recommends sizing accordingly, and why its documentation discusses write load during a save. The mitigations are collectors that keep mark state outside the object — separate mark bitmaps — or avoiding fork-based patterns in GC languages altogether. It is a good example of two reasonable optimisations interacting badly.

Because the cgroup limit counts far more than the heap. A JVM process consumes heap plus metaspace, thread stacks at around 1 MB each, the JIT code cache, GC structures, direct and mapped byte buffers, and native allocations from libraries such as compression or crypto. Total RSS commonly exceeds the heap by 25 to 50 percent. The cgroup also charges page cache to the container. A process writing large files fills the page cache, and although that memory is reclaimable, it counts toward the limit and can trigger a kill under pressure. So setting -Xmx equal to the container limit guarantees an eventual kill. The heap alone can legitimately reach its maximum while everything else has nowhere to go. The fixes: use -XX:MaxRAMPercentage around 70 to 75 rather than a fixed -Xmx, bound thread counts since stacks are per-thread, enable Native Memory Tracking to attribute the non-heap portion, and set -XX:MaxDirectMemorySize explicitly if the application uses NIO buffers heavily. Exit code 137 in Kubernetes is SIGKILL, and the evidence is in dmesg rather than application logs.

An inode is the data structure describing a file: its metadata and the pointers to its data blocks. It holds the file type, permissions, owner and group, size, timestamps, link count, and block pointers. What it notably does not hold is the filename — names live in directory entries, which map a name to an inode number. That separation explains several behaviours. Hard links are multiple directory entries pointing at the same inode, which is why they share content and permissions and why deleting one does not remove the file. Renaming is cheap because only the directory entry changes. And a file can be unlinked while still open: the directory entry goes, but the inode survives until the link count and the open file count both reach zero. That last point is why deleting a large log file does not free space if a process still holds it open — a classic production surprise where df shows a full disk and du does not account for it. Inodes are a finite resource allocated at filesystem creation, so a partition can report "no space left" with free bytes but no free inodes.

A hard link is an additional directory entry pointing at the same inode. A symbolic link is a small file whose contents are a path to another file. Hard links are indistinguishable from the original — there is no "original", just multiple names for one inode. Deleting one leaves the data intact until the link count reaches zero. They cannot cross filesystems, because inode numbers are only meaningful within one filesystem, and conventionally cannot point at directories, since that would allow cycles. Symlinks store a path, so they can cross filesystems and point at directories. But they break if the target is moved or deleted, leaving a dangling link. Resolving one costs an extra lookup. The practical distinction is what happens on deletion. Remove the target of a symlink and the link is broken; remove one hard link and the data is fine. This is why deployment schemes use a symlink to a versioned directory — atomic switching by repointing the link — and why backup tools have to decide explicitly whether to follow symlinks or copy them.

A small non-negative integer identifying an open file within a process — an index into the process's file descriptor table. That table points to entries in a system-wide open file table, which holds the current offset and access mode, and those point to inodes. The three-level structure explains subtle behaviour: two separate open() calls on the same file get independent offsets, while a descriptor duplicated with dup() or inherited across fork() shares the offset, so writes from parent and child interleave rather than overwrite. Descriptors 0, 1 and 2 are stdin, stdout and stderr by convention, which is what makes shell redirection work. They are not only files. Sockets, pipes, devices, epoll instances and timers are all descriptors, which is what allows select and epoll to wait on heterogeneous sources uniformly — the "everything is a file" idea. The operational concern is exhaustion. Each process has a limit, and leaking descriptors by not closing connections eventually produces "too many open files". lsof and /proc/PID/fd are how you find the leak, and it is one of the most common causes of a service degrading over days.

fsync forces all buffered writes for a file descriptor out of the page cache and onto durable storage, and does not return until the device confirms. It matters because ordinary write() only updates the page cache. The call returns immediately and the data may sit in memory for seconds before writeback. If the machine loses power in that window, acknowledged writes are gone. Any database claiming durability must fsync its write-ahead log before acknowledging a commit. That is precisely what makes commits expensive — you are waiting on physical storage, not memory. The complications are notorious. Some drives lie, acknowledging a flush while data sits in a volatile drive cache. fdatasync is a cheaper variant that skips metadata when only content changed. And on some filesystems fsync on a file does not guarantee the directory entry is durable, so creating a file requires fsyncing the parent directory too. There is also the "fsyncgate" issue where a failed fsync on Linux could clear the error state, so a retry appeared to succeed while data was lost — which changed how PostgreSQL handles fsync errors.

A journal is a log of pending metadata changes written before the changes are applied, so a crash mid-update can be recovered. Without it, a crash during a multi-step operation leaves the filesystem inconsistent — a block marked allocated but referenced by nothing, or a directory entry pointing at an incomplete inode. Recovery meant fsck scanning the entire filesystem, which took hours on large volumes. With a journal, recovery replays or discards the log, which takes seconds regardless of filesystem size. The modes trade safety against speed. Journal mode logs both data and metadata, which is safest and slowest since everything is written twice. Ordered mode, the ext4 default, journals metadata but writes data blocks before the metadata that references them, so you never see a file pointing at stale content. Writeback mode journals metadata only with no ordering guarantee, which is fastest and can expose garbage in a file after a crash. Ordered is the sensible default because it prevents the genuinely dangerous outcome — reading someone else's deleted data — without the double-write cost.

The kernel keeps recently accessed file pages in memory, so repeated reads are served without touching the disk. On read, the kernel checks the cache first; a hit is a memory copy, a miss is a disk read that also populates the cache. On write, data goes into the cache and is marked dirty, with writeback happening later in the background — so write() returns long before anything reaches the disk. The kernel also does readahead, detecting sequential access and prefetching pages before they are requested, which is why sequential reads are dramatically faster than random ones. The cache uses all otherwise-free memory and is reclaimed under pressure, which is why Linux appears to have little free memory. That is healthy, not a problem. The consequences for application design: your own in-process cache may be duplicating the page cache, wasting memory. Direct I/O with O_DIRECT bypasses it, which databases use because they manage their own buffer pool and know their access patterns better than the kernel does. And benchmarks are meaningless unless you drop the cache or account for it.

Buffered I/O goes through the page cache. Reads may be served from memory, writes return once cached. Simple, fast for repeated access, and the default. Direct I/O with O_DIRECT bypasses the page cache and transfers straight between the device and a user buffer. It avoids double caching and gives the application full control over what is cached — which is why databases use it, since they have a buffer pool tuned to their access patterns. The cost is strict alignment requirements and no readahead, so it is slower for naive sequential access. Asynchronous I/O submits a request and returns immediately, with completion delivered later. It lets one thread keep many I/O operations in flight without blocking, which is essential for high-throughput servers. Linux AIO worked only with O_DIRECT and had awkward edges; io_uring is the modern replacement and supports buffered I/O properly with far lower overhead. The three are largely orthogonal concerns — caching versus blocking — which is a distinction interviewers like to hear made explicitly, since people often conflate "async" with "fast".

The directory entry is removed, so the name disappears and ls no longer shows it, but the data remains until the last open descriptor is closed. The inode tracks two counts: hard links and open descriptors. unlink decrements the link count; the inode and its blocks are freed only when both counts reach zero. The practical consequence is a classic production incident. Someone deletes a large log file to free disk space, df still reports the disk full, and du cannot find the space because the file has no name any more. The space returns only when the process holding it is restarted or the descriptor closed. The diagnosis is lsof +L1, which lists open files with a link count of zero. If restarting the process is unacceptable, truncating through /proc/PID/fd/N frees the blocks immediately. The same behaviour is used deliberately: creating a temp file and immediately unlinking it gives a file that is guaranteed to be cleaned up when the process exits, even on a crash.

Block devices transfer fixed-size blocks and support random access — disks, SSDs, USB storage. The kernel buffers them through the page cache and can reorder and merge requests for efficiency. Character devices transfer a stream of bytes with no inherent block structure and generally no seeking — terminals, serial ports, /dev/random. Access is unbuffered and sequential. The distinction determines which kernel subsystem handles them and what operations make sense. Seeking on a block device is natural; on a character device it usually is not. ls -l shows the type in the first character: b for block, c for character. The practical relevance is mostly in understanding what you can do with a device file. Mounting requires a block device. /dev/null and /dev/zero are character devices, which is why they stream indefinitely rather than having a size. Loop devices bridge the two, presenting a regular file as a block device — which is how you mount a disk image, and part of how container image layers work.

A journaling filesystem writes intended changes to a log first, then applies them in place. A copy-on-write filesystem never overwrites live data — it writes modified blocks to new locations and atomically updates pointers to reference them. Copy-on-write gives crash consistency without a separate journal: either the pointer update happened or it did not, and the old data is untouched either way. It also makes snapshots nearly free, since a snapshot is just a retained set of pointers to blocks that will not be reused. That is why ZFS and Btrfs support instant snapshots and cheap clones, and why container storage drivers like overlayfs use copy-on-write layering. The cost is fragmentation. Blocks are relocated on every write, so a file that is repeatedly modified in place becomes scattered — which is bad for databases doing random updates, and is why running a database on a copy-on-write filesystem often needs the copy-on-write behaviour disabled for the data directory. Write amplification is the related concern: updating one byte may require rewriting a block plus the pointer chain above it.

Blocking I/O suspends the calling thread until the operation completes. Simple to reason about, but it requires one thread per concurrent connection, and threads are expensive. Non-blocking I/O returns immediately, with an error if the operation would block. The caller must retry, so naive use becomes a busy loop burning CPU. Multiplexed I/O solves that: select, poll or epoll let one thread wait on many descriptors at once and be told which are ready. One thread can then service thousands of connections, handling each only when it actually has data. That is the model behind Nginx, Node.js and Netty, and it is why they handle high connection counts on few threads. select and poll are O(n) in the number of watched descriptors, because they rescan the whole set each call. epoll is O(1) for ready notification since the kernel maintains the ready list, which is what makes the C10K problem tractable. The trade is programming model. Blocking code reads linearly; event-driven code fragments into callbacks, which is what virtual threads and async/await try to give back.

Direct Memory Access lets a device transfer data to or from main memory without the CPU copying each byte. Without it, programmed I/O has the CPU read from a device register and write to memory in a loop, consuming the processor for the entire transfer. Reading a gigabyte would saturate a core doing nothing but shuttling bytes. With DMA, the CPU programs a controller with a source, destination and length, then goes and does something else. The controller performs the transfer and raises an interrupt on completion. That is why a machine can copy files at gigabytes per second while remaining responsive. The complication is cache coherence: the device writes directly to memory, so the CPU's caches may hold stale copies. Hardware or the driver must invalidate appropriately, which is why DMA buffers have alignment and coherency requirements. DMA underpins zero-copy techniques. sendfile moves data from the page cache to a socket without it ever entering user space, which is a large part of why Kafka and static file servers achieve the throughput they do.

An interrupt is a hardware signal that causes the CPU to suspend its current work and run a handler. Polling is software repeatedly checking whether a device needs attention. Interrupts are efficient when events are infrequent: the CPU does useful work and is notified only when something happens. Polling wastes cycles checking a device that is usually idle. But interrupts have overhead — saving state, entering the handler, restoring — and at very high event rates that overhead dominates. A 10 Gbps network card can generate millions of packets per second, and one interrupt per packet would consume the entire machine. That is called an interrupt storm. So high-throughput drivers switch adaptively. Linux NAPI takes the first interrupt then disables interrupts and polls until the queue drains, getting the best of both. Interrupt coalescing similarly batches several events into one interrupt. Handlers are split into a top half that runs with interrupts disabled and does the minimum, and a bottom half deferred to a softirq or workqueue — because time spent with interrupts disabled adds latency to everything else on the system.

Zero-copy eliminates redundant copies of data between kernel and user space. Sending a file over a socket conventionally takes four copies and four context switches: disk to page cache via DMA, page cache to a user buffer, user buffer to the socket buffer, socket buffer to the network card. Two of those copies pass through user space for no reason — the application never inspects the bytes. sendfile transfers directly from the page cache to the socket inside the kernel, removing both user-space copies and two context switches. With scatter-gather DMA support, the data can go from the page cache to the network card without even being copied into the socket buffer, leaving genuinely zero CPU copies. The throughput and CPU savings are large — often more than double. The constraint is that the application cannot touch the data, since it never enters user space. So it works for serving static files or replicating a log, and not when you need to transform or encrypt in userspace. Kafka's consumer path relies on this, which is a substantial part of its performance story.

They exist to minimise seek time on rotating disks by reordering pending requests. FCFS services in arrival order — fair but with terrible seek behaviour, since the head jumps randomly. SSTF picks the nearest request to the current head position. Good average seek time, but it starves distant requests when nearby ones keep arriving. SCAN, the elevator algorithm, sweeps the head from one end to the other servicing everything in its path, then reverses. No starvation, but requests just behind the head wait for a full sweep. C-SCAN sweeps in one direction only, then jumps back without servicing. This gives more uniform waiting times, at the cost of the return trip. The important modern caveat is that all of this matters far less on SSDs, which have no seek time — random access costs roughly the same as sequential. Linux offers the noop or none scheduler precisely for flash, where reordering adds overhead for no benefit. What still matters on SSDs is write amplification and queue depth, not head position, so the classic algorithms are increasingly historical.

An anonymous pipe is a unidirectional byte stream between related processes, created before a fork so the child inherits the descriptor. It has no name in the filesystem and disappears when both ends close — this is what the shell's | operator creates. A named pipe, or FIFO, is the same mechanism with a filesystem entry, so unrelated processes can open it by path. Still unidirectional and still local. A socket is bidirectional and can be local (Unix domain) or networked. Unix domain sockets are faster than TCP for local communication since they skip the network stack entirely, and they can pass file descriptors between processes, which pipes cannot. The choice follows the requirement. Pipes for simple parent-child streaming. FIFOs when the processes are unrelated but local. Unix sockets when you need bidirectional communication or descriptor passing locally. TCP sockets when the processes may be on different machines. The common gotcha with pipes is the buffer limit — typically 64 KB — after which a writer blocks. A reader that never drains deadlocks the writer, which is a frequent bug when spawning subprocesses.

Work through the resources systematically rather than guessing. Start with top or htop for overall CPU and load. High user CPU means application computation; high system CPU means excessive syscalls or context switching; high iowait means blocked on disk. For memory, free -h distinguishes used from available, and vmstat's si/so columns reveal swapping. Sustained swap activity explains almost any slowness on its own. For disk, iostat -x shows utilisation and await. A device near 100% utilisation with rising await is saturated. For network, ss -s for connection counts and states, and check for retransmissions. A large number of sockets in TIME_WAIT or CLOSE_WAIT indicates specific problems — the latter usually means the application is not closing connections. The pattern that catches people is low CPU with high latency, which almost always means waiting: on I/O, on a lock, or on a downstream service. That is when you go to a thread dump or distributed trace rather than to system metrics. The USE method — utilisation, saturation, errors for every resource — is a good structure to name.

iowait is the percentage of time the CPU was idle while at least one I/O request was outstanding. It means the CPU had nothing to run because everything was blocked on I/O. The crucial subtlety is that it is a form of idle time, not busy time. High iowait means the CPU is available and work is blocked — it is not itself a problem, it is a symptom. And it can be misleading in both directions. A machine with a slow disk but plenty of other work to run shows low iowait, because the CPU is never idle. So low iowait does not mean I/O is healthy. Conversely, a mostly idle machine with one slow I/O can show high iowait without any real issue. So iowait should never be read alone. Pair it with iostat: look at device utilisation and await, which measure the device directly rather than inferring from CPU idleness. On multi-core systems the number is also averaged across cores, which dilutes it further. The useful conclusion from high iowait is "look at the storage layer", not "the disk is the bottleneck".

For CPU, top sorted by CPU, then narrow to threads with top -H -p PID. For a JVM, map the thread ID to a Java thread by converting to hex and matching the nid in a jstack dump — that is how you find which thread is spinning. For memory, ps aux sorted by RSS, or smem for PSS which handles shared pages correctly. /proc/PID/status gives detailed breakdowns. For disk I/O, iotop shows per-process read and write rates, which iostat cannot since it is per-device. pidstat -d is the non-interactive equivalent. For network, ss -tp shows sockets with owning processes. nethogs gives per-process bandwidth. For file descriptors, lsof -p PID counts open handles, and ls /proc/PID/fd is the quick version. The general point worth making is that /proc is the underlying source for nearly all of these, so when a tool is unavailable — which is common in a minimal container — you can read /proc directly. That is often the difference between diagnosing an incident and waiting for someone to install a package.

They are the two kernel features containers are built from, and they do different jobs. Namespaces provide isolation — what a process can see. There are several: PID namespaces so a container has its own process tree with its own PID 1, network namespaces for separate interfaces and routing tables, mount namespaces for a distinct filesystem view, UTS for hostname, IPC for shared memory, and user namespaces for UID mapping. cgroups provide limits — what a process can use. CPU shares and quotas, memory limits, block I/O throttling, and device access. So a container is a process running with namespaces for isolation and cgroups for resource control, plus a filesystem image. There is no "container" object in the kernel — that is the key insight, and it explains why containers are so much lighter than VMs. No guest kernel, no hardware emulation, just a normal process with restricted visibility. It also explains the limits: containers share the host kernel, so isolation is weaker than a VM, and a kernel exploit crosses the boundary.

CPU requests map to cgroup CPU shares — a relative weight used only when the CPU is contended. CPU limits map to CFS quota: a maximum number of microseconds of CPU time per 100 ms period. When a container exhausts its quota within a period, it is throttled — descheduled entirely until the next period begins. That is the part that surprises people: throttling is not gentle slowing, it is a hard stop for the remainder of the window. So a limit of 1 CPU means 100 ms of CPU per 100 ms period. A multithreaded application with four threads can burn that in 25 ms of wall clock and then sit idle for 75 ms, producing severe latency spikes while average CPU usage looks low. That is why container_cpu_cfs_throttled_seconds_total is a metric worth alarming on, and why many teams set requests without limits for latency-sensitive services. The JVM complication is that it sizes thread pools and GC threads from the visible core count. Modern JVMs read the cgroup quota, but older ones see all host cores and create wildly oversized pools.

SIGTERM is the polite request to terminate. It can be caught, so the process can flush buffers, close connections, deregister from a load balancer, and exit cleanly. It is the default for kill and what orchestrators send first. SIGKILL cannot be caught, blocked or ignored. The kernel destroys the process immediately with no opportunity to clean up. Open files are closed by the kernel, but application-level state is lost — in-flight requests dropped, buffers unflushed. SIGSTOP suspends the process without terminating it, and also cannot be caught. SIGCONT resumes it. Useful for debugging and for job control, and it is what Ctrl+Z does via SIGTSTP, which unlike SIGSTOP can be caught. The sequence matters operationally. Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds, then sends SIGKILL. If your application ignores SIGTERM or takes too long, it gets killed mid-request. So handling SIGTERM properly — stop accepting new work, finish in-flight requests, then exit — is what makes a rolling deployment invisible to users rather than a burst of errors.

PID 1 has two special responsibilities the kernel assigns it, and application processes generally implement neither. First, it must reap orphaned children. When a process dies and its parent is gone, it is re-parented to PID 1, which must call wait to collect the exit status. An application that does not do this accumulates zombie processes until the PID table is exhausted. Second, signal handling differs. The kernel does not apply default signal actions to PID 1, so a process that has not installed a SIGTERM handler will simply ignore it. The container then never shuts down gracefully and is SIGKILLed after the grace period — which looks like slow, unclean deployments. A third practical problem is shell form in Dockerfiles: CMD in shell form runs your process under /bin/sh, which becomes PID 1 and does not forward signals to its child. The fixes are to use exec form so your process is PID 1 directly, install a proper signal handler, and use a minimal init such as tini via docker run --init when the process spawns children.

Identify the thread, then find out what it is executing. top -H -p PID lists threads with their individual CPU usage, which immediately narrows it from the process to one or two threads. For a JVM, convert the offending thread ID to hexadecimal and search a jstack dump for that nid — that gives the exact Java stack. Taking three dumps a few seconds apart and comparing shows whether the thread is stuck in one place or making progress through a loop. For native code, perf top or perf record gives a profile by symbol, and gdb can attach for a stack trace. The distinction to establish early is whether it is genuinely working or spinning. A tight retry loop, a busy-wait, or a regular expression with catastrophic backtracking all look identical to a legitimate computation from outside. One specific pattern worth knowing in Java: an unsynchronised HashMap corrupted by concurrent writes can produce a cycle in a bucket, causing get to loop forever at 100% CPU. It is a classic, and the stack trace points straight at HashMap.get with no obvious cause.

strace traces the system calls a process makes, showing each call with its arguments, return value, and optionally timing. It is the tool for questions about the boundary between your application and the kernel. Which file did it fail to open? What address is it connecting to? Why is startup slow? Is it making thousands of tiny writes instead of buffering? Common invocations: strace -p PID to attach to a running process, -f to follow child processes, -T to show time spent in each call, -c to summarise counts and times per syscall, and -e trace=openat to filter. The -c summary is often the fastest win — it immediately shows whether a process is dominated by one syscall. The important caveat is overhead. strace uses ptrace and stops the process on every syscall, which can slow it by an order of magnitude. That makes it dangerous on a busy production service and can itself change timing-dependent behaviour. For production, perf trace or eBPF tools such as bpftrace give similar visibility at far lower cost, which is why they have largely displaced strace for live systems.

The soft limit is the value currently enforced. The hard limit is the ceiling the soft limit may be raised to. An unprivileged process can raise its soft limit up to the hard limit, and can lower either, but only a privileged process can raise the hard limit. This lets administrators set a maximum while allowing applications to opt into more within it. ulimit -n shows the soft limit, ulimit -Hn the hard one. In /etc/security/limits.conf they are configured separately, and systemd services use LimitNOFILE. The practical problem is that defaults are often low — 1024 is common — while a server handling many concurrent connections needs far more. Each socket is a descriptor, so a few thousand connections exhausts it and accept starts failing with "too many open files". The confusing part in production is that raising the limit in a shell does not affect an already-running service, and systemd units ignore limits.conf entirely. That mismatch is why the setting appears to have no effect, which is a very common source of wasted debugging time.

The shell parses the line into a command and arguments, expanding globs, variables and quotes. If it is a builtin such as cd, the shell executes it directly — cd must be a builtin because changing directory in a child process would not affect the shell. Otherwise the shell searches PATH for an executable, then forks. The child sets up redirections and pipes by manipulating file descriptors — dup2 to point stdout at a file or pipe — and then calls exec, replacing itself with the new program. The kernel loads the executable, maps its segments, resolves dynamic libraries via the loader, and jumps to the entry point. The parent shell calls wait to collect the exit status, unless the command was backgrounded with &. The details worth knowing are why the fork-then-exec split exists — it gives the child a window to configure redirection before the new program starts — and that a pipeline creates a pipe and forks once per stage, connecting them by descriptor. Understanding this makes shell behaviour like "why does cd in a script not affect my shell" obvious rather than mysterious.

Both are about deadlines rather than speed. A real-time system is one whose correctness depends on meeting timing constraints, not on being fast on average. In a hard real-time system, missing a deadline is a system failure. Anti-lock brakes, pacemakers, flight control. The system must guarantee worst-case timing, which means bounded interrupt latency, no unpredictable garbage collection, no demand paging, and often no dynamic allocation at all. In a soft real-time system, missing a deadline degrades quality but is tolerable. Video playback dropping a frame, a game stuttering, a VoIP call glitching. The engineering consequence is that hard real-time optimises for predictability at the cost of throughput. A general-purpose OS does the opposite — it will happily make the average case fast and the worst case unbounded. That is why standard Linux is not hard real-time, and why PREEMPT_RT exists to bound preemption latency. Most backend work is soft real-time in effect: a p99 latency target is a soft deadline, which is why tail latency matters more than mean and why GC pauses are worth engineering around.

Several effects compound, and they all get worse with thread count. Context switching: more runnable threads than cores means the scheduler switches constantly, and each switch costs registers plus cache and TLB pollution. Lock contention: threads serialise on shared locks, so past a point adding threads only lengthens the queue. Amdahl's law bounds the benefit by the serial fraction — a workload 5% serial cannot exceed a 20x speedup regardless of core count. Cache pressure: each thread has a working set, and more threads mean more competition for a fixed cache, lowering hit rates for everyone. False sharing: threads writing to adjacent memory invalidate each other's cache lines even without logical sharing. Memory: each thread has a stack, typically 1 MB in Java, so ten thousand threads is ten gigabytes before any heap. The practical answer is to size pools to the workload — cores for CPU-bound, higher for I/O-bound — and to measure rather than assume. The throughput curve rises, plateaus, then falls, and the peak is usually lower than people expect.

Because most production incidents are ultimately resource problems, and the abstractions leak precisely when things go wrong. A container killed with exit 137 is a cgroup memory limit. A service that degrades over days is usually a file descriptor or thread leak. Unexplained latency spikes are often CFS throttling or GC interacting with page faults. A deployment that drops requests is SIGTERM being ignored. A service that stops accepting connections is a descriptor limit. None of those are visible in application code, and none are diagnosable without knowing what the OS is doing. It also changes design decisions. Thread pool sizing follows from how the scheduler works. Buffering follows from syscall cost. Choosing between blocking and event-driven I/O follows from how threads and epoll behave. Caching strategy interacts with the page cache. The honest framing for an interview is that OS knowledge is not about reciting scheduling algorithms — it is that when something is slow or dying and the application logs look fine, the answer is almost always one level down, and you need the vocabulary to go and look.