Performance Overview

1. Introduction: The Science of Linux Performance

Performance tuning is not a haphazard collection of "magic" sysctl settings or random StackOverflow tweaks. In enterprise infrastructure, performance tuning is an empirical, disciplined engineering methodology aimed at optimizing system resource utilization, maximizing throughput, and minimizing latency without sacrificing system stability.

Every computing workload in Linux ultimately interacts with four core physical resources:

1. CPU (Compute)
2. Memory (RAM / Page Cache)
3. Storage (Block I/O / Filesystems)
4. Network (Sockets / Network Interface Controllers)

+--------------------------------------------------------------------------+
|                     Linux Performance Observability Map                  |
+--------------------------------------------------------------------------+
|  Applications & Runtimes (Postgres, JVM, Node.js, Go, Nginx)             |
|                                |                                         |
|  System Call Interface (VFS, Sockets, sys_clone)                         |
|  - Tracing Tools: strace, perf, bpftrace, eBPF                           |
|                                |                                         |
|  Kernel Subsystems & Tunables                                            |
|  +--------------------+   +--------------------+   +-------------------+ |
|  | Scheduler / CFS    |   | Virtual Memory / VM|   | Network TCP Stack | |
|  | - sysctl kernel.*  |   | - sysctl vm.*      |   | - sysctl net.*    | |
|  +---------+----------+   +---------+----------+   +---------+---------+ |
|            |                        |                        |           |
|  Hardware  v                        v                        v           |
|  +--------------------+   +--------------------+   +-------------------+ |
|  | CPU Cores & Caches |   | Physical RAM / DIMM|   | NIC / 100GbE / TCP| |
|  +--------------------+   +--------------------+   +-------------------+ |
+--------------------------------------------------------------------------+

2. Brendan Gregg's USE Method for Root-Cause Analysis

Created by legendary performance architect Brendan Gregg (Netflix / Intel), the USE Method (Utilization, Saturation, Errors) is the industry-standard checklist for isolating hardware bottlenecks:

For every hardware and operating system resource, measure:

1. Utilization: The percentage of time the resource was actively performing work over a given interval (e.g., Disk %util in iostat, CPU user/system time in mpstat).
2. Saturation: The degree to which extra work is queued waiting for the resource to become available (e.g., CPU runqueue length r in vmstat, network socket backlog in ss).
3. Errors: Hardware faults, dropped packets, and retry counters (e.g., rx_dropped on NICs, read errors in SMART).

+-------------------+--------------------+--------------------+
| Resource          | Utilization Metric | Saturation Metric  |
+-------------------+--------------------+--------------------+
| **CPU**           | %user + %system    | Runqueue length (r)|
| **Memory**        | Used physical RAM  | Swap paging (si/so)|
| **Storage / Disk**| Disk %util         | Queue depth (aqu-sz|
| **Network**       | Throughput / Bandw.| Dropped packets    |
+-------------------+--------------------+--------------------+

3. The Performance Tuning Golden Rules

1. Measure Before and After: Never apply a kernel sysctl parameter without establishing a repeatable benchmark baseline. If you cannot measure the difference, do not touch the setting.
2. One Variable at a Time: Change exactly one parameter per test cycle. Changing multiple sysctl values simultaneously makes it impossible to identify which setting caused an improvement or regression.
3. Understand the Trade-Off: Every performance optimization incurs a trade-off. For example:
   • Reducing latency often costs CPU cycles (polling instead of interrupts).
   • Increasing network throughput often consumes higher memory per TCP socket.
   • Disabling filesystem sync (fsync) increases write speed but risks catastrophic data loss on power failure.
4. Tune the Application First: 80% of performance bottlenecks stem from poor application design (missing database indexes, N+1 query patterns, memory leaks). Tuning the Linux kernel cannot fix broken application code!

4. The Modern Linux Performance Toolkit

5. Summary Takeaways

• Performance tuning is an empirical science: establish baselines and measure before/after.
• Apply the USE Method (Utilization, Saturation, Errors) across all resources.
• Focus on application and architectural tuning before tweaking kernel parameters.
• Be aware of the trade-offs: latency vs. throughput, speed vs. data safety.