8.5 Introduction to Resource Monitoring
Now we’ll discuss some topics in resource monitoring, including processor (CPU) time, memory, and disk I/O. We’ll examine utilization on a systemwide scale, as well as on a per-process basis.
Many people touch the inner workings of the Linux kernel in the interest of improving performance. However, most Linux systems perform well under a distribution’s default settings, and you can spend days trying to tune your machine’s performance without meaningful results, especially if you don’t know what to look for. So rather than think about performance as you experiment with the tools in this chapter, think about seeing the kernel in action as it divides resources among processes.
8.5.1 Measuring CPU Time
To monitor one or more specific processes over time, use the -p option to top, with this syntax:
$ top -p pid1 [-p pid2 ...]
To find out how much CPU time a command uses during its lifetime, use time. Unfortunately, there is some confusion here, because most shells have a built-in time command that doesn’t provide extensive statistics, and there’s a system utility at /usr/bin/time. You’ll probably encounter the bash shell built-in first, so try running time with the ls command:
$ time ls
After ls terminates, time should print output like the following:
real 0m0.442s
user 0m0.052s
sys 0m0.091s
User time (user) is the number of seconds that the CPU has spent running the program’s own code. Some commands run so quickly that the CPU time is close to 0. The system time (sys or system) is how much time the kernel spends doing the process’s work (for example, reading files and directories). Finally, real time (real) (also called elapsed time) is the total time it took to run the process from start to finish, including the time that the CPU spent doing other tasks. This number is normally not very useful for performance measurement, but subtracting the user and system time from elapsed time can give you a general idea of how long a process spends waiting for system and external resources. For example, the time spent waiting for a network server to respond to a request would show up in the elapsed time, but not in the user or system time.
8.5.2 Adjusting Process Priorities
You can change the way the kernel schedules a process in order to give the process more or less CPU time than other processes. The kernel runs each process according to its scheduling priority, which is a number between –20 and 20, with –20 being the foremost priority. (Yes, this can be confusing.)
The ps -l command lists the current priority of a process, but it’s a little easier to see the priorities in action with the top command, as shown here:
$ top
Tasks: 244 total, 2 running, 242 sleeping, 0 stopped, 0 zombie
Cpu(s): 31.7%us, 2.8%sy, 0.0%ni, 65.4%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 6137216k total, 5583560k used, 553656k free, 72008k buffers
Swap: 4135932k total, 694192k used, 3441740k free, 767640k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28883 bri 20 0 1280m 763m 32m S 58 12.7 213:00.65 chromium-browse
1175 root 20 0 210m 43m 28m R 44 0.7 14292:35 Xorg
4022 bri 20 0 413m 201m 28m S 29 3.4 3640:13 chromium-browse
4029 bri 20 0 378m 206m 19m S 2 3.5 32:50.86 chromium-browse
3971 bri 20 0 881m 359m 32m S 2 6.0 563:06.88 chromium-browse
5378 bri 20 0 152m 10m 7064 S 1 0.2 24:30.21 xfce4-session
3821 bri 20 0 312m 37m 14m S 0 0.6 29:25.57 soffice.bin
4117 bri 20 0 321m 105m 18m S 0 1.8 34:55.01 chromium-browse
4138 bri 20 0 331m 99m 21m S 0 1.7 121:44.19 chromium-browse
4274 bri 20 0 232m 60m 13m S 0 1.0 37:33.78 chromium-browse
4267 bri 20 0 1102m 844m 11m S 0 14.1 29:59.27 chromium-browse
2327 bri 20 0 301m 43m 16m S 0 0.7 109:55.65 xfce4-panel
In this top output, the PR (priority) column lists the kernel’s current schedule priority for the process. The higher the number, the less likely the kernel is to schedule the process if others need CPU time. The schedule priority alone doesn’t determine the kernel’s decision to give CPU time to a process, however, and the kernel may also change the priority during program execution according to the amount of CPU time the process consumes.
Next to the priority column is the NI (nice value) column, which gives a hint to the kernel’s scheduler. This is what you care about when trying to influence the kernel’s decision. The kernel adds the nice value to the current priority to determine the next time slot for the process. When you set the nice value higher, you’re being “nicer” to other processes because the kernel prioritizes them.
By default, the nice value is 0. Now, say you’re running a big computation in the background that you don’t want to bog down your interactive session. To make that process take a back seat to other processes and run only when the other tasks have nothing to do, you can change the nice value to 20 with the renice command (where pid is the process ID of the process that you want to change):
$ renice 20 pid
If you’re the superuser, you can set the nice value to a negative number, but doing so is almost always a bad idea because system processes may not get enough CPU time. In fact, you probably won’t need to alter nice values much because many Linux systems have only a single user, and that user doesn’t perform much real computation. (The nice value was much more important back when there were many users on a single machine.)
8.5.3 Measuring CPU Performance with Load Averages
Overall CPU performance is one of the easier metrics to measure. The load average is the average number of processes currently ready to run. That is, it is an estimate of the number of processes that are capable of using the CPU at any given time—this includes processes that are running and those that are waiting for a chance to use the CPU. When thinking about a load average, keep in mind that most processes on your system are usually waiting for input p. 209 (from the keyboard, mouse, or network, for example), meaning they’re not ready to run and shouldn’t contribute anything to the load average. Only processes that are actually doing something affect the load average.
Using uptime
The uptime command tells you three load averages in addition to how long the kernel has been running:
$ uptime
... up 91 days, ... load average: 0.08, 0.03, 0.01
The three bolded numbers are the load averages for the past 1 minute, 5 minutes, and 15 minutes, respectively. As you can see, this system isn’t very busy: an average of only 0.01 processes have been running across all processors for the past 15 minutes. In other words, if you had just one processor, it was running user-space applications for only 1 percent of the last 15 minutes.
Traditionally, most desktop systems would exhibit a load average of about 0 when you were doing anything except compiling a program or playing a game. A load average of 0 is usually a good sign, because it means that your processor isn’t being challenged and you’re saving power.
However, user interface components on current desktop systems tend to occupy more of the CPU than those in the past. In particular, certain websites (and especially their advertisements) cause web browsers to become resource hogs.
If a load average goes up to around 1, a single process is probably using the CPU nearly all of the time. To identify that process, use the top command; the process will usually rise to the top of the display.
Most modern systems have more than one processor core or CPU, so multiple processes can easily run simultaneously. If you have two cores, a load average of 1 means that only one of the cores is likely active at any given time, and a load average of 2 means that both cores have just enough to do all of the time.
Managing High Loads
A high load average doesn’t necessarily mean that your system is having trouble. A system with enough memory and I/O resources can easily handle many running processes. If your load average is high and your system still responds well, don’t panic; the system just has a lot of processes sharing the CPU. The processes have to compete with one another for processor time, and as a result, they’ll take longer to perform their computations than they would if they were each allowed to use the CPU all the time. Another case where a high load average might be normal is with a web or compute server, where processes can start and terminate so quickly that the load average measurement mechanism can’t function effectively.
However, if the load average is very high and you sense that the system is slowing down, you might be running into memory performance p. 210 problems. When the system is low on memory, the kernel can start to thrash, or rapidly swap memory to and from the disk. When this happens, many processes will become ready to run, but their memory might not be available, so they’ll remain in the ready-to-run state (contributing to the load average) for much longer than they normally would. Next we’ll look at why this can happen by exploring memory in more detail.
8.5.4 Monitoring Memory Status
One of the simplest ways to check your system’s memory status as a whole is to run the free command or view /proc/meminfo to see how much real memory is being used for caches and buffers. As just mentioned, performance problems can arise from memory shortages. If not much cache/buffer memory is being used (and the rest of the real memory is taken), you may need more memory. However, it’s too easy to blame a shortage of memory for every performance problem on your machine.
How Memory Works
As Chapter 1 explained, the CPU has a memory management unit (MMU) to add flexibility in memory access. The kernel assists the MMU by breaking down the memory used by processes into smaller chunks called pages. The kernel maintains a data structure, called a page table, that maps a process’s virtual page addresses to real page addresses in memory. As a process accesses memory, the MMU translates the virtual addresses used by the process into real addresses based on the kernel’s page table.
A user process doesn’t actually need all of its memory pages to be immediately available in order to run. The kernel generally loads and allocates pages as a process needs them; this system is known as on-demand paging or just demand paging. To see how this works, consider how a program starts and runs as a new process:
- The kernel loads the beginning of the program’s instruction code into memory pages.
- The kernel may allocate some working-memory pages to the new process.
- As the process runs, it might reach a point where the next instruction in its code isn’t in any of the pages that the kernel initially loaded. At this point, the kernel takes over, loads the necessary page into memory, and then lets the program resume execution.
- Similarly, if the program requires more working memory than was initially allocated, the kernel handles it by finding free pages (or by making room) and assigning them to the process.
You can get a system’s page size by looking at the kernel configuration:
$ getconf PAGE_SIZE
4096
This number is in bytes, and 4k is typical for most Linux systems.
The kernel does not arbitrarily map pages of real memory to virtual addresses; that is, it does not put all of the available pages into one big pool and allocate from there. Real memory has many divisions that depend on hardware limitations, kernel optimization of contiguous pages, and other factors. However, you shouldn’t worry about any of this when you’re just getting started.
Page Faults
If a memory page isn’t ready when a process wants to use it, the process triggers a page fault. In the event of a page fault, the kernel takes control of the CPU from the process in order to get the page ready. There are two kinds of page faults: minor and major.
Minor page faults
A minor page fault occurs when the desired page is actually in main memory, but the MMU doesn’t know where it is. This can happen when the process requests more memory or when the MMU doesn’t have enough space to store all of the page locations for a process (the MMU’s internal mapping table is usually quite small). In this case, the kernel tells the MMU about the page and permits the process to continue. Minor page faults are nothing to worry about, and many occur as a process runs.
Major page faults
A major page fault occurs when the desired memory page isn’t in main memory at all, which means that the kernel must load it from the disk or some other slow storage mechanism. A lot of major page faults will bog the system down, because the kernel must do a substantial amount of work to provide the pages, robbing normal processes of their chance to run.
Some major page faults are unavoidable, such as those that occur when you load the code from disk when running a program for the first time. The biggest problems happen when you start running out of memory, which forces the kernel to start swapping pages of working memory out to the disk in order to make room for new pages and can lead to thrashing.
You can drill down to the page faults for individual processes with the ps, top, and time commands. You’ll need to use the system version of time (/usr/bin/time) instead of the shell built-in. The following shows a simple example of how the time command displays page faults (the output of the cal command is irrelevant, so we’re discarding it by redirecting it to /dev/null):
$ /usr/bin/time cal > /dev/null
0.00user 0.00system 0:00.06elapsed 0%CPU (0avgtext+0avgdata 3328maxresident)k
648inputs+0outputs (2major+254minor)pagefaults 0swaps
As you can see from the bolded text, when this program ran, there were 2 major page faults and 254 minor ones. The major page faults occurred when the kernel needed to load the program from the disk for the first time. If you ran this command again, you probably wouldn’t get any major page faults because the kernel would have cached the pages from the disk.
If you’d rather see the page faults of processes as they’re running, use top or ps. When running top, use f to change the displayed fields and select nMaj as one of the columns to display the number of major page faults. Selecting vMj (the number of major page faults since the last update) can be helpful if you’re trying to track down a process that might be misbehaving.
When using ps, you can use a custom output format to view the page faults for a particular process. Here’s an example for PID 20365:
$ ps -o pid,min_flt,maj_flt 20365
PID MINFL MAJFL
20365 834182 23
The MINFL and MAJFL columns show the numbers of minor and major page faults. Of course, you can combine this with any other process selection options, as described in the ps(1) manual page.
Viewing page faults by process can help you zero in on certain problematic components. However, if you’re interested in your system performance as a whole, you need a tool to summarize CPU and memory action across all processes.
8.5.5 Monitoring CPU and Memory Performance with vmstat
Among the many tools available to monitor system performance, the vmstat command is one of the oldest, with minimal overhead. You’ll find it handy for getting a high-level view of how often the kernel is swapping pages in and out, how busy the CPU is, and how I/O resources are being utilized.
The trick to unlocking the power of vmstat is to understand its output. For example, here’s some output from vmstat 2, which reports statistics every two seconds:
$ vmstat 2
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 320416 3027696 198636 1072568 0 0 1 1 2 0 15 2 83 0
2 0 320416 3027288 198636 1072564 0 0 0 1182 407 636 1 0 99 0
1 0 320416 3026792 198640 1072572 0 0 0 58 281 537 1 0 99 0
0 0 320416 3024932 198648 1074924 0 0 0 308 318 541 0 0 99 1
0 0 320416 3024932 198648 1074968 0 0 0 0 208 416 0 0 99 0
0 0 320416 3026800 198648 1072616 0 0 0 0 207 389 0 0 100 0
The output falls into categories: procs for processes, memory for memory usage, swap for the pages pulled in and out of swap, io for disk usage, system for the number of times the kernel switches into kernel code, and cpu for the time used by different parts of the system.
The preceding output is typical for a system that isn’t doing much. You’ll usually start looking at the second line of output—the first one is an average for the entire uptime of the system. For example, here the system has 320,416KB of memory swapped out to the disk (swpd) and around 3,027,000KB (3GB) of real memory free. Even though some swap space is in use, the zero-valued si (swap-in) and so (swap-out) columns report that the kernel is not currently swapping anything in or out from the disk. The buff column indicates the amount of memory that the kernel is using for disk buffers (see Section 4.2.5).
On the far right, under the CPU heading, you can see the distribution of CPU time in the us, sy, id, and wa columns. Respectively, these list the percentage of time the CPU is spending on user tasks, system (kernel) tasks, idle time, and waiting for I/O. In the preceding example, there aren’t too many user processes running (they’re using a maximum of 1 percent of the CPU); the kernel is doing practically nothing, and the CPU is sitting around doing nothing 99 percent of the time.
Listing 8-3 shows what happens when a big program starts up.
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 320412 2861252 198920 1106804 0 0 0 0 2477 4481 25 2 72 0 ❶
1 0 320412 2861748 198924 1105624 0 0 0 40 2206 3966 26 2 72 0
1 0 320412 2860508 199320 1106504 0 0 210 18 2201 3904 26 2 71 1
1 1 320412 2817860 199332 1146052 0 0 19912 0 2446 4223 26 3 63 8
2 2 320284 2791608 200612 1157752 202 0 4960 854 3371 5714 27 3 51 18 ❷
1 1 320252 2772076 201076 1166656 10 0 2142 1190 4188 7537 30 3 53 14
0 3 320244 2727632 202104 1175420 20 0 1890 216 4631 8706 36 4 46 14
Listing 8-3: Memory activity
As you can see at ❶ in Listing 8-3, the CPU starts to see some usage for an extended period, especially from user processes. Because there is enough free memory, the amount of cache and buffer space used starts to increase as the kernel uses the disk more.
Later on, we see something interesting: notice at ❷ that the kernel pulls some pages into memory that were once swapped out (the si column). This means the program that just ran probably accessed some pages shared by another process, which is common—many processes use the code in certain shared libraries only when starting up.
Also notice from the b column that a few processes are blocked (prevented from running) while waiting for memory pages. Overall, the amount of free memory is decreasing, but it’s nowhere near being depleted. There’s also a fair amount of disk activity, as indicated by the increasing numbers in the bi (blocks in) and bo (blocks out) columns.
The output is quite different when you run out of memory. As the free space depletes, both the buffer and cache sizes decrease because the kernel increasingly needs the space for user processes. Once there is nothing left, you’ll see activity in the so (swapped out) column as the kernel starts moving pages onto the disk, at which point nearly all of the other output columns change to reflect the amount of work the kernel is doing. You see p. 214 more system time, more data going in and out of the disk, and more processes blocked because the memory they want to use isn’t available (it has been swapped out).
We haven’t explored all of the vmstat output columns. You can dig deeper into them in the vmstat(8) manual page, but you might need to learn more about kernel memory management first from a class or a book like Silberschatz, Gagne, and Galvin’s Operating System Concepts, 10th edition (Wiley, 2018), in order to understand them.
8.5.6 I/O Monitoring
By default, vmstat provides some general I/O statistics. Although you can get very detailed per-partition resource usage with vmstat -d, you might be overwhelmed by the amount of output resulting from this option. Instead, try a tool just for I/O called iostat.
Many of the I/O utilities we’ll discuss here aren’t built into most distributions by default, but they’re easily installed.
Using iostat
Like vmstat, when run without any options, iostat shows the statistics for your machine’s current uptime:
$ iostat
[kernel information]
avg-cpu: %user %nice %system %iowait %steal %idle
4.46 0.01 0.67 0.31 0.00 94.55
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 4.67 7.28 49.86 9493727 65011716
sde 0.00 0.00 0.00 1230 0
The avg-cpu part at the top reports the same CPU utilization information as other utilities that you’ve seen in this chapter, so skip down to the bottom, which shows you the following for each device:
tps Average number of data transfers per second
kB_read/s Average number of kilobytes read per second
kB_wrtn/s Average number of kilobytes written per second
kB_read Total number of kilobytes read
kB_wrtn Total number of kilobytes written
Another similarity to vmstat is that you can provide an interval argument, such as iostat 2, to give an update every two seconds. When using an interval, you might want to display only the device report by using the -d option (such as iostat -d 2).
By default, the iostat output omits partition information. To show all of the partition information, use the -p ALL option. Because a typical system has many partitions, you’ll get a lot of output. Here’s part of what you might see:
$ iostat -p ALL
--snip--
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
--snip--
sda 4.67 7.27 49.83 9496139 65051472
sda1 4.38 7.16 49.51 9352969 64635440
sda2 0.00 0.00 0.00 6 0
sda5 0.01 0.11 0.32 141884 416032
scd0 0.00 0.00 0.00 0 0
--snip--
sde 0.00 0.00 0.00 1230 0
In this example, sda1, sda2, and sda5 are all partitions of the sda disk, so the read and written columns will have some overlap. However, the sum of the partition columns won’t necessarily add up to the disk column. Although a read from sda1 also counts as a read from sda, keep in mind that you can read from sda directly, such as when reading the partition table.
Per-Process I/O Utilization and Monitoring: iotop
If you need to dig even deeper to see I/O resources used by individual processes, the iotop tool can help. Using iotop is similar to using top. It generates a continuously updating display that shows the processes using the most I/O, with a general summary at the top:
# iotop
Total DISK READ: 4.76 K/s | Total DISK WRITE: 333.31 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
260 be/3 root 0.00 B/s 38.09 K/s 0.00 % 6.98 % [jbd2/sda1-8]
2611 be/4 juser 4.76 K/s 10.32 K/s 0.00 % 0.21 % zeitgeist-daemon
2636 be/4 juser 0.00 B/s 84.12 K/s 0.00 % 0.20 % zeitgeist-fts
1329 be/4 juser 0.00 B/s 65.87 K/s 0.00 % 0.03 % soffice.b~ash-
pipe=6
6845 be/4 juser 0.00 B/s 812.63 B/s 0.00 % 0.00 % chromium-browser
19069 be/4 juser 0.00 B/s 812.63 B/s 0.00 % 0.00 % rhythmbox
Along with the user, command, and read/write columns, notice that there’s a TID column instead of a PID column. The iotop tool is one of the few utilities that displays threads instead of processes.
The PRIO (priority) column indicates the I/O priority. It’s similar to the CPU priority that you’ve already seen, but it affects how quickly the kernel schedules I/O reads and writes for the process. In a priority such as be/4, the be part is the scheduling class, and the number is the priority level. As with CPU priorities, lower numbers are more important; for example, the kernel allows more I/O time for a process with priority be/3 than one with priority be/4.
The kernel uses the scheduling class to add more control for I/O scheduling. You’ll see three scheduling classes from iotop:
be Best effort. The kernel does its best to schedule I/O fairly for this class. Most processes run under this I/O scheduling class.
rt Real time. The kernel schedules any real-time I/O before any other class of I/O, no matter what.
idle Idle. The kernel performs I/O for this class only when there is no other I/O to be done. The idle scheduling class has no priority level.
You can check and change the I/O priority for a process with the ionice utility; see the ionice(1) manual page for details. You’ll probably never need to worry about the I/O priority, though.
8.5.7 Per-Process Monitoring with pidstat
You’ve seen how you can monitor specific processes with utilities such as top and iotop. However, this display refreshes over time, and each update erases the previous output. The pidstat utility allows you to see the resource consumption of a process over time in the style of vmstat. Here’s a simple example for monitoring process 1329, updating every second:
$ pidstat -p 1329 1
Linux 5.4.0-48-generic (duplex) 11/09/2020 _x86_64_ (4 CPU)
09:26:55 PM UID PID %usr %system %guest %CPU CPU Command
09:27:03 PM 1000 1329 8.00 0.00 0.00 8.00 1 myprocess
09:27:04 PM 1000 1329 0.00 0.00 0.00 0.00 3 myprocess
09:27:05 PM 1000 1329 3.00 0.00 0.00 3.00 1 myprocess
09:27:06 PM 1000 1329 8.00 0.00 0.00 8.00 3 myprocess
09:27:07 PM 1000 1329 2.00 0.00 0.00 2.00 3 myprocess
09:27:08 PM 1000 1329 6.00 0.00 0.00 6.00 2 myprocess
The default output shows the percentages of user and system time and the overall percentage of CPU time, and it even tells you on which CPU the process was running. (The %guest column here is somewhat odd—it’s the percentage of time that the process spent running something inside a virtual machine. Unless you’re running a virtual machine, don’t worry about this.)
Although pidstat shows CPU utilization by default, it can do much more. For example, you can use the -r option to monitor memory and -d to turn on disk monitoring. Try them out, and then look at the pidstat(1) manual page to see even more options for threads, context switching, or just about anything else that we’ve talked about in this chapter.