Chapter #27: System Usage, Utilization, and Troubleshooting
In this chapter, learn how to monitor and troubleshoot Linux systems, including CPU, memory, disk usage, processes, and log analysis using both classic and modern tools.
Although Linux is very reliable, wise system administrators should find a way to keep an eye on the system’s behavior and utilization always.
Ensuring an uptime as close to 100% as possible and the availability of resources are critical needs in many environments.
Examining the past and status of the system will allow us to foresee and most likely prevent possible issues.
In this chapter, we will present a list of a few tools that are available in most upstream distributions to check on the system status, analyze outages, and troubleshoot ongoing issues.
Specifically, of the myriad of available data, we will focus on CPU, storage space, and memory utilization, basic process management, and log analysis.
Storage Space Utilization
There are 2 well-known commands in Linux that are used to inspect storage space usage: df and du.
The first one, df (which stands for disk free), is typically used to report overall disk space usage by the file system.
Example 1: Reporting Disk Space Usage in Human-Readable Format
Without options, df reports disk space usage in bytes. With the -h flag, it will display the same information using MB or GB instead.
Note that this report also includes the total size of each file system (in 1-K blocks), the free and available spaces, and the mount point of each storage device:
df -h
Or
df -hT
That’s certainly nice, but there’s another limitation that can render a file system unusable, and that is running out of inodes. All files in a file system are mapped to an inode that contains their metadata.
Example 2: Inspect Inode Usage by File System in Human-Readable Format
With the following command, you can see the amount of used and available inodes:
df -hTi
According to the above image, there are 146 used inodes (1%) in /home, which means that you can still create 226K files in that file system.
Example 3: Finding and Deleting Empty Files and Directories
Note that you can run out of storage space long before running out of inodes, and vice versa.
For that reason, you need to monitor not only the storage space utilization but also the number of inodes used by the file system.
Use the following commands to find empty files or directories (which occupy 0 B) that are using inodes without a reason:
find /home -type f -empty
find /home -type d -empty
Also, you can add the -delete flag at the end of each command if you also want to delete those empty files and directories:
find /home -type f -empty -delete
find /home -type d -empty -delete
The previous procedure deleted 4 files. Let’s check again the number of used/available nodes in /home:
df -hTi /home
As you can see, there are 142 used inodes now (4 fewer than before).