Linux monitor resources from terminal

Resources

mtr - My Traceroute

Combines ping and traceroute into interactive tool. It's often already included in the most of distributions.

Usage:

$ mtr wp.pl

Key bindings:

• d - to switch Display modes:

1. tracert+ping,
2. Cisco lost packets
3. Latency

iotop

$sudo iotop -aoP

• -a Will show accumulated output
• -o Will only output
• -P Will only show processes instead of threads

sysstat

sudo apt-get install sysstat

$ sar
09:25:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
09:35:01 AM     all      0.11      0.00      0.01      0.00      0.00     99.88
09:45:01 AM     all      0.12      0.00      0.01      0.00      0.00     99.86

The %iowait is the time spent waiting on I/O. Using the Debian package, you must enable the stat collector via the /etc/default/sysstat config file after package installation. To see current utilization broken out by device, you can use the iostat

$sudo iostat -x 1
Linux 3.13.0-43-generic (piotr-x220) 	21/05/15 	_x86_64_	(4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.37    0.03   17.66    3.68    0.00   74.26

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               8.23    39.52   78.63   45.53  3546.76  4702.75   132.88     6.22   50.06    9.87  119.45   3.09  38.37

dstat

$ sudo apt-get install dstat
$ dstat -tdD total,sda,sdb,sdc,md1 60        #colourful
----system---- -dsk/total----dsk/sda--
     time     | read  writ: read  writ
21-05 22:13:04|3441k 4478k:3441k 4478k
21-05 22:13:18| 109k  242k: 109k  242k
21-05 22:13:29|  62k 1404k:  62k 1404k
21-05 22:13:34|  81k 1263k:  81k 1263k

• -t for timestamps
• -d for disk statistics
• -D to specify the exact devices to report
• 60 to average over 60 seconds. The display is updated every second, but only once per 60 seconds a new line will be started.
• -c can report wait IO percentage, which in most cases is related to the CPU waiting for data from the disks

Saidar

sudo apt-get install saidar
Hostname  : linux-ubuntu  Uptime : 00:14:52          Date : 2015-01-25 11:56:42

Load 1    :   0.19   CPU Idle  :  96.01%  Running   :     2   Zombie    :     0
Load 5    :   0.27   CPU System:   1.12%  Sleeping  :   198   Total     :   200
Load 15   :   0.22   CPU User  :   2.87%  Stopped   :     0   No. Users :     2 

Mem Total :   3729M  Swap Total:   3869M  Mem Used  : 22.04%  Paging in :     0
Mem Used  :    822M  Swap Used :      0B  Swap Used :  0.00%  Paging out:     0
Mem Free  :   2907M  Swap Free :   3869M  Total Used: 10.82%

Disk Name      Read         Write         Network Interface        rx        tx
ram0             0B            0B         eth0                     0B        0B
ram1             0B            0B         lo                       0B        0B
ram2             0B            0B         wlan0                  373B       84B
ram3             0B            0B
ram4             0B            0B         Mount Point            Free      Used
ram5             0B            0B         /                      245G     10.59%
ram6             0B            0B         /sys                     0B      -nan%
ram7             0B            0B         /proc                    0B      -nan%
ram8             0B            0B         /dev                  1854M      0.00%
ram9             0B            0B         /dev/pts                 0B      -nan%
ram10            0B            0B         /run                   371M      0.34%
ram11            0B            0B         /                      245G     10.59%

Glances also shows containers

Written in Python and light on resources text based and API aware monitoring tool

$ curl -L https://bit.ly/glances | /bin/bash
$ wget -O- https://bit.ly/glances | /bin/bash
$ pip install glances

Ncdu - NCurses Disk Usage

Ncdu is a disk usage analyzer with an ncurses interface.

sudo apt install ncdu

Key bindings: e - show hidden files g - show % i - show info r - rescan b - spawn shell s - sort by size n - sort by name

Discus - Disk Space Usage

sudo apt install discus	#Debian/UBuntu, it takes ~60kB
sudo yum install discus	#RHEL/CentOS
sudo dnf install discus	#Fedora 22+

Network

Check for tcp open port

No nc, or any other tools, you can check if tcp socket is open using Linux native /dev/tcp device.

PORT=22; SERVER=myserver
$> timeout 2 bash -c "</dev/tcp/${SERVER}/${PORT}" && echo port-open || echo port-closed

$> </dev/tcp/localhost/22  && echo port-open || echo port-closed
$> </dev/tcp/10.244.2.2/80 && echo port-open || echo port-closed
$> </dev/tcp/k8sservice/80 && echo port-open || echo port-closed

netstat - network statistics

In CentOS 7 net-tools package has been replaced by iproute2 therefore in course to use netstat needs installing:

[piotr@vmcent7 ~]$ sudo yum install net-tools

$ sudo netstat -pant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1771/master         
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1434/sshd           
tcp        0      0 192.168.1.148:22        192.168.1.89:40653      ESTABLISHED 35910/sshd: piotr [ 
tcp        0      0 192.168.1.148:22        192.168.1.89:40614      ESTABLISHED 35831/sshd: piotr [ 
tcp        0      0 192.168.1.148:22        192.168.1.64:4334       ESTABLISHED 35679/sshd: piotr [ 
tcp6       0      0 ::1:25                  :::*                    LISTEN      1771/master         
tcp6       0      0 :::22                   :::*                    LISTEN      1434/sshd

Show UDP, notice switch -u used to show only UDP protocol

$ netstat -anup
udp        0      0 0.0.0.0:514      0.0.0.0:*     3111/rsyslogd
udp        0      0 0.0.0.0:42528    0.0.0.0:*     3111/rsyslogd
udp        0      0 0.0.0.0:33498    0.0.0.0:*     3111/rsyslogd
udp        0      0 :::514           :::*          3111/rsyslogd

ss - socket viewer

ss tool is by defult shipped in a minimal CentOS 7, similar to the netstat provides more detailed information about sockets.

$ sudo ss -pant
State       Recv-Q Send-Q             Local Address:Port          Peer Address:Port 
LISTEN      0      100                    127.0.0.1:25                       *:*      users:(("master",1771,13))
LISTEN      0      128                            *:22                       *:*      users:(("sshd",1434,3))
ESTAB       0      0                  192.168.1.148:22            192.168.1.89:40653  users:(("sshd",35914,3),("sshd",35910,3))
ESTAB       0      0                  192.168.1.148:22            192.168.1.89:40614  users:(("sshd",35835,3),("sshd",35831,3))
ESTAB       0      0                  192.168.1.148:22            192.168.1.64:4334   users:(("sshd",35683,3),("sshd",35679,3))
LISTEN      0      100                          ::1:25                      :::*      users:(("master",1771,14))
LISTEN      0      128                           :::22                      :::*      users:(("sshd",1434,4))

Show UDP ports

$ ss -a -A udp -n  #or  ss -luna
State       Recv-Q Send-Q     Local Address:Port     Peer Address:Port
UNCONN      0      0                      *:55040               *:*
UNCONN      10600  0                      *:514                 *:*
UNCONN      0      0                      *:34827               *:*
UNCONN      0      0                      *:43157               *:*
UNCONN      0      0                      *:52377               *:*
ESTAB       0      0              127.0.0.1:35869       127.0.0.1:35869

Show Unix SOCKETS

$ ls -la /proc/$(pidof rsyslog)/fd

tcptrack

apt-get install tcptrack
tcptrack -i <interface>
tcptrack -i eth0 "dst port 80"

CBM Color Bandwidth Meter

The Color Bandwidth Meter (CBM) is a small program to display the traffic currently flowing through the network devices in a simple curses-based GUI. The traffic for all interfaces include values as receive, transfer and total Bytes/s or bits/s (or its multiples as KB/s and Kb/s).

sudo apt install cbm

Pktstat

Pktstat displays all the active connections in real time, and the speed at which data is being transferred through them. It also displays the type of the connection, i.e. tcp or udp and also details about http requests if involved.

$ sudo pktstat -i eth0 -nt

nload - network load

apt-get install nload # Ubuntu 22.04 docker image
nload                 # use left right arrow to switch view between interfaces

sar -n KEYWORD <every_sec> <count> - report network statistics

This reports various network statistics. For example: number of packets received (transmitted) through the network card, statistics of packet failure etc.,. “1 3” reports for every 1 seconds a total of 3 times.

$ sar -n DEV 1 10

KEYWORD can be one of the following:

• DEV – Displays network devices vital statistics for eth0, eth1, etc.,
• EDEV – Display network device failure statistics
• NFS – Displays NFS client activities
• NFSD – Displays NFS server activities
• SOCK – Displays sockets in use for IPv4
• IP – Displays IPv4 network traffic
• EIP – Displays IPv4 network errors
• ICMP – Displays ICMPv4 network traffic
• EICMP – Displays ICMPv4 network errors
• TCP – Displays TCPv4 network traffic
• ETCP – Displays TCPv4 network errors
• UDP – Displays UDPv4 network traffic
• SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6
• ALL – This displays all of the above information. The output will be very long.

dnstop - domain queries stats

./dnstop eth0  #download http://dnstop.measurement-factory.com

Available statistics can be toggled using key shortcuts

Query Type     Count      %
---------- --------- ------
A?               202   97.1
NS?                4    1.9
SRV?               1    0.5
AAAA?              1    0.5

Source      Query Name     Count      %        Destinations     Count      %
----------- ---------- --------- ------        ------------ --------- ------
10.0.20.197 com              106   46.1        195.60.0.5         173   67.8
10.0.20.197 net              106   46.1        195.60.0.1          82   32.2
10.0.20.197 uk                 8    3.5
..

2nd level Query Names                        TopLevelDomain - queries 
Query Name           Count      %            Query Name     Count      %
---------------- --------- ------            ---------- --------- ------
akamaiedge.net           5   38.5            net               38   73.1
akamai.net               3   23.1            com               12   23.1
adobe.com                1    7.7d?          .                  1    1.9
twitter.com              1    7.7            uk                 1    1.9
..

Key shortcuts

 s - Sources list        t - Query types    r - Rcodes  ^R - Reset counters
 d - Destinations list   o - Opcodes                    ^X - Exit
 1 - 1st level Query Names      ! - with Sources
 2 - 2nd level Query Names      @ - with Sources
 3 - 3rd level Query Names      # - with Sources
 4 - 4th level Query Names      $ - with Sources
 5 - 5th level Query Names      % - with Sources
 6 - 6th level Query Names      ^ - with Sources
 7 - 7th level Query Names      & - with Sources
 8 - 8th level Query Names      * - with Sources
 9 - 9th level Query Names      ( - with Sources

Hardware

inxi

inxi -v4 -F -xxADGMN

List of resource monitoring tools

Performance and resources monitors

• ctop top for containers
• htop
• lsof
• iotop
• vmstat -S m #displays stats in Mb
• glances

Bootup time

• systemd-analyze blame
• systemd-analyze critical-chain
• systemd-analyze plot > boot_analysis.svg; xviewer boot_analysis.svg

Web servers

• ngxtop -monitor nGinx logs/count/responses

SSL

• sslscan
• testssl.sh

Network monitoring tools

• bmon
• bwbar
• bwm
• bwm-ng
• iftraf
• iftop
• iperf
• ipfm
• speedometer
• cbm
• ibmonitor
• pktstat
• mactrack
• nload -u K wlan0
• vnstat -d -i wlan0
• sudo tcptraceroute
• trace 198.6.1.2
• traceroute
• tracepath
• tracepath6
• mtr 198.6.1.2
• ngrep -searches network streams, it's tcpdump compatible syntax

Space utilization

• ncdu
• discus
• dutree

Hardware

• lspci
• inxi -v4 -F -xxADGMN

Logs

• lnav
• multitail
• journalctl -systemd tool

Network statistics scripts

connscount.sh

#!/bin/bash
 netstat -an | awk '/^tcp/ {A[$(NF)]++} END {for (I in A) {printf "%5d %s\n", A[I], I}}'

dnscheck.pl

#!/usr/bin/perl -w
 use strict;
 while () {
 my $date = localtime(time());
         open(CH,"dig \@195.60.0.1 www.bbc.co.uk |") || die "cannot run dig: $!\n";
         while (<CH>) {
                 chomp;
                 next unless /query time/i;
                 print $date . "$_\n";
         };
         close CH;
         open(CH,"dig \@195.60.0.5 www.bbc.co.uk |") || die "cannot run dig: $!\n";
         while (<CH>) {
                 chomp;
                 next unless /query time/i;
                 print $date . "$_\n";
         };
         close CH; 
 
         sleep 10;
 };

Run

./dnscheck.pl
Fri Dec 11 21:50:15 2015;; Query time: 5 msec
Fri Dec 11 21:50:15 2015;; Query time: 4 msec

Service control managers

• Ubuntu: System > Administrator > services
• bum - GUI bootup manager
• rcconf
• sysv-rc-conft

References

• Command line tools to monitor Linux performance Tecmint
• Linux System IO monitoring