One liners

df -h   #displays filesystem disk space usage for all mounted partitions
du -skh * | sort -n #displays the disk usage summary for each directory 
free -m   #displays the amount of free and used memory in the system
lsb_release -a #prints version information for the Linux release you're running
tload -draws   #system load on text based graph

Copy with progress bar

rsync and cp

rsync -aP - copy with progress can be also aliased alias cp='rsync -aP'
cp -rv old-directory new-directory - shows progress bar

PV does not preserve permissions and does not handle attributes

pv ~/kali.iso | cat - /media/usb/kali.iso equals cp ~/kali.iso /media/usb/kali.iso
pv ~/kali.iso > /media/usb/kali.iso equals cp ~/kali.iso /media/usb/kali.iso
pv access.log | gzip > access.log.gz shows gzip compressing progress.

PV can be imagined as CAT command piping '|' output to another command with a bar progress and ETA times. -c makes sure one pv output is not use to write over to another, -N creates a named stream. Find more at How to use PV pipe viewer to add progress bar to cp, tar, etc..

$ pv -cN source access.log | gzip | pv -cN gzip > access.log.gz
source:  760MB 0:00:15 [37.4MB/s] [=>     ] 19% ETA 0:01:02
gzip: 34.5MB 0:00:15 [1.74MB/s] [  <=>  ]

Copy files between remote systems quick

List SSH MACs, Ciphers, and KexAlgorithms

ssh -Q cipher; ssh -Q mac; ssh -Q kex

Rsync

rsync -OPTION SOURCE DEST
rsync -axvPW  user@remote-srv:/remote/path/from /local/path/to
rsync -axvPWz user@remote-srv:/remote/path/from /local/path/to  #compresses before transfer

Rsync over ssh

rsync -avPW -e ssh $SOURCE $USER@$REMOTE:$DEST
rsync -avPW -e ssh /local/path/from remote@server.com:/remote/path/to

-e, --rsh=COMMAND  -specify the remote shell to use
-P --progress      -progress
-a, --archive      -archive mode; equals -rlptgoD (no -H,-A,-X)
-v, --verbose  
-W, --whole-file   -copy files whole (w/o delta-xfer algorithm)
-x, --one-file-system     -don't cross filesystem boundaries
-z                 -compress files before transfer, consumes ~70% of CPU

Tar over ssh

Copy from a local server (data source) to a remote server. It TARs a folder but we do not specify an archive name "-" so it redirects (tar stream) via the pipe "|" to ssh, where extracts the tarball at the remote server.

tar -cf - /path/to/dir | ssh user@remote-srv-copy-to 'tar -xvf - -C /path/to/remotedir'

Coping from local server (the data source) to a remote server as a single compressed .tar.gz file

tar czf - -C /path/to/source files-and-folders | ssh user@remote-srv-copy-to "cat - > /path/to/archive/backup.tar.gz"

Coping from a remote server to local server (where you execute the command). This will execute tar on remote server and redirects "-" to STDOUT to extract locally.

ssh user@remote-srv-copy-from "tar czpf - /path/to/data" | tar xzpf - -C /path/to/extract/data

-c; -f --file; -     -create a new archive; archive name; 'dash' means STDOUT
-                    -redirect to STDOUT
-C, --directory=DIR  -change to directory DIR, cd to the specified directory at the destination 
-x -v -f             -extract; -dispaly files on a screen; archive_name

References

How to copy a large number of files quickly between two servers Server-fault

Listing a directory in the form of a tree

$ tree ~
$ ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/   /' -e 's/-/|/'
$ alias lst='ls -R | grep ":$" | sed -e '"'"'s/:$//'"'"' -e '"'"'s/[^-][^\/]*\//--/g'"'"' -e '"'"'s/^/   /'"'"' -e '"'"'s/-/|/'"'"    
$ ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\// /g' -e 's/^/ /'   #using spaces, doesn't list .git

A directory statistics: size, files count and files types based on an extension

find . -type f | sed 's/.*\.//' | sort | uniq -c | sort -n | tail -20; echo "Total files: " | tr --delete '\n'; find . -type f | wc -l; echo "Total size: " | tr --delete '\n' ; du -sh

Constantly print tcp connections count in line

while true; do echo -n `ss -at | wc -l`" " ; sleep 3; done

Check MySQL current connections

watch -n1 "mysql -u root -pPASSWORD -te \"select id, command, host, time, left(info,60) from information_schema.processlist where info is not null or time > 300 order by time asc;\" 2>/dev/null"

Stop/start multiple services at the same time

cd /etc/init.d
for i in $(ls servicename-*); do service $i status; done
for i in $(ls servicename-*); do service $i restart; done

Unlock a user on the FTP server

pam_tally2 --user <uid>  #this will show you the number of failed logins
pam_tally2 --user <uid> --reset  #this will reset the count and let the user in

Tail log files

tail-f-the-output-of-dmesg or install multitail

tail -f /var/log/{messages,kernel,dmesg,syslog} - old school but not perfect
watch 'dmesg | tail -50' - approved by man dmesg
watch 'sudo dmesg -c >> /tmp/dmesg.log; tail -n 40 /tmp/dmesg.log' - tested, but experimental

Big log files

Clear a file content

Therefore clearing logs from this location, will release space on / partition cd /chroot/httpd/usr/local/apache2/logs > mod_jk.log #zeroize the file

Clear a part of a file

You can use time commands to measure time lapsed to execute the command

$ wc -l catalina.out     #count lines
3156616 catalina.out
$ time split -d -l 1000000 catalina.out.tmp catalina.out.tmp-   #split tmp file every 1000000th line prefixing files 
                                                                #with catalina.out.tmp-##, -d specify ## numeric sequence

$ time tail -n 300000 catalina.out > catalina.out.tmp     #creates a copy of the file with 300k last lines
$ time cat catalina.out.tmp > catalina.out                #clears and appends tmp file content to the current open file

Locked file by a process does not release free space back to a file system

When you delete a file you in fact deleting an inode pointer to a disk block. If there is still a file handle attached to it the files system will not see the file but the space will not be freed up. One way to re evaluate the space is to send HUP signal to the process occupying the file handle. This is due to how Unix works, if a process is still using a file - the system system shouldn't be trying to get rid of it.

kill -HUP <PID>

Diagram is showing File Descriptor (fd) table, File table and Inode table, finally pointing to a block device where data is stored.

File descriptor table and file table and inode table

Search for deleted files are still held open.

$ lsof | grep deleted
COMMAND    PID    TID   USER   FD   TYPE   DEVICE  SIZE/OFF   NODE NAME
mysqld     2230         mysql  4u   REG    253,2   793825     /var/tmp/ibXwbu5H (deleted)
mysqld     2230         mysql  5u   REG    253,2   793826     /var/tmp/ibfsqdZz (deleted)

$ lsof | grep DEL
COMMAND    PID   TID    USER    FD  TYPE   DEVICE  SIZE/OFF   NODE NAME
httpd      54290        apache  DEL  REG    0,4     32769     /SYSV011246e6
httpd      54290        apache  DEL  REG    0,4    262152     /SYSV0112b646

Timeout commands after certain time

Ping multiple hosts and terminate ping after 2 seconds, helpful when a server is behind firewall and no responses to ICMP returns

$ for srv in `cat nodes.txt`;do timeout 2 ping -c 1 $srv; done

Replace unix timestamps in logs to human readable date format

user@laptop:/var/log$ tail dmesg | perl -pe 's/(\d+)/localtime($1)/e'
[   Thu Jan  1 01:00:29 1970.168088] b43-phy0: Radio hardware status changed to DISABLED
[   Thu Jan  1 01:00:29 1970.308597] tg3 0000:09:00.0: irq 44 for MSI/MSI-X
[   Thu Jan  1 01:00:29 1970.344378] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   Thu Jan  1 01:00:29 1970.344745] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
 
user@laptop:/var/log$ tail dmesg
[   29.168088] b43-phy0: Radio hardware status changed to DISABLED
[   29.308597] tg3 0000:09:00.0: irq 44 for MSI/MSI-X
[   29.344378] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   29.344745] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

References

Transforming Unix Timestamps in logs to date

Sed - Stream Editor

Replace, substitute

sed -i 's/ugly/beautiful/g' ~/sue.txt  #substitutes ugly with beautiful in whole file
sed    's/A//g'     #substitutes 'A' with an empty string '/g' - globally; all it means it will remove 'A' from whole string

Substitute every file grepp'ed (-l returns a list of matched files)

for i in `grep -iRl 'dummy is undefined or' *`;do sed -i 's/dummy is undefined or/dummy is defined and/' $i; sleep 1; done

Replace a string between XML tags. It will find String Pattern between <Tag> and </Tag> then it will substitute /s findstring with replacingstring globally /g within the Pattern String.

sed -i '/<Tag>/,/<\/Tag>/s/findstring/replacingstring/g' file.xml

Awk - language for processing text files

Remove duplicate keys from known_host file

awk '!seen[$0]++' ~/.ssh/known_hosts > /tmp/known_hosts; mv -f /tmp/known_hosts ~/.ssh/

Useful packages

ARandR Screen Layout Editor - 0.1.7.1

Add user to a group

In ubuntu adding a user to group admin will grant the root privileges. Adding them to sudo group will allow to execute any command

sudo usermod -aG nameofgroup nameofuser #requires to login again

In RedHat/CentOS add a user to a group 'wheel' to grant him sudo access

sudo usermod -aG wheel nameofuser   #requires to login again

Show USB devices

lsusb -t     #shows USB tree

Copy and Paste in terminal

In Linux X graphical interface this works different then in Windows you can read more in X Selections, Cut Buffers, and Kill Rings. When you select some text this becomes the Primary selection (not the Clipboard selection) then Primary selection can be pasted using the middle mouse button. Note however that if you close the application offering the selection, in your case the terminal, the selection is essentially "lost".

Option 1 works in X

select text to copy then use your mouse middle button or press a wheel

Option 2 works in Gnome Terminal

Ctrl+Shift+C - copy
Ctrl+Shift+V or Shift+Insert - paste

Option 3 Install Parcellite GTK+ clipboard manager

sudo apt-get install parcellite

then in the settings check "use primary" and "synchronize clipboards"

Generate random password

cat /dev/urandom|tr -dc "a-zA-Z0-9"|fold -w 48|head -n1
openssl rand -base64 24

Localization - Change a keyboard layout

setxkbmap gb

at, atd - schedule a job

At can execute command at given time. It's important to remember 'at' can only take one line and by default it uses limited /bin/sh shell.

service atd status #check if at demon is running
at 01:05 AM
atq -list job queue
at -c <job number> -cat the job_number
atrm <job_number> -deletes job
mail -at command emails a user who scheduled a job with its output

getops - parameters

while getopts ":k:h" opt; do allows you to pass parameters into a script

:  - first colon has special meaning
k: - parameter k requires argument value to be passed on because ':' follows 'k'
h  - parameter h does not require a value to be passed on

AWS Cli commands

List all instances and their status

aws ec2 describe-instances --query 'Reservations[*].Instances[*].[State.Name,InstanceId,Tags[?Key==`Name`].Value]' --output text

Linux Shell

http://wiki.bash-hackers.org/start

dirname, basename, $0

#/bin/bash
PRG=$0  #relative path with program name
BASENAME=$(basename $0)  #strip directory and suffix from filenames, here it's own script name
DIRNAME=$(dirname $0)    #strip last component from file name, returns relative path to the script
printf "PRG=$0      --> computed full path: $PRG\n"
printf "BASENAME=\$(basename \$0) --> computed name of script: $BASENAME\n"
printf "DIRNAME=\$(dirname \$0)   --> computed  dir of script: $DIRNAME\n"

Bash shell - for loop

for I in {1..10}; do echo $I; done
for I in 1 2 3 4 5 6 7 8 9 10; do echo $I; done
for I in $(seq 1 10); do echo $I; done
for ((I=1; I <= 10 ; I++)); do echo $I; done

for host in $(cat hosts.txt); do ssh "$host" "$command" >"output.$host"; done
   a01.prod.com
   a02.prod.com
for host in $(cat hosts.txt); do ssh "$host" 'hostname; grep Certificate /etc/httpd/conf.d/*; echo ====='; done

Dash shell - for loop

#!/bin/dash
wso2List="esb dss am mb das greg"  #by default white space separates items in the list structures
for i in $wso2List; do
  echo "Product: $i"
done

BASH While Loop

Example 1

 while true; do tail /etc/passwd; sleep 2; clear; done

Example 2 - reads a file line by line

FILE="hostlist.txt"
while read -r HOSTNAME; do    #-r read a file line by line
  echo "Host name: $HOSTNAME"
done < "$FILE"  #redirect a file content to the while loop

Arrays

hostlist=("am-mgr-1" "am-wkr-1" "am-wkr-2" "esb-mgr-1") #array what is actually just a list data type
for INDEX in ${hostlist[@]}; do    #@ expands to every element in the list
                                   #* expands to match any string, so it mates every element as well
  printf "${hostlist[INDEX]}\n"
done

if statement

if [ "$VALUE" -eq 1 ] || [ "$VALUE" -eq 5 ] && [ "$VALUE" -gt 4 ]; then
  echo "Hello True"
elif [ "$VALUE" -eg 7 ] 2>/dev/null; then
  echo "Hello 7"
else
  echo "False" 
fi

case statement

case $MENUCHOICE in
  1)
    echo "Good choice!"
    ;; #end of that case statements so it does not loop infinite
  2)
    echo "Better choice"
    ;;
  *)
    echo "Help: wrong choice";;
esac

LOCAL UBTUNU MIRROR

rsync -a  --progress rysnc://archive.ubuntu.com/ubuntu /opt/mirror/ubuntu - command to create local mirror.
ls /opt/mirror/ubuntu - shows all files

NETWORK INSTALL VIA MIRROR

192.168.1.240 (Local web server) /ubuntu/ (Local Folder) - enter mirror manually

OR Online mirrors:

select 'enter mirror manually' http://mirrors.sonic.net/centos/6/os/i386 - etc

Boot INIT Sequence

Process

Bios ensures everything is intact - peripherals etc.
1st bootable disk GRUB responsible for setting up environment INITRD (Ensures kernel friends root file system) - kernel etc.
Load INIT first process by kernel - sets default run level of 2 unless specified.
INIT process - loads appropriate daemons from default run level.

7 run-levels

0 - halt

1 - single

2 - multi user - default

3 - multi user

4 - multi user

5 - multi user

6 - reboot

Use init + number to enter different run level

cd /boot/ - should see grub directory.
cd /etc/ - ls-l rc - shows different run levels
cd /etc/init.d - contains daemons, service scripts that are referenced from run level.

Partitions / Raid / LVM etc

Disk Commands

df -h - shows disk space
sudo fdisk -l - shows hard drive partitions
ls -l /dev/sd* - SHOWS ALL DRIVES
cat /proc/mdstat - SHOWS LINUX RAID DRIVES IN USE
pvdisplay - SHOWS PHYSICAL VOLUMES
lvdisplay - SHOW LOGICAL VOLUM DISPLAY

Provisioning Filesystems (Extra Storage)

Provision storage while sudoserver is online.

fdisk -l - reveals connected disks and partitions
df -h - shows amount of memory used
/dev/sdb - unpartitioned
mklabel - type MSDOs - if needed.
sudo parted - partition tool
select /dev/sdb - selects disk -
mkpart primary 1 10GB - 10GB partition
print - shows disks in parted
quit - to leave parted

mke2fs - overlay filesystem on new partition.
mke2fs -t ext4 -j /dev/sdb1 - creates file system.

or

sudo mkfs.ext4 -j /dev/sdb1/ - same as above.

mount /dev/sdb1 /projectx/10gb. - create mount point
mount - shows all system mounts.
dd if=/dev/zero bs=1024 count-10240 of=/projectx/10GB/test.file.1 - creates 10MB in new mount
sudo blkid - shows partition uuid for stab.
sudo nano /etc/fstab - edits stab. UUID="number" /projectx/10GB ext4 defaults - stores in fstab.

Provision SWAP storage on demand

Ability for kernel to extend RAM via disk.

free -m - determines current stare of storage.
top - also shows SWAP info.

sudo fdisk -l - to identify partition space
parted /dev/sdb - places in context of /dev/sdv/
print - to show partition table.
mkpart primary linux-swap 10GB 12GB starting from the 10GB first block moving up.
set 2 swap on - turns partitions to swap and on.
sudo fdisk -l /dev/sdb - confirms swap allocation.
sudo mkswap /dev/sdb2 - overlays SWAP filesystem and displays UUID (FSTAB)
sudo blkid - shows all UUID's
sudo nano /etc/fstab - opens stab for editing for SWAP reference using UUID.
UUID "" NONE swap sw 0 0 - for nano file.
swap on -s - displays current swap situation
sudo swap on -a - turns on swap storage.
free -m || top || swapon -s - to confirm configuration.

Option SWAP creation which is file based

dd if=/dev/zero of=/projectx/10GB/swapfile3GB count=3G bs=1024 - creates dummy swap file.
sudo mkswap /projectx/10GB/swapfile3GB - overlays SWAP file system.
sudo nano /etc/fstab - opens stab for editing for SWAP reference using path /projectx/10GB/swapfile3GB
sudo swapon -a - tuns on all swap storage.

Storage Management LVM (Logical Volume Management)

Volume sets based on various disparate storage technologies.

Common configuration - raid hardware (redundancy) / LVM overlaying RAID config (aggregation)

Ability to extend, reduce, manipulate storage on demand.

LVM storage hierarchy:

Volume Group (Consists of 1 or more physical volumes)

Logical Volume(s)
File System(s)

6 steps to LVM setup

Appropirate 1 or more LVM partitions

sudo parted /dev/sdb
mkpart extended 13GB 20GB
print
mkpart logical lvm 13GB 20GB
1. select /dev/sdc/
mklabel msdos
mkpart primary 1GB 20GB
1. set 1 lvm on
2. print
20GB on SDC and 7GB on SDB

Partition

sudo pvcreate /dev/sdb5 /dev/sdc1 - allocates LVM partitions as physical volumes
sudo pvdisplay - shows LVM physical volumes.
sudo vgcreate volgroup001 /dev/sdb5 /dev/sdc1 - aggregates volumes to volume group
sudo lvcreate -L 200GB volgroup001 -n logvol001 - creates logical volumes
sudo lvdisplay - shows logical volumes
sudo mk3fs -t ext -j /dev/volgroup001/logvol001 - overlays EXT4 filesytem.
Mount filesystem and commit changes to fstab

LVM related tasks

sudo lvrename volgroup001 logvol001 volgroup002 - renames volume group. (Remember to edit fstab file or unmount)
sudo lvresize -L 25GB /dev/volgroup001/logvolvar - Resize logical volume by 5GB from 20GB
sudo resize2fs /dev/mapper/volgroup001-logvolvar 25G - Resize filesystem after increasing memory.
sudo lvremove /dev/volgroup001/logvolvar - Remove volume completely. (Umount first)
sudo parted /dev/sdc - Add or assign more partitions to volume group LVM.
print
mkpart primary 20GB 25GB
print
set 2 lvm on
print
sudo pvcreate /dev/sdc2
sudo pvdisplay
sudo vgextend volgroup001 /dev/sdc2 - add new PV to volume group

Bash

Delete key gives ~ ? Add the following line to your $HOME/.inputrc (might not work if added to /etc/inputrc )

"\e[3~": delete-char

Linux shell/Commands