Linux shell/Commands
One liners
df -h #displays filesystem disk space usage for all mounted partitions du -skh * | sort -n #displays the disk usage summary for each directory free -m #displays the amount of free and used memory in the system lsb_release -a #prints version information for the Linux release you're running tload -draws #system load on text based graph
Copy with progress bar
- rsync and cp
- rsync -aP - copy with progress can be also aliased alias cp='rsync -aP'
- cp -rv old-directory new-directory - shows progress bar
- PV does not preserve permissions and does not handle attributes
- pv ~/kali.iso | cat - /media/usb/kali.iso equals cp ~/kali.iso /media/usb/kali.iso
- pv ~/kali.iso > /media/usb/kali.iso equals cp ~/kali.iso /media/usb/kali.iso
- pv access.log | gzip > access.log.gz shows gzip compressing progress.
PV can be imagined as CAT command piping '|' output to another command with a bar progress and ETA times. -c makes sure one pv output is not use to write over to another, -N creates a named stream. Find more at How to use PV pipe viewer to add progress bar to cp, tar, etc..
$ pv -cN source access.log | gzip | pv -cN gzip > access.log.gz source: 760MB 0:00:15 [37.4MB/s] [=> ] 19% ETA 0:01:02 gzip: 34.5MB 0:00:15 [1.74MB/s] [ <=> ]
Copy files between remote systems quick
List SSH MACs, Ciphers, and KexAlgorithms
ssh -Q cipher; ssh -Q mac; ssh -Q kex
Rsync
rsync -OPTION SOURCE DEST rsync -axvPW user@remote-srv:/remote/path/from /local/path/to rsync -axvPWz user@remote-srv:/remote/path/from /local/path/to #compresses before transfer
Rsync over ssh
rsync -avPW -e ssh $SOURCE $USER@$REMOTE:$DEST rsync -avPW -e ssh /local/path/from remote@server.com:/remote/path/to
-e, --rsh=COMMAND -specify the remote shell to use -P --progress -progress -a, --archive -archive mode; equals -rlptgoD (no -H,-A,-X) -v, --verbose -W, --whole-file -copy files whole (w/o delta-xfer algorithm) -x, --one-file-system -don't cross filesystem boundaries -z -compress files before transfer, consumes ~70% of CPU
Tar over ssh
Copy from a local server (data source) to a remote server. It TARs a folder but we do not specify an archive name "-" so it redirects (tar stream) via the pipe "|" to ssh, where extracts the tarball at the remote server.
tar -cf - /path/to/dir | ssh user@remote-srv-copy-to 'tar -xvf - -C /path/to/remotedir'
Coping from local server (the data source) to a remote server as a single compressed .tar.gz file
tar czf - -C /path/to/source files-and-folders | ssh user@remote-srv-copy-to "cat - > /path/to/archive/backup.tar.gz"
Coping from a remote server to local server (where you execute the command). This will execute tar on remote server and redirects "-" to STDOUT to extract locally.
ssh user@remote-srv-copy-from "tar czpf - /path/to/data" | tar xzpf - -C /path/to/extract/data
-c; -f --file; - -create a new archive; archive name; 'dash' means STDOUT - -redirect to STDOUT -C, --directory=DIR -change to directory DIR, cd to the specified directory at the destination -x -v -f -extract; -dispaly files on a screen; archive_name
- References
Listing a directory in the form of a tree
$ tree ~ $ ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/' $ alias lst='ls -R | grep ":$" | sed -e '"'"'s/:$//'"'"' -e '"'"'s/[^-][^\/]*\//--/g'"'"' -e '"'"'s/^/ /'"'"' -e '"'"'s/-/|/'"'" $ ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\// /g' -e 's/^/ /' #using spaces, doesn't list .git
A directory statistics: size, files count and files types based on an extension
find . -type f | sed 's/.*\.//' | sort | uniq -c | sort -n | tail -20; echo "Total files: " | tr --delete '\n'; find . -type f | wc -l; echo "Total size: " | tr --delete '\n' ; du -sh
Constantly print tcp connections count in line
while true; do echo -n `ss -at | wc -l`" " ; sleep 3; done
Check MySQL current connections
watch -n1 "mysql -u root -pPASSWORD -te \"select id, command, host, time, left(info,60) from information_schema.processlist where info is not null or time > 300 order by time asc;\" 2>/dev/null"
Stop/start multiple services at the same time
cd /etc/init.d for i in $(ls servicename-*); do service $i status; done for i in $(ls servicename-*); do service $i restart; done
Unlock a user on the FTP server
pam_tally2 --user <uid> #this will show you the number of failed logins pam_tally2 --user <uid> --reset #this will reset the count and let the user in
Tail log files
tail-f-the-output-of-dmesg or install multitail
tail -f /var/log/{messages,kernel,dmesg,syslog}
- old school but not perfectwatch 'dmesg | tail -50'
- approved by man dmesgwatch 'sudo dmesg -c >> /tmp/dmesg.log; tail -n 40 /tmp/dmesg.log'
- tested, but experimental
Big log files
Clear a file content
Therefore clearing logs from this location, will release space on / partition
cd /chroot/httpd/usr/local/apache2/logs > mod_jk.log #zeroize the file
Clear a part of a file
You can use time commands to measure time lapsed to execute the command
$ wc -l catalina.out #count lines 3156616 catalina.out $ time split -d -l 1000000 catalina.out.tmp catalina.out.tmp- #split tmp file every 1000000th line prefixing files #with catalina.out.tmp-##, -d specify ## numeric sequence
$ time tail -n 300000 catalina.out > catalina.out.tmp #creates a copy of the file with 300k last lines $ time cat catalina.out.tmp > catalina.out #clears and appends tmp file content to the current open file
Locked file by a process does not release free space back to a file system
When you delete a file you in fact deleting an inode pointer to a disk block. If there is still a file handle attached to it the files system will not see the file but the space will not be freed up. One way to re evaluate the space is to send HUP signal to the process occupying the file handle. This is due to how Unix works, if a process is still using a file - the system system shouldn't be trying to get rid of it.
kill -HUP <PID>
Diagram is showing File Descriptor (fd) table, File table and Inode table, finally pointing to a block device where data is stored.
- Search for deleted files are still held open.
$ lsof | grep deleted COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME mysqld 2230 mysql 4u REG 253,2 793825 /var/tmp/ibXwbu5H (deleted) mysqld 2230 mysql 5u REG 253,2 793826 /var/tmp/ibfsqdZz (deleted)
$ lsof | grep DEL COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME httpd 54290 apache DEL REG 0,4 32769 /SYSV011246e6 httpd 54290 apache DEL REG 0,4 262152 /SYSV0112b646
Timeout commands after certain time
Ping multiple hosts and terminate ping after 2 seconds, helpful when a server is behind firewall and no responses to ICMP returns
$ for srv in `cat nodes.txt`;do timeout 2 ping -c 1 $srv; done
Replace unix timestamps in logs to human readable date format
user@laptop:/var/log$ tail dmesg | perl -pe 's/(\d+)/localtime($1)/e' [ Thu Jan 1 01:00:29 1970.168088] b43-phy0: Radio hardware status changed to DISABLED [ Thu Jan 1 01:00:29 1970.308597] tg3 0000:09:00.0: irq 44 for MSI/MSI-X [ Thu Jan 1 01:00:29 1970.344378] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ Thu Jan 1 01:00:29 1970.344745] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready user@laptop:/var/log$ tail dmesg [ 29.168088] b43-phy0: Radio hardware status changed to DISABLED [ 29.308597] tg3 0000:09:00.0: irq 44 for MSI/MSI-X [ 29.344378] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 29.344745] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
- References
Sed - Stream Editor
Replace, substitute
sed -i 's/ugly/beautiful/g' ~/sue.txt #substitutes ugly with beautiful in whole file sed 's/A//g' #substitutes 'A' with an empty string '/g' - globally; all it means it will remove 'A' from whole string
Substitute every file grepp'ed (-l returns a list of matched files)
for i in `grep -iRl 'dummy is undefined or' *`;do sed -i 's/dummy is undefined or/dummy is defined and/' $i; sleep 1; done
Replace a string between XML tags. It will find String Pattern between <Tag> and </Tag> then it will substitute /s
findstring with replacingstring globally /g
within the Pattern String.
sed -i '/<Tag>/,/<\/Tag>/s/findstring/replacingstring/g' file.xml
Awk - language for processing text files
Remove duplicate keys from known_host file
awk '!seen[$0]++' ~/.ssh/known_hosts > /tmp/known_hosts; mv -f /tmp/known_hosts ~/.ssh/
Useful packages
- ARandR Screen Layout Editor - 0.1.7.1
Add user to a group
In ubuntu adding a user to group admin will grant the root privileges. Adding them to sudo group will allow to execute any command
sudo usermod -aG nameofgroup nameofuser #requires to login again
In RedHat/CentOS add a user to a group 'wheel' to grant him sudo access
sudo usermod -aG wheel nameofuser #requires to login again
Show USB devices
lsusb -t #shows USB tree
Copy and Paste in terminal
In Linux X graphical interface this works different then in Windows you can read more in X Selections, Cut Buffers, and Kill Rings. When you select some text this becomes the Primary selection (not the Clipboard selection) then Primary selection can be pasted using the middle mouse button. Note however that if you close the application offering the selection, in your case the terminal, the selection is essentially "lost".
Option 1 works in X
- select text to copy then use your mouse middle button or press a wheel
Option 2 works in Gnome Terminal
- Ctrl+Shift+C - copy
- Ctrl+Shift+V or Shift+Insert - paste
Option 3 Install Parcellite GTK+ clipboard manager
sudo apt-get install parcellite
then in the settings check "use primary" and "synchronize clipboards"
Generate random password
cat /dev/urandom|tr -dc "a-zA-Z0-9"|fold -w 48|head -n1 openssl rand -base64 24
Localization - Change a keyboard layout
setxkbmap gb
at, atd - schedule a job
At can execute command at given time. It's important to remember 'at' can only take one line and by default it uses limited /bin/sh shell.
service atd status #check if at demon is running at 01:05 AM atq -list job queue at -c <job number> -cat the job_number atrm <job_number> -deletes job mail -at command emails a user who scheduled a job with its output
AWS Cli commands
List all instances and their status
aws ec2 describe-instances --query 'Reservations[*].Instances[*].[State.Name,InstanceId,Tags[?Key==`Name`].Value]' --output text
Linux Shell
http://wiki.bash-hackers.org/start
dirname, basename, $0
#/bin/bash PRG=$0 #relative path with program name BASENAME=$(basename $0) #strip directory and suffix from filenames, here it's own script name DIRNAME=$(dirname $0) #strip last component from file name, returns relative path to the script printf "PRG=$0 --> computed full path: $PRG\n" printf "BASENAME=\$(basename \$0) --> computed name of script: $BASENAME\n" printf "DIRNAME=\$(dirname \$0) --> computed dir of script: $DIRNAME\n"
file descriptors
File descriptors are numbers, refereed also as a numbered streams, the first three FDs are reserved by system:
0 - STDIN, 1 - STDOUT, 2 - STDERR
echo "Enter a file name to read: "; read FILE exec 5<>$FILE #open/assign a file for '>' writing and reading '<' while read -r SUPERHERO; do #-r read a file echo "Superhero Name: $SUPERHERO" done <&5 #'&' describes that '5' is a FD, reads the file and redirects into while loop echo "File Was Read On: `date`" >&5 #write date to the file exec 5>&- #close '-' FD and close out all connections to it
Bash shell - for loop
for I in {1..10}; do echo $I; done for I in 1 2 3 4 5 6 7 8 9 10; do echo $I; done for I in $(seq 1 10); do echo $I; done for ((I=1; I <= 10 ; I++)); do echo $I; done
for host in $(cat hosts.txt); do ssh "$host" "$command" >"output.$host"; done a01.prod.com a02.prod.com for host in $(cat hosts.txt); do ssh "$host" 'hostname; grep Certificate /etc/httpd/conf.d/*; echo ====='; done
IFS and delimiting
You can change the delimiter variable in your subshell
echo $IFS #env variable that contains standard delimiter 'space'
Dash shell - for loop
#!/bin/dash wso2List="esb dss am mb das greg" #by default white space separates items in the list structures for i in $wso2List; do echo "Product: $i" done
BASH While Loop
Example 1
while true; do tail /etc/passwd; sleep 2; clear; done
Example 2 - reads a file line by line
FILE="hostlist.txt" while read -r HOSTNAME; do #-r read a file line by line echo "Host name: $HOSTNAME" done < "$FILE" #redirect a file content to the while loop
Arrays
hostlist=("am-mgr-1" "am-wkr-1" "am-wkr-2" "esb-mgr-1") #array what is actually just a list data type for INDEX in ${hostlist[@]}; do #@ expands to every element in the list #* expands to match any string, so it mates every element as well printf "${hostlist[INDEX]}\n" done
if statement
if [ "$VALUE" -eq 1 ] || [ "$VALUE" -eq 5 ] && [ "$VALUE" -gt 4 ]; then echo "Hello True" elif [ "$VALUE" -eg 7 ] 2>/dev/null; then echo "Hello 7" else echo "False" fi
case statement
case $MENUCHOICE in 1) echo "Good choice!" ;; #end of that case statements so it does not loop infinite 2) echo "Better choice" ;; *) echo "Help: wrong choice";; esac
traps
trap 'echo "Press Q to exit(what to do parameter"' SIGINT SIGTERM
debug mode
You can enable debugging mode anywhere in your scrip and disable many times
set -x #starts debug echo "Command to debug" set +x #stops debug
Or debug in sub-shell by running with -x option eg.
$ bash -x ./script.sh
Additionally you can disable globing with -f
LOCAL UBTUNU MIRROR
rsync -a --progress rysnc://archive.ubuntu.com/ubuntu /opt/mirror/ubuntu - command to create local mirror. ls /opt/mirror/ubuntu - shows all files
Boot INIT Sequence
Process
- Bios ensures everything is intact - peripherals etc.
- 1st bootable disk GRUB responsible for setting up environment INITRD (Ensures kernel friends root file system) - kernel etc.
- Load INIT first process by kernel - sets default run level of 2 unless specified.
- INIT process - loads appropriate daemons from default run level.
7 run-levels
- 0 - halt
- 1 - single
- 2 - multi user - default
- 3 - multi user
- 4 - multi user
- 5 - multi user
- 6 - reboot
Use init + number to enter different run level
cd /boot/ - should see grub directory. cd /etc/ - ls-l rc - shows different run levels cd /etc/init.d - contains daemons, service scripts that are referenced from run level.
Partitions / Raid / LVM etc
Disk Commands
df -h
- shows disk spacesudo fdisk -l
- shows hard drive partitionsls -l /dev/sd*
- SHOWS ALL DRIVEScat /proc/mdstat
- SHOWS LINUX RAID DRIVES IN USEpvdisplay
- SHOWS PHYSICAL VOLUMESlvdisplay
- SHOW LOGICAL VOLUM DISPLAY
Provisioning Filesystems (Extra Storage)
Provision storage while sudoserver is online.
fdisk -l
- reveals connected disks and partitionsdf -h
- shows amount of memory used/dev/sdb
- unpartitionedmklabel
- type MSDOs - if needed.sudo parted
- partition toolselect /dev/sdb
- selects disk -mkpart primary 1 10GB
- 10GB partitionprint
- shows disks in partedquit
- to leave parted
mke2fs
- overlay filesystem on new partition.mke2fs -t ext4 -j /dev/sdb1
- creates file system.
or
sudo mkfs.ext4 -j /dev/sdb1/
- same as above.
mount /dev/sdb1 /projectx/10gb.
- create mount pointmount
- shows all system mounts.dd if=/dev/zero bs=1024 count-10240 of=/projectx/10GB/test.file.1
- creates 10MB in new mountsudo blkid
- shows partition uuid for stab.sudo nano /etc/fstab
- edits stab.UUID="number" /projectx/10GB ext4 defaults
- stores in fstab.
Provision SWAP storage on demand
Ability for kernel to extend RAM via disk.
free -m
- determines current stare of storage.top
- also shows SWAP info.
sudo fdisk -l
- to identify partition spaceparted /dev/sdb
- places in context of /dev/sdv/print
- to show partition table.mkpart primary linux-swap 10GB 12GB
starting from the 10GB first block moving up.set 2 swap on
- turns partitions to swap and on.sudo fdisk -l /dev/sdb
- confirms swap allocation.sudo mkswap /dev/sdb2
- overlays SWAP filesystem and displays UUID (FSTAB)sudo blkid
- shows all UUID'ssudo nano /etc/fstab
- opens stab for editing for SWAP reference using UUID.UUID "" NONE swap sw 0 0
- for nano file.swap on -s
- displays current swap situationsudo swap on -a
- turns on swap storage.free -m || top || swapon -s
- to confirm configuration.
Option SWAP creation which is file based
dd if=/dev/zero of=/projectx/10GB/swapfile3GB count=3G bs=1024
- creates dummy swap file.sudo mkswap /projectx/10GB/swapfile3GB
- overlays SWAP file system.sudo nano /etc/fstab
- opens stab for editing for SWAP reference using path /projectx/10GB/swapfile3GBsudo swapon -a
- tuns on all swap storage.
Storage Management LVM (Logical Volume Management)
Volume sets based on various disparate storage technologies.
Common configuration - raid hardware (redundancy) / LVM overlaying RAID config (aggregation)
Ability to extend, reduce, manipulate storage on demand.
LVM storage hierarchy:
Volume Group (Consists of 1 or more physical volumes)
- Logical Volume(s)
- File System(s)
6 steps to LVM setup
Appropirate 1 or more LVM partitions
sudo parted /dev/sdb
mkpart extended 13GB 20GB
print
mkpart logical lvm 13GB 20GB
select /dev/sdc/
mklabel msdos
- mkpart primary 1GB 20GB
set 1 lvm on
print
20GB on SDC and 7GB on SDB
Partition
sudo pvcreate /dev/sdb5 /dev/sdc1
- allocates LVM partitions as physical volumessudo pvdisplay
- shows LVM physical volumes.sudo vgcreate volgroup001 /dev/sdb5 /dev/sdc1
- aggregates volumes to volume groupsudo lvcreate -L 200GB volgroup001 -n logvol001
- creates logical volumessudo lvdisplay
- shows logical volumessudo mk3fs -t ext -j /dev/volgroup001/logvol001
- overlays EXT4 filesytem.Mount filesystem and commit changes to fstab
sudo lvrename volgroup001 logvol001 volgroup002
- renames volume group. (Remember to edit fstab file or unmount)sudo lvresize -L 25GB /dev/volgroup001/logvolvar
- Resize logical volume by 5GB from 20GBsudo resize2fs /dev/mapper/volgroup001-logvolvar 25G
- Resize filesystem after increasing memory.sudo lvremove /dev/volgroup001/logvolvar
- Remove volume completely. (Umount first)sudo parted /dev/sdc
- Add or assign more partitions to volume group LVM.print
mkpart primary 20GB 25GB
print
set 2 lvm on
print
sudo pvcreate /dev/sdc2
sudo pvdisplay
sudo vgextend volgroup001 /dev/sdc2
- add new PV to volume group
Bash
Delete key gives ~ ? Add the following line to your $HOME/.inputrc (might not work if added to /etc/inputrc )
"\e[3~": delete-char