Difference between revisions of "Linux shell/Commands"

From Ever changing code
Jump to navigation Jump to search
 
(284 intermediate revisions by the same user not shown)
Line 1: Line 1:
= One liners =
= One liners and aliases =
<source lang=bash>
<source lang=bash>
df -h   #displays filesystem disk space usage for all mounted partitions
find . -type d -exec chmod 0755 {} \; #find dirs and change their permissions
du -skh * | sort -n #displays the disk usage summary for each directory  
df -h                               # disk free shows available disk space for all mounted partitions
free -m   #displays the amount of free and used memory in the system
du -sch .[^.]* * |sort -h          # disk usage summary including hidden files
lsb_release -a #prints version information for the Linux release you're running
du -skh * --exclude /data | sort -h # disk usage summary for each directory and sort using human readable size
tload -draws  #system load on text based graph
free -m             # displays the amount of free and used memory in the system
lsb_release -a     # prints version information for the Linux release you're running
tload               # system load on text terminal
 
# Aliases
alias ll='ls -al --file-type --color --group-directories-first' #group dirs first
 
# SSH
ssh-keygen -o -a 100 -t ed25519
# -o :- save the key in new format (the old one is really weak) by using -o and additionally to specify to use many KDF function rounds to secure the key using
 
# Sudo
sudo su - jenkins  #switch user taking current environment variables
 
# List processes
ps -ef --forest | grep nginx
 
# Create 100% cpu load, no I/O overloaded so your terminal is still accessible
timeout 10 yes>/dev/null&
</source>
 
= cp, mv, mkdir with {,} brace expansion =
<source lang=bash>
cp /etc/httpd/httpd.conf{,.bak}    #-> cp /etc/httpd/httpd.conf /etc/httpd/httpd.conf.bak
mv /etc/httpd/httpd.conf{.bak,}    #-> mv /etc/httpd/httpd.conf.bak /etc/httpd/httpd.conf
mv /etc/httpd/httpd.{conf.bak,conf} #-> mv /etc/httpd/httpd.conf.bak /etc/httpd/httpd.conf
mkdir -p /apache-jail/{usr,bin,lib64,dev}
echo foo{1,2,3}.txt                #-> foo1.txt foo2.txt foo3.txt
</source>
 
= Tests =
Bash test
<source lang=bash>
# [ -v varname ] True if the shell variable varname is set (has been assigned a value); bash>4.2
# [ -z string  ]  True if the length of string is zero.
</source>
</source>


= Copy with progress bar =
= Copy with progress bar =
; scp with progress bar
<source lang=bash>
scp -v ~/local/file.tar.gz ubuntu@10.46.77.156:~ #note '-v' option
rsync -r -v --progress -e ssh ~/local/file.tar.gz ubuntu@remote-server.exe:~
sending incremental file list
file.tar.gz
      6,389,760  10%  102.69kB/s    0:08:40
</source>
;rsync and cp
;rsync and cp
<source lang=bash>
rsync -aP                          #copy with progress can be also aliased
alias cp='rsync -aP'
cp -rv old-directory new-directory #shows progress bar
</source>


*<span style="font-weight: bold; color: green"><tt>rsync -aP</tt></span> - copy with progress can be also aliased <span style="font-weight: bold; color: green"><tt>alias cp='rsync -aP'</tt></span>
*<span style="font-weight: bold; color: green"><tt>cp -rv old-directory new-directory</tt></span> - shows progress bar


;PV does not preserve permissions and does not handle attributes
;PV does not preserve permissions and does not handle attributes
<source lang=bash>
pv ~/kali.iso | cat - /media/usb/kali.iso #equals cp ~/kali.iso /media/usb/kali.iso
pv ~/kali.iso > /media/usb/kali.iso      #equals cp ~/kali.iso /media/usb/kali.iso
pv access.log | gzip > access.log.gz      #shows gzip compressing progress
</source>


*<span style="font-weight: bold; color: green"><tt>pv ~/kali.iso | cat - /media/usb/kali.iso</tt></span> equals <span style="font-weight: bold; color: green"><tt>cp ~/kali.iso /media/usb/kali.iso</tt></span>
*<span style="font-weight: bold; color: green"><tt>pv ~/kali.iso > /media/usb/kali.iso</tt></span> equals <span style="font-weight: bold; color: green"><tt>cp ~/kali.iso /media/usb/kali.iso</tt></span>
*<span style="font-weight: bold; color: green"><tt>pv access.log | gzip > access.log.gz</tt></span> shows gzip compressing progress. 


'''PV''' can be imagined as CAT command piping '|' output to another command with a bar progress and ETA times. '''-c''' makes sure one pv output is not use to write over to another, '''-N''' creates a named stream. Find more at [http://www.catonmat.net/blog/unix-utilities-pipe-viewer/ How to use PV pipe viewer to add progress bar to cp, tar, etc.].
'''PV''' can be imagined as CAT command piping '|' output to another command with a bar progress and ETA times. '''-c''' makes sure one pv output is not use to write over to another, '''-N''' creates a named stream. Find more at [http://www.catonmat.net/blog/unix-utilities-pipe-viewer/ How to use PV pipe viewer to add progress bar to cp, tar, etc.].
<source lang=bash>
$ pv -cN source access.log | gzip | pv -cN gzip > access.log.gz
source:  760MB 0:00:15 [37.4MB/s] [=>    ] 19% ETA 0:01:02
gzip: 34.5MB 0:00:15 [1.74MB/s] [  <=>  ]
</source>
= Copy files between remote systems quick =
List SSH  MACs, Ciphers, and KexAlgorithms
ssh -Q cipher; ssh -Q mac; ssh -Q kex
== Rsync ==
Basic syntax of rsync command and examples of comping over ssh.
<source lang="bash">
rsync -options source destination
rsync -axvPW  user@remote-srv:/remote/path/from /local/path/to
rsync -axvPWz user@remote-srv:/remote/path/from /local/path/to  #compresses before transfer
# -a :archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps
# -x, --one-file-system  :don't cross filesystem boundaries
# -P, --progress        :progress bar
# -W, --whole-file      :copy files whole (w/o delta-xfer algorithm)
# -z, -compress          :files before transfer, consumes ~70% of CPU
# -r :copies data recursively (but don’t preserve timestamps and permission while transferring data
# -h :human-readable, output numbers in a human-readable format
# -v :verbose
# --remove-source-files  :removes source file once copied
</source>
Rsync over ssh
<source lang="bash">
rsync -avPW -e ssh $SOURCE $USER@$REMOTE:$DEST
rsync -avPW -e ssh /local/path/from remote@server.com:/remote/path/to
# -e, --rsh=COMMAND      :specify the remote shell to use
# -a, --archive          :archive mode; equals -rlptgoD (no -H,-A,-X) 
</source>
== Tar over ssh ==
Copy from a local server (data source) to a remote server. It TARs a folder but we do not specify an archive name "-" so it redirects (tar stream) via the pipe "|" to ssh, where extracts the tarball at the remote server.
tar -cf - /path/to/dir | ssh user@remote-srv-copy-to 'tar -xvf - -C /path/to/remotedir'
Coping from local server (the data source) to a remote server as a single compressed .tar.gz file
tar czf - -C /path/to/source files-and-folders | ssh user@remote-srv-copy-to "cat - > /path/to/archive/backup.tar.gz"
Coping from a remote server to local server (where you execute the command). This will execute tar on remote server and redirects "-" to STDOUT to extract locally.
ssh user@remote-srv-copy-from "tar czpf - /path/to/data" | tar xzpf - -C /path/to/extract/data


  $ pv -cN source access.log | gzip | pv -cN gzip > access.log.gz
  -c; -f --file; -    -create a new archive; archive name; 'dash' means STDOUT
  '''source:''' 760MB 0:00:15 [37.4MB/s] [=>    ] 19% ETA 0:01:02
  -                    -redirect to STDOUT
  '''gzip:''' 34.5MB 0:00:15 [1.74MB/s] [  <=>  ]
  -C, --directory=DIR  -change to directory DIR, cd to the specified directory at the destination
  -x -v -f            -extract; -dispaly files on a screen; archive_name
 
;References
*[http://serverfault.com/questions/18125/how-to-copy-a-large-number-of-files-quickly-between-two-servers How to copy a large number of files quickly between two servers] Server-fault


= Listing a directory in the form of a tree =
= Listing a directory in the form of a tree =
  $ tree ~
  $ tree ~
$ ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/  /' -e 's/-/|/'
  $ alias lst='ls -R | grep ":$" | sed -e '"'"'s/:$//'"'"' -e '"'"'s/[^-][^\/]*\//--/g'"'"' -e '"'"'s/^/  /'"'"' -e '"'"'s/-/|/'"'"     
  $ alias lst='ls -R | grep ":$" | sed -e '"'"'s/:$//'"'"' -e '"'"'s/[^-][^\/]*\//--/g'"'"' -e '"'"'s/^/  /'"'"' -e '"'"'s/-/|/'"'"     
  $ ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\// /g' -e 's/^/ /'  #using spaces, doesn't list .git
  $ ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\// /g' -e 's/^/ /'  #using spaces, doesn't list .git


= Check MySQL current connections =
A directory statistics: size, files count and files types based on an extension
  watch -n1 "mysql -u root -pPASSWORD -te \"select id, command, host, time, left(info,60) from information_schema.processlist where info is not null or time > 300 order by time asc;\" 2>/dev/null"
  find . -type f | sed 's/.*\.//' | sort | uniq -c | sort -n | tail -20; echo "Total files: " | tr --delete '\n'; find . -type f | wc -l; echo "Total size: " | tr --delete '\n' ; du -sh


= Stop/start multiple services at the same time =
= Constantly print tcp connections count in line =
while true; do echo -n `ss -at | wc -l`" " ; sleep 3; done
 
= Stop/start multiple services in a loop =
  cd /etc/init.d
  cd /etc/init.d
  for i in $(ls servicename-*); do service $i status; done
  for i in $(ls servicename-*); do service $i status; done
  for i in $(ls servicename-*); do service $i restart; done
  for i in $(ls servicename-*); do service $i restart; done
= Linux processes =
== List all signals ==
<source>
kill -l
1) SIGHUP      2) SIGINT      3) SIGQUIT      4) SIGILL      5) SIGTRAP
6) SIGABRT      7) SIGBUS      8) SIGFPE      9) SIGKILL    10) SIGUSR1
11) SIGSEGV    12) SIGUSR2    13) SIGPIPE    14) SIGALRM    15) SIGTERM
16) SIGSTKFLT  17) SIGCHLD    18) SIGCONT    19) SIGSTOP    20) SIGTSTP
21) SIGTTIN    22) SIGTTOU    23) SIGURG      24) SIGXCPU    25) SIGXFSZ
26) SIGVTALRM  27) SIGPROF    28) SIGWINCH    29) SIGIO      30) SIGPWR
31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX
</source>


= Unlock a user on the FTP server =
= Unlock a user on the FTP server =
Line 45: Line 167:
= Tail log files =
= Tail log files =
[http://superuser.com/questions/289239/is-it-possible-to-tail-f-the-output-of-dmesg tail-f-the-output-of-dmesg] or install <tt>multitail</tt>
[http://superuser.com/questions/289239/is-it-possible-to-tail-f-the-output-of-dmesg tail-f-the-output-of-dmesg] or install <tt>multitail</tt>
*<code>tail -f /var/log/{messages,kernel,dmesg,syslog}</code>  - old school but not perfect
<source lang="bash">
*<code> watch 'dmesg | tail -50'</code>  - approved by man dmesg
tail -f /var/log/{messages,kernel,dmesg,syslog} #old school but not perfect
*<code> watch 'sudo dmesg -c >> /tmp/dmesg.log; tail -n 40 /tmp/dmesg.log'</code> - tested, but experimental
less +F /var/log/syslog  #equivalent tail -f but allows for scrolling
watch 'dmesg | tail -50' # approved by man dmesg
watch 'sudo dmesg -c >> /tmp/dmesg.log; tail -n 40 /tmp/dmesg.log' #tested, but experimental
</source>
;less: multifile monitor to see what’s happening in the second file, you need to first <code>Ctrl-c</code> to go to normal mode, then type <code>:n</code> to go to the next buffer, and then <code>F</code> again to go back to the watching mode.
= [https://serverfault.com/questions/53894/colorize-monitoring-of-logs Colourize log files] =
;[https://github.com/sharkdp/bat#installation BAT], supports syntax highlighting for a large number of programming and markup languages
<source lang=bash>
# Ubuntu <19.10
wget https://github.com/sharkdp/bat/releases/download/v0.12.1/bat_0.12.1_amd64.deb
sudo dpkg -i bat_0.12.1_amd64.deb
bat deployment.yaml -l yaml
 
# Ubuntu 19.10+, notice the command is 'batcat' due to collision with another command 'bat'
sudo apt install bat # Ubuntu Eoan 19.10 or Debian unstable sid
batcat deployment.yaml -l yaml
 
# Usage
bat kubernetes/application.yml
</source>
 
;CCZE, a fast log colorizer written in C, intended to be a drop-in replacement for colorize
<source lang="bash">
sudo apt install ccze # log colorizer
# -c, --color KEY=COLOR :- set the color of the keyword KEY to COLOR, like one would do in one of the configuration files
# -A --raw-ansi        :- ANSI instead curses
# -m --mode            :- mode curses, ansi, html
# -h --html            :- instead of colorising the input onto the console, output it in HTML format instead
# -l                    :- list of plugins
</source>
 
 
;GRC, Generic Colouriser by default for these commands <tt>ping, traceroute, gcc, make, netstat, diff, last, ldap, and cvs</tt>
<source lang="bash">
sudo apt install grc # frontend for generic colouriser grcat(1)
grc tail -f /var/log/apache/access.log /var/log/apache/error.log
</source>


= Big log files =
= Big log files =
== Clear a file content ==
== Clear a file content ==
Therefore clearing logs from this location, will release space on / partition
Therefore clearing logs from this location, will release space on / partition
cd /chroot/httpd/usr/local/apache2/logs
cd /chroot/httpd/usr/local/apache2/logs
> mod_jk.log    #zeroize the file
> mod_jk.log    #zeroize the file


= Clear a part of a file =
== Clear a part of a file ==
You can use ''time'' commands to measure time lapsed to execute the command
You can use ''time'' commands to measure time lapsed to execute the command
  $ wc -l catalina.out    #count lines
  $ wc -l catalina.out    #count lines
Line 66: Line 224:


= Locked file by a process does not release free space back to a file system =
= Locked file by a process does not release free space back to a file system =
When you delete a file you in fact deleting an inode pointer to a disk block. If there is still a file handle attached to it the files system will not see the file but the space will not be freed up. One way to re evaluate the space is to send HUP signal to the process occupying the file handle. This is due to if a process is still using a file Unix shouldn't be trying to get rid of it.
When you delete a file you in fact deleting an inode pointer to a disk block. If there is still a file handle attached to it the files system will not see the file but the space will not be freed up. One way to re evaluate the space is to send HUP signal to the process occupying the file handle. This is due to how Unix works, if a process is still using a file - the system system shouldn't be trying to get rid of it.
  kill -HUP <PID>
  kill -HUP <PID>


Diagram is showing File Descriptor (fd) table, File table and Inode table, finally pointing to a block device where data is stored.
[[File:File-descriptor.jpg|none|left|File descriptor table and file table and inode table]]
[[File:File-descriptor.jpg|none|left|File descriptor table and file table and inode table]]


;Search for deleted files are still held open.
;Search for deleted files are still held open.
  $ lsof | grep deleted
  $ lsof | grep deleted
  mysqld    2230     mysql   4u  REG    253,2  0     793825 /var/tmp/ibXwbu5H (<span style="color: red">deleted</span>)
COMMAND    PID    TID  USER  FD  TYPE  DEVICE  SIZE/OFF  NODE NAME
  mysqld    2230     mysql   5u  REG    253,2  0     793826 /var/tmp/ibfsqdZz (<span style="color: red">deleted</span>)
  mysqld    2230         mysql 4u  REG    253,2  793825     /var/tmp/ibXwbu5H (<span style="color: red">deleted</span>)
  mysqld    2230         mysql 5u  REG    253,2  793826     /var/tmp/ibfsqdZz (<span style="color: red">deleted</span>)


  $ lsof | grep DEL
  $ lsof | grep DEL
  httpd     54290     apache  <span style="color: red">DEL</span>  REG    0,4         32769 /SYSV011246e6
COMMAND    PID  TID    USER    FD  TYPE  DEVICE  SIZE/OFF  NODE NAME
  httpd     54290     apache  <span style="color: red">DEL</span>  REG    0,4       262152 /SYSV0112b646
  httpd     54290       apache  <span style="color: red">DEL</span>  REG    0,4     32769     /SYSV011246e6
  httpd     54290       apache  <span style="color: red">DEL</span>  REG    0,4   262152     /SYSV0112b646


= Timeout commands after certain time =
= Timeout commands after certain time =
Line 84: Line 245:
  $ for srv in `cat nodes.txt`;do timeout 2 ping -c 1 $srv; done
  $ for srv in `cat nodes.txt`;do timeout 2 ping -c 1 $srv; done


= Replace unix timestamps in logs to human readable date format  =
= Time manipulation =
 
== Replace unix timestamps in logs to human readable date format  ==
<source lang=bash>
<source lang="bash">
user@laptop:/var/log$ tail dmesg | perl -pe 's/(\d+)/localtime($1)/e'
user@laptop:/var/log$ tail dmesg | perl -pe 's/(\d+)/localtime($1)/e'
[  Thu Jan  1 01:00:29 1970.168088] b43-phy0: Radio hardware status changed to DISABLED
[  Thu Jan  1 01:00:29 1970.168088] b43-phy0: Radio hardware status changed to DISABLED
Line 99: Line 260:
[  29.344745] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  29.344745] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
</source>
</source>
== Convert seconds to human readable ==
<source lang="bash">
sec=4717.728; eval "echo $(date -ud "@$sec" +'$((%s/3600/24))d:%-Hh:%Mm:%Ss')"  # '-' hyphen at %-H it means do not pad
0d:1h:18m:37s
</source>
;References:
*[https://alexlurthu.wordpress.com/2007/07/22/transforming-unix-timestamps-in-logs-to-date/ Transforming Unix Timestamps in logs to date]
== <code>tai64n</code> [https://cr.yp.to/daemontools/tai64n.html Toward a Unified Timestamp with explicit precision] ==
tai64n reads lines from stdin and prefixes with a timestamp in <code>12-byte TAI64N</code>  labels to stdout
<source>
sudo apt-get install daemontools
</source>
;tai64n: puts a precise timestamp on each line
;tai64nlocal: scans stdin for '@' followed precise TAI64N timestamps and converts to a human-readable format; timestamp is converted to local time in ISO format: YYYY-MM-DD HH:MM:SS.SSSSSSSSS
{| class="wikitable"
|+ Usage
|-
! tai64n
! tai64nlocal
|- style="vertical-align:top;"
| <source>
for i in {1..5}; do echo hello $i | tai64n; done
@400000005e36d56736b9e14c hello 1
@400000005e36d56737330cd4 hello 2
@400000005e36d56737bc5944 hello 3
@400000005e36d56738081fb4 hello 4
@400000005e36d567383e1c04 hello 5
</source>
| <source>
for i in {1..5}; do echo hello $i | tai64n; done | tai64nlocal
2020-02-02 13:57:49.918151500 hello 1
2020-02-02 13:57:49.926092500 hello 2
2020-02-02 13:57:49.935090500 hello 3
2020-02-02 13:57:49.940056500 hello 4
2020-02-02 13:57:49.943594500 hello 5
</source>
|}


;References:
;References:
*[https://alexlurthu.wordpress.com/2007/07/22/transforming-unix-timestamps-in-logs-to-date/ Transforming Unix Timestamps in logs to date]
*[https://stackoverflow.com/questions/50907211/what-is-a-tai64-time-format What is a TAI64 time format?] Stackoverfolow
== Time stamp with precision log() function ==
<source lang="bash">
log() {
    local msg="$1"
    if [ -n "$msg" ]; then
        level="${2:-INFO}"
        printf "%s - %s - %s\n" "$(date +"%Y-%m-%dT%T.%3N")" "$level" "$msg" >&2
    fi
}
</source>
 
The <code>.%3N</code> is not supported by POSIX /bin/sh.
 
== Timestamp in the filename  ==
<source lang="bash">
echo $(date +"%Y%m%d-%H%M")    #20180618-1136
echo $(date +"%Y%m%dT%H%M")    #20180618T1136
echo -n $(date +"%Y%m%d-%H%M") # no new line
date +"%Y%m%d-%H%M"|tr -d "\n" # no new line
</source>
 
 
Examples of use
<source lang=bash>
# Backup dir with all Kubernetes manifests
$> cp -r manifests_dir backup/dist-$(git rev-parse --abbrev-ref HEAD)-$(date +"%Y%m%dT%H%M")-$(git rev-parse HEAD | cut -c1-8)
 
# Kubernetes manifest tagging for multiple iteration comparison
$> cp application.yml application-$(git rev-parse --abbrev-ref HEAD)-$(date +"%Y%m%dT%H%M")-$(git rev-parse HEAD | cut -c1-8).yml
 
$> ls -l                                  # ___name____ _____branch____________ ___time______ _commit_
-rw-rw-r-- 1 piotr piotr 275158 Jan 23 12:23 application-IPN-1000_splitHeartbeat-20200123T1223-aaaabbbb.yml
 
# Terraform, note "!!:1" it's bash expansion to recall a previous command 1st argument, so it will be action eg. apply
$> terraform apply -var-file=<FILENAME> | tee "!!:1"-$(git rev-parse --abbrev-ref HEAD)-$(date +"%Y%m%dT%H%M")-$(git rev-parse HEAD | cut -c1-8).log
 
$> ls -l                                  #    action _________branch_______ _____time____ _commit_
-rw-rw-r-- 1 vagrant vagrant 29794 Jan 31 15:24  apply-IPN-1111_acmValidation-20200131T1524-aaaabbbb.log
</source>
 
Reference:
* [https://stackoverflow.com/questions/34942796/shell-cmd-date-without-new-line-in-the-end shell-cmd-date-without-new-line-in-the-end] Stackoverflow
 
= Replace numbers into human readable units =
<code>numfmt</code> is a standard Ubuntu package that allows to convert numbers into the human readable units.
<source lang=bash>
numfmt --to iec 1024      # -> 1.0K
numfmt --to iec 1024000    # -> 1000K
numfmt --to iec 1024000000 # -> 977M
 
# Convert numbers in column 5
kubectl get vpa -A --no-headers | tr -s " " | numfmt --to iec --field=5 --invalid=ignore | column -t
</source>


= Sed - Stream Editor =
= Sed - Stream Editor =
Replace, substitute
Replace, substitute
sed -i 's/ugly/beautiful/g' ~/sue.txt  #substitutes ''ugly'' with ''beautiful'' in whole file
<source lang="bash">
sed    's/A//g'    #substitutes 'A' with an empty string '/g' - globally; all it means it will remove 'A' from whole string
sed -i 's/pattern/substituteWith/g' ./file.txt  #substitutes ''pattern'' with ''substituteWith'' in each matching line
sed    's/A//g'    #substitutes 'A' with an empty string '/g' - globally; all it means it will remove 'A' from whole string
 
##Type of operation
# s/ -substitute
 
##Commands
# /p printing modified content, print only the changes to the console
# /w substitute and write matching patterns into a file
$ sed -n "s/(name)/Mark/pw log2.txt" birthday.txt
# /i -insensitive search
# /g -global replace, all matches on all lines in a file
# /2 -nth replace, replace 2nd occurrence in each line
sed "s/cha/foo/2" log.txt
# /d -delete any line matching a pattern
 
##Option flags
# -i editing in place
# -i.bak saves original file with .bak extension
# -n suppress the default output to STD out
sed 's/pattern/substituteWith/w matchingPattern.txt' ./file.txt
</source>
 
 
Substitute every file grepp'ed (-l returns a list of matched files)
<source lang="bash">
for i in `grep -iRl 'dummy is undefined or' *`;do sed -i 's/dummy is undefined or/dummy is defined and/' $i; sleep 1; done
</source>
 
 
;Matching lines
<source lang="bash">
#line matching pattern
#    \      _ then perform subsequent /s search and replace 
#      \    / 
$ sed "/foo/s/./*/g" birthday.txt #replace any line containing 'foo' with '*'
</source>
 
 
Replace a string between XML tags. It will find String Pattern between <Tag> and </Tag> then it will substitute <code>/s</code> '''findstring''' with '''replacingstring''' globally <code>/g</code> within the Pattern String.
<source lang="bash">
sed -i '/<Tag>/,/<\/Tag>/s/findstring/replacingstring/g' file.xml
</source>
 
Replace 1st occurance
<source lang="bash">
sed '0,/searchPattern/s/searchPattern/substituteWith/g' ./file.txt #0,/searchPattern -index0 of the pattern first occurrence
</source>
 
Remove all lines until first match
<source lang="bash">
sed -r -i -n -e '/Dynamic,$p' resolv.conf
#-r regex, -i inline, so no need to print lines(quiet,silient)
#-e expression, $p print to the EOFile
</source>
 
Regex search and remove (replace with empty string) all xml tags.
<source lang="bash">
sed 's/<[^>]*>//' ./file.txt
# [] -regex characters class, [^>] -negate, so following character is not > and <> won't be matched,
</source>
== References ==
[https://www.gnu.org/software/sed/manual/sed.html Sed manual] Official GNU


= Awk - language for processing text files =
= Awk - language for processing text files =
Line 115: Line 433:
*ARandR Screen Layout Editor - 0.1.7.1
*ARandR Screen Layout Editor - 0.1.7.1


=Add user to a group=
= Manage users =
;Interactive
<source lang=bash>
sudo adduser piotr
</source>
 
;Scripting
<source lang=bash>
sudo useradd -m piotr
# -m creates $HOME directory if does not exists
sudo useradd -m piotr -s /bin/bash -d /sftp/piotr
# -s define default shell
# -d define $HOME directory
</source>
 
= User groups =
==Add user to a group==
In ubuntu adding a user to group '''admin''' will grant the '''root''' privileges. Adding them to '''sudo''' group will allow to execute any command
In ubuntu adding a user to group '''admin''' will grant the '''root''' privileges. Adding them to '''sudo''' group will allow to execute any command
sudo usermod -aG nameofgroup nameofuser #requires to login again
<source lang=bash>
sudo usermod -a -G nameofgroup nameofuser #requires to login again
</source>
 


In RedHat/CentOS add a user to a group 'wheel' to grant him '''sudo access'''
In RedHat/CentOS add a user to a group 'wheel' to grant him '''sudo access'''
sudo usermod -aG wheel nameofuser  #requires to login again
<source lang=bash>
sudo usermod -a -G wheel <nameofuser>   #requires to login again
# -a add
# -G list of supplementary groups
# -g primary group
groups <nameofuser> # list all groups, 1st is a primary-group
</source>
 
==Change user primary group and other groups==
To assign a primary group to an user
<source lang=bash>
sudo usermod -g primarygroupname username
</source>
 
 
To assign secondary groups to a user (-a keeps already existing secondary groups intact otherwise they'll be removed)
<source lang=bash>
sudo usermod -a -G secondarygroupname username
 
From man-page:
...
-g (primary group assigned to the users)
-G (Other groups the user belongs to), supplementary groups
-a (Add the user to the supplementary group(s))
...
</source>
 
==Modify group==
Rename a group name <code>-n</code>
<source lang=bash>
sudo groupmod -n <new-group-name> <old-group>
</source>


= Show USB devices =
= Show USB devices =
Line 131: Line 499:
*select text to copy then use your mouse '''middle button''' or press ''' a wheel'''
*select text to copy then use your mouse '''middle button''' or press ''' a wheel'''


Option 2 works in Gnome Terminal
 
*'''Ctrl+Shift+C''' - copy
Option 2 works in Gnome Terminal and a webterminal
*'''Ctrl+Shift+V''' or '''Shift+Insert''' - paste
*<code>Ctrl</code>+<code>Shift</code>+<code>c</code> or <code>Ctrl</code>+<code>Insert</code> - copy
*<code>Ctrl</code>+<code>Shift</code>+<code>v</code> or <code>Shift</code>+<code>Insert</code> - paste
 


Option 3 Install Parcellite GTK+ clipboard manager
Option 3 Install Parcellite GTK+ clipboard manager
sudo apt-get install parcellite
<source lang=bash>
sudo apt-get install parcellite
</source>
then in the settings check "use primary" and "synchronize clipboards"
then in the settings check "use primary" and "synchronize clipboards"


= Generate random password =
= Generate random password =
  cat /dev/urandom|tr -dc "a-zA-Z0-9"|fold -w 48|head -n1
  cat /dev/urandom|tr -dc "a-zA-Z0-9"|fold -w 48|head -n1
openssl rand -base64 24
pwgen 32 1 #pwgen <length> <num_of_passwords>
= xargs - running command once per item in array =
* <code>-p</code> - print all commands and prompt for confirmation
* <code>-t</code> - print commands while executing
<source lang=bash>
declare -a ARRAY=(item1 item2 item2)
echo "${ARRAY[@]}" | xargs -0 -t -n1 -I {} . "${CWD}/{}"
# Insert args in the middle
dig wp.pl +short | xargs -I% nc -zv % 443
# | -I replace-str - replace  occurrences  of replace-str in the initial-arguments with names read from standard input
</source>


= Localization - Change a keyboard layout =
= Localization - Change a keyboard layout =
Line 147: Line 535:
= at, atd - schedule a job =
= at, atd - schedule a job =
At can execute command at given time. It's important to remember 'at' can only take one line and by default it uses limited /bin/sh shell.
At can execute command at given time. It's important to remember 'at' can only take one line and by default it uses limited /bin/sh shell.
service atd status #check if at demon is running
<source lang=bash>
at 01:05 AM
service atd status # check if at demon is running
atq -list job queue
at 01:05 AM       # schedule a job, this will drop into at> prompt, use C^d to exit
at -c <job number> -cat the job_number
atq               # list job queue
  atrm <job_number> -deletes job
at -c <job number> # cat the job_number
mail -at command emails a user who scheduled a job with its output
atrm <job_number> # deletes job
mail               # at command emails a user who scheduled a job with its output
</source>
 
= whoami, who and w =
<source lang=bash>
# 'whoami' - print effective userid
vagrant@ubuntu-bionic ~  $ whoami
vagrant
vagrant@ubuntu-bionic ~  $ sudo su -
root@ubuntu-bionic:~$ whoami # print effective userid
root
 
# 'who' - show all users who has a login-tty, shows who is logged on, even if 'su' has been used it will print the logged on userid
# who = whoami(userid) + login-tty
root@ubuntu-bionic:~$ who
vagrant  pts/0        2020-02-02 19:02 (10.0.2.2)
 
# 'who' vs 'who am i'
root@ubuntu-bionic:~$ who      # shows all logged sessions for the user, but this shows multiple termianl logins
vagrant  pts/0        2020-02-02 19:02 (10.0.2.2)
vagrant  pts/1        2020-02-02 19:13 (10.0.2.2)
root@ubuntu-bionic:~$ who am i # shows only the current terminal session
vagrant  pts/0        2020-02-02 19:02 (10.0.2.2)
## 'who arg1 arg2' 2 parameters followed by who, it will print the current login-tty user, no matters what the params are
 
# 'w' - show who is logged on and what they are doing
# w = who(logged on sessions) + what they are doing
root@ubuntu-bionic:~$ w
19:19:17 up 45 min,  2 users,  load average: 0.02, 0.02, 0.00
USER    TTY      FROM            LOGIN@  IDLE  JCPU  PCPU WHAT
vagrant  pts/0    10.0.2.2        19:02    0.00s  0.24s  0.00s sshd: vagrant [priv]
vagrant  pts/1    10.0.2.2        19:13    5:50  0.08s  0.08s -bash
</source>
 
= which and whereis =
<source lang"bash">
which pwd  #returns binary path only
whereis pwd #returns binary path and paths to man pages
</source>
 
= Linux Shell =
*[http://wiki.bash-hackers.org/start wiki.bash-hackers.org]
*[http://tldp.org/LDP/abs/html/refcards.html Reference Cards]  tldp.org/LDP
*[https://unix.stackexchange.com/questions/131766/why-does-my-shell-script-choke-on-whitespace-or-other-special-characters How to handle white spaces] Explains white spacing, eval and how to deal with building complex commands in scripts
 
== Syntax checker ==
Common tool is to use [https://github.com/koalaman/shellcheck/ shellchecker]. Here below it's a reference how to install stable version on Ubuntu
<source lang="bash">
sudo apt install xz-utils #pre req
export scversion="stable" # or "v0.4.7", or "latest"
wget "https://storage.googleapis.com/shellcheck/shellcheck-${scversion}.linux.x86_64.tar.xz"
tar --xz -xvf shellcheck-"${scversion}".linux.x86_64.tar.xz
cp shellcheck-"${scversion}"/shellcheck /usr/bin/
shellcheck --version
</source>
 
== Special variables ==
*<code>$_</code> The last argument of the previous command.
*<code>$-</code> Flags passed to script (using set)
*<code>$?</code> Exit status of a command, function, or the script itself last executed
*<code>$$</code> The process ID of the current shell. For shell scripts, this is the process ID under which they are executing.
*<code>$!</code> PID (process ID) of last job run in background
*<code>$0</code> The filename of the current script.
*<code>$#</code> arguments vount supplied to a script.
*<code>$@</code> All the arguments are individually double quoted. If a script receives two arguments, $@ is equivalent to $1 $2.
*<code>$*</code> All the arguments are double quoted. If a script receives two arguments, $* is equivalent to $1 $2.
 
;[https://dev.to/rpalo/handling-arguments-in-bash-scripts-3o5m Note] on <code>$*</code> and <code>$@</code>
<code>$*</code> and <code>$@</code> when unquoted are identical and expand into the arguments.
<code>"$*"</code> is a single word, comprising all the arguments to the shell, joined together with spaces. For example <code>'1 2' 3</code> becomes <code>"1 2 3"</code>.
<code>"$@"</code> is identical to the arguments received by the shell, the resulting list of words completely match what was given to the shell. For example <code>'1 2' 3</code> becomes <code>"1 2" "3"</code>
 
 
Example
<source lang=bash>
#!/bin/bash
 
function print_args() {
  echo "-- Unquoted, asterisk --" ; for i in  $*  ;do echo $i; done
  echo "-- Quoted,  asterisk --" ; for i in "$*" ;do echo $i; done
  echo "-- Unquoted, atpersand --"; for i in  $@  ;do echo $i; done
  echo "-- Quoted,  atpersand --"; for i in "$@" ;do echo $i; done
}
print_args "a" "b c" "d e f"
</source>
 
 
Special Bash variables
*<code>PROMPT_COMMAND</code> :- run a shell command (value of this variable) every time your prompt is displayed
*<code>HISTTIMEFORMAT</code> :- once this variable is set, new history entries record the time along with the command; eg. <code>HISTTIMEFORMAT='%d/%m/%y %T '</code>
*<code>SHLVL</code> :- variable tracks how deeply nested you are in the bash shell, useful in scripts where you’re not sure whether you should <code>exit</code>
*<code>LINENO</code> :- reports the number of commands that have been run in the session so far. This is most often used in debugging scripts. By inserting lines like: <code>echo DEBUG:$LINENO</code> you can quickly determine where in the script you are (or are not) getting to.
*<code>REPLY</code> records <code>read INPUT</code>, so no need to read into a variable as <code>echo $INPUT</code> is the same as <code>echo $REPLY</code>
*<code>TMOUT</code> :- if nothing is typed in for the number of seconds this is set to, then the shell will exit
 
=== Resources ===
* [http://tldp.org/LDP/abs/html/internalvariables.html TLDP] Internal Variables
 
== dirname, basename, $0 ==
Option 1
<source lang="bash">
#/bin/bash
PRG=$0                  #relative path with program name
BASENAME=$(basename $0) #strip directory and suffix from filenames, here it's own script name
DIRNAME=$(dirname  $0) #strip last component from file name, returns relative path to the script
printf "PRG=$0                    --> computed full path: $PRG\n"
printf "BASENAME=\$(basename \$0) --> computed name of script: $BASENAME\n"
printf "DIRNAME=\$(dirname \$0)  --> computed  dir of script: $DIRNAME\n"
 
# Path from sourced file, this is bash only so won't work in containers /bin/sh
source $(dirname "${BASH_SOURCE[0]}")/../../env
</source>
 
 
Option 2
<source lang="bash">
CWD="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
CWD="$(dirname $(realpath $0))"
</source>
 
 
Option 3 [[Linux_expect | TLK and Expect]]
 
== Shebang ==
The shebang line has never been specified as part of POSIX, SUS, LSB or any other specification. AFAIK, it hasn't even been properly documented. There is a rough consensus about what it does: take everything between the ! and the \n and exec it. The assumption is that everything between the ! and the \n is a full absolute path to the interpreter. There is no consensus about what happens if it contains whitespace.
 
#Some operating systems simply treat the entire thing as the path. After all, in most operating systems, whitespace or dashes are legal in a path.
#Some operating systems split at whitespace and treat the first part as the path to the interpreter and the rest as individual arguments.
#Some operating systems split at the first whitespace and treat the front part as the path to the interpeter and the rest as a single argument (which is what you are seeing).
#Some even don't support shebang lines at all.
Thankfully, 1. and 4. seem to have died out, but 3. is pretty widespread, so you simply cannot rely on being able to pass more than one argument.
 
And since the location of commands is also not specified in POSIX or SUS, you generally use up that single argument by passing the executable's name to env so that it can determine the executable's location; e.g.:
#!/usr/bin/env gawk
 
This still assumes a particular path for ''env'', but there are only very few systems where it lives in ''/bin'', so this is generally safe. The location of env is a lot more standardized than the location of gawk or even worse something like python or ruby or spidermonkey. Which means that you cannot actually use any arguments at all.
 
 
Starting with coreutils 8.30 you can use
#!/usr/bin/env -S command arg1 arg2 ...
 
[https://stackoverflow.com/questions/4303128/how-to-use-multiple-arguments-for-awk-with-a-shebang-i-e/4304187#4304187 Read more...]
 
== Bash shell good practices ==
*<code>#!/usr/bin/env bash</code> is more portable than <code>#!/bin/bash</code>
*<code>set -x, set -o xtrace</code> -trace what gets executed
*<code>set -u, set -o nounset</code> -exit a script if you try to use an uninitialised variable
*<code>set -e, set -o errexit</code> -make your script exit when a command fails, then add <code>|| true</code> to commands that you allow to fail
*<code>set -o pipefail</code> -On a slightly related note, by default bash takes the error status of the last item in a pipeline, which may not be what you want. For example, false | true will be considered to have succeeded. If you would like this to fail, then you can use set -o pipefail to make it fail.
 
=== Unofficial bash [http://redsymbol.net/articles/unofficial-bash-strict-mode/ strict mode] ===
<source lang=bash>
#!/bin/bash
set -o errexit
set -o nounset
set -o pipefail
 
set -euo pipefail # one liner
</source>
 
 
Explain:
* <code>set -e</code> - immediately exit if any command has a non-zero exit status
* <code>set -u</code> - affects variables. When set, a reference to any variable you haven't previously defined - with the exceptions of $* and $@ - is an error
* <code>set -o pipefail</code> - any command in a pipeline fails, that return code will be used as the return code of the whole pipeline. By default, the pipeline's return code is that of the last command.
 
=== Script path - execute a script not in current working directory ===
When executing a script not in current working directory then you may want the script do following:
;Change CWD to the script directory. Because the script will run in sub-shell your CWD after the script exits out does not change.
<source lang="bash">
# Option 1
SCRIPT_PATH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$SCRIPT_PATH" || exit 1
 
# Option 2 - from K8s repo tree
SCRIPT_ROOT=$(dirname ${BASH_SOURCE})/..
 
# Option 3
DIR="$(dirname $(realpath $0))" # this works also if you reference symlinks and will return hardfile path
 
# Option 4 - magic variables for current file & dir
__dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
__file="${__dir}/$(basename "${BASH_SOURCE[0]}")"
__base="$(basename ${__file} .sh)"
__root="$(cd "$(dirname "${__dir}")" && pwd)" # <-- change this as it depends on your app
 
arg1="${1:-}"
</source>
 
== Testing frameworks ==
*[https://github.com/niieani/bash-oo-framework Bash Infinity] standard library and a boilerplate framework for writing tools using bash
 
== file descriptors ==
File descriptors are numbers, refereed also as a numbered streams, the first three FDs are reserved by system:
<source lang="bash">0 - STDIN, 1 - STDOUT, 2 - STDERR</source>
 
<source lang="bash" line>
echo "Enter a file name to read: "; read FILE
exec 5<>$FILE              #open/assign a file for '>' writing and reading '<'
while read -r SUPERHERO; do #-r read a file
  echo "Superhero Name: $SUPERHERO"
done <&5                            #'&' describes that '5' is a FD, reads the file and redirects into while loop
echo "File Was Read On: `date`" >&5 #write date to the file
exec 5>&-                          #close '-' FD and close out all connections to it
</source>
== Remove empty lines ==
 
<source lang-"bash">
cat blanks.txt | awk NF                  # [1]
cat blanks.txt | grep -v '^[[:blank:]]*$' # [2]
grep -e "[[:blank:]]" blanks
</source>
* [1] The command <code>awk NF</code> is shorthand for <code>awk 'NF != 0'</code>, or 'print those lines where the number of fields is not zero':
* [2] The POSIX character class [:blank:] includes the whitespace and tab characters. The asterisk (*) after the class means 'any number of occurrences of that expression'. The initial caret (^) means that the character class begins the line. The dollar sign ($) means that any number of occurrences of the character class also finish the line, with no additional characters in between.
=== References ===
*[http://www.thelinuxrain.com/articles/how-to-kill-blank-lines-elegantly how-to-kill-blank-lines-elegantly]
*[https://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat) UUOC (Useless Use Of Cat)]
 
== Here-doc | heredoc - cat redirection to create a file ==
Example of how to use heredoc and redirect STDOUT to create a file.
 
{| class="wikitable"
|+ Heredocs redirect
|-
! Standard
! Literal
! Remove white-spaces
|-
| <source lang="bash">
cat >file.txt <<EOF
cat << EOF > file.txt
wp.pl
bbc.co.uk
EOF
</source>
| <source lang="bash">
$ cat >file <<'EOF'
echo "$ABC=home_dir"
EOF
</source>
If you don't want ANY of the doc to be expanded or any special characters interpreted, quote the label with single quotes.
| <source lang="bash">
cat >file.txt <<-EOF
wp.pl
bbc.co.uk
EOF
</source>
Change the here-doc operator to <code><<-</code> . You can then indent both the here-doc and the delimiter with tabs. It must be tabs.
|}
 
Explanation
* <code><<EOF</code> indicates everything following is to be treated as input until the string <code>EOF</code> is reached.
* the <code>cat</code> command is used to print out a block of text
* <code>>hosts.txt</code> redirects the output from <code>cat</code> to the file
 
== echo | print on-screen ==
[http://www.etalabs.net/sh_tricks.html Printing the value of a variable]. The <code>\\n</code> may be omitted if a following newline is not desired. The quotation marks are essential.
<source lang="bash">
printf %s\\n "$var"
echo "$var"        # is NOT a valid substitute, NEVER use echo like this
</source>
== Expansions - string manipulation ==
*[http://www.tldp.org/LDP/abs/html/parameter-substitution.html parameter-substitution] tldp.org/LDP
*[https://unix.stackexchange.com/questions/122845/using-a-b-for-variable-assignment-in-scripts/122878 Using "${a:-b}" for variable assignment in scripts] unix.stackexchange.com
 
=== String manipulation | parameter substitution ===
<source lang="bash">
# Strip the longest match ## from the beginning(hashbang) of the string matching pattern(`* `). It matches any number of characters and a <space> .
host="box-1.acme.cloud 22"; echo "${host##* }" # -> 22              # pattern `* `
host="https://google.com" ; echo "${host##*/}" # -> google.com      # pattern `*/`
 
# Strip the longest match %% from end(percentage) of the string, matching pattern `<space>`  and `*` anything (fileglob )
host="box-1.acme.cloud 22"; echo "${host%% *}" # -> box-1.acme.cloud # pattern ` *`
host="api-uat-1"          ; echo "${host%%-*}" # -> api              # pattern `-*`
 
# Strip the longest match (%%) from end of string (percentage), [0-9]* regex matching any digit
vpc=dev02;      echo "${vpc%%[0-9]*}"          # -> dev              # pattern `[0-9]*`
 
# Strip end of string '.aws.company.*'; file globing allowed or ".* " is a regEx unsure
host="box-1.prod1.midw.prod20.aws.acme.cloud"; echo ${host%.aws.acme.*}
</source>
 
=== Default value and parameter substitution ===
<source lang="bash">
PASSWORD="${1:-data123}" #assign $1 arg if exists and is not zero length, otherwise expand to default value "data123" string
PASSWORD="${1:=data123}" #if $1 is not set or is empty, evaluate expression to "data123"
PASSWORD="${var:=$DEFAULT}"
</source>
 
=== Search and Replace ===
Expansion operators:
<code>/</code>  - substitutes 1st match
<code>//</code> - substitutes all matches
 
<source lang="bash">
# Example 1. Replace all '_' with '/' which is a common way turning a string into a path
STRING=secret_loc-a_branch1_branch2_key1
echo "path: ${STRING//_//}"
path: secret/loc-a/branch1/branch2/key1
 
# Example 2. Replace 1st match '-' with '_'
STRING=gke-dev
echo ${STRING/-/_} # -> gke_dev
</source>
 
=== tr - translate ===
Replaces/translates one set of characters from STDIN to how are they look in STDOUT; eg: char classes are:
<source lang="bash">
echo "Who is the standard text editor?" |tr [:lower:] [:upper:]
echo 'ed, of course!' |tr -d aeiou #-d deletes characters from a string
echo 'The ed utility is the standard text editor.' |tr -s astu ' ' # translates characters into spaces
echo ‘extra      tabs – 2’ | tr -s [:blank:] #supress white spaces
tr -d '\r' < dosfile.txt > unixfile.txt #remove carriage returns <CR>
</source>
=== <code>sort</code> ===
<source lang=bash>
# sort TSV by second column
sort -t$'\t' -k2
# -t$'\t' tab as a column/key delimiter, note single quotes
# to insert literal TAB do Ctrl+v, then TAB
</source>
 
 
Skip 1st line to preserve a header.
<source lang=bash>
ps aux | (sed -u 1q; sort)  | column -s, -t
# -u  -unbuffered
# 1q  -print first line (header) and quit (leaving the rest of the input to sort)
</source>
The -u switch (unbuffered) is required for those seds (notably, GNU's) that would otherwise read the input in chunks, thereby consuming data that you want to go through sort instead.
 
 
;[https://www.gnu.org/software/bash/manual/html_node/ANSI_002dC-Quoting.html#ANSI_002dC-Quoting ANSI-C Quoting]
Words of the form <code>$'string'</code> are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.
 
=== <code>uniq</code> ===
This command removes repeated lines. Therefore the file need to be sorted first.
<source lang=bash>
cat file | uniq -u        # remove only reoccuring duplications
cat file | sort | uniq -u # display only uniq lines
</source>
 
=== Continue ===
https://www.ibm.com/developerworks/aix/library/au-unixtext/index.html
Next colrm
 
=== paste ===
Paste can join 2 files side by side to provide a horizontal concatenation.
<source lang=bash>
paste -d$';' -s
# -d delimiter to be used in the output. It'll replace \n or TAB
# -s serialization, if imput are files or a list of new lines, this will join all lines
</source>
 
== Functions ==
Syntax:
* the standard <code>Bourne shell /bin/sh</code> uses the <tt>function name</tt> followed immediately by a pair of parentheses () and curly brackets {} around the code itself
* <code>bash</code> and <code>ksh</code> the function name is preceded by the keyword function <code>function myFunc</code>. The <code>function</code> keyword is optional unless the function name is also an alias. If the <code>function</code> keyword is used, parentheses are then optional.
 
Parameters are passed to functions in the same manner as to scripts using $1, $2, $3 positional parameters and using them inside the function. <code>$@</code> returns the array of all positional parameters $1, $2, etc.
<source lang="bash">
funcLastLoginInDays () {
  echo "$USERNAME - logged in $1 days ago"
  return 147
}
#eg. of calling a function with a parameter
funcLastLoginInDay $USERNAME; echo $? #last commands will show a return numeric code
</source>
=== Execution scope ===
Functions can be declared to run in the current shell using curly braces to scope the body.
<source lang="bash">
my_func () {
# Executes within the current environment
}
</source>
 
 
Whereas functions declared using brackets run in a new subshell.
<source lang="bash">
my_func () (
# Executes within a new shell
)
</source>
=== Forward declaration ===
Functions need to be defined before they are used. This is commonly achieved using the following pattern:
<source lang="bash">
main() { foo; }          #main function code, includes other functions. Visually it's at the top of a script
foo () { echo "Step_1"; } #other functions block
main "$@"                #execute main() function
</source>
=== Private functions ===
Functions that are private to a script should be named with a __ prefix and unset at the end of the script.
<source lang="bash">
__my_func () {
}
unset -f __my_func
</source>
=== Bash variable scoping ===
*Environment variable - defined in ENV of a shell
*Script Global Scope variable - defined within a script
*Function local variable - defined within a function and will become available/defined in Global Scope only after the function has been called
 
==== Variables global to shell and all subshells ====
Name the variable in upper case and using underscores.
<source lang="bash">
export MY_VARIABLE
</source>
==== Variables local within a shell script ====
Variables declared with the local keyword are local to the declaring function and all functions called from that function.
<source lang="bash">
child_function () {
        echo "$foo"
        local foo=2
        echo "$foo"
}
main () {
        local foo=1
        echo "$foo"
        child_function
        echo "$foo"
}
main
 
#### Actual output
# 1
# 1
# 2
# 1
</source>
==== Variables global within a shell script ====
Two mechanisms can be used to prevent variables declared within a script from leaking into the calling shell environment.
 
;<code>unset</code> each variable at the end of the script
<source lang="bash">
    g_variable=something
    ...
    unset g_variable
</source>
;Declare each variable from within a subshell scoped main
<source lang="bash">
    # Scope to subshell
    main () (
    # Global to current subshell
    g_variable=something
    )
</source>
==== <code>unset</code> all environment variables starting with a leading string ====
;<code>unset ${!prefix@}</code>: expands to the names of variables whose names begin with <code>prefix [...]</code> When <code>@</code> is used and the expansion appears within double quotes, each variable name expands to a separate word
;<code>unset $(compgen -v prefix)</code>: expands to all variables matching the prefix
;<code>unset $(env | grep string |awk -F'=' '{print $1}')</code>: search with string grep will fetch both environment and fetch the name of environment variable then pass to unset command
 
=== Environment scoping ===
Querying variables and functions in scope
<source lang="bash">
# List all functions declared within the current environment
compgen -A function
# List all variables declared wtihin the current environment
compgen -v
</source>
 
== <code>getopt(s)</code> - [https://dustymabe.com/2013/05/17/easy-getopt-for-a-bash-script/ argument parameters parsers] ==
;<code>getopt</code> - a command line utility provided by the util-linux package
<source lang=bash>
#TODO example
</source>
 
 
;<code>getopts</code> - Bash buildin, does not support long options, uses the <code>$OPTARG</code> variable for options that have arguments
<source lang=bash>
# parse parameters into the script
while getopts ":u:h" OPTIONS; do
  case "${OPTIONS}" in
    u) URL=${OPTARG} ;;
    h) HELP="true"
  esac
done
 
# :  - first colon has special meaning
# u: - parameter u requires argument value to be passed on because ':' follows 'u'
# h  - parameter h does not require a value to be passed on
</source>
 
;References
*[http://www.bahmanm.com/blogs/command-line-options-how-to-parse-in-bash-using-getopt command-line-options-how-to-parse-in-bash-using-getopt]
 
== set options ==
<source lang="bash">
set -o noclobber # > prevents overriding a file if already exists
set +o noclobber # enable clobber back on
</source>
 
== short version of checking command line parameters ==
Checking 3 positional parameters. Run with: <tt>./check one</tt>
<source lang="bash">
#!/bin/bash
: ${3?"Usage: $1 ARGUMENT $2 ARGUMENT $3 ARGUMENT"}
</source>
 
== IFS and delimiting ==
<code>$IFS</code> Internal Field Separator is an environment variable defining default delimiter. It can be changed at any time in a global scope of within subshell by assigning a new value, eg:
<source lang="bash">
echo $IFS # displays the current delimiter, by default it's 'space'
$IFS=','  #changes delimiter into ',' coma
</source>
 
== Counters ==
These counters do not require invocation, they always start from 1.
<source lang="bash">
#!/bin/bash
COUNTER=0  #defined only to satisfy while test function; otherwise exception [: -lt: unary operator expected
while [ $COUNTER -lt 5 ]; do
  (( COUNTER++ ))
  # COUNTER=$((COUNTER+1))
  # let COUNTER++
  echo $COUNTER
done
</source>
 
== Loops ==
=== Bash shell - for i loop ===
<source lang="bash">
for i in {1..10};    do echo $i; done
for i in {1..10..2}; do echo $i; done #step function {start..end..increment} will print 1 3 5 7 9
for i in 1 2 3 4 5 6 7 8 9 10; do echo $i; done
for i in $(seq 1 10); do echo $i; done
for ((i=1; i <= 10 ; i++)); do echo $i; done
</source>
<source lang="bash">
for host in $(cat hosts.txt); do ssh "$host" "$command" >"output.$host"; done
  a01.prod.com
  a02.prod.com
for host in $(cat hosts.txt); do ssh "$host" 'hostname; grep Certificate /etc/httpd/conf.d/*; echo -e "\n"'; done
</source>
 
=== Dash shell - for i loop ===
<source lang="bash">
#!/bin/dash
wso2List="esb dss am mb das greg"  #by default white space separates items in the list structures
for i in $wso2List; do
  echo "Product: $i"
done
</source>
=== Bash shell - while loop ===
<source lang="bash">
while true; do tail /etc/passwd; sleep 2; clear; done
</source>
 
 
{| class="wikitable"
|+ Read a file or variable line by line
|-
!  Read a file
! Read a variable
|-
| <source lang="bash">
FILE="hostlist.txt" #a list of hostnames, 1 host per line
# optional: IFS= -r
while IFS= read -r LINE; do
  (( COUNTER++ ))
  echo "Line: $COUNTER - Host_name: $LINE"
done < "$FILE" #redirect the file content into loop
</source>
| <source lang="bash">
PLACES='Warsaw
London'
while read LINE; do
  (( COUNTER++ ))
  echo "Line: $COUNTER - Line_content: $LINE"
  done <<< "$PLACES" #inject multi-line string variable, ref. heredoc
</source>
|}
* <code>-r</code> option passed to read command prevents backslash escapes from being interpreted
* <code>IFS= </code> option before read command to prevent leading/trailing whitespace from being trimmed -
 
 
Loop until ''counter'' expression is true
<source lang="bash">
while [ $COUNT -le $DISPLAYNUMBER ]; do
    echo "Hello World - $COUNT"
    COUNT="$(expr $COUNT + 1)"
done
</source>
 
== Arrays ==
Simple array
<source lang="bash">
hostlist=("am-mgr-1" "am-wkr-1" "am-wkr-2" "esb-mgr-1") #array that is actually just a list data type
for INDEX in ${hostlist[@]}; do    #@ expands to every element in the list
  printf "${hostlist[INDEX]}\n"    #* expands to match any string, so it matches every element as well
done
</source>
 
 
*<code>arr=()</code> Create an empty array
*<code>arr=(1 2 3)</code> Initialize array
*<code>${arr[2]}</code> Retrieve third element
*<code>${arr[@]}</code> Retrieve all elements
*<code>${!arr[@]}</code> Retrieve array indices
*<code>${#arr[@]}</code> Calculate array size
*<code>arr[0]=3</code> Overwrite 1st element
*<code>arr+=(4)</code> Append value(s)
*<code>str=$(ls)</code> Save <code>ls</code> output as a string
*<code>arr=( $(ls) )</code> Save <code>ls</code> output as an array of files
*<code>${arr[@]:s:n}</code> Retrieve n elements starting at index s
*<code>echo ${array[@]/*[aA]*/}</code> Remove strings[items] that contain 'a' or 'A'
 
 
Display array all items
<source lang="bash">
#!/bin/bash
array=($(cat -))
echo "${array[@]}" # display array without '\n' because STDIN is line by line
 
# Test
$(for i in {1..5}; do echo $i; done) | ./script.sh
</source>
 
 
Display specific indexes
<source lang="bash">
echo "${array[@]:3:5}"
 
</source>
 
[https://linuxhint.com/associative_array_bash/ Associative arrays], key=value pairs
<source lang="bash">
#!/bin/bash
ec2type () {  #function declaration
  declare -A array=(  #associative array declaration, bash 4
    ["c1.medium"]="1.7 GiB,2 vCPUs" #double quotation is required
    ["c1.xlarge"]="7.0 GiB,8 vCPUs"
    ["c3.2xlarge"]="15.0 GiB,8 vCPUs"
  )
  echo -e "${array[$1]:-NA,NA}"  #lookups for a value $1 (eg. c1.medium) in the array,
}                                #if nothing found (:-) expands to default "NA,NA" string
ec2type $1 #invoke the function passing 1st argument received from a command line
 
### Usage
$ ./ec2types.sh c1.medium
1.7 GiB,2 vCPUs
</source>
 
 
Convert a string into array, remember to leave the variable unquoted. This will fail if <code>$string</code> has globbing characters in. It can be avoided by <code>set -f && array=($string) && set +f</code>. The array items will be delimited by a space character.
<source lang="bash">
array=($string)
read -a array <<< $string
IFS=' ' read -a array <<< "$string" #avoid fileglob expansions for *,? and [], variable is double-quoted
</source>
 
== if statement ==
<source lang="bash">
if [ "$VALUE" -eq 1 ] || [ "$VALUE" -eq 5 ] && [ "$VALUE" -gt 4 ]; then
  echo "Hello True"
elif [ "$VALUE" -eg 7 ] 2>/dev/null; then
  echo "Hello 7"
else
  echo "False"
fi
</source>
 
== case statement ==
<source lang="bash">
case $MENUCHOICE in
  1)
    echo "Good choice!"
    ;; #end of that case statements so it does not loop infinite
  2)
    echo "Better choice"
    ;;
  *)
    echo "Help: wrong choice";;
esac
</source>
 
== traps ==
First argument is a command to execute, followed by events to be trapped
<source bash="bash">
trap 'echo "Press Q to exit"' SIGINT SIGTERM
trap 'funcMyExit' EXIT          #run a function on script exit
trap "{ rm -f $LOCKFILE; }" EXIT #delete a file on exit
</source>
 
 
<source bash="bash">
#!/bin/bash
trap ctrl_c INT # trap ctrl-c and call ctrl_c()
 
function ctrl_c() {
  echo "** Trapped CTRL-C **"
}
</source>


= BASH For Loop Sequence =
== debug mode ==
for I in {1..10}; do echo $I; done
You can enable debugging mode anywhere in your scrip and disable many times
  for I in 1 2 3 4 5 6 7 8 9 10; do echo $I; done
<source lang="bash">
  for I in $(seq 1 10); do echo $I; done
set -x #starts debug
for ((I=1; I <= 10 ; I++)); do echo $I; done
echo "Command to debug"
set +x #stops debug
</source>


for host in $(cat hosts.txt); do ssh "$host" "$command" >"output.$host"; done
    a01.prod.com
    a02.prod.com
    a03.prod.com
    a04.prod.com
for host in $(cat hosts.txt); do ssh "$host" 'hostname; grep Certificate /etc/httpd/conf.d/*; echo ====='; done


= BASH While Loop =
Or debug in sub-shell by running with -x option eg.
  while true; do tail /etc/passwd; sleep 2; clear; done
<source lang="bash">
$ bash -x ./script.sh
</source>
Additionally you can disable globbing with <code>-f</code>


= LOCAL UBTUNU MIRROR =
;Execute a script line by line
rsync -a --progress rysnc://archive.ubuntu.com/ubuntu /opt/mirror/ubuntu - command to create local mirror.
<source lang="bash">
ls /opt/mirror/ubuntu - shows all files
set -x
trap read debug
< YOUR CODE HERE >
</source>
 
;Execute a script line by line (improved version)
If your script is reading content from files, the above listed will not work. A workaround could look like the following example.
<source lang="bash">
#!/usr/bin/env bash
echo "Press CTRL+C to proceed."
trap "pkill -f 'sleep 1h'" INT
trap "set +x ; sleep 1h ; set -x" DEBUG


= NETWORK INSTALL VIA MIRROR =
< YOUR CODE HERE >
192.168.1.240 (Local web server)  /ubuntu/ (Local Folder) - enter mirror manually
</source>


OR Online mirrors:


select 'enter mirror manually' <tt>http://mirrors.sonic.net/centos/6/os/i386 - etc </tt>
Install Bash debugger
<source lang="bash">
apt-get install bashdb
bashdb your_command.sh #type step, and then hit carriage return after that
</source>
=== References ===
* [https://stackoverflow.com/questions/9080431/how-execute-bash-script-line-by-line Execute Bash script line by line]


= Boot INIT Sequence =
== errors ==
Process
=== syntax error: unexpected end of file ===
#Bios ensures everything is intact - peripherals etc.
This often can be caused by CRLF line terminators. Just run <code>dos2unix script.sh</code>.
#1st bootable disk GRUB responsible for setting up environment INITRD (Ensures kernel friends root file system) - kernel etc.
#Load INIT first process by kernel  - sets default run level of 2 unless specified.
#INIT process - loads appropriate daemons from default run level.


7 run-levels
:0 - halt
:1 - single
:2 - multi user - default
:3 - multi user
:4 - multi user
:5 - multi user
:6 - reboot


Use init + number to enter different run level
Another thing to check:
cd /boot/ - should see grub directory.
* terminate bodies of single-line functions with semicolon
cd /etc/ - ls-l rc - shows different run levels
cd /etc/init.d - contains daemons, service scripts that are referenced from run level.


= Partitions / Raid / LVM etc =
I.e. this innocent-looking snippet will cause the same error:
<source lang="bash">
die () { test -n "$@" && echo "$@"; exit 1 }
</source>
To make the dumb parser happy:
<source lang="bash">
die () { test -n "$@" && echo "$@"; exit 1; }
</source>


==Disk Commands==


*<code>df -h</code> - shows disk space
<source lang="bash">
*<code>sudo fdisk -l</code> - shows hard drive partitions
# Applies also to below, lack of last ; will produce the error
*<code>ls -l /dev/sd*</code> - SHOWS ALL DRIVES
[[ "$#" == 1 ]] && [[ "$arg" == [1,2,3,4] ]] && printf "%s\n" "blah" || { printf "%s\n" "blahblah"; usage; }
*<code>cat /proc/mdstat</code> - SHOWS LINUX RAID DRIVES IN USE
if [ -f ~/.git-completion.bash ]; then . ~/.git-completion.bash; fi;
*<code>pvdisplay</code> - SHOWS PHYSICAL VOLUMES
</source>
*<code>lvdisplay</code> - SHOW LOGICAL VOLUM DISPLAY


==Provisioning Filesystems (Extra Storage)==
== Create a big file ==
Provision storage while sudoserver is online.
Use <code>dd</code> tool that will create a file of size '''count*bs bytes'''
* 1024 bytes * 1024 count = 1048576 Bytes = 1Mb
* 1024 bytes * 10240 count = 10Mb
* 1024 bytes * 102400 count = 100Mb
* 1024 bytes * 1024000 count = 1G
<source lang="bash">
dd if=/dev/zero    bs=1024  count=10240 of=/tmp/10mb.zero #creates 10MB zero'd
dd if=/dev/urandom bs=1048576 count=100 of=/tmp/100mb.bin #creates 100MB random file
dd if=/dev/urandom bs=1048576 count=1000 of=1G.bin status=progress
</source>
dd in GNU Coreutils 8.24+ (Ubuntu 16.04 and newer) got a new status option to display the progress:
<source lang="bash">
dd if=/dev/zero bs=1048576 count=1000 of=1G.bin status=progress
1004535808 bytes (1.0 GB, 958 MiB) copied, 6.01483 s, 167 MB/s


*<code>fdisk -l</code> - reveals connected disks and partitions
1000+0 records in
*<code>df -h</code> - shows amount of memory used
1000+0 records out
*<code>/dev/sdb</code> - unpartitioned
1048576000 bytes (1.0 GB, 1000 MiB) copied, 15.4451 s, 67.9 MB/s
*<code>mklabel</code> - type MSDOs - if needed.
</source>
*<code>sudo parted</code> - partition tool
*<code>select /dev/sdb</code> - selects disk -
*<code>mkpart primary 1 10GB</code> - 10GB partition
*<code>print</code> - shows disks in parted
*<code>quit</code> - to leave parted


*<code>mke2fs</code> - overlay filesystem on new partition.
== Colours ==
*<code>mke2fs -t ext4 -j /dev/sdb1</code> - creates file system.
<source lang="bash">
# Option 1
echo_b(){ echo -e "\e[1;34m${@}\e[0m"; }
echo_g(){ echo -e "\e[1;32m${@}\e[0m"; }
echo_p(){ echo -e "\e[1;35m${@}\e[0m"; }
echo_r(){ echo -e "\e[1;31m${@}\e[0m"; }
echo_b " -h Print this message"


or
# Option 2
        red="\e[31m"
    red_bold="\e[1;31m" # '1' makes it bold
  blue_bold="\e[1;34m"
  green_bold="\e[1;32m"
light_yellow="\e[93m"
      reset="\e[0m"
echo -e "${green_bold}Hello world!${reset}"


*<code>sudo mkfs.ext4 -j /dev/sdb1/</code> - same as above.
# If you wish to unset, add the common prefix to above variables eg 'c_', then:
unset ${!c_@}


*<code>mount /dev/sdb1 /projectx/10gb.</code> - create mount point
# Option 3
*<code>mount</code> - shows all system mounts.
red=$'\e[1;31m' # '1' makes it bold
*<code>dd if=/dev/zero bs=1024 count-10240 of=/projectx/10GB/test.file.1</code> - creates 10MB in new mount
grn=$'\e[1;32m'
*<code>sudo blkid</code> - shows partition uuid for stab.
yel=$'\e[1;33m'
*<code>sudo nano /etc/fstab</code> - edits stab. <code>UUID="number" /projectx/10GB ext4 defaults</code> - stores in fstab.
blu=$'\e[1;34m'
mag=$'\e[1;35m'
cyn=$'\e[1;36m'
end=$'\e[0m'
printf "%s\n" "Text in ${red}red${end}, white and ${blu}blue${end}.
</source>


==Provision SWAP storage on demand==
Ability for kernel to extend RAM via disk.


*<code>free -m</code> - determines current stare of storage.
Colours function using a terminal settings rather than ANSI color control sequences
*<code>top</code> - also shows SWAP info.
<source lang="bash">
if test -t 1; then # if terminal
    ncolors=$(which tput > /dev/null && tput colors) # supports color
    if test -n "$ncolors" && test $ncolors -ge 8; then
        termcols=$(tput cols)
        bold="$(    tput bold)"
        underline="$(tput smul)"
        standout="$( tput smso)"
        normal="$(  tput sgr0)"    # reset to default font
        black="$(    tput setaf 0)"
        red="$(      tput setaf 1)"
        green="$(    tput setaf 2)"
        yellow="$(  tput setaf 3)"
        blue="$(    tput setaf 4)"
        magenta="$(  tput setaf 5)"
        cyan="$(    tput setaf 6)"
        white="$(    tput setaf 7)"
    fi
fi
</source>
How to use it's the same as using ANSI color control sequences. Use <code>${colorName}Text${normal}</code>
<source lang="bash">
print_bold() {
    title="$1"
    text="$2"
    echo "${red}================================${normal}"
    echo -e "  ${bold}${yellow}${title}${normal}"
    echo -en "  ${text}"
    echo "${red}================================${normal}"
}
</source>
=== References ===
*[https://misc.flogisoft.com/bash/tip_colors_and_formatting Bash colours] ANSI/VT100 Control sequences


*<code>sudo fdisk -l</code> - to identify partition space
== Run scripts from website ==
*<code>parted /dev/sdb</code> - places in context of /dev/sdv/
The script available via http/s as usual needs begin with shebang and can be executed in fly without a need to download to a file system using cURL or wget.
*<code>print</code> - to show partition table.
<source lang="bash">
*<code>mkpart primary linux-swap 10GB 12GB </code> starting from the 10GB first block moving up.
curl -sL  https://raw.githubusercontent.com/pio2pio/project/master/setup_sh | sudo -E bash -
*<code>set 2 swap on</code> - turns partitions to swap and on.
wget -qO- https://raw.githubusercontent.com/pio2pio/project/master/setup_sh | sudo -E bash -
*<code>sudo fdisk -l /dev/sdb</code> - confirms swap allocation.
</source>
*<code>sudo mkswap /dev/sdb2</code> - overlays SWAP filesystem and displays UUID (FSTAB)
== Run local scripts remotly ==
*<code>sudo blkid</code> - shows all UUID's
Use the <code>-s</code> option, which forces bash (or any POSIX-compatible shell) to read its command from standard input, rather than from a file named by the first positional argument. All arguments are treated as parameters to the script instead. If you wish to use option parameters like eg. <code>--silent true</code> make sure you put <code>--</code> before arg so it is interpreted as an argument to test.sh instead of bash.  
*<code>sudo nano /etc/fstab</code> - opens stab for editing for SWAP reference using UUID.
<source lang="bash">
*<code>UUID "" NONE swap sw 0 0</code> - for nano file.
ssh user@remote-addr 'bash -s arg' < test.sh
*<code>swap on -s</code> - displays current swap situation
ssh user@remote-addr bash -sx -- -p am -c /tmp/conf < ./support/scripts/sanitizer.sh
*<code>sudo swap on -a</code> - turns on swap storage.
# -- signals the end of options and disables further option processing.
*<code>free -m || top || swapon -s</code> - to confirm configuration.
#    any arguments after the -- are treated as filenames and arguments
# -p am -c /tmp/conf -these are arguments passed onto the sanitizer.sh script
</source>
References
*[https://unix.stackexchange.com/questions/87405/how-can-i-execute-local-script-on-remote-machine-and-include-arguments how-can-i-execute-local-script-on-remote-machine-and-include-arguments]


==Option SWAP creation which is file based==
== Scripts  - solutions ==
*<code>dd if=/dev/zero of=/projectx/10GB/swapfile3GB count=3G bs=1024</code> - creates dummy swap file.
Check connectivity
*<code>sudo mkswap /projectx/10GB/swapfile3GB</code> - overlays SWAP file system.
<source lang="bash">
*<code>sudo nano /etc/fstab</code> - opens stab for editing for SWAP reference using path <tt>/projectx/10GB/swapfile3GB</tt>
#!/bin/bash
*<code>sudo swapon -a</code> - tuns on all swap storage.


==Storage Management LVM (Logical Volume Management)==
red="\e[31m"; green="\e[32m"; blue="\e[34m"; light_yellow="\e[93m"; reset="\e[0m"
Volume sets based on various disparate storage technologies.


Common configuration - raid hardware (redundancy) / LVM overlaying RAID config (aggregation)
if [ "$1" != "onprem" ]; then
    echo -e "${blue}Connectivity test to servers: on-prem${reset}"
    declare -a hosts=(  
    wp.pl
    rzeczpospolita.pl
    )
else
    echo -e "${blue}Connectivity test to Polish news servers: polish-news${reset}"
    declare -a hosts=(  
    wp.pl
    rzeczpospolita.pl
    )
fi


Ability to extend, reduce, manipulate storage on demand.
#################################################


LVM storage hierarchy:
echo -e "From server: ${blue}$(hostname -f) $(hostname -I)${reset}"


Volume Group (Consists of 1 or more physical volumes)
for ((i = 0; i < ${#hosts[@]}; ++i)); do
*Logical Volume(s)
  RET=$(timeout 3 nc -z ${hosts[$i]} 22 2>&1) \
*File System(s)
  && echo -e "[OK] $i ${hosts[$i]}" \
  || echo -e "${red}[ER] $i ${hosts[$i]}\t[ERROR: \
  $(if [ $? -eq 124 ]; then echo "timeout maybe filtered traffic"; else \
  if [ -z "$RET" ]; then echo "unknown"; else echo -e "$RET"; fi; fi)]${reset}"
  sleep 0.3
done


==6 steps to LVM setup==
# Working as well
Appropirate 1 or more LVM partitions
#for host in "${hosts[@]}"; do
#<code>sudo parted /dev/sdb</code>
# RET=$(timeout 3 nc -z ${host} 22 2>&1) && echo -e "[OK] ${host}" || echo -e "${red}[ER] ${host}\t[ERROR: \
#<code>mkpart extended 13GB 20GB</code>
# $(if [ $? -eq 124 ]; then echo "timeout maybe filtered traffic"; else \
#<code>print</code>
# if [ -z "$RET" ]; then echo "unknown"; else echo -e "$RET"; fi; fi)]${reset}"
#<code>mkpart logical lvm 13GB 20GB</code>
# sleep 0.3
##<code>select /dev/sdc/</code>
#done
#<code>mklabel msdos</code>
</source>
#mkpart primary 1GB 20GB</code>
##<code>set 1 lvm on</code>
##<code>print</code>
#<code>20GB on SDC and 7GB on SDB</code>


==Partition==
= LOCAL UBTUNU MIRROR =
*<code>sudo pvcreate /dev/sdb5 /dev/sdc1</code> - allocates LVM partitions as physical volumes
rsync -a  --progress rysnc://archive.ubuntu.com/ubuntu /opt/mirror/ubuntu - command to create local mirror.
*<code>sudo pvdisplay</code> - shows LVM physical volumes.  
ls /opt/mirror/ubuntu - shows all files
*<code>sudo vgcreate volgroup001 /dev/sdb5 /dev/sdc1</code> - aggregates volumes to volume group
*<code>sudo lvcreate -L 200GB volgroup001 -n logvol001</code> - creates logical volumes
*<code>sudo lvdisplay</code> - shows logical volumes
*<code>sudo mk3fs -t ext -j /dev/volgroup001/logvol001</code> - overlays EXT4 filesytem.
*<code>Mount filesystem and commit changes to fstab</code>


==LVM related tasks==
#<code>sudo lvrename volgroup001 logvol001 volgroup002</code> - renames volume group. (Remember to edit fstab file or unmount)
#<code>sudo lvresize -L 25GB /dev/volgroup001/logvolvar </code> - Resize logical volume by 5GB from 20GB
#<code>sudo resize2fs /dev/mapper/volgroup001-logvolvar 25G</code> - Resize filesystem after increasing memory.
#<code>sudo lvremove /dev/volgroup001/logvolvar</code> - Remove volume completely. (Umount first)
#<code>sudo parted /dev/sdc  </code> - Add or assign more partitions to volume group LVM.
#<code>print</code>
#<code>mkpart primary 20GB 25GB </code>
#<code>print</code>
#<code>set 2 lvm on</code>
#<code>print</code>
#<code>sudo pvcreate /dev/sdc2</code>
#<code>sudo pvdisplay</code>
#<code>sudo vgextend volgroup001 /dev/sdc2</code> - add new PV to volume group
= Bash =
= Bash =
Delete key gives ~ ? Add the following line to your $HOME/.inputrc (might not work if added to /etc/inputrc )
Delete key gives ~ ? Add the following line to your $HOME/.inputrc (might not work if added to /etc/inputrc )
  "\e[3~": delete-char
  "\e[3~": delete-char


[[Category:linux]]
=[https://www.linuxjournal.com/content/shell-process-redirection  Process substitution]=
Process substitution expands output of commands to a file. Depends on OS default /.../fd descriptor or temporary file is used to store the content. But the syntax works without using any file names thus looks simpler. There are also situations that a command requires a file and | pipe cannot be used.
source <(kubectl completion bash)
 
The example sources a file but syntaticly there is no file when using the process substitution construction.
 
= References =
* [https://mywiki.wooledge.org/BashPitfalls BashPitfalls] awesome!


[[Category:Linux]]
[[Category:bash]] [[Category:shell]] [[Category:linux]]

Latest revision as of 16:15, 19 December 2023

One liners and aliases

find . -type d -exec chmod 0755 {} \; #find dirs and change their permissions
df -h                               # disk free shows available disk space for all mounted partitions
du -sch .[^.]* * |sort -h           # disk usage summary including hidden files
du -skh * --exclude /data | sort -h # disk usage summary for each directory and sort using human readable size
free -m             # displays the amount of free and used memory in the system
lsb_release -a      # prints version information for the Linux release you're running
tload               # system load on text terminal

# Aliases
alias ll='ls -al --file-type --color --group-directories-first' #group dirs first

# SSH
ssh-keygen -o -a 100 -t ed25519
# -o :- save the key in new format (the old one is really weak) by using -o and additionally to specify to use many KDF function rounds to secure the key using 

# Sudo
sudo su - jenkins   #switch user taking current environment variables

# List processes
ps -ef --forest | grep nginx

# Create 100% cpu load, no I/O overloaded so your terminal is still accessible
timeout 10 yes>/dev/null&

cp, mv, mkdir with {,} brace expansion

cp /etc/httpd/httpd.conf{,.bak}     #-> cp /etc/httpd/httpd.conf /etc/httpd/httpd.conf.bak
mv /etc/httpd/httpd.conf{.bak,}     #-> mv /etc/httpd/httpd.conf.bak /etc/httpd/httpd.conf
mv /etc/httpd/httpd.{conf.bak,conf} #-> mv /etc/httpd/httpd.conf.bak /etc/httpd/httpd.conf
mkdir -p /apache-jail/{usr,bin,lib64,dev}
echo foo{1,2,3}.txt                 #-> foo1.txt foo2.txt foo3.txt

Tests

Bash test

# [ -v varname ] True if the shell variable varname is set (has been assigned a value); bash>4.2
# [ -z string  ]  True if the length of string is zero.

Copy with progress bar

scp with progress bar
scp -v ~/local/file.tar.gz ubuntu@10.46.77.156:~ #note '-v' option
rsync -r -v --progress -e ssh ~/local/file.tar.gz ubuntu@remote-server.exe:~
sending incremental file list
file.tar.gz
      6,389,760  10%  102.69kB/s    0:08:40


rsync and cp
rsync -aP                          #copy with progress can be also aliased 
alias cp='rsync -aP'
cp -rv old-directory new-directory #shows progress bar


PV does not preserve permissions and does not handle attributes
pv ~/kali.iso | cat - /media/usb/kali.iso #equals cp ~/kali.iso /media/usb/kali.iso
pv ~/kali.iso > /media/usb/kali.iso       #equals cp ~/kali.iso /media/usb/kali.iso
pv access.log | gzip > access.log.gz      #shows gzip compressing progress


PV can be imagined as CAT command piping '|' output to another command with a bar progress and ETA times. -c makes sure one pv output is not use to write over to another, -N creates a named stream. Find more at How to use PV pipe viewer to add progress bar to cp, tar, etc..

$ pv -cN source access.log | gzip | pv -cN gzip > access.log.gz
source:  760MB 0:00:15 [37.4MB/s] [=>     ] 19% ETA 0:01:02
gzip: 34.5MB 0:00:15 [1.74MB/s] [  <=>  ]

Copy files between remote systems quick

List SSH MACs, Ciphers, and KexAlgorithms

ssh -Q cipher; ssh -Q mac; ssh -Q kex

Rsync

Basic syntax of rsync command and examples of comping over ssh.

rsync -options source destination
rsync -axvPW  user@remote-srv:/remote/path/from /local/path/to
rsync -axvPWz user@remote-srv:/remote/path/from /local/path/to  #compresses before transfer
# -a :archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps
# -x, --one-file-system  :don't cross filesystem boundaries
# -P, --progress         :progress bar
# -W, --whole-file       :copy files whole (w/o delta-xfer algorithm)
# -z, -compress          :files before transfer, consumes ~70% of CPU
# -r :copies data recursively (but don’t preserve timestamps and permission while transferring data
# -h :human-readable, output numbers in a human-readable format
# -v :verbose
# --remove-source-files  :removes source file once copied


Rsync over ssh

rsync -avPW -e ssh $SOURCE $USER@$REMOTE:$DEST
rsync -avPW -e ssh /local/path/from remote@server.com:/remote/path/to
# -e, --rsh=COMMAND      :specify the remote shell to use
# -a, --archive          :archive mode; equals -rlptgoD (no -H,-A,-X)

Tar over ssh

Copy from a local server (data source) to a remote server. It TARs a folder but we do not specify an archive name "-" so it redirects (tar stream) via the pipe "|" to ssh, where extracts the tarball at the remote server.

tar -cf - /path/to/dir | ssh user@remote-srv-copy-to 'tar -xvf - -C /path/to/remotedir'

Coping from local server (the data source) to a remote server as a single compressed .tar.gz file

tar czf - -C /path/to/source files-and-folders | ssh user@remote-srv-copy-to "cat - > /path/to/archive/backup.tar.gz"

Coping from a remote server to local server (where you execute the command). This will execute tar on remote server and redirects "-" to STDOUT to extract locally.

ssh user@remote-srv-copy-from "tar czpf - /path/to/data" | tar xzpf - -C /path/to/extract/data
-c; -f --file; -     -create a new archive; archive name; 'dash' means STDOUT
-                    -redirect to STDOUT
-C, --directory=DIR  -change to directory DIR, cd to the specified directory at the destination 
-x -v -f             -extract; -dispaly files on a screen; archive_name
References

Listing a directory in the form of a tree

$ tree ~
$ ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/   /' -e 's/-/|/'
$ alias lst='ls -R | grep ":$" | sed -e '"'"'s/:$//'"'"' -e '"'"'s/[^-][^\/]*\//--/g'"'"' -e '"'"'s/^/   /'"'"' -e '"'"'s/-/|/'"'"    
$ ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\// /g' -e 's/^/ /'   #using spaces, doesn't list .git

A directory statistics: size, files count and files types based on an extension

find . -type f | sed 's/.*\.//' | sort | uniq -c | sort -n | tail -20; echo "Total files: " | tr --delete '\n'; find . -type f | wc -l; echo "Total size: " | tr --delete '\n' ; du -sh

Constantly print tcp connections count in line

while true; do echo -n `ss -at | wc -l`" " ; sleep 3; done

Stop/start multiple services in a loop

cd /etc/init.d
for i in $(ls servicename-*); do service $i status; done
for i in $(ls servicename-*); do service $i restart; done

Linux processes

List all signals

kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT      7) SIGBUS       8) SIGFPE       9) SIGKILL     10) SIGUSR1
11) SIGSEGV     12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM
16) SIGSTKFLT   17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU     25) SIGXFSZ
26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGIO       30) SIGPWR
31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX

Unlock a user on the FTP server

pam_tally2 --user <uid>  #this will show you the number of failed logins
pam_tally2 --user <uid> --reset  #this will reset the count and let the user in

Tail log files

tail-f-the-output-of-dmesg or install multitail

tail -f /var/log/{messages,kernel,dmesg,syslog} #old school but not perfect
less +F /var/log/syslog  #equivalent tail -f but allows for scrolling
watch 'dmesg | tail -50' # approved by man dmesg
watch 'sudo dmesg -c >> /tmp/dmesg.log; tail -n 40 /tmp/dmesg.log' #tested, but experimental
less
multifile monitor to see what’s happening in the second file, you need to first Ctrl-c to go to normal mode, then type :n to go to the next buffer, and then F again to go back to the watching mode.

Colourize log files

BAT, supports syntax highlighting for a large number of programming and markup languages
# Ubuntu <19.10
wget https://github.com/sharkdp/bat/releases/download/v0.12.1/bat_0.12.1_amd64.deb
sudo dpkg -i bat_0.12.1_amd64.deb
bat deployment.yaml -l yaml

# Ubuntu 19.10+, notice the command is 'batcat' due to collision with another command 'bat'
sudo apt install bat # Ubuntu Eoan 19.10 or Debian unstable sid
batcat deployment.yaml -l yaml

# Usage
bat kubernetes/application.yml
CCZE, a fast log colorizer written in C, intended to be a drop-in replacement for colorize
sudo apt install ccze # log colorizer
# -c, --color KEY=COLOR :- set the color of the keyword KEY to COLOR, like one would do in one of the configuration files
# -A --raw-ansi         :- ANSI instead curses
# -m --mode             :- mode curses, ansi, html
# -h --html             :- instead of colorising the input onto the console, output it in HTML format instead
# -l                    :- list of plugins


GRC, Generic Colouriser by default for these commands ping, traceroute, gcc, make, netstat, diff, last, ldap, and cvs
sudo apt install grc # frontend for generic colouriser grcat(1)
grc tail -f /var/log/apache/access.log /var/log/apache/error.log

Big log files

Clear a file content

Therefore clearing logs from this location, will release space on / partition

cd /chroot/httpd/usr/local/apache2/logs
> mod_jk.log     #zeroize the file

Clear a part of a file

You can use time commands to measure time lapsed to execute the command

$ wc -l catalina.out     #count lines
3156616 catalina.out
$ time split -d -l 1000000 catalina.out.tmp catalina.out.tmp-   #split tmp file every 1000000th line prefixing files 
                                                                #with catalina.out.tmp-##, -d specify ## numeric sequence
$ time tail -n 300000 catalina.out > catalina.out.tmp     #creates a copy of the file with 300k last lines
$ time cat catalina.out.tmp > catalina.out                #clears and appends tmp file content to the current open file

Locked file by a process does not release free space back to a file system

When you delete a file you in fact deleting an inode pointer to a disk block. If there is still a file handle attached to it the files system will not see the file but the space will not be freed up. One way to re evaluate the space is to send HUP signal to the process occupying the file handle. This is due to how Unix works, if a process is still using a file - the system system shouldn't be trying to get rid of it.

kill -HUP <PID>

Diagram is showing File Descriptor (fd) table, File table and Inode table, finally pointing to a block device where data is stored.

File descriptor table and file table and inode table
Search for deleted files are still held open.
$ lsof | grep deleted
COMMAND    PID    TID   USER   FD   TYPE   DEVICE  SIZE/OFF   NODE NAME
mysqld     2230         mysql  4u   REG    253,2   793825     /var/tmp/ibXwbu5H (deleted)
mysqld     2230         mysql  5u   REG    253,2   793826     /var/tmp/ibfsqdZz (deleted)
$ lsof | grep DEL
COMMAND    PID   TID    USER    FD  TYPE   DEVICE  SIZE/OFF   NODE NAME
httpd      54290        apache  DEL  REG    0,4     32769     /SYSV011246e6
httpd      54290        apache  DEL  REG    0,4    262152     /SYSV0112b646

Timeout commands after certain time

Ping multiple hosts and terminate ping after 2 seconds, helpful when a server is behind firewall and no responses to ICMP returns

$ for srv in `cat nodes.txt`;do timeout 2 ping -c 1 $srv; done

Time manipulation

Replace unix timestamps in logs to human readable date format

user@laptop:/var/log$ tail dmesg | perl -pe 's/(\d+)/localtime($1)/e'
[   Thu Jan  1 01:00:29 1970.168088] b43-phy0: Radio hardware status changed to DISABLED
[   Thu Jan  1 01:00:29 1970.308597] tg3 0000:09:00.0: irq 44 for MSI/MSI-X
[   Thu Jan  1 01:00:29 1970.344378] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   Thu Jan  1 01:00:29 1970.344745] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
 
user@laptop:/var/log$ tail dmesg
[   29.168088] b43-phy0: Radio hardware status changed to DISABLED
[   29.308597] tg3 0000:09:00.0: irq 44 for MSI/MSI-X
[   29.344378] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   29.344745] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Convert seconds to human readable

sec=4717.728; eval "echo $(date -ud "@$sec" +'$((%s/3600/24))d:%-Hh:%Mm:%Ss')"  # '-' hyphen at %-H it means do not pad 
0d:1h:18m:37s
References

tai64n Toward a Unified Timestamp with explicit precision

tai64n reads lines from stdin and prefixes with a timestamp in 12-byte TAI64N labels to stdout

sudo apt-get install daemontools


tai64n
puts a precise timestamp on each line
tai64nlocal
scans stdin for '@' followed precise TAI64N timestamps and converts to a human-readable format; timestamp is converted to local time in ISO format: YYYY-MM-DD HH:MM:SS.SSSSSSSSS


Usage
tai64n tai64nlocal
for i in {1..5}; do echo hello $i | tai64n; done
@400000005e36d56736b9e14c hello 1
@400000005e36d56737330cd4 hello 2
@400000005e36d56737bc5944 hello 3
@400000005e36d56738081fb4 hello 4
@400000005e36d567383e1c04 hello 5
for i in {1..5}; do echo hello $i | tai64n; done | tai64nlocal
2020-02-02 13:57:49.918151500 hello 1
2020-02-02 13:57:49.926092500 hello 2
2020-02-02 13:57:49.935090500 hello 3
2020-02-02 13:57:49.940056500 hello 4
2020-02-02 13:57:49.943594500 hello 5


References

Time stamp with precision log() function

log() {
    local msg="$1"
    if [ -n "$msg" ]; then
        level="${2:-INFO}"
        printf "%s - %s - %s\n" "$(date +"%Y-%m-%dT%T.%3N")" "$level" "$msg" >&2
    fi
}

The .%3N is not supported by POSIX /bin/sh.

Timestamp in the filename

echo $(date +"%Y%m%d-%H%M")    #20180618-1136
echo $(date +"%Y%m%dT%H%M")    #20180618T1136
echo -n $(date +"%Y%m%d-%H%M") # no new line
date +"%Y%m%d-%H%M"|tr -d "\n" # no new line


Examples of use

# Backup dir with all Kubernetes manifests
$> cp -r manifests_dir backup/dist-$(git rev-parse --abbrev-ref HEAD)-$(date +"%Y%m%dT%H%M")-$(git rev-parse HEAD | cut -c1-8)

# Kubernetes manifest tagging for multiple iteration comparison
$> cp application.yml application-$(git rev-parse --abbrev-ref HEAD)-$(date +"%Y%m%dT%H%M")-$(git rev-parse HEAD | cut -c1-8).yml

$> ls -l                                   # ___name____ _____branch____________ ___time______ _commit_
-rw-rw-r-- 1 piotr piotr 275158 Jan 23 12:23 application-IPN-1000_splitHeartbeat-20200123T1223-aaaabbbb.yml

# Terraform, note "!!:1" it's bash expansion to recall a previous command 1st argument, so it will be action eg. apply
$> terraform apply -var-file=<FILENAME> | tee "!!:1"-$(git rev-parse --abbrev-ref HEAD)-$(date +"%Y%m%dT%H%M")-$(git rev-parse HEAD | cut -c1-8).log

$> ls -l                                   #    action _________branch_______ _____time____ _commit_ 
-rw-rw-r-- 1 vagrant vagrant 29794 Jan 31 15:24  apply-IPN-1111_acmValidation-20200131T1524-aaaabbbb.log

Reference:

Replace numbers into human readable units

numfmt is a standard Ubuntu package that allows to convert numbers into the human readable units.

numfmt --to iec 1024       # -> 1.0K
numfmt --to iec 1024000    # -> 1000K
numfmt --to iec 1024000000 # -> 977M

# Convert numbers in column 5
kubectl get vpa -A --no-headers | tr -s " " | numfmt --to iec --field=5 --invalid=ignore | column -t

Sed - Stream Editor

Replace, substitute

sed -i 's/pattern/substituteWith/g' ./file.txt  #substitutes ''pattern'' with ''substituteWith'' in each matching line
sed    's/A//g'     #substitutes 'A' with an empty string '/g' - globally; all it means it will remove 'A' from whole string

##Type of operation
# s/ -substitute

##Commands
# /p printing modified content, print only the changes to the console
# /w substitute and write matching patterns into a file
$ sed -n "s/(name)/Mark/pw log2.txt" birthday.txt
# /i -insensitive search
# /g -global replace, all matches on all lines in a file
# /2 -nth replace, replace 2nd occurrence in each line
sed "s/cha/foo/2" log.txt
# /d -delete any line matching a pattern

##Option flags
# -i editing in place
# -i.bak saves original file with .bak extension
# -n suppress the default output to STD out
sed 's/pattern/substituteWith/w matchingPattern.txt' ./file.txt


Substitute every file grepp'ed (-l returns a list of matched files)

for i in `grep -iRl 'dummy is undefined or' *`;do sed -i 's/dummy is undefined or/dummy is defined and/' $i; sleep 1; done


Matching lines
#line matching pattern
#     \       _ then perform subsequent /s search and replace  
#       \    /  
$ sed "/foo/s/./*/g" birthday.txt #replace any line containing 'foo' with '*'


Replace a string between XML tags. It will find String Pattern between <Tag> and </Tag> then it will substitute /s findstring with replacingstring globally /g within the Pattern String.

sed -i '/<Tag>/,/<\/Tag>/s/findstring/replacingstring/g' file.xml

Replace 1st occurance

sed '0,/searchPattern/s/searchPattern/substituteWith/g' ./file.txt #0,/searchPattern -index0 of the pattern first occurrence

Remove all lines until first match

sed -r -i -n -e '/Dynamic,$p' resolv.conf
#-r regex, -i inline, so no need to print lines(quiet,silient)
#-e expression, $p print to the EOFile

Regex search and remove (replace with empty string) all xml tags.

sed 's/<[^>]*>//' ./file.txt
# [] -regex characters class, [^>] -negate, so following character is not > and <> won't be matched,

References

Sed manual Official GNU

Awk - language for processing text files

Remove duplicate keys from known_host file

awk '!seen[$0]++' ~/.ssh/known_hosts > /tmp/known_hosts; mv -f /tmp/known_hosts ~/.ssh/

Useful packages

  • ARandR Screen Layout Editor - 0.1.7.1

Manage users

Interactive
sudo adduser piotr
Scripting
sudo useradd -m piotr
# -m creates $HOME directory if does not exists
sudo useradd -m piotr -s /bin/bash -d /sftp/piotr
# -s define default shell
# -d define $HOME directory

User groups

Add user to a group

In ubuntu adding a user to group admin will grant the root privileges. Adding them to sudo group will allow to execute any command

sudo usermod -a -G nameofgroup nameofuser #requires to login again


In RedHat/CentOS add a user to a group 'wheel' to grant him sudo access

sudo usermod -a -G wheel <nameofuser>   #requires to login again
# -a add
# -G list of supplementary groups
# -g primary group 
groups <nameofuser> # list all groups, 1st is a primary-group

Change user primary group and other groups

To assign a primary group to an user

sudo usermod -g primarygroupname username


To assign secondary groups to a user (-a keeps already existing secondary groups intact otherwise they'll be removed)

sudo usermod -a -G secondarygroupname username

From man-page:
 ...
 -g (primary group assigned to the users)
 -G (Other groups the user belongs to), supplementary groups
 -a (Add the user to the supplementary group(s))
 ...

Modify group

Rename a group name -n

sudo groupmod -n <new-group-name> <old-group>

Show USB devices

lsusb -t     #shows USB tree

Copy and Paste in terminal

In Linux X graphical interface this works different then in Windows you can read more in X Selections, Cut Buffers, and Kill Rings. When you select some text this becomes the Primary selection (not the Clipboard selection) then Primary selection can be pasted using the middle mouse button. Note however that if you close the application offering the selection, in your case the terminal, the selection is essentially "lost".

Option 1 works in X

  • select text to copy then use your mouse middle button or press a wheel


Option 2 works in Gnome Terminal and a webterminal

  • Ctrl+Shift+c or Ctrl+Insert - copy
  • Ctrl+Shift+v or Shift+Insert - paste


Option 3 Install Parcellite GTK+ clipboard manager

sudo apt-get install parcellite

then in the settings check "use primary" and "synchronize clipboards"

Generate random password

cat /dev/urandom|tr -dc "a-zA-Z0-9"|fold -w 48|head -n1
openssl rand -base64 24
pwgen 32 1 #pwgen <length> <num_of_passwords>

xargs - running command once per item in array

  • -p - print all commands and prompt for confirmation
  • -t - print commands while executing


declare -a ARRAY=(item1 item2 item2)
echo "${ARRAY[@]}" | xargs -0 -t -n1 -I {} . "${CWD}/{}"

# Insert args in the middle
dig wp.pl +short | xargs -I% nc -zv % 443
# | -I replace-str - replace  occurrences  of replace-str in the initial-arguments with names read from standard input

Localization - Change a keyboard layout

setxkbmap gb

at, atd - schedule a job

At can execute command at given time. It's important to remember 'at' can only take one line and by default it uses limited /bin/sh shell.

service atd status # check if at demon is running
at 01:05 AM        # schedule a job, this will drop into at> prompt, use C^d to exit
atq                # list job queue
at -c <job number> # cat the job_number
atrm  <job_number> # deletes job
mail               # at command emails a user who scheduled a job with its output

whoami, who and w

# 'whoami' - print effective userid
vagrant@ubuntu-bionic ~  $ whoami
vagrant
vagrant@ubuntu-bionic ~  $ sudo su -
root@ubuntu-bionic:~$ whoami # print effective userid
root

# 'who' - show all users who has a login-tty, shows who is logged on, even if 'su' has been used it will print the logged on userid
# who = whoami(userid) + login-tty
root@ubuntu-bionic:~$ who
vagrant  pts/0        2020-02-02 19:02 (10.0.2.2)

# 'who' vs 'who am i'
root@ubuntu-bionic:~$ who      # shows all logged sessions for the user, but this shows multiple termianl logins
vagrant  pts/0        2020-02-02 19:02 (10.0.2.2)
vagrant  pts/1        2020-02-02 19:13 (10.0.2.2)
root@ubuntu-bionic:~$ who am i # shows only the current terminal session
vagrant  pts/0        2020-02-02 19:02 (10.0.2.2)
## 'who arg1 arg2' 2 parameters followed by who, it will print the current login-tty user, no matters what the params are

# 'w' - show who is logged on and what they are doing
# w = who(logged on sessions) + what they are doing
root@ubuntu-bionic:~$ w
 19:19:17 up 45 min,  2 users,  load average: 0.02, 0.02, 0.00
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
vagrant  pts/0    10.0.2.2         19:02    0.00s  0.24s  0.00s sshd: vagrant [priv]
vagrant  pts/1    10.0.2.2         19:13    5:50   0.08s  0.08s -bash

which and whereis

which pwd   #returns binary path only
whereis pwd #returns binary path and paths to man pages

Linux Shell

Syntax checker

Common tool is to use shellchecker. Here below it's a reference how to install stable version on Ubuntu

sudo apt install xz-utils #pre req
export scversion="stable" # or "v0.4.7", or "latest"
wget "https://storage.googleapis.com/shellcheck/shellcheck-${scversion}.linux.x86_64.tar.xz"
tar --xz -xvf shellcheck-"${scversion}".linux.x86_64.tar.xz
cp shellcheck-"${scversion}"/shellcheck /usr/bin/
shellcheck --version

Special variables

  • $_ The last argument of the previous command.
  • $- Flags passed to script (using set)
  • $? Exit status of a command, function, or the script itself last executed
  • $$ The process ID of the current shell. For shell scripts, this is the process ID under which they are executing.
  • $! PID (process ID) of last job run in background
  • $0 The filename of the current script.
  • $# arguments vount supplied to a script.
  • $@ All the arguments are individually double quoted. If a script receives two arguments, $@ is equivalent to $1 $2.
  • $* All the arguments are double quoted. If a script receives two arguments, $* is equivalent to $1 $2.
Note on $* and $@

$* and $@ when unquoted are identical and expand into the arguments. "$*" is a single word, comprising all the arguments to the shell, joined together with spaces. For example '1 2' 3 becomes "1 2 3". "$@" is identical to the arguments received by the shell, the resulting list of words completely match what was given to the shell. For example '1 2' 3 becomes "1 2" "3"


Example

#!/bin/bash

function print_args() {
  echo "-- Unquoted, asterisk --" ; for i in  $*  ;do echo $i; done
  echo "-- Quoted,   asterisk --" ; for i in "$*" ;do echo $i; done
  echo "-- Unquoted, atpersand --"; for i in  $@  ;do echo $i; done
  echo "-- Quoted,   atpersand --"; for i in "$@" ;do echo $i; done
}
print_args "a" "b c" "d e f"


Special Bash variables

  • PROMPT_COMMAND :- run a shell command (value of this variable) every time your prompt is displayed
  • HISTTIMEFORMAT :- once this variable is set, new history entries record the time along with the command; eg. HISTTIMEFORMAT='%d/%m/%y %T '
  • SHLVL :- variable tracks how deeply nested you are in the bash shell, useful in scripts where you’re not sure whether you should exit
  • LINENO :- reports the number of commands that have been run in the session so far. This is most often used in debugging scripts. By inserting lines like: echo DEBUG:$LINENO you can quickly determine where in the script you are (or are not) getting to.
  • REPLY records read INPUT, so no need to read into a variable as echo $INPUT is the same as echo $REPLY
  • TMOUT :- if nothing is typed in for the number of seconds this is set to, then the shell will exit

Resources

  • TLDP Internal Variables

dirname, basename, $0

Option 1

#/bin/bash
PRG=$0                  #relative path with program name
BASENAME=$(basename $0) #strip directory and suffix from filenames, here it's own script name
DIRNAME=$(dirname   $0) #strip last component from file name, returns relative path to the script
printf "PRG=$0                    --> computed full path: $PRG\n"
printf "BASENAME=\$(basename \$0) --> computed name of script: $BASENAME\n"
printf "DIRNAME=\$(dirname \$0)   --> computed  dir of script: $DIRNAME\n"

# Path from sourced file, this is bash only so won't work in containers /bin/sh
source $(dirname "${BASH_SOURCE[0]}")/../../env


Option 2

CWD="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
CWD="$(dirname $(realpath $0))"


Option 3 TLK and Expect

Shebang

The shebang line has never been specified as part of POSIX, SUS, LSB or any other specification. AFAIK, it hasn't even been properly documented. There is a rough consensus about what it does: take everything between the ! and the \n and exec it. The assumption is that everything between the ! and the \n is a full absolute path to the interpreter. There is no consensus about what happens if it contains whitespace.

  1. Some operating systems simply treat the entire thing as the path. After all, in most operating systems, whitespace or dashes are legal in a path.
  2. Some operating systems split at whitespace and treat the first part as the path to the interpreter and the rest as individual arguments.
  3. Some operating systems split at the first whitespace and treat the front part as the path to the interpeter and the rest as a single argument (which is what you are seeing).
  4. Some even don't support shebang lines at all.

Thankfully, 1. and 4. seem to have died out, but 3. is pretty widespread, so you simply cannot rely on being able to pass more than one argument.

And since the location of commands is also not specified in POSIX or SUS, you generally use up that single argument by passing the executable's name to env so that it can determine the executable's location; e.g.:

#!/usr/bin/env gawk

This still assumes a particular path for env, but there are only very few systems where it lives in /bin, so this is generally safe. The location of env is a lot more standardized than the location of gawk or even worse something like python or ruby or spidermonkey. Which means that you cannot actually use any arguments at all.


Starting with coreutils 8.30 you can use

#!/usr/bin/env -S command arg1 arg2 ...

Read more...

Bash shell good practices

  • #!/usr/bin/env bash is more portable than #!/bin/bash
  • set -x, set -o xtrace -trace what gets executed
  • set -u, set -o nounset -exit a script if you try to use an uninitialised variable
  • set -e, set -o errexit -make your script exit when a command fails, then add || true to commands that you allow to fail
  • set -o pipefail -On a slightly related note, by default bash takes the error status of the last item in a pipeline, which may not be what you want. For example, false | true will be considered to have succeeded. If you would like this to fail, then you can use set -o pipefail to make it fail.

Unofficial bash strict mode

#!/bin/bash
set -o errexit
set -o nounset
set -o pipefail

set -euo pipefail # one liner


Explain:

  • set -e - immediately exit if any command has a non-zero exit status
  • set -u - affects variables. When set, a reference to any variable you haven't previously defined - with the exceptions of $* and $@ - is an error
  • set -o pipefail - any command in a pipeline fails, that return code will be used as the return code of the whole pipeline. By default, the pipeline's return code is that of the last command.

Script path - execute a script not in current working directory

When executing a script not in current working directory then you may want the script do following:

Change CWD to the script directory. Because the script will run in sub-shell your CWD after the script exits out does not change.
# Option 1
SCRIPT_PATH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$SCRIPT_PATH" || exit 1

# Option 2 - from K8s repo tree
SCRIPT_ROOT=$(dirname ${BASH_SOURCE})/..

# Option 3
DIR="$(dirname $(realpath $0))" # this works also if you reference symlinks and will return hardfile path

# Option 4 - magic variables for current file & dir
__dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
__file="${__dir}/$(basename "${BASH_SOURCE[0]}")"
__base="$(basename ${__file} .sh)"
__root="$(cd "$(dirname "${__dir}")" && pwd)" # <-- change this as it depends on your app

arg1="${1:-}"

Testing frameworks

  • Bash Infinity standard library and a boilerplate framework for writing tools using bash

file descriptors

File descriptors are numbers, refereed also as a numbered streams, the first three FDs are reserved by system:

0 - STDIN, 1 - STDOUT, 2 - STDERR
echo "Enter a file name to read: "; read FILE
exec 5<>$FILE               #open/assign a file for '>' writing and reading '<'
while read -r SUPERHERO; do #-r read a file
  echo "Superhero Name: $SUPERHERO"
done <&5                            #'&' describes that '5' is a FD, reads the file and redirects into while loop
echo "File Was Read On: `date`" >&5 #write date to the file
exec 5>&-                           #close '-' FD and close out all connections to it

Remove empty lines

cat blanks.txt | awk NF                   # [1]
cat blanks.txt | grep -v '^[[:blank:]]*$' # [2]
grep -e "[[:blank:]]" blanks
  • [1] The command awk NF is shorthand for awk 'NF != 0', or 'print those lines where the number of fields is not zero':
  • [2] The POSIX character class [:blank:] includes the whitespace and tab characters. The asterisk (*) after the class means 'any number of occurrences of that expression'. The initial caret (^) means that the character class begins the line. The dollar sign ($) means that any number of occurrences of the character class also finish the line, with no additional characters in between.

References

Here-doc | heredoc - cat redirection to create a file

Example of how to use heredoc and redirect STDOUT to create a file.

Heredocs redirect
Standard Literal Remove white-spaces
cat >file.txt <<EOF
cat << EOF > file.txt
wp.pl
bbc.co.uk
EOF
$ cat >file <<'EOF'
echo "$ABC=home_dir"
EOF

If you don't want ANY of the doc to be expanded or any special characters interpreted, quote the label with single quotes.

cat >file.txt <<-EOF
	wp.pl
	bbc.co.uk
	EOF

Change the here-doc operator to <<- . You can then indent both the here-doc and the delimiter with tabs. It must be tabs.

Explanation

  • <<EOF indicates everything following is to be treated as input until the string EOF is reached.
  • the cat command is used to print out a block of text
  • >hosts.txt redirects the output from cat to the file

echo | print on-screen

Printing the value of a variable. The \\n may be omitted if a following newline is not desired. The quotation marks are essential.

printf %s\\n "$var"
echo "$var"         # is NOT a valid substitute, NEVER use echo like this

Expansions - string manipulation

String manipulation | parameter substitution

# Strip the longest match ## from the beginning(hashbang) of the string matching pattern(`* `). It matches any number of characters and a <space> .
host="box-1.acme.cloud 22"; echo "${host##* }" # -> 22               # pattern `* `
host="https://google.com" ; echo "${host##*/}" # -> google.com       # pattern `*/`

# Strip the longest match %% from end(percentage) of the string, matching pattern `<space>`  and `*` anything (fileglob )
host="box-1.acme.cloud 22"; echo "${host%% *}" # -> box-1.acme.cloud # pattern ` *`
host="api-uat-1"          ; echo "${host%%-*}" # -> api              # pattern `-*`

# Strip the longest match (%%) from end of string (percentage), [0-9]* regex matching any digit
vpc=dev02;      echo "${vpc%%[0-9]*}"          # -> dev              # pattern `[0-9]*`

# Strip end of string '.aws.company.*'; file globing allowed or ".* " is a regEx unsure
host="box-1.prod1.midw.prod20.aws.acme.cloud"; echo ${host%.aws.acme.*}

Default value and parameter substitution

PASSWORD="${1:-data123}" #assign $1 arg if exists and is not zero length, otherwise expand to default value "data123" string
PASSWORD="${1:=data123}" #if $1 is not set or is empty, evaluate expression to "data123"
PASSWORD="${var:=$DEFAULT}"

Search and Replace

Expansion operators: / - substitutes 1st match // - substitutes all matches

# Example 1. Replace all '_' with '/' which is a common way turning a string into a path
STRING=secret_loc-a_branch1_branch2_key1
echo "path: ${STRING//_//}" 
path: secret/loc-a/branch1/branch2/key1

# Example 2. Replace 1st match '-' with '_'
STRING=gke-dev
echo ${STRING/-/_} # -> gke_dev

tr - translate

Replaces/translates one set of characters from STDIN to how are they look in STDOUT; eg: char classes are:

echo "Who is the standard text editor?" |tr [:lower:] [:upper:]
echo 'ed, of course!' |tr -d aeiou #-d deletes characters from a string
echo 'The ed utility is the standard text editor.' |tr -s astu ' ' # translates characters into spaces
echo ‘extra       tabs – 2’ | tr -s [:blank:] #supress white spaces
tr -d '\r' < dosfile.txt > unixfile.txt #remove carriage returns <CR>

sort

# sort TSV by second column
sort -t$'\t' -k2
# -t$'\t' tab as a column/key delimiter, note single quotes
# to insert literal TAB do Ctrl+v, then TAB


Skip 1st line to preserve a header.

ps aux | (sed -u 1q; sort)  | column -s, -t
# -u  -unbuffered 
# 1q  -print first line (header) and quit (leaving the rest of the input to sort)

The -u switch (unbuffered) is required for those seds (notably, GNU's) that would otherwise read the input in chunks, thereby consuming data that you want to go through sort instead.


ANSI-C Quoting

Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.

uniq

This command removes repeated lines. Therefore the file need to be sorted first.

cat file | uniq -u        # remove only reoccuring duplications
cat file | sort | uniq -u # display only uniq lines

Continue

https://www.ibm.com/developerworks/aix/library/au-unixtext/index.html Next colrm

paste

Paste can join 2 files side by side to provide a horizontal concatenation.

paste -d$';' -s
# -d delimiter to be used in the output. It'll replace \n or TAB
# -s serialization, if imput are files or a list of new lines, this will join all lines

Functions

Syntax:

  • the standard Bourne shell /bin/sh uses the function name followed immediately by a pair of parentheses () and curly brackets {} around the code itself
  • bash and ksh the function name is preceded by the keyword function function myFunc. The function keyword is optional unless the function name is also an alias. If the function keyword is used, parentheses are then optional.

Parameters are passed to functions in the same manner as to scripts using $1, $2, $3 positional parameters and using them inside the function. $@ returns the array of all positional parameters $1, $2, etc.

funcLastLoginInDays () {
  echo "$USERNAME - logged in $1 days ago"
  return 147
}
#eg. of calling a function with a parameter
funcLastLoginInDay $USERNAME; echo $? #last commands will show a return numeric code

Execution scope

Functions can be declared to run in the current shell using curly braces to scope the body.

my_func () {
	# Executes within the current environment
}


Whereas functions declared using brackets run in a new subshell.

my_func () (
	# Executes within a new shell
)

Forward declaration

Functions need to be defined before they are used. This is commonly achieved using the following pattern:

main() { foo; }           #main function code, includes other functions. Visually it's at the top of a script
foo () { echo "Step_1"; } #other functions block
main "$@"                 #execute main() function

Private functions

Functions that are private to a script should be named with a __ prefix and unset at the end of the script.

__my_func () {
}
unset -f __my_func

Bash variable scoping

  • Environment variable - defined in ENV of a shell
  • Script Global Scope variable - defined within a script
  • Function local variable - defined within a function and will become available/defined in Global Scope only after the function has been called

Variables global to shell and all subshells

Name the variable in upper case and using underscores.

export MY_VARIABLE

Variables local within a shell script

Variables declared with the local keyword are local to the declaring function and all functions called from that function.

child_function () {
        echo "$foo"
        local foo=2
        echo "$foo"
}
main () {
        local foo=1
        echo "$foo"
        child_function
        echo "$foo"
}
main

#### Actual output
# 1
# 1
# 2
# 1

Variables global within a shell script

Two mechanisms can be used to prevent variables declared within a script from leaking into the calling shell environment.

unset each variable at the end of the script
g_variable=something
    ...
    unset g_variable
Declare each variable from within a subshell scoped main
# Scope to subshell
    main () (
    	# Global to current subshell
    	g_variable=something
    )

unset all environment variables starting with a leading string

unset ${!prefix@}
expands to the names of variables whose names begin with prefix [...] When @ is used and the expansion appears within double quotes, each variable name expands to a separate word
unset $(compgen -v prefix)
expands to all variables matching the prefix
unset $(env | grep string |awk -F'=' '{print $1}')
search with string grep will fetch both environment and fetch the name of environment variable then pass to unset command

Environment scoping

Querying variables and functions in scope

# List all functions declared within the current environment
compgen -A function
# List all variables declared wtihin the current environment
compgen -v

getopt(s) - argument parameters parsers

getopt - a command line utility provided by the util-linux package
#TODO example


getopts - Bash buildin, does not support long options, uses the $OPTARG variable for options that have arguments
# parse parameters into the script
while getopts ":u:h" OPTIONS; do
  case "${OPTIONS}" in
    u) URL=${OPTARG} ;;
    h) HELP="true"
  esac
done

# :  - first colon has special meaning
# u: - parameter u requires argument value to be passed on because ':' follows 'u'
# h  - parameter h does not require a value to be passed on
References

set options

set -o noclobber # > prevents overriding a file if already exists
set +o noclobber # enable clobber back on

short version of checking command line parameters

Checking 3 positional parameters. Run with: ./check one

#!/bin/bash
: ${3?"Usage: $1 ARGUMENT $2 ARGUMENT $3 ARGUMENT"}

IFS and delimiting

$IFS Internal Field Separator is an environment variable defining default delimiter. It can be changed at any time in a global scope of within subshell by assigning a new value, eg:

echo $IFS # displays the current delimiter, by default it's 'space'
$IFS=','  #changes delimiter into ',' coma

Counters

These counters do not require invocation, they always start from 1.

#!/bin/bash
COUNTER=0   #defined only to satisfy while test function; otherwise exception [: -lt: unary operator expected
while [ $COUNTER -lt 5 ]; do
  (( COUNTER++ ))
  # COUNTER=$((COUNTER+1))
  # let COUNTER++ 
  echo $COUNTER
done

Loops

Bash shell - for i loop

for i in {1..10};    do echo $i; done
for i in {1..10..2}; do echo $i; done #step function {start..end..increment} will print 1 3 5 7 9
for i in 1 2 3 4 5 6 7 8 9 10; do echo $i; done
for i in $(seq 1 10); do echo $i; done
for ((i=1; i <= 10 ; i++)); do echo $i; done
for host in $(cat hosts.txt); do ssh "$host" "$command" >"output.$host"; done
   a01.prod.com
   a02.prod.com
for host in $(cat hosts.txt); do ssh "$host" 'hostname; grep Certificate /etc/httpd/conf.d/*; echo -e "\n"'; done

Dash shell - for i loop

#!/bin/dash
wso2List="esb dss am mb das greg"  #by default white space separates items in the list structures
for i in $wso2List; do
  echo "Product: $i"
done

Bash shell - while loop

while true; do tail /etc/passwd; sleep 2; clear; done


Read a file or variable line by line
Read a file Read a variable
FILE="hostlist.txt" #a list of hostnames, 1 host per line
# optional: IFS= -r 
while IFS= read -r LINE; do 
  (( COUNTER++ ))
  echo "Line: $COUNTER - Host_name: $LINE"
done < "$FILE" #redirect the file content into loop
PLACES='Warsaw
London'
while read LINE; do
  (( COUNTER++ ))
  echo "Line: $COUNTER - Line_content: $LINE"
  done <<< "$PLACES" #inject multi-line string variable, ref. heredoc
  • -r option passed to read command prevents backslash escapes from being interpreted
  • IFS= option before read command to prevent leading/trailing whitespace from being trimmed -


Loop until counter expression is true

while [ $COUNT -le $DISPLAYNUMBER ]; do
    echo "Hello World - $COUNT"
    COUNT="$(expr $COUNT + 1)"
done

Arrays

Simple array

hostlist=("am-mgr-1" "am-wkr-1" "am-wkr-2" "esb-mgr-1") #array that is actually just a list data type
for INDEX in ${hostlist[@]}; do    #@ expands to every element in the list
  printf "${hostlist[INDEX]}\n"    #* expands to match any string, so it matches every element as well
done


  • arr=() Create an empty array
  • arr=(1 2 3) Initialize array
  • ${arr[2]} Retrieve third element
  • ${arr[@]} Retrieve all elements
  • ${!arr[@]} Retrieve array indices
  • ${#arr[@]} Calculate array size
  • arr[0]=3 Overwrite 1st element
  • arr+=(4) Append value(s)
  • str=$(ls) Save ls output as a string
  • arr=( $(ls) ) Save ls output as an array of files
  • ${arr[@]:s:n} Retrieve n elements starting at index s
  • echo ${array[@]/*[aA]*/} Remove strings[items] that contain 'a' or 'A'


Display array all items

#!/bin/bash
array=($(cat -))
echo "${array[@]}" # display array without '\n' because STDIN is line by line

# Test
$(for i in {1..5}; do echo $i; done) | ./script.sh


Display specific indexes

echo "${array[@]:3:5}"

Associative arrays, key=value pairs

#!/bin/bash 
ec2type () {  #function declaration
  declare -A array=(  #associative array declaration, bash 4
    ["c1.medium"]="1.7 GiB,2 vCPUs" #double quotation is required
    ["c1.xlarge"]="7.0 GiB,8 vCPUs"
    ["c3.2xlarge"]="15.0 GiB,8 vCPUs"
  )
  echo -e "${array[$1]:-NA,NA}"  #lookups for a value $1 (eg. c1.medium) in the array, 
}                                #if nothing found (:-) expands to default "NA,NA" string
ec2type $1 #invoke the function passing 1st argument received from a command line

### Usage
$ ./ec2types.sh c1.medium
1.7 GiB,2 vCPUs


Convert a string into array, remember to leave the variable unquoted. This will fail if $string has globbing characters in. It can be avoided by set -f && array=($string) && set +f. The array items will be delimited by a space character.

array=($string)
read -a array <<< $string
IFS=' ' read -a array <<< "$string" #avoid fileglob expansions for *,? and [], variable is double-quoted

if statement

if [ "$VALUE" -eq 1 ] || [ "$VALUE" -eq 5 ] && [ "$VALUE" -gt 4 ]; then
   echo "Hello True"
 elif [ "$VALUE" -eg 7 ] 2>/dev/null; then
   echo "Hello 7"
 else
   echo "False" 
 fi

case statement

case $MENUCHOICE in
  1)
    echo "Good choice!"
    ;; #end of that case statements so it does not loop infinite
  2)
    echo "Better choice"
    ;;
  *)
    echo "Help: wrong choice";;
esac

traps

First argument is a command to execute, followed by events to be trapped

trap 'echo "Press Q to exit"' SIGINT SIGTERM
trap 'funcMyExit' EXIT           #run a function on script exit
trap "{ rm -f $LOCKFILE; }" EXIT #delete a file on exit


#!/bin/bash
trap ctrl_c INT # trap ctrl-c and call ctrl_c()

function ctrl_c() {
  echo "** Trapped CTRL-C **"
}

debug mode

You can enable debugging mode anywhere in your scrip and disable many times

set -x  #starts debug
echo "Command to debug"
set +x  #stops debug


Or debug in sub-shell by running with -x option eg.

$ bash -x ./script.sh

Additionally you can disable globbing with -f

Execute a script line by line
set -x
trap read debug
< YOUR CODE HERE >
Execute a script line by line (improved version)

If your script is reading content from files, the above listed will not work. A workaround could look like the following example.

#!/usr/bin/env bash
echo "Press CTRL+C to proceed."
trap "pkill -f 'sleep 1h'" INT
trap "set +x ; sleep 1h ; set -x" DEBUG

< YOUR CODE HERE >


Install Bash debugger

apt-get install bashdb
bashdb your_command.sh #type step, and then hit carriage return after that

References

errors

syntax error: unexpected end of file

This often can be caused by CRLF line terminators. Just run dos2unix script.sh.


Another thing to check:

  • terminate bodies of single-line functions with semicolon

I.e. this innocent-looking snippet will cause the same error:

die () { test -n "$@" && echo "$@"; exit 1 }

To make the dumb parser happy:

die () { test -n "$@" && echo "$@"; exit 1; }


# Applies also to below, lack of last ; will produce the error
[[ "$#" == 1 ]] && [[ "$arg" == [1,2,3,4] ]] && printf "%s\n" "blah" || { printf "%s\n" "blahblah"; usage; }
if [ -f ~/.git-completion.bash ]; then . ~/.git-completion.bash; fi;

Create a big file

Use dd tool that will create a file of size count*bs bytes

  • 1024 bytes * 1024 count = 1048576 Bytes = 1Mb
  • 1024 bytes * 10240 count = 10Mb
  • 1024 bytes * 102400 count = 100Mb
  • 1024 bytes * 1024000 count = 1G
dd if=/dev/zero    bs=1024  count=10240 of=/tmp/10mb.zero #creates 10MB zero'd
dd if=/dev/urandom bs=1048576 count=100 of=/tmp/100mb.bin #creates 100MB random file
dd if=/dev/urandom bs=1048576 count=1000 of=1G.bin status=progress

dd in GNU Coreutils 8.24+ (Ubuntu 16.04 and newer) got a new status option to display the progress:

dd if=/dev/zero bs=1048576 count=1000 of=1G.bin status=progress
1004535808 bytes (1.0 GB, 958 MiB) copied, 6.01483 s, 167 MB/s

1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 15.4451 s, 67.9 MB/s

Colours

# Option 1
echo_b(){ echo -e "\e[1;34m${@}\e[0m"; }
echo_g(){ echo -e "\e[1;32m${@}\e[0m"; }
echo_p(){ echo -e "\e[1;35m${@}\e[0m"; }
echo_r(){ echo -e "\e[1;31m${@}\e[0m"; }
echo_b " -h Print this message"

# Option 2
         red="\e[31m"
    red_bold="\e[1;31m" # '1' makes it bold
   blue_bold="\e[1;34m"
  green_bold="\e[1;32m"
light_yellow="\e[93m"
       reset="\e[0m"
echo -e "${green_bold}Hello world!${reset}"

# If you wish to unset, add the common prefix to above variables eg 'c_', then:
unset ${!c_@}

# Option 3
red=$'\e[1;31m' # '1' makes it bold
grn=$'\e[1;32m'
yel=$'\e[1;33m'
blu=$'\e[1;34m'
mag=$'\e[1;35m'
cyn=$'\e[1;36m'
end=$'\e[0m'
printf "%s\n" "Text in ${red}red${end}, white and ${blu}blue${end}.


Colours function using a terminal settings rather than ANSI color control sequences

if test -t 1; then # if terminal
    ncolors=$(which tput > /dev/null && tput colors) # supports color
    if test -n "$ncolors" && test $ncolors -ge 8; then
        termcols=$(tput cols)
        bold="$(     tput bold)"
        underline="$(tput smul)"
        standout="$( tput smso)"
        normal="$(   tput sgr0)"    # reset to default font
        black="$(    tput setaf 0)"
        red="$(      tput setaf 1)"
        green="$(    tput setaf 2)"
        yellow="$(   tput setaf 3)"
        blue="$(     tput setaf 4)"
        magenta="$(  tput setaf 5)"
        cyan="$(     tput setaf 6)"
        white="$(    tput setaf 7)"
    fi
fi

How to use it's the same as using ANSI color control sequences. Use ${colorName}Text${normal}

print_bold() {
    title="$1"
    text="$2"
    echo "${red}================================${normal}"
    echo -e "  ${bold}${yellow}${title}${normal}"
    echo -en "  ${text}"
    echo "${red}================================${normal}"
}

References

Run scripts from website

The script available via http/s as usual needs begin with shebang and can be executed in fly without a need to download to a file system using cURL or wget.

curl -sL  https://raw.githubusercontent.com/pio2pio/project/master/setup_sh | sudo -E bash -
wget -qO- https://raw.githubusercontent.com/pio2pio/project/master/setup_sh | sudo -E bash -

Run local scripts remotly

Use the -s option, which forces bash (or any POSIX-compatible shell) to read its command from standard input, rather than from a file named by the first positional argument. All arguments are treated as parameters to the script instead. If you wish to use option parameters like eg. --silent true make sure you put -- before arg so it is interpreted as an argument to test.sh instead of bash.

ssh user@remote-addr 'bash -s arg' < test.sh
ssh user@remote-addr bash -sx -- -p am -c /tmp/conf < ./support/scripts/sanitizer.sh
# -- signals the end of options and disables further option processing. 
#    any arguments after the -- are treated as filenames and arguments
# -p am -c /tmp/conf -these are arguments passed onto the sanitizer.sh script

References

Scripts - solutions

Check connectivity

#!/bin/bash 

red="\e[31m"; green="\e[32m"; blue="\e[34m"; light_yellow="\e[93m"; reset="\e[0m"

if [ "$1" != "onprem" ]; then
    echo -e "${blue}Connectivity test to servers: on-prem${reset}"
    declare -a hosts=( 
    wp.pl
    rzeczpospolita.pl
    )
else
    echo -e "${blue}Connectivity test to Polish news servers: polish-news${reset}"
    declare -a hosts=( 
    wp.pl
    rzeczpospolita.pl
    )
fi

#################################################

echo -e "From server: ${blue}$(hostname -f) $(hostname -I)${reset}"

for ((i = 0; i < ${#hosts[@]}; ++i)); do
  RET=$(timeout 3 nc -z ${hosts[$i]} 22 2>&1) \
  && echo -e "[OK] $i ${hosts[$i]}" \
  || echo -e "${red}[ER] $i ${hosts[$i]}\t[ERROR: \
  $(if [ $? -eq 124 ]; then echo "timeout maybe filtered traffic"; else \
  if [ -z "$RET" ]; then echo "unknown"; else echo -e "$RET"; fi; fi)]${reset}"
  sleep 0.3
done

# Working as well
#for host in "${hosts[@]}"; do
#  RET=$(timeout 3 nc -z ${host} 22 2>&1) && echo -e "[OK] ${host}" || echo -e "${red}[ER] ${host}\t[ERROR: \
#  $(if [ $? -eq 124 ]; then echo "timeout maybe filtered traffic"; else \
#  if [ -z "$RET" ]; then echo "unknown"; else echo -e "$RET"; fi; fi)]${reset}"
#  sleep 0.3
#done

LOCAL UBTUNU MIRROR

rsync -a  --progress rysnc://archive.ubuntu.com/ubuntu /opt/mirror/ubuntu - command to create local mirror.
ls /opt/mirror/ubuntu - shows all files

Bash

Delete key gives ~ ? Add the following line to your $HOME/.inputrc (might not work if added to /etc/inputrc )

"\e[3~": delete-char

Process substitution

Process substitution expands output of commands to a file. Depends on OS default /.../fd descriptor or temporary file is used to store the content. But the syntax works without using any file names thus looks simpler. There are also situations that a command requires a file and | pipe cannot be used.

source <(kubectl completion bash)

The example sources a file but syntaticly there is no file when using the process substitution construction.

References