Linux shell/Commands

From Ever changing code
Jump to navigation Jump to search

One liners

df -h   #displays filesystem disk space usage for all mounted partitions
du -skh * | sort -n #displays the disk usage summary for each directory 
free -m   #displays the amount of free and used memory in the system
lsb_release -a #prints version information for the Linux release you're running
tload -draws   #system load on text based graph

Copy with progress bar

rsync and cp
  • rsync -aP - copy with progress can be also aliased alias cp='rsync -aP'
  • cp -rv old-directory new-directory - shows progress bar
PV does not preserve permissions and does not handle attributes
  • pv ~/kali.iso | cat - /media/usb/kali.iso equals cp ~/kali.iso /media/usb/kali.iso
  • pv ~/kali.iso > /media/usb/kali.iso equals cp ~/kali.iso /media/usb/kali.iso
  • pv access.log | gzip > access.log.gz shows gzip compressing progress.

PV can be imagined as CAT command piping '|' output to another command with a bar progress and ETA times. -c makes sure one pv output is not use to write over to another, -N creates a named stream. Find more at How to use PV pipe viewer to add progress bar to cp, tar, etc.. 

$ pv -cN source access.log | gzip | pv -cN gzip > access.log.gz
source:  760MB 0:00:15 [37.4MB/s] [=>     ] 19% ETA 0:01:02
gzip: 34.5MB 0:00:15 [1.74MB/s] [  <=>  ]

Tail log files

tail-f-the-output-of-dmesg or install multitail

  • tail -f /var/log/{messages,kernel,dmesg,syslog} - old school but not perfect
  • watch 'dmesg | tail -50' - approved by man dmesg
  • watch 'sudo dmesg -c >> /tmp/dmesg.log; tail -n 40 /tmp/dmesg.log' - tested, but experimental

Replace unix timestamps in logs to human readable date format

user@laptop:/var/log$ tail dmesg | perl -pe 's/(\d+)/localtime($1)/e'
[   Thu Jan  1 01:00:29 1970.168088] b43-phy0: Radio hardware status changed to DISABLED
[   Thu Jan  1 01:00:29 1970.308597] tg3 0000:09:00.0: irq 44 for MSI/MSI-X
[   Thu Jan  1 01:00:29 1970.344378] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   Thu Jan  1 01:00:29 1970.344745] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
 
user@laptop:/var/log$ tail dmesg
[   29.168088] b43-phy0: Radio hardware status changed to DISABLED
[   29.308597] tg3 0000:09:00.0: irq 44 for MSI/MSI-X
[   29.344378] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   29.344745] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
References

Sed - Stream Editor - replace, substitute

sed -i 's/ugly/beautiful/g' ~/sue.txt  #substitutes ugly with beautiful in whole file

at, atd - schedule a job

At can execute command at given time. It's important to remember 'at' can only take one line and by default it uses limited /bin/sh shell. 

service atd status #check if at demon is running
at 01:05 AM
atq -list job queue
at -c <job number> -cat the job_number
atrm <job_number> -deletes job
mail -at command emails a user who scheduled a job with its output

Useful packages

  • ARandR Screen Layout Editor - 0.1.7.1

Add user to a group

In ubuntu adding a user to group admin will grant the root privileges. Adding them to sudo group will allow to execute any command 

sudo usermod -aG nameofgroup nameofuser #requires to login again

Add user to sudo

sudo usermod -aG sudo nameofuser  #enough for Ubuntu, logout required

Edit safely /etc/sudoers file 

sudo visudo

User rules fields explained 

        The first ALL is the users allowed
         |    The second one is the hosts; on all hosts (if you distribute the same sudoers file to many computers)
         |     |
   piotr ALL=(ALL:ALL) ALL
                  /     |
                 /   The last one is the commands allowed
         The third one is the user as you are running the command

In examples below names beginning with a "%" indicate group names in /etc/group 

root  ALL=(ALL)    ALL              #standard root entry
piotr ALL=(ALL)    NOPASSWD: ALL    #user can run as root without password

piotr ALL=         NOPASSWD: ALL      #piotr will not be prompted for password, just another format to above
piotr ALL= NOPASSWD: /usr/bin/service #piotr will not be prompt for password while running 'service' command
%wheel ALL=(ALL)   NOPASSWD: ALL      #members of 'wheel' group can run without a password

If you find a number of entries applied to your user, the last entry takes precedence 

sudo -l #list all of the rules in the /etc/sudoers that apply to your user
sudo -k #clear the timer

Show USB devices

lsusb -t     #shows USB tree

Copy and Paste in terminal

In Linux X graphical interface this works different then in Windows you can read more in X Selections, Cut Buffers, and Kill Rings. When you select some text this becomes the Primary selection (not the Clipboard selection) then Primary selection can be pasted using the middle mouse button. Note however that if you close the application offering the selection, in your case the terminal, the selection is essentially "lost".

Option 1 works in X

  • select text to copy then use your mouse middle button or press a wheel

Option 2 works in Gnome Terminal

  • Ctrl+Shift+C - copy
  • Ctrl+Shift+V or Shift+Insert - paste

Option 3 Install Parcellite GTK+ clipboard manager 

sudo apt-get install parcellite

then in the settings check "use primary" and "synchronize clipboards"

Generate random password

cat /dev/urandom|tr -dc "a-zA-Z0-9"|fold -w 48|head -n1

Localization

Change a keyboard layout
setxkbmap gb

at, atd - schedule a job

At can execute command at given time. It's important to remember 'at' can only take one line and by default it uses limited /bin/sh shell. 

service atd status #check if at demon is running
at 01:05 AM
atq -list job queue
at -c <job number> -cat the job_number
atrm <job_number> -deletes job
mail -at command emails a user who scheduled a job with its output

BASH For Loop Sequence

for I in {1..10}; do echo $I; done
for I in 1 2 3 4 5 6 7 8 9 10; do echo $I; done
for I in $(seq 1 10); do echo $I; done
for ((I=1; I <= 10 ; I++)); do echo $I; done

for host in $(cat hosts.txt); do ssh "$host" "$command" >"output.$host"; done
   a01.prod.com
   a02.prod.com
   a03.prod.com
   a04.prod.com
for host in $(cat hosts.txt); do ssh "$host" 'hostname; grep Certificate /etc/httpd/conf.d/*; echo ====='; done

BASH While Loop

 while true; do tail /etc/passwd; sleep 2; clear; done

LOCAL UBTUNU MIRROR

rsync -a  --progress rysnc://archive.ubuntu.com/ubuntu /opt/mirror/ubuntu - command to create local mirror.
ls /opt/mirror/ubuntu - shows all files

NETWORK INSTALL VIA MIRROR

192.168.1.240 (Local web server) /ubuntu/ (Local Folder) - enter mirror manually

OR Online mirrors:

select 'enter mirror manually' http://mirrors.sonic.net/centos/6/os/i386 - etc

KICKSTART FILES

sudo apt-get install system-config-kickstart - installs kickstart config generator
system-config-kickstart - start application

Go through all the menu options on the left hand side and select right language and from local mirror, network devices etc 

FILE - SAVE FILE - DESKTOP- ks.cfg (Save File) - double click it and can edit text file.

ADD PACKAGES TO FILE

# PACKAGE INFORMATION
@ ubuntu-desktop - will install ubtunu desktop, openssh etc
openssh-server
vim
nfs-client

INITIAL SETUP CONSIDERATIONS

cd /etc/apt - NAVIGATE TO SEE APT FILES
cat sources.list or less sources.list - SHOWS ALL REPOSITORIES (REMOVE HASH TO ENABLE)
Install
  • vim - alternative to vi.
  • openssh-server - enable ssh
  • screen - able to use multiple screens
  • postfix - allows for email notifications if server fails etc
  • mailutils - able to mail things directly from command line
  • sysstat - to help with troubleshooting
  • iotop - top for input/output

Boot INIT Sequence

Process

  1. Bios ensures everything is intact - peripherals etc.
  2. 1st bootable disk GRUB responsible for setting up environment INITRD (Ensures kernel friends root file system) - kernel etc.
  3. Load INIT first process by kernel - sets default run level of 2 unless specified.
  4. INIT process - loads appropriate daemons from default run level.

7 run-levels

0 - halt
1 - single
2 - multi user - default
3 - multi user
4 - multi user
5 - multi user
6 - reboot

Use init + number to enter different run level 

cd /boot/ - should see grub directory.
cd /etc/ - ls-l rc - shows different run levels
cd /etc/init.d - contains daemons, service scripts that are referenced from run level.

Partitions / Raid / LVM etc

Disk Commands

  • df -h - shows disk space
  • sudo fdisk -l - shows hard drive partitions
  • ls -l /dev/sd* - SHOWS ALL DRIVES
  • cat /proc/mdstat - SHOWS LINUX RAID DRIVES IN USE
  • pvdisplay - SHOWS PHYSICAL VOLUMES
  • lvdisplay - SHOW LOGICAL VOLUM DISPLAY

Provisioning Filesystems (Extra Storage)

Provision storage while sudoserver is online.

  • fdisk -l - reveals connected disks and partitions
  • df -h - shows amount of memory used
  • /dev/sdb - unpartitioned
  • mklabel - type MSDOs - if needed.
  • sudo parted - partition tool
  • select /dev/sdb - selects disk -
  • mkpart primary 1 10GB - 10GB partition
  • print - shows disks in parted
  • quit - to leave parted
  • mke2fs - overlay filesystem on new partition.
  • mke2fs -t ext4 -j /dev/sdb1 - creates file system.

or

  • sudo mkfs.ext4 -j /dev/sdb1/ - same as above.
  • mount /dev/sdb1 /projectx/10gb. - create mount point
  • mount - shows all system mounts.
  • dd if=/dev/zero bs=1024 count-10240 of=/projectx/10GB/test.file.1 - creates 10MB in new mount
  • sudo blkid - shows partition uuid for stab.
  • sudo nano /etc/fstab - edits stab. UUID="number" /projectx/10GB ext4 defaults - stores in fstab.

Provision SWAP storage on demand

Ability for kernel to extend RAM via disk.

  • free -m - determines current stare of storage.
  • top - also shows SWAP info.
  • sudo fdisk -l - to identify partition space
  • parted /dev/sdb - places in context of /dev/sdv/
  • print - to show partition table.
  • mkpart primary linux-swap 10GB 12GB starting from the 10GB first block moving up.
  • set 2 swap on - turns partitions to swap and on.
  • sudo fdisk -l /dev/sdb - confirms swap allocation.
  • sudo mkswap /dev/sdb2 - overlays SWAP filesystem and displays UUID (FSTAB)
  • sudo blkid - shows all UUID's
  • sudo nano /etc/fstab - opens stab for editing for SWAP reference using UUID.
  • UUID "" NONE swap sw 0 0 - for nano file.
  • swap on -s - displays current swap situation
  • sudo swap on -a - turns on swap storage.
  • free -m || top || swapon -s - to confirm configuration.

Option SWAP creation which is file based

  • dd if=/dev/zero of=/projectx/10GB/swapfile3GB count=3G bs=1024 - creates dummy swap file.
  • sudo mkswap /projectx/10GB/swapfile3GB - overlays SWAP file system.
  • sudo nano /etc/fstab - opens stab for editing for SWAP reference using path /projectx/10GB/swapfile3GB
  • sudo swapon -a - tuns on all swap storage.

Storage Management LVM (Logical Volume Management)

Volume sets based on various disparate storage technologies.

Common configuration - raid hardware (redundancy) / LVM overlaying RAID config (aggregation)

Ability to extend, reduce, manipulate storage on demand.

LVM storage hierarchy:

Volume Group (Consists of 1 or more physical volumes)

  • Logical Volume(s)
  • File System(s)

6 steps to LVM setup

Appropirate 1 or more LVM partitions

  1. sudo parted /dev/sdb
  2. mkpart extended 13GB 20GB
  3. print
  4. mkpart logical lvm 13GB 20GB
    1. select /dev/sdc/
  5. mklabel msdos
  6. mkpart primary 1GB 20GB
    1. set 1 lvm on
    2. print
  7. 20GB on SDC and 7GB on SDB

Partition

  • sudo pvcreate /dev/sdb5 /dev/sdc1 - allocates LVM partitions as physical volumes
  • sudo pvdisplay - shows LVM physical volumes.
  • sudo vgcreate volgroup001 /dev/sdb5 /dev/sdc1 - aggregates volumes to volume group
  • sudo lvcreate -L 200GB volgroup001 -n logvol001 - creates logical volumes
  • sudo lvdisplay - shows logical volumes
  • sudo mk3fs -t ext -j /dev/volgroup001/logvol001 - overlays EXT4 filesytem.
  • Mount filesystem and commit changes to fstab

LVM related tasks

  1. sudo lvrename volgroup001 logvol001 volgroup002 - renames volume group. (Remember to edit fstab file or unmount)
  2. sudo lvresize -L 25GB /dev/volgroup001/logvolvar - Resize logical volume by 5GB from 20GB
  3. sudo resize2fs /dev/mapper/volgroup001-logvolvar 25G - Resize filesystem after increasing memory.
  4. sudo lvremove /dev/volgroup001/logvolvar - Remove volume completely. (Umount first)
  5. sudo parted /dev/sdc - Add or assign more partitions to volume group LVM.
  6. print
  7. mkpart primary 20GB 25GB
  8. print
  9. set 2 lvm on
  10. print
  11. sudo pvcreate /dev/sdc2
  12. sudo pvdisplay
  13. sudo vgextend volgroup001 /dev/sdc2 - add new PV to volume group

General Admin

  • cat /etc/*rele* - shows version
  • runlevel - shows run level
  • ifconfig - shows network configuration
  • exit - to log off servers
  • sudo shutdown -h now - shutdowns server straight away
  • top - show programs running
  • ps -ef - show processes
  • uptime - shows uptime
  • ps -ef | grep -a spa - Shows if apache is running
  • cat /etc/resolv.conft - shows DNS resolving

Sudo

  • Elevated privileges for non privileged users.
  • Admin group has sudo privileges.
  • All actions logged via authpriv or /var/log/auth.log
  • /etc/sudousers - default policy file.
  • grep adm /etc/group shows admin users.
  • grep sudo /etc/group shows admin users.
  • sudo -l - lists commands current user may run
  • sudo tail /var/log/auth.log - shows authorisation log
  • sudo ls -l ~dean - shows someone else's directory
  • sudo -k flushes cached credentials - forces password
  • sudo passwd root - changes root password

Bash

Delete key gives ~ ? Add the following line to your $HOME/.inputrc (might not work if added to /etc/inputrc ) 

"\e[3~": delete-char
Retrieved from ‘http://wiki.ciscolinux.co.uk/index.php?title=Linux_shell/Commands&oldid=1935’