Difference between revisions of "Linux shell/Commands"
Line 54: | Line 54: | ||
*[https://alexlurthu.wordpress.com/2007/07/22/transforming-unix-timestamps-in-logs-to-date/ Transforming Unix Timestamps in logs to date] | *[https://alexlurthu.wordpress.com/2007/07/22/transforming-unix-timestamps-in-logs-to-date/ Transforming Unix Timestamps in logs to date] | ||
= Sed - Stream Editor | = Sed - Stream Editor = | ||
Replace, substitute | |||
sed -i 's/ugly/beautiful/g' ~/sue.txt #substitutes ''ugly'' with ''beautiful'' in whole file | sed -i 's/ugly/beautiful/g' ~/sue.txt #substitutes ''ugly'' with ''beautiful'' in whole file | ||
sed 's/A//g' #substitutes 'A' with an empty string '/g' - globally; all it means it will remove 'A' from whole string | |||
= Useful packages = | = Useful packages = |
Revision as of 18:29, 11 April 2016
One liners
df -h #displays filesystem disk space usage for all mounted partitions du -skh * | sort -n #displays the disk usage summary for each directory free -m #displays the amount of free and used memory in the system lsb_release -a #prints version information for the Linux release you're running tload -draws #system load on text based graph
Copy with progress bar
- rsync and cp
- rsync -aP - copy with progress can be also aliased alias cp='rsync -aP'
- cp -rv old-directory new-directory - shows progress bar
- PV does not preserve permissions and does not handle attributes
- pv ~/kali.iso | cat - /media/usb/kali.iso equals cp ~/kali.iso /media/usb/kali.iso
- pv ~/kali.iso > /media/usb/kali.iso equals cp ~/kali.iso /media/usb/kali.iso
- pv access.log | gzip > access.log.gz shows gzip compressing progress.
PV can be imagined as CAT command piping '|' output to another command with a bar progress and ETA times. -c makes sure one pv output is not use to write over to another, -N creates a named stream. Find more at How to use PV pipe viewer to add progress bar to cp, tar, etc..
$ pv -cN source access.log | gzip | pv -cN gzip > access.log.gz source: 760MB 0:00:15 [37.4MB/s] [=> ] 19% ETA 0:01:02 gzip: 34.5MB 0:00:15 [1.74MB/s] [ <=> ]
Listing a directory in the form of a tree
$ alias lst='ls -R | grep ":$" | sed -e '"'"'s/:$//'"'"' -e '"'"'s/[^-][^\/]*\//--/g'"'"' -e '"'"'s/^/ /'"'"' -e '"'"'s/-/|/'"'" $ ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\// /g' -e 's/^/ /' #using spaces, doesn't list .git
Tail log files
tail-f-the-output-of-dmesg or install multitail
tail -f /var/log/{messages,kernel,dmesg,syslog}
- old school but not perfectwatch 'dmesg | tail -50'
- approved by man dmesgwatch 'sudo dmesg -c >> /tmp/dmesg.log; tail -n 40 /tmp/dmesg.log'
- tested, but experimental
Replace unix timestamps in logs to human readable date format
user@laptop:/var/log$ tail dmesg | perl -pe 's/(\d+)/localtime($1)/e' [ Thu Jan 1 01:00:29 1970.168088] b43-phy0: Radio hardware status changed to DISABLED [ Thu Jan 1 01:00:29 1970.308597] tg3 0000:09:00.0: irq 44 for MSI/MSI-X [ Thu Jan 1 01:00:29 1970.344378] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ Thu Jan 1 01:00:29 1970.344745] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready user@laptop:/var/log$ tail dmesg [ 29.168088] b43-phy0: Radio hardware status changed to DISABLED [ 29.308597] tg3 0000:09:00.0: irq 44 for MSI/MSI-X [ 29.344378] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 29.344745] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
- References
Sed - Stream Editor
Replace, substitute
sed -i 's/ugly/beautiful/g' ~/sue.txt #substitutes ugly with beautiful in whole file sed 's/A//g' #substitutes 'A' with an empty string '/g' - globally; all it means it will remove 'A' from whole string
Useful packages
- ARandR Screen Layout Editor - 0.1.7.1
Add user to a group
In ubuntu adding a user to group admin will grant the root privileges. Adding them to sudo group will allow to execute any command
sudo usermod -aG nameofgroup nameofuser #requires to login again
Sudo - grant a user with root privilages
sudo usermod -aG sudo nameofuser #enough for Ubuntu, logout required sudo passwd root #sets root password, so it can be used to login
Edit safely /etc/sudoers file
sudo visudo
User rules fields explained
The first ALL is the users allowed | The second one is the hosts; on all hosts (if you distribute the same sudoers file to many computers) | | piotr ALL=(ALL:ALL) ALL / | / The last one is the commands allowed The third one is the user as you are running the command
In examples below names beginning with a "%" indicate group names in /etc/group
root ALL=(ALL) ALL #standard root entry piotr ALL=(ALL) NOPASSWD: ALL #user can run as root without password piotr ALL= NOPASSWD: ALL #piotr will not be prompted for password, just another format to above piotr ALL= NOPASSWD: /usr/bin/service #piotr will not be prompt for password while running 'service' command %wheel ALL=(ALL) NOPASSWD: ALL #members of 'wheel' group can run without a password
If you find a number of entries applied to your user, the last entry takes precedence
sudo -l #list all of the rules in the /etc/sudoers that apply to your user sudo -k #clear the timer
Show USB devices
lsusb -t #shows USB tree
Copy and Paste in terminal
In Linux X graphical interface this works different then in Windows you can read more in X Selections, Cut Buffers, and Kill Rings. When you select some text this becomes the Primary selection (not the Clipboard selection) then Primary selection can be pasted using the middle mouse button. Note however that if you close the application offering the selection, in your case the terminal, the selection is essentially "lost".
Option 1 works in X
- select text to copy then use your mouse middle button or press a wheel
Option 2 works in Gnome Terminal
- Ctrl+Shift+C - copy
- Ctrl+Shift+V or Shift+Insert - paste
Option 3 Install Parcellite GTK+ clipboard manager
sudo apt-get install parcellite
then in the settings check "use primary" and "synchronize clipboards"
Generate random password
cat /dev/urandom|tr -dc "a-zA-Z0-9"|fold -w 48|head -n1
Localization
- Change a keyboard layout
setxkbmap gb
at, atd - schedule a job
At can execute command at given time. It's important to remember 'at' can only take one line and by default it uses limited /bin/sh shell.
service atd status #check if at demon is running at 01:05 AM atq -list job queue at -c <job number> -cat the job_number atrm <job_number> -deletes job mail -at command emails a user who scheduled a job with its output
BASH For Loop Sequence
for I in {1..10}; do echo $I; done for I in 1 2 3 4 5 6 7 8 9 10; do echo $I; done for I in $(seq 1 10); do echo $I; done for ((I=1; I <= 10 ; I++)); do echo $I; done
for host in $(cat hosts.txt); do ssh "$host" "$command" >"output.$host"; done a01.prod.com a02.prod.com a03.prod.com a04.prod.com for host in $(cat hosts.txt); do ssh "$host" 'hostname; grep Certificate /etc/httpd/conf.d/*; echo ====='; done
BASH While Loop
while true; do tail /etc/passwd; sleep 2; clear; done
LOCAL UBTUNU MIRROR
rsync -a --progress rysnc://archive.ubuntu.com/ubuntu /opt/mirror/ubuntu - command to create local mirror. ls /opt/mirror/ubuntu - shows all files
NETWORK INSTALL VIA MIRROR
192.168.1.240 (Local web server) /ubuntu/ (Local Folder) - enter mirror manually
OR Online mirrors:
select 'enter mirror manually' http://mirrors.sonic.net/centos/6/os/i386 - etc
KICKSTART FILES
sudo apt-get install system-config-kickstart - installs kickstart config generator system-config-kickstart - start application
Go through all the menu options on the left hand side and select right language and from local mirror, network devices etc
FILE - SAVE FILE - DESKTOP- ks.cfg (Save File) - double click it and can edit text file.
ADD PACKAGES TO FILE
# PACKAGE INFORMATION @ ubuntu-desktop - will install ubtunu desktop, openssh etc openssh-server vim nfs-client
INITIAL SETUP CONSIDERATIONS
cd /etc/apt - NAVIGATE TO SEE APT FILES cat sources.list or less sources.list - SHOWS ALL REPOSITORIES (REMOVE HASH TO ENABLE)
- Install
- vim - alternative to vi.
- openssh-server - enable ssh
- screen - able to use multiple screens
- postfix - allows for email notifications if server fails etc
- mailutils - able to mail things directly from command line
- sysstat - to help with troubleshooting
- iotop - top for input/output
Boot INIT Sequence
Process
- Bios ensures everything is intact - peripherals etc.
- 1st bootable disk GRUB responsible for setting up environment INITRD (Ensures kernel friends root file system) - kernel etc.
- Load INIT first process by kernel - sets default run level of 2 unless specified.
- INIT process - loads appropriate daemons from default run level.
7 run-levels
- 0 - halt
- 1 - single
- 2 - multi user - default
- 3 - multi user
- 4 - multi user
- 5 - multi user
- 6 - reboot
Use init + number to enter different run level
cd /boot/ - should see grub directory. cd /etc/ - ls-l rc - shows different run levels cd /etc/init.d - contains daemons, service scripts that are referenced from run level.
Partitions / Raid / LVM etc
Disk Commands
df -h
- shows disk spacesudo fdisk -l
- shows hard drive partitionsls -l /dev/sd*
- SHOWS ALL DRIVEScat /proc/mdstat
- SHOWS LINUX RAID DRIVES IN USEpvdisplay
- SHOWS PHYSICAL VOLUMESlvdisplay
- SHOW LOGICAL VOLUM DISPLAY
Provisioning Filesystems (Extra Storage)
Provision storage while sudoserver is online.
fdisk -l
- reveals connected disks and partitionsdf -h
- shows amount of memory used/dev/sdb
- unpartitionedmklabel
- type MSDOs - if needed.sudo parted
- partition toolselect /dev/sdb
- selects disk -mkpart primary 1 10GB
- 10GB partitionprint
- shows disks in partedquit
- to leave parted
mke2fs
- overlay filesystem on new partition.mke2fs -t ext4 -j /dev/sdb1
- creates file system.
or
sudo mkfs.ext4 -j /dev/sdb1/
- same as above.
mount /dev/sdb1 /projectx/10gb.
- create mount pointmount
- shows all system mounts.dd if=/dev/zero bs=1024 count-10240 of=/projectx/10GB/test.file.1
- creates 10MB in new mountsudo blkid
- shows partition uuid for stab.sudo nano /etc/fstab
- edits stab.UUID="number" /projectx/10GB ext4 defaults
- stores in fstab.
Provision SWAP storage on demand
Ability for kernel to extend RAM via disk.
free -m
- determines current stare of storage.top
- also shows SWAP info.
sudo fdisk -l
- to identify partition spaceparted /dev/sdb
- places in context of /dev/sdv/print
- to show partition table.mkpart primary linux-swap 10GB 12GB
starting from the 10GB first block moving up.set 2 swap on
- turns partitions to swap and on.sudo fdisk -l /dev/sdb
- confirms swap allocation.sudo mkswap /dev/sdb2
- overlays SWAP filesystem and displays UUID (FSTAB)sudo blkid
- shows all UUID'ssudo nano /etc/fstab
- opens stab for editing for SWAP reference using UUID.UUID "" NONE swap sw 0 0
- for nano file.swap on -s
- displays current swap situationsudo swap on -a
- turns on swap storage.free -m || top || swapon -s
- to confirm configuration.
Option SWAP creation which is file based
dd if=/dev/zero of=/projectx/10GB/swapfile3GB count=3G bs=1024
- creates dummy swap file.sudo mkswap /projectx/10GB/swapfile3GB
- overlays SWAP file system.sudo nano /etc/fstab
- opens stab for editing for SWAP reference using path /projectx/10GB/swapfile3GBsudo swapon -a
- tuns on all swap storage.
Storage Management LVM (Logical Volume Management)
Volume sets based on various disparate storage technologies.
Common configuration - raid hardware (redundancy) / LVM overlaying RAID config (aggregation)
Ability to extend, reduce, manipulate storage on demand.
LVM storage hierarchy:
Volume Group (Consists of 1 or more physical volumes)
- Logical Volume(s)
- File System(s)
6 steps to LVM setup
Appropirate 1 or more LVM partitions
sudo parted /dev/sdb
mkpart extended 13GB 20GB
print
mkpart logical lvm 13GB 20GB
select /dev/sdc/
mklabel msdos
- mkpart primary 1GB 20GB
set 1 lvm on
print
20GB on SDC and 7GB on SDB
Partition
sudo pvcreate /dev/sdb5 /dev/sdc1
- allocates LVM partitions as physical volumessudo pvdisplay
- shows LVM physical volumes.sudo vgcreate volgroup001 /dev/sdb5 /dev/sdc1
- aggregates volumes to volume groupsudo lvcreate -L 200GB volgroup001 -n logvol001
- creates logical volumessudo lvdisplay
- shows logical volumessudo mk3fs -t ext -j /dev/volgroup001/logvol001
- overlays EXT4 filesytem.Mount filesystem and commit changes to fstab
sudo lvrename volgroup001 logvol001 volgroup002
- renames volume group. (Remember to edit fstab file or unmount)sudo lvresize -L 25GB /dev/volgroup001/logvolvar
- Resize logical volume by 5GB from 20GBsudo resize2fs /dev/mapper/volgroup001-logvolvar 25G
- Resize filesystem after increasing memory.sudo lvremove /dev/volgroup001/logvolvar
- Remove volume completely. (Umount first)sudo parted /dev/sdc
- Add or assign more partitions to volume group LVM.print
mkpart primary 20GB 25GB
print
set 2 lvm on
print
sudo pvcreate /dev/sdc2
sudo pvdisplay
sudo vgextend volgroup001 /dev/sdc2
- add new PV to volume group
General Admin
cat /etc/*rele*
- shows versionrunlevel
- shows run levelifconfig
- shows network configurationexit
- to log off serverssudo shutdown -h now
- shutdowns server straight awaytop
- show programs runningps -ef
- show processesuptime
- shows uptimeps -ef | grep -a spa
- Shows if apache is runningcat /etc/resolv.conft
- shows DNS resolving
Bash
Delete key gives ~ ? Add the following line to your $HOME/.inputrc (might not work if added to /etc/inputrc )
"\e[3~": delete-char