= Cheatsheet Linux and Bash = **Summary**: Linux Management & Bash hints, tips, oneliners and best practices. \\ **Date**: 8 December 2024 \\ {{tag>cheatsheet linux bash}} == OS == > See release cat /etc/os-release cat /etc/SuSE-release \\ > See kernel version # Kernel version uname -r # Full version uname -a === Runlevel === This is the runlevel config as defined on SuSE: * 0 System halt * 1 Single user mode * 2 Local multiuser mode zonder netwerk * 3 Full multiuser mode met netwerk * 4 Niet gebruikt * 5 Full multiser met netwerk en GUI * 6 System Reboot The default runlevel is set in /etc/inittab. When the box is already running you can issue the command "init x" while replacing the x for the appropriate runlevel number to change the runlevel. If you want to change the runlevel while booting issue the same command as a boot parameter. == Environment info == * .profile * Set variables (must be exported), aliases and configure the shell * ENV: special variable, used to set a file that will be executed on starting the shell (usually .bashrc en .kshrc ) * Defines extra variabels and aliases. * .dtprofile * Set GUI, DTSOURCEPROFILE must be TRUE to also load .profile *.exrc * Loaded on starting vi * Allows for stuff like "set number" and "set showmode" * .hushlogin * Add the empty .hushlogin in your HOME to discard the MOTD (Message Of The Day) when logging in. == Users == > Check who is logged in: who \\ > Check who you are with session information: who am i \\ > Check your own userid and group information: whoami id \\ > Check the user information of a specific user: id userid finger userid \\ > Check the last login of a user: last > Show last # Show the last 20 logins last -20 # Show the last logins of a specific user last userid == Vi == === Configure vi === * Turn off syntax {{{:syntax off}}} * Show line numbers {{{:set number}}} * Show mode (input or command) {{{:set showmode}}} === Vi commands === * Insert text before the cursor {{{i}}} * Append text after the cursor {{{a}}} * Insert text at the beginning of the line {{{I}}} * Append text at the end of the line {{{A}}} * Replace character {{{r}}} * Replace text {{{R}}} * Delete character {{{x}}} * Delete 5 characters {{{5x}}} * Delete current line {{{dd}}} * Delete the next word {{{dw}}} * Delete 10 lines {{{10dd}}} * Delete lines 2 to 4 {{{2,4dd}}} * Delete all below lines, including current: {{{dG}}} * Copy 10 lines {{{10yy}}} * Paste the copied lines {{{p}}} * Undo last change {{{u}}} * New line below the current line and insert {{{o}}} * New line above the current line and insert {{{O}}} === Vi search === * Search for a string downwards {{{/"string"}}} * Search for a string upwards {{{?"string"}}} * Search and replace in the complete document {{{:%s/stringold/stringnew/}}} * Search and replace in the complete document, confirm each change {{{:%s/stringold/stringnew/gc}}} === Vi navigation === * Go to the next word {{{w}}} * Go to the previous word {{{b}}} * Go to the end of the line {{{$}}} * Go to the beginning of the line {{{0}}} * Go to the last line of the file {{{G}}} * Go to line 1 {{{1G}}} * Go to line 206 {{{206G}}} === Other Vi commands === * Past text from windows clipboard through putty: {{{SHIFT + INS}}} == Certificate Management == > Create csr with new private key (preferred) openssl req -nodes -newkey rsa:2048 -keyout *.shiftwiki.nl.key -out *.shiftwiki.nl.key.csr \\ > Create csr with existing key openssl req -new -key /data/keys/httpd-wild.shiftwiki.nl/oeserver.pem -out *.shiftwiki.nl.key.csr \\ > View certificate openssl x509 -text -noout -in certificate.crt \\ View csr openssl req -text -in request.csr == Logfile Management == Truncate a logfile that has grown too big sudo truncate -s 0 odoo.log \\ > Show and follow a logfile tail -f "logfile" \\ > Show and follow a logfile with a specific number of lines tail -f -n 100 "logfile" \\ > Show and follow a logfile starting from a specific line tail -f +100 "logfile" \\ > Show and follow a logfile starting from a specific line counted from the end tail -f -100 "logfile" \\ > Show the first 8 lines from a file head -8 "logfile" === Kernel log === > Kernel log file: /var/log/messages dmesg == Disks == See link between device and filesystem for AWS EBS disks. From the instance, go to Storage tab and see the device name, then use these commands: [sjoerd@server ~]$ df -hT /dev/sdf Filesystem Type Size Used Avail Use% Mounted on /dev/nvme1n1 xfs 20G 14G 6.8G 67% /var/lib/docker [sjoerd@server ~]$ df -hT /dev/sdg Filesystem Type Size Used Avail Use% Mounted on /dev/nvme2n1 xfs 20G 15G 5.8G 72% /data === iNodes === Understanding UNIX / Linux filesystem Inodes The inode (index node) is a fundamental concept in the Linux filesystem. Each object in the filesystem is represented by an inode. But what are the objects? Let us try to understand it in simple words. Each and every file under Linux (and UNIX) has following attributes: * File type (executable, block special etc) * Permissions (read, write etc) * Owner * Group * File Size * File access, change and modification time (remember UNIX or Linux never stores file creation time, this is favorite question asked in UNIX/Linux sys admin job interview) * File deletion time * Number of links (soft/hard) * Extended attribute such as append only or no one can delete file including root user (immutability) * Access Control List (ACLs) All the above information stored in an inode. In short the inode identifies the file and its attributes (as above). Each inode is identified by a unique inode number within the file system. Inode is also know as index number. How do I see file inode number? $ ls -i /etc/passwd You can also use stat command to find out inode number and its attribute: $ stat /etc/passwd Find and remove file using find command (necessary with special characters), type the command as follows: $ find . -inum 782263 -exec rm -i {} \; Additional info: A directory is nothing more than a table of the underlying files and a pointer to the inode information of this file. To "jump" to this inode table you need execute rights on the direcory. So if you want to open a file you'll also need execute rights on the directory. == Finding MD5 Hash on Linux == Find the MD5 hash of files on Linux by entering the command below and replace the "Your-File-Here" with the name of the file: echo -n "Your-File-Here" | md5sum == Netwerk Commands == > Show the hostname hostname hostname -s \\ > Use scp to securely copy files between hosts scp "source" "target" scp host:/home/file /targetfile # Copy a directory scp -r host:/home/dir /targetdir \\ > Check all open ports netstat -a # -n only show port number # -r Route tabel # -p Including processes # Search for open port 389 with the corresponding process netstat -nap | grep :389 # Show the routing table netstat -rn == Processes == > Using top to get a list of running processes and their resource usage top # c Expand processes # 1 separate all processes # s set refresh rate # M sort on memory usage # P sort on processor usage \\ > Using ps to get a list of running processes # all running processes ps -e # all running processes with full info excluding kernel processes ps -ef # all running processes with full info ps -Af # all running processes for a specific user ps -u "userid" # Start a process in the background and keep it running after the session is closed nohup command & == Searching == === Grep === > grep file without empty lines and comments grep -v '^\s*$\|^\s*\#' postgresql.conf \\ > grep options # -i : case insensitive # -v : invert match # -w : match whole word # -r : recursive # -l : only filenames # -n : show line numbers # -c : count matches # Search for test in all files in the current directory and subdirectories and only show the filenames grep -rlw test . \\ > Find the Linenumber of a Specific Match grep -n \\ > Find Specific String in Files - Recursive grep -r '172.18' * === Find === > Find is always recursive # find file in current and subdirectories sudo find . -name main.cf # find all files starting with a and execute ls -l on them find . -name 'a*' -exec ls -l {} \; # find all files starting with a and execute ls -l on them, but ask for confirmation find . -name 'a*' -ok ls -l {} \; # Set read permissions on other for all files in /var/news find /var/news -type f -exec chmod o+r {} \; # Rename all files in /tmp to filename.old find /tmp -type f -exec mv {} {}.old \; # Find only files older than 31 days and remove them find /tmp/archive/. -type f -mtime +31 -exec rm {} \; ==== Find and Disk Space Info ==== # Check disk usage df -H # Find largest directories sudo du -Sh / | sort -rh | head -5 # Find largest files in /data sudo find /data -type f -exec du -Sh {} + | sort -rh | head -n 10 # Find largest files in /var/lib/docker sudo find /var/lib/docker -type f -exec du -Sh {} + | sort -rh | head -n 10 # Find largest files except in /data sudo find / -type f -not -path "/data/*" -exec du -Sh {} + | sort -rh | head -n 10 == Crontab == > display all crontabs for all users as root {{{sudo su -}}} for user in $(cut -f1 -d: /etc/passwd); do echo $user; crontab -u $user -l; done == Shell Management == > Note this is especially useful in pipelines \\ >set # -e : exit the script if any command fails # -u : treat unset variables as errors and exit # -o pipefail : ensure the entire pipeline fails if any command within it fails # -x : print all the commands to the terminal # -v : print all lines without variable resolution # - For all, to disable use +, for example `set +v` set -euo pipefail == File Management == > Long listing of files ls -l \\ > List files ordered by change date ls -t \\ > Reverse file listing order ls -r \\ > List files including hidden files (files starting with a dot) ls -a \\ > Count files in a directory # wc: word count; -l: count lines ls -l | wc -l \\ > Compare two files diff file file2 \\ > Compare two files next to each other, only differences diff -y --suppress-common-lines file1 file2 === File Management with Regex === > Show file type file "filename" \\ > Show file type, inode number and all three dates (modify, access, change) stat "filename" \\ > List files with a, b or c in the name ls [abc] \\ > List files without a, b or c in the name ls [!abc] \\ > List files that start with a, b, c or d ls [a-d]* \\ > List files that start with a or d ls [ad]* \\ > List files that start with a or d and have a or e as second letter ls [ad][ae]* \\ > List files that start with a, A, b or B ls [a-bA-B]* \\ > List files that start with a or b and have at least one more character ls [ab]?* \\ > List files that don't end with a letter ls *[!a-zA-Z] == Disk Management == > List disk usage # show diskspace of all directories 1 level deep in human readable format du --max-depth=1 -h -c # do 'du' for every existing directory ## ls -1 lists all files and directories each on a separate line ## [[ -d ]] is true if listed variable is a directory for i in `ls -1`; do if [[ -d $i ]]; then sudo du -sm $i; fi; done == If File in Directory Then == for i in `ls -1 | grep was-`; do if [ -f $i/.run ]; then echo start slot $i; fi; done * ls -1 lists all files and directories each on a separate line * [ -f ] is true if listed file is a regular file == Variables == # Standard variables # Prompt PS1 # Home HOME # Path PATH # User USER # Add text directly after a variable (useful in scripts) echo ${var}tekst # Assign output of command to variable x=$(command) # Create a readonly variable readonly var=waarde # Remove / unset a variable unset var # Export a variable so it is also available in subshells export var == Regular Expressions == . Any single character * 0,1 or more of any character .* 0,1 or more of any character [aA] a or A [a-f] a to and including f ^a Starts with a a$ Ends with a ^a.*b$ Everything starting with a and ending with b ^$ Empty line > Combined with grep you can use regular expressions to filter out comments and empty lines out of config files. However, this should take you two regular expressions: {{{grep -v '^&' file | grep -v '^#'}}}. Using egrep allows you to do that in one go: {{{egrep -v '^$|^#' file}}} == Output Redirection == # Run command in the background (output and errors still go to the screen) & 'command' # Input redirection < inputfile # Output redirection > outputfile # Error redirection 2> errorfile # Combine output and errors for redirection to a command or a file 2>&1 # Output redirection append >> outputfile # reads the input, then writes the output of a program to standard output and simultaneously copies it into the specified file or files, with example: The output of command is written to file1 and also used as input for command2: command | tee file1 | command2 == Sed == > Basic sed search and replace line: sed "s/$FIND/$REPLACE/g" $inputfile > /tmp/outputfile \\ > Code line to make email adresses unusable by adding the extension .local to the complete email address: sed "s/^mail.*$/&\.local/g" inputfile > outputfile * s : substitute * {{{^mail}}} : all lines that start with mail * {{{.*$}}} : and match everything after the initial match * & : everything that has been matched * \.local : Add .local (\ is for escaping the '.') * g : global == Running a script == There are three ways to run scripts: * bash script * Script runs in subshell and read permissions are sufficient * (./)script * Script runs in subshell and execute permissions are required * . (./)script * Script runs in current shell and read permissions are sufficient (used to export variables) === Script Variables === * $$ = The PID number of the process executing the shell. * $? = Exit status variable. * $0 = The name of the command you used to call a program. * $1 = The first argument on the command line. * $2 = The second argument on the command line. * $n = The nth argument on the command line. * $* = All the arguments on the command line. * $# = The number of command line arguments. * $@ = All arguments The "shift" command can be used to shift command line arguments to the left, ie $1 becomes the value of $2, $3 shifts into $2, etc. The command, "shift 2" will shift 2 places meaning the new value of $1 will be the old value of $3 and so forth. === If Options === |[ -a FILE ]|True if FILE exists. | |[ -b FILE ]|True if FILE exists and is a block-special file. | |[ -c FILE ]|True if FILE exists and is a character-special file. | |[ -d FILE ]|True if FILE exists and is a directory. | |[ -e FILE ]|True if FILE exists. | |[ -f FILE ]|True if FILE exists and is a regular file. | |[ -g FILE ]|True if FILE exists and its SGID bit is set. | |[ -h FILE ]|True if FILE exists and is a symbolic link. | |[ -k FILE ]|True if FILE exists and its sticky bit is set. | |[ -p FILE ]|True if FILE exists and is a named pipe (FIFO). | |[ -r FILE ]|True if FILE exists and is readable. | |[ -s FILE ]|True if FILE exists and has a size greater than zero. | |[ -t FD ]|True if file descriptor FD is open and refers to a terminal. | |[ -u FILE ]|True if FILE exists and its SUID (set user ID) bit is set. | |[ -w FILE ]|True if FILE exists and is writable. | |[ -x FILE ]|True if FILE exists and is executable. | |[ -O FILE ]|True if FILE exists and is owned by the effective user ID.| |[ -G FILE ]|True if FILE exists and is owned by the effective group ID. | |[ -L FILE ]|True if FILE exists and is a symbolic link. | |[ -N FILE ]|True if FILE exists and has been modified since it was last read. | |[ -S FILE ]|True if FILE exists and is a socket. | |[ FILE1 -nt FILE2 ]|True if FILE1 has been changed more recently than FILE2, or if FILE1 exists and FILE2 does not. | |[ FILE1 -ot FILE2 ]|True if FILE1 is older than FILE2, or is FILE2 exists and FILE1 does not. | |[ FILE1 -ef FILE2 ]|True if FILE1 and FILE2 refer to the same device and inode numbers. | |[ -o OPTIONNAME ]|True if shell option "OPTIONNAME" is enabled. | |[ -z STRING ]|True of the length if "STRING" is zero. | |[ -n STRING ] or [ STRING ]|True if the length of "STRING" is non-zero. | |[ STRING1 == STRING2 ] |True if the strings are equal. "=" may be used instead of "==" for strict POSIX compliance. | |[ STRING1 != STRING2 ] |True if the strings are not equal. | |[ STRING1 < STRING2 ] |True if "STRING1" sorts before "STRING2" lexicographically in the current locale. | |[ STRING1 > STRING2 ] |True if "STRING1" sorts after "STRING2" lexicographically in the current locale. | |[ ARG1 OP ARG2 ]|"OP" is one of -eq, -ne, -lt, -le, -gt or -ge. These arithmetic binary operators return true if "ARG1" is equal to, not equal to, less than, less than or equal to, greater than, or greater than or equal to "ARG2", respectively. "ARG1" and "ARG2" are integers. | |[ ! EXPR ]|True if EXPR is false. | |[ ( EXPR ) ]|Returns the value of EXPR. This may be used to override the normal precedence of operators.| |[ EXPR1 -a EXPR2 ]|True if both EXPR1 and EXPR2 are true. | |[ EXPR1 -o EXPR2 ]|True if either EXPR1 or EXPR2 is true. | |command |True if command returns a 0 (zero). | == Useful links == * [[https://www.ii.com/git-bash-is-my-preferred-windows-shell/|Git Bash Is My Preferred Windows Shell]]