Grow Your Sysadmin-Fu

There are a few commands that the Linux sysadmin simply can't live without.  The need to see exactly what is going on with your army of *nixs and what any one is doing at any particular time is a must.  Here are a few commands that I would seriously miss, and ones that I make sure are installed on every box I administer.  So, in no particular order....


kill
As well as being a great command to terminate processes, by far my favourite use of this command is to pause running processes.  Imagine you have something really resource-hungry running, for example you're compiling a new piece of software, and suddenly you have a need to do an update on the MySQL databases stored and running on the box.  To pause your compilation, first find the pid(s) of the processes you want to pause.  Then issue a kill -STOP <pid1> <pid2> <pid.... 
To restart your compilation after your database fiddling, it's kill -CONT <pid1> <pid2>....

netstat
Netstat is a wonderful little command that gives you a list of everything that has a network socket, port or other connection to the outside world.  Probably the best incantation is netstat -anp.  The a  is for all, n is for don't resolve hostnames (just use IP addresses), and the p tells netstat to list name of the process that has opened/is listening to the port.

top
Top is brilliant.  It's basically a process monitor that lists all running processes in a neat little list and shows overall CPU and memory useage.  Top has loads of switches/commands - press ? for a list of them all.  To start with, try pressing 1 and 2  if you have a multiprocessor machine.  Also try m and t to see some pretty ascii-art graphs.

vmstat
Vmstat is one of my favourites, it's particularly useful for spotting I/O bottlenecks and finding I/O bound issues.  Officially, vmstat is for monitoring virtual memory stats, but it shows a lot more.  You can spot I/O issues by getting large wait states in the processor info - vmstat by default displays how long the CPU spends running different code: user is time spent running non-kernel code (including nice'd processes), system is time spent running - you guessed it - kernel code (system time).  idle is, surprise surprise, time the processor spends idle (prior to 2.5.41 this also included IO wait time).  wait is the time spent by the processor waiting for IO requests to complete.  If this number is persistently high (anything above about 20-30%) then you should consider taking action - the system is becoming IO bound.  This can also be an indicator of a failing hard drive or failing RAI24 D array.  I like to use vmstat -awSM 3 - this tells vmstat to report active/inactive memory statistics, use wide mode for outputting the data and then to output stats in SM megabytes instead of bytes, and to update every 3 seconds.  Vmstat will continue forever until you Ctrl^C it or kill it.

ip
One of my all-time favourite command.  ip allows you inspect and change the IP address assignments on the box and also to inspect and modify the routing table.  All commands can be abbreviated, for example ip address list can also be written as ip a l. ip route list can be written as ip r l.  ip is a wonderfully flexible and powerful command.  It has waaaay too many options to go into it in detail, but it's main use for me is to establish a static route, for example: ip route add 192.168.15.0/24 via 192.168.1.1 dev eth0 adds a static route to the 192.168.15.0/24 network via a router at 192.168.1.1 using eth0.

iptraf
If you're using a linux box as a router, you must install this little beauty.  iptraf is a wonderful little program with a basic GUI frontend that allows you to easily see what traffic is going where on your linux router.  It has lots of modes, from a LAN-station monitor that shows which active connections go where and to what port.  Here's a couple of screenshots:

iftop
Top but for network activity instead of system activity.  Gives a list of all active connections, sorted by current bandwidth  usage.  I must aadmit to not using iftop much, I tend to get by with netstat, top and iptraf.  I do like it's layout though, and it's super easy to fire it up and just check on the screen every couple of seconds or however long is required, with easy to read bar graphs showing current network bandwidth.  Help is accessed by hitting the ? like in top.  Open it up, have a play.  You may like it more than me!

lsof
Lsof stands for list open files.  It does exactly what it says on the tin.  Usually you would put a command/process name after lsof, but lsof on it's own just shows every directory/file handle currently in use.

w
I guess this stands for who/what.  I use it just because it's easier to type than uptime or other similar bollox.  It returns the uptime, load averages and a list of currently logged in users and what they are doing (ie which command is being executed).

dmesg
This is where the kernel spits out all it's info.  Super useful for finding hardware problems and physical faults.  Use it how you want, but I like to do a dmesg -L -H.  The L instructs dmesg to use pretty colours.  The H tells it to output in a human-readable format.taking up.  

df/du
Stands for disk free and disk use(d).  Displays how much space is left on a drive or how much space a filesystem is taking up.  Super useful for seeing, for example, how big a web site is - du -h /var/www/html/mywebsite, or how much free space is on your root partition (on SCSI drive A) df -h /dev/sda1.
 
grep
Dunno why I'm including this - you should be well versed in using grep.  Grep - or get regular expression - is basically a filter.  It will return anything that matches the given regular expression from the list of file(s) given.  For example, to look for all references to cron.hourly in syslog you'd write: cat /var/log/syslog | grep cron.hourly and this would return all the lines in syslog that mention cron.hourly.  Useful switches are -A<num> which also returns <num> number of lines after the match and -B<num> which returns <num> number of lines previous to the match.  If you want a "not" then -v returns everything -except- lines matching the pattern.

<, >, |, &
Although not commands, these are useful.  These characters will redirect output from commands.  For example, ls -l > dir.txt this will list the current directory and will redirect the output to a text file called dir.txt.  The pipe character | is easily the most used special character, used like in the example above.  If the example above returned loads of information that scrolled off the screen, you could issue cat /var/log/syslog | grep cron.hourly | more or, to be super pernickety, you don't need the cat, so it becomes grep cron.hourly /var/log/syslog | more.  The ampersand - & - when placed at the end of a command will fork the process and return the TTY to the user with the pid of the forked process.  Two ampersands will run processes consecutively, waiting for the first to finish before starting the second, eg; command1 && command2 && command3 - this will run command1 and wait for it to complete before running command2 which must complete before command3 is executed.


vim
I can't stress how important it is to have a decent text editor or a proper IDE when programming.  I really recommend you get to grips with either EMACS or VIM, the time you spend learning the foibles, oddities and weird-as-fuck commands and key combinations will be time that you will make back doing something like a global find and replace, and you'll feel awesome after.  I much prefer vim to emacs, but it's personal preference.  Use both.  Find which one you're more comfortable with and adopt it.  Both vim and emacs have syntax highlighting for spotting mistakes and easier programming and scripting.  I love vim so much that I now automatically hit <esc><:x> to try and save and exit when typing and on auto-pilot.  Having to go back to the arduous task of moving the mouse and going to <menu><File><Save/save as> suddenly seems like such an unnecessary long-winded way of doing things...! There's an XML file I'm editing that has definitions of servers that are exactly 18 lines long - I want to take out a definition here and there so with vim you get your cursor at the start of the definition and hit <esc><18dd> and voila - a whole definition disappeared.  If you want it back, it's stored in vim's internal paste buffer, so just hit <p>.  Cocked up the whole file? Hit <esc><:q!>.  Once you get the hang of emacs or vim, you'll never look back.  So learn one of 'em - it's a must.

nmap
As well as being the hacker's number one essential tool, nmap is also useful in diagnosing problems across the network. Network Mapper will scan an IP address (or range of IP addresses) and return a list of open ports.  There are several different scan types that I won't go into here, but at it's most informative it will even attempt to spit out version information of the program running behind the port - try a -sV or -A as a switch.

--==MORE TO BE ADDED, CHECK BACK HERE SOON (LAST UPDATE: 22-10-19 15:15BST)==--