My Sysadmin Toolkit

When I started my job I often struggled to troubleshoot problems. It is awfully difficult to fix problems when you can't see what is going on, and all too often I did not know what tools were available to help me out. I longed for a good list of debugging resources.

This is my contribution to such a list. It is intentionally terse; I figure you can learn about basic command usage on your own.

In addition to troubleshooting I will include some utilities I have found useful at work.

Windows

Most of this list is oriented towards troubleshooting Windows XP, although there are often equivalents for later versions of Windows.

runas: Launch admin programs from cmd

The runas command is "sudo for Windows". While logged in as a regular user, you can log in as adminlogin on domain mydomain by typing

runas /user:mydomain\adminlogin cmd

This will prompt for credentials and then open a second window. In that window you can type commands that will launch Windows programs and utilities in administrator mode. There is a useful list here: http://en.kioskea.net/faq/403-command-prompts-for-windows

From that list, I often use:

services.msc : Windows services, which can be restarted.
appwiz.cpl : Add/Remove programs
lusrmgr.msc : Local users and groups

One task that is difficult is launching a Windows Explorer window as an administrative user. To do this, type

runas /user:mydomain\adminlogin "explorer.exe /separate"

To change permissions from the commandline, use the CACLS utility. See CACLS writeup

Managing printers

I frequently need to add a local printer, which can't be done with a restricted account. The following command launches the Add Printer Wizard from the command line:

rundll32 printui.dll,PrintUIEntry /il

The following command lists printers:

cscript c:\windows\system32\prnmngr.vbs -l | more

And the following can delete a printer named "HP Laserjet 6L":

rundll32 printui.dll,PrintUIEntry /dl /n "HP Laserjet 6L"

but usually I delete printers using the registry. There are instructions here: http://www.windowsnetworking.com/kbase/WindowsTips/WindowsNT/RegistryTips/Print/RemovePrinterDefinitionfromtheRegistry.html. The synopsis is that there are two places to look. Local printers (including ones created by RDP) are here:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Print\Printers

You need to run regedit as an administrator to delete these.

Network printers are here:

HKEY_CURRENT_USER\Printers\Connections

You have to run regedit as the affected user to delete these.

Remote Desktop Connection

You can launch RDP from the cmd window if you can't find it in the Start Menu (which seems to happen from time to time). The executable name is mstsc.exe

If you have remoted into a computer and want to shut down that computer remotely, you can press <ctrl>+<alt>+<end> to bring up the "Lock Workstation/Shut Down/Change Password" screen you usually get when pressing <ctrl>+<alt>+<del>.

There is a version of RDP which runs in the MMC console, which is handy when you have a list of machines to remote into.

PuTTY Sessions

When you have many PuTTY windows open, your screen can get cluttered. There is no good tab interface for PuTTY, but the best of a bad lot appears to be SuperPuTTY

Outlook Autocompletions

Outlook keeps its address autocompletions in an NK2 file, which is often in the file

C:\Documents and Settings\<username>\Application Data\Microsoft\Outlook\Outlook.NK2

The NK2Edit utility can be used to edit NK2 files, which is useful when somebody has misspelled an address and it keeps appearing in the autocompletion list. Nirsoft has a lot of other useful utilities too.

If you only want to delete one autocompletion, go into Outlook, start typing the bad address, and use the down arrow key to select the bad address in the autocompletion list. Then hit the <del> key.

Packaging and Updates

To check for Windows Updates and make the little yellow exclamation mark shield icon appear in the corner, type:

wuauclt.exe /detectnow

When setting up computers for individuals I find the Ninite service invaluable. You can download an installer that will keep all those annoying plugins (Flash, Java, etc) up to date easily, which makes it the closest equivalent I have found to APT for Windows. This tool is not appropriate for use in an organization unless that organization has a Ninite Pro subscription, however (which we don't).

As of this writing, Mozilla STILL does not provide MSI files or Group Policy templates for Firefox. Another company called FrontMotion does provide these, however: http://www.frontmotion.com/FMFirefoxCE/index.htm

There are excellent collections of Windows drivers at http://www.driverpacks.net . We unpack drivers onto a network server and then let clients browse to the share and run the driver installation utility. The one type of driver we consistently have problems with are sound card drivers, because vastly different sound cards all identify as AC'97.

To automate and script Windows XP installations we use http://unattended.sourceforge.net , but apparently the Windows 7 Microsoft installation toolkit is pretty good now.

Windows Server Utilities

To view Active Directory LDAP attributes, use the adsiedit.msc utility.

To see Exchange's weird folder structure, use PFDAVAdmin.exe . See my Exchange Resource Mailbox writeup.

Local Networks

These are tools I use in the LAN (as opposed to the wider internet, although they have applications there as well).

Network sniffers

Sniffers inspect the traffic going between two hosts. Sometimes you can install a sniffer on the host you are monitoring, but I usually insert a small network hub (not switch) and hook up a laptop running Wireshark.

Wireshark and tcpdump are useful when diagnosing firewalling problems. Wireshark has a lot of advanced features, but I don't know how to use them.

In Wireshark, use the following to filter out all traffic NOT communicating with 192.168.1.133:

ip.addr == 192.168.1.133

If you are wanting to sniff SSL connections in Windows, Fiddler2 http://www.fiddler2.com/fiddler2 is pretty easy to use.

Once I needed to monitor a Windows machine that appeared to be sending out traffic to a lot of weird places. I installed TCPView from Microsoft and left it running on the machine. It recorded all the hosts that the Windows machine communicated with. It was not a great solution, but it worked.

Host identification

nmap and its Windows GUI Zenmap are helpful to identify strange machines on your network.

To list all hosts in the 192.168.1.x subnet that are up, I use:

nmap -sP 192.168.1.0/24

(but apparently -sP has been replaced with -sn in newer versions of nmap)

To intensely scan the host 192.168.1.133 (telling me what ports are open, guessing the operating system, etc) I use:

nmap -v -T Aggressive -A 192.168.1.133

MAC addresses

To find the MAC address of a strange device, first ping the device by IP address and then type

arp -a

To identify the manufacturer of the network card with that MAC address, you can use a MAC address lookup website: http://coffer.com/mac_find

There are also lists of manufacturers by MAC address hidden in the Linux filesystem when you install wireshark. Look in /usr/share/wireshark/manuf .

Internet

These are tools that I tend to use when diagnosing wider internet access problems (although they have applications to local networks as well).

Traceroute

tracert in Windows or traceroute in Linux will let you trace the route that a packet takes to reach its destination.

tracert google.ca

Note that this is not perfect: a traceroute run can fail and yet traffic can get through.

DNS Lookups

To test whether your DNS server is having trouble resolving a host, use nslookup with a DNS server parameter:

nslookup pnijjar.freeshell.org 8.8.8.8

Internet Reliability Websites

http://www.speakeasy.net/speedtest/ : a standard test to measure upload and download bandwidth. I have heard that this test has problems, but it is a good rough approximation.
http://netalyzr.icsi.berkeley.edu/ : Another comprehensive test suite, with comprehensible output.
http://myspeed.visualware.com/indexvoip.php : VoIP specific
http://www.downforeveryoneorjustme.com (and similar websites) are useful for distinguishing local network issues from ones out of your control.
http://www.measurementlab.net/measurement-lab-tools : Has a number of tests for different aspects of network health and net neutrality.

Hardware

Some of these tools are for manipulating hardware in addition to testing it.

Component Testing

memtest86+ is included on many LiveCDs, and it works well for diagnosing memory problems. With older hardware I have found that older versions of memtest sometimes work better than newer ones.

To view SMART hard drive information I find the hdat2 tool (Available at http://www.hdat2.com, or on many utilities disks) helpful. It lets you view SMART information and can run some simple hard drive tests. When possible, however, I run manufacturer-specific hard drive tools to diagnose drive problems.

There is a small freeware Windows utility called Nokia Monitor test that I have used to test CRT monitors. It is available from http://www.prad.de/en/monitore/testsoftware/nokia_monitor_test.html .

System Information

The lshw command in Linux is the best free and legal tool I know of for listing information. It is not hard to script this command to pick out the specific information that you want.

Cloning

The best free and legal cloning tool I have found is Clonezilla . Clonezilla has problems resizing NTFS partitions, but GParted handles this well. Clonezilla also has issues cloning big hard drives to smaller ones, even if the big hard drive has lots of blank space.

If you are reimaging an entire lab, using the DRBL functionality of Clonezilla is helpful. Note that you probably want to disconnect the lab in question from your wider network before you do this, because DRBL wants to run its own DHCP server.

Drive Wiping

We use DBAN to wipe donated drives. It also serves as a reasonable (but not perfect) drive test.

Physical Tools

A good cable tester is invaluable for tracing cables hidden behind walls and for identifying crimping problems. I like the Ideal VDV tester, which has jacks to test phone, network and coax connections. It costs about $120. Other testers are fancier (we have one that calculates the length of a network cable, which is nice), but being able to test both phone and ethernet cables easily is really nice.

A digital power supply tester costs about $25. A digital power supply tester will be able to diagnose fluctuations in power and inappropriately high/low voltages. The disadvantage of such a device is that you quickly see how many power supplies deliver unclean power. We have a power supply tester branded "CoolMax" that looks like this:

Live CDs and Distros

Knoppix

For recovering Linux installations I inevitably turn to that old reliable, KNOPPIX. It still seems to work better than most other LiveCDs I have run into. Often I use it to access the hard drive of a target system:

mount /media/sda1
mount --bind /dev /media/sda1/dev
mount --bind /sys /media/sda1/sys
chroot /media/sda1

At this point I can reinstall GRUB, reset the root password, and (if home directories are not encrypted) access user files. I can even install software to the chroot if I so choose.

Hirens

http://www.hiren.info/pages/bootcd

I don't even know how people obtain this CD. It has a whack of useful utilities, but not all of them are legal to use. If I am stuck looking for a helpful tool or utility, this is one of the first places I look.

GPXE

Setting up a PXE server and network booting is incredibly useful. We use our PXE server for installations, running utilities, drive wiping, and much more. However, many network cards do not support network booting, and some of the ones that do have buggy PXE implementations. GPXE solves this problem easily: you boot into the GPXE CD, and it boots over the network.

You can generate an ISO here: http://rom-o-matic.net/gpxe/gpxe-1.0.1/contrib/rom-o-matic/ .

Related Resources

http://ss64.com : Home of the best cmd programming reference I have found.
http://sectools.org : Comprehensive (and somewhat frightening) collection of security tools, with ratings.

- http://unixsadm.blogspot.com/2007/10/digital-forensic-tools-imaging.html: good list of utilities, with focuses on virtualization and digital forensics.

What's Missing?

Something to stress test network switches. There's iperf, but it can't detect flaky network switches well enough.
Something to mimic strace on Windows in a useful way.
Something to help test CD and DVD drives.