In my professional life so far I have aquired a skill to takle difficult problems that other find impossible to solve and make them possible.
Most of the time I found it involves good verbose logging and some creative thinking.
There are definite limits when you are using closed and/or proprietary systems, that have a high interest to not let you understand how the system works and being forced to delegate support out to the manufacturer.
I pride myself to provide real support, given that I do not have to work with a system that only allows troubleshooting with the "Voodoo Informatics" approach.
Here some important sequence points when troubleshooting a Linux system:
Listen to the Kernel
When connecting hardware, like USB devices and such, it is always interesting what the kernel is saying.
The simple command dmesg (in Debian) and cat /var/log/messages will show you what the system core is complaining about.
I love the way Linux is handling this, since you can spot many problems from the onset. Having a faulty USB flash drive? Just happend to me recently, the kernel spewing tons of read and write errors.
Logs, Logs and more Logs
Unlike Windows, where you will not find much useful data in the event-viewer, you can find a bounty of information on any Linux System in the directory /var/log/
There is a log for anything you can imagine, the general one being messages.
It differs from system to system, what is listed where, but when you check out the filenames, you will be able to figure out what is what.
What's going on on the bus?
The commands lsusb and lspci will show you what devices are connected to the USB or PCI bus, very handy to see what you got in there.
Missing a harddisk? Check out the starup of the kernel, when it recognizes all the hardware. If it is not there, chances are it is broken or no driver exists for it (very rare, as a Linux system will recognize pretty much any controler and disk there is). But yet again, if you have compiled your own kernel (which you should NOT do for an production system), you might have forgotten the module in the first place.
Most things are automatic, modules (drivers in Linux) get loaded automatically and configuration too, so if you connect your firewire card (PCMCIA for example) and the kernel does not say anything, you can be pretty sure that this is the problem.
Stop the daemon from mumbling in the back
Many daemons (the UNIX/Linux term for service or server) can be made to speak up and run in the foreground, instead of background.
Check the manpage, and usualy it is the -d or -D option.
Usualy they have a d at the back of the name and let's not forget the verbose or double verbose, a very handy tool in troubleshooting. I wish all closed software had this, my life would be so much easier.
Add -v to pretty much any command or programm, and it will be more verbose, tell you more about what it is doing....
-vv or -vvv produces more output, sometimes too much. But better too much than too little, right?
Not all commands use the same way, check the man page.
Seeing if you can reach the server at least over TCP/IP helps to identify problems. Most clients as servers do not log properly, sometimes not at all. You will never be able to find out why it is not working.
You might be getting a ACCESS DENIED, a nontransparent error that should make cracking the system difficult. Now what is denied, access to the user, access to the server, or is the password wrong?
Using the port- and securityscanner NMAP, you can always find out why a server is not responding. NMAP offers tons of options, for to many to quote here, but a simple
nmap -p 389 -sV hostname
will do wonders if you LDAP server cannot be reached for some reason. The above command does a portscan for port 289, querying also the version of the server hiding behind 389, which will show you if it is active at all.
Now omitting the port number, NMAP will do a full range port scan, which is not really a noninvasive thing to do and many IDSes will detect this, as they should. Now if you want to see if somebody is home, you also don't go and knock on every door and window on the house, would you?
this document was created on:
10. Oct. 2006
07. Dec. 2006