The Art of Triage

To many, troubleshooting seems to be a gift that either you have or you don't. For instance, my father is a mechanic. When he owned his own repair shop he would hire young guys who would spend hours troubleshooting a problem, but then within minutes from my old man getting involved he could quickly diagnose the cause of a problem. The timing belt, carburetor issues, whatever it was, he was quick to pinpoint. Inevitably once the problem was found there was the “oh, of course” from the Junior grease monkey.

I was too clumsy to be a mechanic, so my dad fired me and forced me into Computers. However, I didn’t forget what I had learned about troubleshooting.

First, troubleshooting is not something you are born with. It is a skill that is harnessed based on 3 common factors:
1) What you know
2) What you don’t know
3) What you are learning

When you piece these 3 factors together you create the framework for discovery. By adding at negative and positive approach will then lead you down a path of what good troubleshooters simply call the process of elimination.

Do you know what is working? Do you know what is not working?
What don’t you know is working? What don’t you know is failing?
What have I proved with this step? What have I disproved with this step?

So when it comes to troubleshooting complex systems, the same principle applies. You just need to analyze them in layers. Here are the layers that VIGILANT has documented as the logical points to eliminate.

Infrastructure: Hardware, Networking, Operating Systems
Application: 3rd party application services
System Interfaces: Connectivity between dependant systems
Business logic: Business rules that cause transactions to operate differently
Business Process: The way the end-user is executing the transaction
Business Service: Dependency on data or other elements for success

For really complex issues, take each of these tiers and apply the 3 principles of discovery to them and you fill find the problem is not as much as a black-hole as you thought it was.

Why are we still fighting fires?

"We have spent so much money on monitoring tools and consulting, why are we still fighting fires?" I heard this from a recent prospect. The simple answer is because we live in a complex and high demand world. As IT proffesionals we are trying to do a lot with a little; little training, little vision, little strategy, little focus. Organizational process alone is not going to cut it. If we want to fight fires, then we have to campaign like Smokey the Bear. "Only you can prevent forest fires!" was his motto. What is yours?

How about these? "Only you can prevent outages from unplanned changes!"- By implementing some more rigourous controls around change and release management.
"Only you can prevent unnecessary downtime by distracting priorities!" - By implementing better incident management procudures, you could avoid the "SWOT" call mentality that takes the engineers attention off of restoring the service and diverts it into CYO for mangaement.
"Only you can prevent business impact from IT service failure!" - By properly planning and validating your infrastrcuture through capacity and load testing, infratstucure validation for fail-over and applicaiton profiling, you can ensure your build and release managment processes are catching issues before your users do.

Why are you still fighting fires? Probably because your processes are like disconnected piles of twigs and your culture is overly reactive.