MisDirection

I've spent a month at work trying to track down a nasty production bug. At the beginning of August one of our web services started blowing up for about a half an hour every morning. I've been living on about four hours of sleep per night, getting up early to attend meetings trying to uncover the root cause. We've looked through all of our production code, nothing has changed. We looked into other jobs that might be running, that we are unaware of. We've scoured the database, the server, the service calls. Nothing yielded anything useful. It looked as though early mornings would be the new normal.

Then in the course of analyzing the database again, a coworker notices one single materialized view. It was on a development machine. Hr dug deeper. There was a dev job that had been updated to point to production. This was causing the view to be recompiled and was most likely the problem we were seeing (We'll know for sure in the morning).

We had looked everywhere but development. This was production after all. But the key lay in a completely different environment. It was a problem of misdirection. Like a magician tricking us to look at his left hand as he manipulated an object in his right.

Remember when debugging, not to let misdirection cause you to overlook what is truly going on!