Importance of Application health sensors in enterprise scale solutions

Imagine a situation-

After having done the best possible development and testing for the proposed application the system is now in a production / BETA environment.

It is subjected to real Variety-Velocity-Veracity of live data.  

It starts creaking & showing a slew of performance issues, slow responses, dropped requests, DB performance drops, Queues choked, etc. 

And, guess what??

These symptoms were never visible in the test labs. 

The system is live and being used by customers and/or end-users. 

There is no option of now introducing debug builds, debugging, or any other intrusive mechanisms, which would further aggravate the already deteriorating end-user experience.

How do we diagnose and fix them?? 

Sensors give you a starting point for diagnosis and analysis. They capture application-specific events, errors, important artifacts like Q lengths, Q wait sizes, request stats (size, latencies, distribution), cache sizes, cache effectiveness, DB hits, etc.

Application Sensors used for specific application counters, have to be identified and planted in the code at the right locations. If done right, they can give you invaluable information about what is happening inside the system.

Do not mistake Application Health Sensors for your diagnostic verbose logging, these are needed in addition to your diagnostic logging.