The God is in the details

The God is in the details

Scalability in prodcut development-Coreview Systems

The system crashed after 7 days of full load testing!

Experts scrambled on-site to hunt the problem; it was a memory issue. The issue was fixed and all the test engineers were called in to reproduce the issue. 24 hours passed, there was no crash.

It crashed again with a full load, this time after 10 days; frustrating yet intriguing!

Our decision of testing the system at full load for a significant amount of time before it was put into production was paying off.

After a meticulous diagnosis, a memory leak was detected that was less than 100 bytes. This would never manifest as a crash during lab testing.

This explains the gravity of getting into finer details, especially when you are building on a large scale.

If the scalability target is a few billion requests/day, every minuscule problem is amplified!

Here are some of the things to watch.

– Eliminating unwanted data bytes/requests could reduce significant bandwidth utilization.

– A small memory leak due to an insignificant resource/request could bring the server down in a few hours; which is difficult to detect during lab testing.

– Small optimizations like eliminating small queries/requests can save billions of cycles of your DB CPA. Freeing up more capacity allows you to handle more requests.

How do you test for scalability?

Share this post