Performance and Scalability Engineering
Customer Challenges:
- Highly complex and interdependent internal structure of the source (PST) data, made it difficult to create parallel threads for execution
- Speed of archival – as new data was generated and archived on a continuous basis in a time critical manner
- Need for a platform agnostic solution, to reduce license cost to customers
Result:
- 25X better performance of the module
- Linearly scalable execution allowed more power
- Able to archive Petabytes of data in a day in real time
- Added facility to measure and monitor progress
- Saved thousands of dollars on distribution license costs
CoreView Solution:
- Redeveloped the specific archiving module using open source tech – Java
- Highly concurrent code
- Scaled using Apache Storm
- Non-IP Contribution to open source libraries
- Heuristics based algorithm to find data/asset ownership
Scope:
- Measure, identify and demonstrate bottlenecks in existing running code
- Complete re-design and development of the module for scalability
- Maintain functional accuracy & demonstrate by comparing with existing results
