Data platform stabilisation and improvement for online gift retailer

We conducted an analysis of the data platform, identifying and removing redundant or obsolete pipelines to reduce the size of the problem space. Monitoring was then introduced to examine the performance of each pipeline and to target optimisation work for maximum benefit.

The data pipelines were running on bespoke, customised infrastructure, which made maintenance and disaster recovery difficult and time consuming. To mitigate this we migrated to newer versions of key elements of the cloud infrastructure, which also reduced the attack surface area and improved security. We then built a robust scheduling solution with AWS Serverless, improving reliability and enabling faster delivery of key reports. Finally, we introduced best-practice DevOps processes to the business and introduced lifecycle management to AWS components, reducing infrastructure costs.

In six months and with a team of four consultants, Infinity Works:

improved stability and performance. All mission-critical dashboards and reports are now available before 9am, including during peak periods;
implemented lifecycle management and reduced AWS costs. Management overhead for pipeline support has also been reduced, resulting in further cost savings and increased capacity for building platform enhancements;
reduced the number of operational incidents, turning a full-time support task into an exceptional occurrence. The result is that teams are now spending more time delivering business value, and less time dealing with issues;
improved insight into business operations, enabling better and more rapid decision making, especially during key peak periods;
introduced operational and infrastructure capabilities to the data engineering team, providing the ability to rapidly iterate on ideas with solutions that span both the data space as well as the infrastructure that runs it.

Impact case study

Data