Photobox is a leading ecommerce retailer of photo printing services and personalised gifts. As Photobox approached their peak period in the run-up to Christmas 2019, they wanted to ensure that up-to-date marketing data was constantly available to maximise their marketing and sales conversion.

The pipelines feeding this data warehouse did not complete before the start of the working day, delaying critical decisions. To further exacerbate the problem, pipelines would often fail, requiring manual intervention to fix, further delaying the availability of important business metrics.

Photobox brought in Infinity Works to help resolve these issues. The solution implemented by Infinity Works included building a more robust scheduling solution using AWS serverless components, improving reliability and allowing the schedules to be tightened.

These changes ensure that key business reports are delivered before the start of the working day. Further best practices were added or improved on, such as DevOps processes, monitoring and alerting, and instance lifecycle management. This solution used a wide variety of AWS components to easily plug functionality together and get the best out of the ecosystem for minimal implementation effort and cost.

With this improved stability and performance, Photobox was able to achieve a very positive business result and is now able to invest more of their time building on the future roadmap and less time supporting pipelines.

Photobox’s business intelligence platform is used to drive customer behaviour and marketing analytics, financial planning, and ad-hoc discovery. Photobox had a large number of data pipelines (over 160) feeding their central business intelligence platform and analytics teams.

A data pipeline is a series of steps used to extract data from source systems, carry out filtering to remove invalid or inconsistent data, transform the data (for example, by enriching postcode data to include latitude and longitude, or by aggregating data), and load it into a target system.

The challenge was that a number of Photobox’s data pipelines were performing increasingly poorly and were unreliable. Photobox were concerned that given the highly seasonal nature of their business, they would not have access to reliable data during peak periods if the performance and reliability issues were not resolved.

Photobox wanted to have access to up-to-date management information at the start of business each day and tasked Infinity Works with achieving this objective.

Photobox’s data pipelines were mainly written in the Python programming language, some running on an outdated version of Amazon’s Elastic Map/Reduce (EMR) product that only supported running data analytics within public Virtual Private Cloud (VPC) subnets. Migrating to a newer version would improve the security of the solution, because newer versions of EMR support private subnets – used to reduce the attack surface area of a solution.

The pipelines were running on AWS Elastic Compute (EC2) instances that had been built manually, and highly customised. Maintenance and disaster recovery of the instances was difficult and time consuming. Infinity Works wanted to make the pipelines reliable and performant and to reduce the amount of time spent managing pipeline failures to achieve this goal.

The mechanism used to schedule jobs was a simple timer (cron job in Linux). This meant that data pipeline dependencies were not managed automatically and there was no automatic retry on failures. A failure of a pipeline would cause a cascading failure and delays, resulting in time being wasted on manual recovery of failures.

Photobox had previously made the decision to build on the AWS ecosystem and had many of its data source systems hosted within AWS, built using AWS components. Both Photobox and Infinity Works teams have a lot of experience building large, scalable systems in AWS and are confident in the ability of AWS to scale to meet demand.

By building this system using AWS components, we were able to enjoy the benefits of integrated components such as authentication and authorization using Identity and Access Management (IAM) and the high scalability and availability of serverless components such as S3, Fargate and Cloud Watch. All of these benefits together meant that we could use a much simplified architecture to transfer data securely and at high speed, when compared to multi-cloud or split cloud and on-premises setups.

By being able to show the end-to-end authentication and authorization flow with IAM, we were able to satisfy internal security and compliance requirements much more easily than using multiple product sets from different vendors, each with distinct authentication mechanisms.

Prior to this engagement, Infinity Works had successfully completed a cloud migration project for another company in Photobox’s group. This project saw Infinity Works using a DevOps culture and approach to help achieve a complex platform migration from an on-premises data centre to AWS, meeting time and budget requirements, while increasing the performance of the solution.

Due to the size, complexity and age of the starting solution, it was not initially clear exactly how the challenge would be met, and some experimentation would be required to find the optimal solution. Flexibility of approach – the ability to pick a solution to a challenge using a wide variety of tools including software, infrastructure, 3rd party services or business process change – was important.

Photobox selected Infinity Works to take on this challenge because they trusted us to be able to work in an agile, DevOps way. Working in an agile way, and maintaining a strong DevOps culture is important to Photobox because that is how they work internally.

Infinity Works proposed first carrying out initial analysis of the pipelines, finding and removing redundant or obsolete pipelines to reduce the size of the problem space. This would reduce the effort required to introduce a better scheduling system to provide retries, dependency management, monitoring and alerting. With monitoring in place, it was then possible to examine the performance of each pipeline and target performance optimization work on the most valuable pipelines.

To improve the scheduling and management of pipelines, Infinity Works proposed the use of Airflow. Airflow is an open source data orchestration tool that can be used for pipeline scheduling, dependency management, and execution of tasks. Airflow was selected, because it is already in use at Photobox, and only required a small amount of customization to meet the requirements.

Airflow has built-in capabilities that make use of the Postgres relational database to store its internal state. Infinity Works felt that this was a good fit for this project, using AWS RDS (Relational Database Service) to host the database engine in order to reduce management overhead (e.g. configuring encryption, backups, clustering etc.).

Figure 1: Airflow UI showing a selection of scheduled data pipeline tasks (pipeline names hidden).

Docker containers are used to build and deploy the Airflow setup. To build these containers, AWS build tooling (Code Pipeline, Code Build and Code Deploy toolchain) were used. The AWS build tooling is well integrated with AWS systems and uses Identity and Access Management (IAM) roles to ensure that pipelines meet the principle of least privilege – i.e. have the minimum amount of permissions required to do their job.

To run the Airflow cluster, AWS Elastic Container Service (ECS) using both Elastic Cloud Compute (EC2) and Fargate were selected as a compute platform.

ECS was selected because it is a flexible platform that allows for scaling up and down of dynamic workloads, allowing Photobox to only pay for what they used.

Fargate is a Serverless container platform that allows a Docker container to be executed on demand. Fargate reduces the amount of time required to manage compute infrastructure, because the underlying operating system is managed by AWS.

For Photobox and Infinity Works, the benefits of Infrastructure as Code – audit, reuse, change control, ability to create test environments and to recover from disasters – mean that developing using Infrastructure as Code techniques is a default position. To not use Infrastructure as Code techniques would be unusual. In this case, Photobox were already using CloudFormation for their projects, and Infinity Works have a lot of experience using the tool.

To monitor the platform and pipelines, Infinity Works proposed the use of CloudWatch. CloudWatch support is built-in to ECS, making the integration simple and cost-effective. Airflow publishes StatsD metrics, and Infinity Works proposed the use of a sidecar container – a container specifically designed to monitor the Airflow container – hosted CloudWatch Agent to gather metrics from the running Airflow cluster and feed these back to CloudWatch.

Dashboards were created in CloudWatch to provide a visual representation of system health.

CloudWatch alarms notify operators about resource usage on Fargate containers. This allows operators to resize container instances appropriately to the CPU and memory requirements of each data pipeline.

DevOps Adoption

Infinity Works helped Photobox further improve their DevOps processes within the Data Engineering space by implementing best practices in Infrastructure as Code and CI/CD by the use of Infrastructure as Code to manage the Airflow cluster.

Infinity Works produced a report containing recommendations to the wider Data Engineering team on where the principle of least privilege, Role Based Access Control (RBAC), and AWS account separation could be used to improve the security of the solution without impacting the solution.

To improve visibility of the health of operational systems, Infinity Works implemented structured logging using CloudWatch logs. This enabled the use of CloudWatch Insights to query and analyse logs data in real-time. Processing and pipeline alerts were connected to Slack channels in order to prompt engineering teams to take action in the case of delays or failures of data pipelines.

With legacy infrastructure, it is unfortunately not always possible to fully utilise infrastructure-as-code and CI/CD to tear down and spin up infrastructure on demand. Some older instances running in EC2 were highly customised and could not easily be rebuilt from a template. For these situations AWS EC2 Lifecycle Manager was used to protect the instances, providing backups, rollback capabilities and the ability to easily recover from disasters, whilst causing minimal change to the instances themselves, thus reducing risk of introducing unforeseen issues in unknown code.

Some data is loaded in bulk (e.g. initial ingests or backfills) using AWS Batch to perform transformations before writing output to S3 for later ingestion. In order to make this modular, extensible, and easy to maintain, Infinity Works implemented a CI/CD pipeline to produce new Docker images for use with AWS Batch.

Figure 2: Extract of architecture for bulk loading data

Performance and reliability were greatly improved. By improving pipeline dependency management, approximately 2 hours and 30 minutes of wasted time during pipeline schedules was eliminated, reducing the duration of pipeline orchestration. The result was that all mission-critical dashboards and reports are now available before 9am throughout peak periods, where some dashboards would previously come in around 12 noon.

The cost of running infrastructure has remained the same, however as management overhead for the pipelines has gone down, this has resulted in cost saving and the ability to redirect this effort to building enhancements instead of fixes.

The improved platform has provided Photobox with improved insight into business operations that allows them to make better business decisions during key periods. This has contributed to Photobox’s excellent business performance over the period.

The number of operational incidents is greatly reduced; where previously this was a near full-time job, this is now an exceptional occurrence. The result is that teams are now spending more time delivering business value, and less time dealing with issues.

One of the key lessons we have taken away from this project is the importance of including operational and infrastructure capabilities in a data engineering team. The ability to rapidly integrate these disciplines has been invaluable to delivering benefits quickly. DevOps is a mentality that can easily be brought into a data engineering capability and will quickly demonstrate benefits.

Next steps

Infinity Works has continued working with Photobox to help build a new architecture for data management that standardizes data structures and events emitted by operational systems. This will allow the simplification of data pipelines, and enable transactional systems to increase their rate of change without affecting the availability of data, and will further reduce the costs of operating and managing the data pipeline.

Photobox: platform stabilisation

Data platform stabilisation