Tech Web Support delivered data platform engineering and development services building an analytics platform that processes 2.5 million events per second and is used by over 30K companies globally.
CUSTOMER: AOL
One of the world’s biggest advertising media conglomerates.
- BUSINESS NEEDSIn 2014 Aol. acquired a video ad platform Tech Web Support had been working on. The new owner needed help with adjusting the solution to the Aol. ecosystem and ensuring that the platform can process highly increased data volumes.
- RESULTOur Big Data engineers elaborated an efficient high-load data system with AWS Cloud-based infrastructure, microservice architecture, and custom reporting tool. The product is used worldwide providing a 24/7 accurate & scalable ad serving environment and processing 26TB of data daily.
MASTERING SUCH MASSIVE DATA VOLUMES IS AN ART RATHER THAN JUST AN ENGINEERING TASK. THE PROJECT WAS A GREAT CHALLENGE AND A FANTASTIC EXPERIENCE FOR OUR DATA ENGINEERS & DEVELOPMENT TEAM.
COLLABORATION OVERVIEW
Key Facts:
The innovative advertising management platform we developed for a start-up called Vidible was acquired by Aol. in 2014. As a Vidible core technology partner, we continued working with Aol. and focused on adapting the platform to the increased loads gradually turning it into the enterprise-scale ad platform.
We created a new Big Data analytics platform and an advanced ad hoc reporting system based on the microservice architecture and AWS Cloud infrastructure. Our team continued working on the project through all the mergers, acquisitions & rebranding (Vidible-Aol.-Oath-Verizon Media).
- Over 2.5M events processed per second
- Smart real-time data analytics and fast custom reporting solution based on over 400 metrics (it takes only 1,5 minutes to generate a report)
- Custom service for automatic best-fit ad selection based on various tags and 100 additional metrics (i.e. theme relevance, popularity)
- Flexible campaigns configuration allowing to easily modify ad units & include new features on the fly
- Rich ad-in-ad technology enabling pre-rolls, interactive mid-rolls, synchronized banners, etc. to be included into campaign itself
Data Platform & Reporting:
The video analytic platform was expected to serve millions of users & billions of video plays, thus the Vidible Data Platform and the reporting layer could no longer cope with the task.
Our team focused on redesigning the AWS Big Data platform’s architecture to reduce the data latency and be able to process bigger data volumes in a near real-time manner. As a result, our experts safeguarded that the data management platform could efficiently handle up to 120TB daily.
- Data warehouse: we redesigned the real-time Data Platform decreasing the data latency from 2h. to 5 min., enabling the ad management platform to process 2.5M events per second, ensuring data governance, smart monitoring & alerting over the infra and platform components
- Ad-hoc reporting solution: our team developed an advanced campaign analytics solution that ensured near real-time reports delivery to the end-users allowing them to keep up with the recent forecasts & trends; the newly developed custom reporting software safeguarded 24/7 reporting services and reduced report design time from days to minutes
Cloud Infrastructure:
The custom analytics platform infra has initially been designed according to the best AWS & DevOps practices and relies on IaC (Infrastructure-as-Code) approach making the Big Data real-time analytics solution flexible and very easy to scale.
After the acquisition by Aol. the BI system infrastructure easily adapted to the drastically increased loads and we continued hardening and fine-tuning it along the way for better availability, security, and robustness.
- Infrastructure built on top of the Amazon Elastic Compute Cloud (EC2) service to ensure fault-tolerance and availability
- AWS Elastic Load Balancer used to automatically spread and drive traffic to the different EC2 instances thus ensuring horizontal scalability for all services
- Amazon Virtual Private Cloud and Availability Zones are implemented and configured to safeguard high security of the whole system and support diversity of services & environments
- Different databases used to handle different types of data (Redshift, RDS, DynamoDB) with Athena DB used as a master backup