This is a guest post from AppLovin's VP of engineering, Basil Shikin, on the infrastructure of its mobile marketing platform. Major brands like Uber, Disney, Yelp and Hotels.com use AppLovin's mobile marketing platform. It processes 30 billion requests a day and 60 terabytes of data a day.

AppLovin's marketing platform provides marketing automation and analytics for brands who want to reach their consumers on mobile. The platform enables brands to use real-time data signals to make effective marketing decisions across one billion mobile consumers worldwide.

Core Stats

30 Billion ad requests per day
300,000 ad requests per second, peaking at 500,000 ad requests per second
5ms average response latency
3 Million events per second
60TB of data processed daily
~1000 servers
9 data centers
~40 reporting dimensions
500,000 metrics data points per minute
1 Pb Spark cluster
15GB/s peak disk writes across all servers
9GB/s peak disk reads across all servers
Founded in 2012, AppLovin is headquartered in Palo Alto, with offices in San Francisco, New York, London and Berlin.

Technology Stack

Third Party Services

Github for code
Asana for project management
HipChat for communication

Data Storage

Aerospike for user profile storage
Vertica for aggregated statistics and real-time reporting
Aggregating 350,000 rows per second and writing to Vertica at 34,000 rows per second
Peak 12,000 user profiles per second written to Aerospike
MySQL for ad data
Spark for offline processing and deep data analysis
Redis for basic caching
Thrift for all data storage and transfers
Each data point replicated in 4 data centers
Each service is replicated at least in 2 data centers (at most in 8)
Amazon Web Services used for long term data storage and backups

Core App And Services

Custom C/C++ Nginx module for high performance ad serving
Java for data processing and auxiliary services
PHP / Javascript for UI
Jenkins for continuous integration and deployment
Zookeeper for distributed locking
HAProxy and IPVS for high availability
Coverity for Java/C++ static code analysis
Checkstyle and PMD for PHP static code analysis
Syslog for DC-centralized log server
Hibernate for transaction-based services

Servers and Provisioning

Ubuntu
Cobbler for bare metal provisioning
Chef for configuring servers
Berkshelf for Chef dependencies
Docker with Test Kitchen for running infrastructure tests
A combination of software (ipvs, haproxy) and hardware load balancers. Plan to gradually move away from traditional hardware load balancers.

Monitoring Stack

Server Monitoring

Icinga for all servers
~100 custom Nagios plugins for deep server monitoring
550 various probes per server
Graphite as data format
Grafana for displaying all graphs
PagerDuty for issue escalation
Smokeping for network mesh monitoring

Application Monitoring

VividCortex for MySQL monitoring
JSON /health endpoint on each service
Cross-DC database consistency monitoring
9 4K 65” TVs for showing all graphs across the office
Statistical deviation monitoring
Fraudulent users monitoring
Third-party systems monitoring
Deployments are recorded in all graphs

Intelligent Monitoring

Intelligent alerting system with a feedback loop: a system that can introspect anything can learn anything
Third-party stats about AppLovin are also monitored
Alerting is a cross-team exercise: developers, ops, business, data scientists are involved

Architecture Overview

General Considerations

Store everything in RAM
If it does not fit, save it to SSD
L2 cache level optimizations matter
Use right tool for the right job
Architecture allows swapping any component
Upgrade only if an alternative is 10x better
Write your own components if there is nothing suitable out there
Replicate important data at least 3x
Make sure every message can be re-played without data corruption
Automate everything
Zero-copy message passing

Message Processing

Custom message processing system that guarantees message delivery
3x replication for each message
Sending a message = writing to disk
Any service may fail, but no data are lost
Message dispatching system connects all components together, provides isolation and extensibility of the system
Cross-DC failure tolerance

Ad Serving

Nginx is really fast: can serve an ad in less than a millisecond
Keep all ad serving data in memory: read only
jemalloc gave a 30% speed improvement
Use Aerospike for user profiles: less than 1ms to fetch a profile
Pre-compute all ad serving data on one box and dispatch across all servers
Torrents are used to propagate serving data across all servers. Using Torrents resulted in 83% network load drop on the originating server compared to HTTP-based distribution.
mmap is used to share ad serving data across nginx processes
XXHash is the fastest hash function with a low collision rate. 75x faster than SHA-1 for computing checksums
5% of real traffic goes to staging environment
Ability to run 3 A/B tests at once (20%/20%/10% of traffic for three separate tests, 50% for control)
A/B test results are available in regular reporting

Data Warehouse

All data are replicated
Running most reports takes under 2 seconds
Aggregation is key to allow fast reports on large amounts of data
Non-aggregated data for the last 48 hours is usually to resolve most issues
7 days of raw logs is usually enough for debug
Some reports must be pre-computed
Always think multiple data centers: every data point goes to a multiple locations
Backup in S3 for catastrophic failures
All raw data are stored in Spark cluster

Team

Structure

70 full-time employees
15 developers (platform, ad serving, frontend, mobile)
4 data scientists
5 dev. ops.
Engineers in Palo Alto, CA
Business in San Francisco, CA
Offices in New York, London and Berlin

Interaction

HipChat to discuss most issues
Asana for project-based communication
All code is reviewed
Frequent group code reviews
Quarterly company outings
Regular town hall meetings with CEO
All engineers (junior to CTO) write code
Interviews are tough: offers are really rare

Development Cycle

Developers, business side or data science team comes up with an idea
Idea is reviewed and scheduled to be executed on a Monday meeting
Feature is implemented in a branch; development environment is used for basic testing
A pull request is created
Code is reviewed and iterated upon
For big features group code reviews are common
The feature gets merged to master
The feature gets deployed to staging with the next build
The feature gets tested on 5% real traffic
Statistics are examined
If the feature is successful it graduates to production
Feature is closely monitored for couple days

Avoiding Issues

The system is designed to handle failure of any component
No failure of a single component can harm ad serving or data consistency
Omniscient monitoring
Engineers watch and analyze key business reports
High quality of code is essential
Some features take multiple code reviews and iterations before graduating
Alarms are triggered when:
- Stats for staging are different from production
- FATAL errors on critical services
- Error rate exceeds threshold
- Any irregular activity is detected
data are never dropped
Most log lines can be easily parsed
Rolling back of any change is easy by design
After every failure: fix, make sure same thing does not happen in the future, and add monitoring

Lessons Learned

Product Development

Being able to swap any component easily is key to growth
Failures drive innovative solutions
Staging environment is essential: always be ready to loose 5%
A/B testing is essential
Monitor everything
Build intelligent alerting system
Engineers should be aware of business goals
Business people should be aware of limitations of engineering
Make builds and continuous integration fast. Jenkins run on a 2 bare metal servers with 32 CPU, 128G RAM and SSD drives

Infrastructure

Monitoring all data points is critical
Automation is important
Every component should support HA by design
Kernel optimizations can have up to 25% performance improvement
Process and IRQ balancing lead to another 20% performance improvement
Power saving features impact performance
Use SSDs as much as possible
When optimizing, profile everything. Flame graphs are great!

On Hacker News