Allen Wang
Allen Wang
Senior Software Engineer
Allen Wang is with Netflix Real-time Data Infrastructure team and a contributor to Apache Kafka and NetflixOSS. He is currently leading the effort of Kafka development and operations and responsible for managing thousands of Kafka brokers in in Amazon AWS. Before joining the team, he contributed significantly to Netflix cloud platform and infrastructure and was responsible for open sourcing Archaius and Ribbon - two NetflixOSS projects. At leisure he likes to travel and playing chess with kids.
Kafka @Scale in Cloud

We recently changed Netflix real time data infrastructure to use Kafka as the gateway for data collection and broker for messaging service. Our infrastructure processes hundreds of billions of messages daily on thousands of Kafka instances in AWS cloud. This session will discuss the following - Use cases and motivation of moving to Kafka - Architecture of Netflix real time data infrastructure - Challenges in deploying Kafka in AWS and Netflix eco system - Deployment strategy - Tools and enhancements we created in both brokers and clients - Methodology, tooling and metrics for broker and consumer monitoring - When things go wrong - our Kafka failover strategy We will also share our vision to build Messaging As A Service on top of Kafka which serves as the backbone for most of our messaging and real time data needs.