Monal Daxini leads the Stream Processing team at Netflix building a scalable and multi-tenant real-time event processing pipeline. He has worked on Netflix's Cassandra & Dynamite infrastructure, and was instrumental in developing the encoding compute infrastructure for Netflix content. He has 15 years of experience building distributed systems at organizations like Netflix, NFL.com, and Cisco.
The need for gleaning answers from data in real-time from unbounded data streams is moving from nicety to a necessity. Netflix is a data driven company and Netflix Keystone Stream Processing Service processes over a Trillion messages a day with at-least once processing semantics in the cloud to derive insights.
To ease extracting intelligence from unbounded streams easily we are building a multi-tenant, self-serve, real-time event processing service infrastructure so that the user can focus on extracting value and not have to worry about building infrastructure. We will share our experience
- Building the service in AWS by leveraging open source software like Kafka, Samza, Flink, Mesos, and Docker
- How to evolve the service inflight without downtime
- Lessons learnt along the way