![How PostNL processes billions of IoT events with Amazon Managed Service for Apache Flink | S02EP21 | Lets talk about data show](https://assets.community.aws/a/2h9HrtnZ8KkZxANk8tB0RQbwRnT/Scre.webp)
How PostNL processes billions of IoT events with Amazon Managed Service for Apache Flink | S02EP21 | Lets talk about data show
In this Twitch session, we talk about the journey of PostNL adopting Apache Flink and practical insights on using Amazon Managed Service for Apache Flink.
Lydia Ray
Amazon Employee
Published May 29, 2024
PostNL, the universal postal provider throughout Belgium, the Netherlands, and Luxembourg, delivers an average of 1.1 million parcels and 6.9 million letters daily. Utilizing an IoT platform, PostNL tracks hundred thousands roller cages across its fulfilment centres, generating billions of raw events each day. To manage this vast amount of data, PostNL employs stream processing techniques to filter and aggregate events.
In this show, we discuss about PostNL's journey migrating their massive IoT platform to Amazon Managed Service for Apache Flink to handle billions of sensor events daily from their logistics operations. One of the key challenge on migration was reproducing the exact behavior of their legacy system to maintain downstream dependencies. Flink's flexible APIs enabled them to leverage the lower-level abstractions to customize the windowing and late event handling logic. The managed service allowed them to focus on core business logic rather than infrastructure management. They employed a phased migration strategy starting with running both old and new systems in parallel before the final switch over. Key lessons learned included optimizing understanding of event time semantics, monitoring for state, and leveraging Flink's local debugging before deployment. The managed service simplified development and autoscaling while ensuring high availability and optimized resource usage.
Some of the key highlights are:
- Leveraging Apache Flink's low level APIs allows greater customization and flexibility.
- Follow a phased migration strategy running dual systems to minimize downtime.
- The granularity of API levels allow flexibility on implementation with Apache Flink.
- Amazon Managed Service for Apache Flink allows developers focus on application code rather than infrastructure.
- Leverage autoscaling on Amazon Managed Service for Apache Flink to optimize resource usage.
Loading...
Hosts of the show 🎤
Lydia Ray - Sr Analytics Solutions Architect @ AWS
Guests 🎤
- Çağrı Çakır - Lead Software Engineer @ PostNL
- Amit Singh - Senior Solution Architect @ AWS
- Lorenzo Nicora - Sr Streaming Solution Architect @ AWS
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.