How PostNL processes billions of IoT events with Amazon Managed Service for Apache Flink | S02EP21 | Lets talk about data show

PostNL, the universal postal provider throughout Belgium, the Netherlands, and Luxembourg, delivers an average of 1.1 million parcels and 6.9 million letters daily. Utilizing an IoT platform, PostNL tracks hundred thousands roller cages across its fulfilment centres, generating billions of raw events each day. To manage this vast amount of data, PostNL employs stream processing techniques to filter and aggregate events.

In this show, we discuss about PostNL's journey migrating their massive IoT platform to Amazon Managed Service for Apache Flink to handle billions of sensor events daily from their logistics operations. One of the key challenge on migration was reproducing the exact behavior of their legacy system to maintain downstream dependencies. Flink's flexible APIs enabled them to leverage the lower-level abstractions to customize the windowing and late event handling logic. The managed service allowed them to focus on core business logic rather than infrastructure management. They employed a phased migration strategy starting with running both old and new systems in parallel before the final switch over. Key lessons learned included optimizing understanding of event time semantics, monitoring for state, and leveraging Flink's local debugging before deployment. The managed service simplified development and autoscaling while ensuring high availability and optimized resource usage.

Some of the key highlights are:

Lydia Ray - Sr Analytics Solutions Architect @ AWS