The timely loading of real-time data from your on-site or cloud-based mission-critical operational systems to your cloud-based analytical systems is assured by a continuous streaming ETL solution. The data loaded for making crucial operational decisions should be reliable due to continuous data flow. By supplying an efficient, end-to-end data integration between the source and the target systems, Striim can guarantee the dependability of the stream ETL solutions. To ensure data reliability and send it to the target systems, the data from the source can be transformed in the real-time data pipeline. Striim applications can be created for a variety of use cases.

About the customer

A power company called Glitre Energi manages the power grid, retails electricity, and offers broadband services. About 90,000 people receive electricity from Glitre Energi. The organization oversees the power lines that pass through heavily populated areas.

Problems with the current design

• Metering data should be loaded to the SQL databases from event-based sources.
• Regardless of any additional parameters in the source events, metering events with the same filename should have the same ID assigned to them.
• Relational database systems have trouble normalizing real-time metering events. Unless all of the previous events have been sent to the target, comparing data in real-time and assigning values becomes difficult.

Solution Provided

• Meter values for the power supply are sent as JSON files from the source applications, which are referred to as meter events, to Azure Event Hubs.
• Due to reporting lags, each file contains n number of events with various timestamps.
• Each event must maintain the link to the file in which it was received in order to maintain traceability back to the source.
• These events are sent to two SQL server tables, one of which contains information about metering and the other of which contains information about metering files.

Also Read: Use Cases for a Unified Data Integration and Streaming Platform like Striim

Usage Of Constituents

Cache

Getting the most unique identifier from the target table is made possible by a memory-based cache of non-real-time historical or reference data that was obtained from an external source.

External Cache The need for data prompts Striim to query an external database. In order to determine whether the incoming data is present in the target table already or not, Striim queries the same data when it joins with real-time data.

Windows

By limiting the data set by a specific number of events, period, or both, Windows will aggregate, join, or perform calculations on the data. This aids in bringing the target database data and real-time data together in one location where the downstream pipeline can carry out the transformations.

Continuous Query

A continuous query that can be used to filter, aggregate, join, enrich, and transform the events specifies the logic of an application. A query facilitates the logic in the data pipeline and the combining of data from various sources.

The use case’s high-level representation is shown in the image below:

Flow Insights

• Striim application must identify the event with new files, get the filename, assign a unique integer Id, and store these values in a separate table in the SQL server database.
• For each event that is processed, the application queries an external cache to see if the filename already exists in the target table.
  • If it exists, the CQ retrieves the Id for that filename, replaces the id value with the incoming event data, and sends it to the target table.
  • If it doesn’t exist, the CQ will increment the id and assign the id to the new filename and send the data to both the target tables.
• Striim cache can be used to load the last received filenames and IDs so that the ID can be incremented.
• Striim cache should be updated regularly depending on what frequency each event has been sent to the target tables, but it effectively needs to be mutable.
• Striim windows help bound the real-time event data and the file data, so the continuous queries can use these data and make decisions accordingly.

Conclusion

Using continuous query components has made it simple to compare event data in real-time load to reach decisions. With the aid of windows and cache components, efficient data that must be retrieved from external sources has been planned out very well. The beauty of Striim allows data to be joined wherever it is needed and the desired output to be obtained, assisting Glitre Energi in achieving the normalization of their metering events in their relational systems.

Please get in touch with us if you still need help or if your needs are still unclear. Our team of professionals will always be available to assist you. Click to do so.

The post Certainty in streaming real-time ETL appeared first on Indium.

Problem Statement 

As we all know, going through each stage to board a flight at an airport can be stressful. As shown in the image, there are typically five steps in the processing process: obtaining a boarding pass, bagging, security inspection, immigration inspection, and finally boarding. To prevent boarding or flight delays, all five of these processes should be finished within a SLA established for each phase. Internally, airport authorities use the message queueing system to populate each stage of the event, but the challenge would be to spot SLA breaches in real-time and report them to address the delay.

Process of Boarding at the Airport

The procedure would be monitored using MQ systems, where each step emits an event with the flight and passenger information. Finding the stage that exceeds the SLA required to pass a certain stage of checking is the challenge here. If any of the processes were to miss the SLA cut-off, the following effects would result.

1. Delaying the current flight and any subsequent flights
2. Panic among the passengers
3. Longer wait times.
4. A process could be missing.
5. There would be a security breach as a result.

The illustration below shows how MQ events are produced as passengers go through the airline’s onboarding procedure.

How does Striim contribute to process improvement?

Striim is a real-time replication tool that enables data streaming from a variety of source systems and aids in event migration to the target systems. Windowing technique is one of its key features, and it can hold data based on options like time, count, and fields to process on-the-fly. The best course of action in this situation is to hold/cache the event until the SAL/cut-off for the boarding process is determined. We can review the events that have occurred after the deadline to force the airport authorities to act right away from the cache. Assumedly, the events that are generated at each stage include the passenger’s boarding and flight information so that the specific passenger can be tracked throughout the boarding process.

Also Read: Use Cases for a Unified Data Integration and Streaming Platform like Striim

Striim Windowing Techniques

Real-time data is constrained within a window by time (for instance, five minutes), event count (for instance, 10,000 events), or both. The creation of a window is necessary for a replication flow to aggregate or process data, fill the dashboard, or send alerts when conditions deviate from expected ranges. An application can only evaluate and respond to individual events without a window to bind the data.

The three types of windows that Striim supports are sliding, jumping, and session windows. When a query’s contents change (sliding), expire (jumping), or there has been a lull in use activity, Windows sends data to the queries that follow (session). Jumping windows, which are regularly updated with an entirely new set of events, are the best fit for our use case out of these three types. Data sets for the hours of 8:00 am–8:04:59 am, 8:05 am–8:09 am, and so on would be produced, for instance, by a five-minute jumping window. A new data set would be produced by a 10,000-event jumping window after each 10,000 events. The window would output a new data set each time it accumulated 10,000 events or five minutes had passed since the previous data set was output if both five minutes and 10,000 events were specified. With the help of this Windows feature, we are putting forth an architecture that will both capture events coming from MQ systems and those that are approaching the cut-off time.

Proposed Architecture for the Airline On-Boarding System

With this suggested architecture, the airline onboarding processing can detect an SLA breach during passenger check-in for a flight with speed and accuracy. By storing the data in a designated window, it operates using the caching technique. The Striim partitioning feature enables us to classify every passenger according to their boarding pass number, allowing us to identify anyone having trouble during the flight. Striim’ s SQL-like queries are used to group and aggregate the events from jumping windows for each stage, from checking in to boarding flights.

CREATE OR REPLACE JUMPING WINDOW Boarding_Data_Window OVER admin.Boarding_Data_Win KEEP 2 ROWS WITHIN 90 SECOND PARTITION BY BoardingPassNo;

CREATE OR REPLACE CQ Passenger_Data_Boarding INSERT INTO admin.NotBoarded SELECT p.flightNo as flightNo, p.boardingPassNo as boardingPassNo , case when b.boardingPassNo is null then “Not Boarded” else “Boarded” END as BoardingStatus, b.boardingPassNo as bagBoardingPassNo FROM Boarding_Data_Window b right join PassengerDataWindow p on p.boardingPassNo = b.boardingPassNo;

CREATE OR REPLACE CQ NotBoarded INSERT INTO admin.NotBoardedResult SELECT * FROM NotBoarded n where BoardingStatus=”Not Boarded” group by n.boardingPassNo having count(*) <2 ;

Here are some benefits of using Striim as a replication tool in this scenario to record and modify SLA beaches during the flight boarding:

1. Real-time data collection that aids in processing the event at every stage.

2. Windowing the events until the designated interval to process and modify.

3. The dashboard and alerting system provides a nearly real-time progress of each passenger’s stages.

4. Quick fixes considerably shorter airport wait times and delays.

5. More accurate reporting.

An elaborate use case for Striim services: Striim-Powered Real-Time Data Integration of Core Banking System with Azure Synapse Analytics

Conclusion

The lengthy part of flying is waiting in line for the boarding process due to the densely populated airport. A more effective tracking system offers a practical way to track individual passengers for a comfortable journey. Using Striim’s windows technique, we can process and change airport authorities at any stage of the boarding process by holding every individual passenger detail in-memory directly from the real-time queuing system. Additionally, Striim aids in the migration of events to alternative target systems for better visual representations.

The Striim experts at Indium have cross-domain experience and can create solutions specifically for each of our clients’ individual needs.

The post Spot Realtime SLA Breaches in Airline On-Boarding Process Using Striim appeared first on Indium.