This will help to effect continuous improvement in performance and create differentiated experiences for customers and employees. It will also enable accelerated development of innovative new solutions.

McKinsey also identifies the current challenges to optimizing data sources as

• Limited capabilities of legacy technologies
• Challenges in modernizing the architecture.
• Demand for high computational resources for real-time processing jobs

This results in only a small part of the data from connected devices being leveraged. As companies balance speed and computational intensity, they are unable to perform complex analyses or implement real-time use cases.

Getting the right data technologies to ingest, process, analyze, and visualize in real-time is going to be a game-changer in improving decision-making, enhancing customer experience, and accelerating growth.

Improved Decision Making

Real-time data is critical for conducting real-time analytics, which helps with faster decision-making. Data is collected from a variety of sources, including sensors, databases, operational systems, cameras, and social media feeds with minimal delay and processed and analyzed quickly. They could be alerts and notifications or inputs from user behavior.

Real-time data can be of two types:

• Event Data: The generation of a collection of data points based on well-defined conditions within a system.
• Stream Data: The continuous generation of a large volume of data without any identifiable beginning or end.

Easy access to data in real-time data enables a quick drawing of insights to make informed decisions and be responsive as events unfold. It helps with capturing trends, both past, and present, and can be analyzed in real-time to decide on the next course of action.

Some of the benefits of real-time data include

Being Proactive

In the absence of real-time data, there is a lag between insights and responses. This reactive approach can prove costly, resulting in losing customers or production-related issues escalating. Real-time data analytics allows enterprises to proactively approach developments and respond appropriately.

Enhance Customer Experience

Visibility and transparency have become key in several client-business relationships. It helps improve decision-making based on project status and enhances customer experience and retention. Responding to customer requirements and empowering them with information in real-time further strengthens the relationship between the two.

To know more about how Indium can help you, please check out more.

Unify Data

Different teams end up creating data silos to suit their requirements. This can distort the view when making strategic decisions at the enterprise level and delay the process. A cloud-based data streaming solution helps to provide a unified view in real-time while allowing different teams access to secure and permission-based data they need to make decisions for their department.

Improve Operational Excellence

Real-time data allows you to manage your organization’s assets proactively. It lets you plan downtimes for maintenance and repair, improves the life of the assets, and take timely steps to replace, where needed, with minimum disruption to operations. This naturally leads to a better quality of products and services and improved profit margins as it lowers overheads.

Striim Power For Real-time Data Analytics

The Striim unified real-time data integration and streaming platform unifies data across multiple sources and targets. It offers built-in adapters and supports more than 125 sources and targets, enabling the management of multiple data pipelines in a Striim cluster. Striim 4.1 offers features such as OJet to let customer applications read multi-terabytes of data per day and a high-performance Oracle Change Data Capture (CDC) reader. It also sends real-time alerts and notifications to identify emerging workload patterns and facilitates collaboration between developers and database administrators.

Striim users can build smart real-time data pipelines quickly for streaming large volumes of events daily. It is scalable and secure, and the features are highly available. It is easy to maintain and allows the rapid adoption of new cloud models, infrastructure modernization, and digitalizing legacy systems.

Striim enables data integration using a streaming-first approach, supporting incremental, real-time views in the cloud database and the streaming layer. It includes Streaming SQL to facilitate real-time analytics of data, as also train machine learning models in real-time.

Business analysts, data scientists, and data engineers can use Streaming SQL to build data pipelines quickly and without the need for custom coding. Striim also allows data movement in real-time, because of which stream processing applications need to operate continuously for years. These further speeds up decision-making as insights can be drawn quickly, without latency between receiving the data and running analytics on it.

Check out our case study on real-time data analytics

Case and Point: Simplifying Healthcare Predictions in 3 Expert Steps

Understanding Symptom Patterns: Our first step involves data acquisition and thorough analysis of historical patient data. We tap into the treasure trove of symptoms, medical records, and outcomes to discern intricate patterns that might remain hidden from traditional analysis.

Feature Engineering with Domain Knowledge: With a team of domain experts, we transform raw symptom data into meaningful features. These features are carefully curated to capture the nuances of various symptoms, their interplay, and potential implications. Our domain knowledge empowers us to create a robust feature set that forms the foundation of accurate predictions.

Advanced Machine Learning Models: Equipped with a rich feature set, we employ advanced machine learning models. From ensemble methods to deep learning architectures, we evaluate and fine-tune models that can effectively map symptoms to probable outcomes. This step requires rigorous experimentation to ensure optimal model performance. 

The utilization of Symptom Pattern Analysis, Feature Engineering, and Advanced Machine Learning Models in the healthcare domain, along with Indium’s implementation of Striim for real-time data migration and processing, brings substantial and quantifiable business value to the table.

Healthcare Providers: Reduced diagnosis time through rapid predictions – from days to hours, thereby accelerating patient care. Enhanced efficiency with streamlined operations leads to quicker decisions and resource allocation. Improved patient care is achieved through early intervention based on predictions, resulting in improved treatment outcomes. Informed resource allocation provides predictive insights that optimize staff schedules, room usage, and equipment availability. Optimized treatment plans driven by personalized treatments yield better outcomes and patient satisfaction. The Cost savings achieved through fewer hospital stays, reduced redundant tests, and efficient resource use contribute to lowering costs. This not only benefits the patients but also benefits the providers by optimizing their resources.

Healthcare Payers and Insurance Companies: The implementation offers a competitive edge for healthcare providers, attracting patients and enhancing the providers’ reputation due to quick and accurate diagnoses. This, in turn, leads to efficient resource utilization, potentially reducing the overall cost of treatments. Cost savings arising from reduced hospital stays and redundant tests contribute to lower healthcare expenditures, benefiting healthcare payers and insurance companies. Healthcare payers such as insurance companies can also reduce fraudulent claims as they will have access to patient diagnosis history in real-time.

Medical Researchers and Innovators: The curated data fosters research opportunities, facilitating medical insights and potential innovation generation. The advanced analytical capabilities of Symptom Pattern Analysis and Machine Learning Models open avenues for new discoveries and improvements in medical practices, benefiting the broader healthcare research community.

Overall, the integration of advanced technologies, real-time data processing, and predictive analytics in the healthcare domain offers benefits that extend to healthcare providers, payers, patients, and the research community. This synergy drives efficiency, quality of care, and cost-effectiveness, ultimately transforming healthcare delivery and outcomes.

Indium for Instant Decisions with Striim

Indium Software, a cutting-edge solution provider, has deep expertise in Striim implementation and can help businesses create exciting digital experiences for their customers.

A private sector bank offering specialized services to 9 million customers across various business verticals and with a presence global presence required data to be updated in real-time from its core banking systems to a reliable destination database for downstream analytics. By migrating the data from legacy systems to Striim in real time, Indium helped the customer improve its responsiveness and operational efficiency apart from other benefits.

Indium’s team of Striim experts have cross-domain experience and can provide custom-built solutions to meet the unique needs of our customers.

FAQs

The post Accelerating Data-Driven Decisions: Empowering Enterprises with Real-Time Insights using Striim appeared first on Indium.

The timely loading of real-time data from your on-site or cloud-based mission-critical operational systems to your cloud-based analytical systems is assured by a continuous streaming ETL solution. The data loaded for making crucial operational decisions should be reliable due to continuous data flow. By supplying an efficient, end-to-end data integration between the source and the target systems, Striim can guarantee the dependability of the stream ETL solutions. To ensure data reliability and send it to the target systems, the data from the source can be transformed in the real-time data pipeline. Striim applications can be created for a variety of use cases.

About the customer

A power company called Glitre Energi manages the power grid, retails electricity, and offers broadband services. About 90,000 people receive electricity from Glitre Energi. The organization oversees the power lines that pass through heavily populated areas.

Problems with the current design

• Metering data should be loaded to the SQL databases from event-based sources.
• Regardless of any additional parameters in the source events, metering events with the same filename should have the same ID assigned to them.
• Relational database systems have trouble normalizing real-time metering events. Unless all of the previous events have been sent to the target, comparing data in real-time and assigning values becomes difficult.

Solution Provided

• Meter values for the power supply are sent as JSON files from the source applications, which are referred to as meter events, to Azure Event Hubs.
• Due to reporting lags, each file contains n number of events with various timestamps.
• Each event must maintain the link to the file in which it was received in order to maintain traceability back to the source.
• These events are sent to two SQL server tables, one of which contains information about metering and the other of which contains information about metering files.

Also Read: Use Cases for a Unified Data Integration and Streaming Platform like Striim

Usage Of Constituents

Cache

Getting the most unique identifier from the target table is made possible by a memory-based cache of non-real-time historical or reference data that was obtained from an external source.

External Cache The need for data prompts Striim to query an external database. In order to determine whether the incoming data is present in the target table already or not, Striim queries the same data when it joins with real-time data.

Windows

By limiting the data set by a specific number of events, period, or both, Windows will aggregate, join, or perform calculations on the data. This aids in bringing the target database data and real-time data together in one location where the downstream pipeline can carry out the transformations.

Continuous Query

A continuous query that can be used to filter, aggregate, join, enrich, and transform the events specifies the logic of an application. A query facilitates the logic in the data pipeline and the combining of data from various sources.

The use case’s high-level representation is shown in the image below:

Flow Insights

• Striim application must identify the event with new files, get the filename, assign a unique integer Id, and store these values in a separate table in the SQL server database.
• For each event that is processed, the application queries an external cache to see if the filename already exists in the target table.
  • If it exists, the CQ retrieves the Id for that filename, replaces the id value with the incoming event data, and sends it to the target table.
  • If it doesn’t exist, the CQ will increment the id and assign the id to the new filename and send the data to both the target tables.
• Striim cache can be used to load the last received filenames and IDs so that the ID can be incremented.
• Striim cache should be updated regularly depending on what frequency each event has been sent to the target tables, but it effectively needs to be mutable.
• Striim windows help bound the real-time event data and the file data, so the continuous queries can use these data and make decisions accordingly.

Conclusion

Using continuous query components has made it simple to compare event data in real-time load to reach decisions. With the aid of windows and cache components, efficient data that must be retrieved from external sources has been planned out very well. The beauty of Striim allows data to be joined wherever it is needed and the desired output to be obtained, assisting Glitre Energi in achieving the normalization of their metering events in their relational systems.

Please get in touch with us if you still need help or if your needs are still unclear. Our team of professionals will always be available to assist you. Click to do so.

The post Certainty in streaming real-time ETL appeared first on Indium.

Problem Statement 

As we all know, going through each stage to board a flight at an airport can be stressful. As shown in the image, there are typically five steps in the processing process: obtaining a boarding pass, bagging, security inspection, immigration inspection, and finally boarding. To prevent boarding or flight delays, all five of these processes should be finished within a SLA established for each phase. Internally, airport authorities use the message queueing system to populate each stage of the event, but the challenge would be to spot SLA breaches in real-time and report them to address the delay.

Process of Boarding at the Airport

The procedure would be monitored using MQ systems, where each step emits an event with the flight and passenger information. Finding the stage that exceeds the SLA required to pass a certain stage of checking is the challenge here. If any of the processes were to miss the SLA cut-off, the following effects would result.

1. Delaying the current flight and any subsequent flights
2. Panic among the passengers
3. Longer wait times.
4. A process could be missing.
5. There would be a security breach as a result.

The illustration below shows how MQ events are produced as passengers go through the airline’s onboarding procedure.

How does Striim contribute to process improvement?

Striim is a real-time replication tool that enables data streaming from a variety of source systems and aids in event migration to the target systems. Windowing technique is one of its key features, and it can hold data based on options like time, count, and fields to process on-the-fly. The best course of action in this situation is to hold/cache the event until the SAL/cut-off for the boarding process is determined. We can review the events that have occurred after the deadline to force the airport authorities to act right away from the cache. Assumedly, the events that are generated at each stage include the passenger’s boarding and flight information so that the specific passenger can be tracked throughout the boarding process.

Also Read: Use Cases for a Unified Data Integration and Streaming Platform like Striim

Striim Windowing Techniques

Real-time data is constrained within a window by time (for instance, five minutes), event count (for instance, 10,000 events), or both. The creation of a window is necessary for a replication flow to aggregate or process data, fill the dashboard, or send alerts when conditions deviate from expected ranges. An application can only evaluate and respond to individual events without a window to bind the data.

The three types of windows that Striim supports are sliding, jumping, and session windows. When a query’s contents change (sliding), expire (jumping), or there has been a lull in use activity, Windows sends data to the queries that follow (session). Jumping windows, which are regularly updated with an entirely new set of events, are the best fit for our use case out of these three types. Data sets for the hours of 8:00 am–8:04:59 am, 8:05 am–8:09 am, and so on would be produced, for instance, by a five-minute jumping window. A new data set would be produced by a 10,000-event jumping window after each 10,000 events. The window would output a new data set each time it accumulated 10,000 events or five minutes had passed since the previous data set was output if both five minutes and 10,000 events were specified. With the help of this Windows feature, we are putting forth an architecture that will both capture events coming from MQ systems and those that are approaching the cut-off time.

Proposed Architecture for the Airline On-Boarding System

With this suggested architecture, the airline onboarding processing can detect an SLA breach during passenger check-in for a flight with speed and accuracy. By storing the data in a designated window, it operates using the caching technique. The Striim partitioning feature enables us to classify every passenger according to their boarding pass number, allowing us to identify anyone having trouble during the flight. Striim’ s SQL-like queries are used to group and aggregate the events from jumping windows for each stage, from checking in to boarding flights.

CREATE OR REPLACE JUMPING WINDOW Boarding_Data_Window OVER admin.Boarding_Data_Win KEEP 2 ROWS WITHIN 90 SECOND PARTITION BY BoardingPassNo;

CREATE OR REPLACE CQ Passenger_Data_Boarding INSERT INTO admin.NotBoarded SELECT p.flightNo as flightNo, p.boardingPassNo as boardingPassNo , case when b.boardingPassNo is null then “Not Boarded” else “Boarded” END as BoardingStatus, b.boardingPassNo as bagBoardingPassNo FROM Boarding_Data_Window b right join PassengerDataWindow p on p.boardingPassNo = b.boardingPassNo;

CREATE OR REPLACE CQ NotBoarded INSERT INTO admin.NotBoardedResult SELECT * FROM NotBoarded n where BoardingStatus=”Not Boarded” group by n.boardingPassNo having count(*) <2 ;

Here are some benefits of using Striim as a replication tool in this scenario to record and modify SLA beaches during the flight boarding:

1. Real-time data collection that aids in processing the event at every stage.

2. Windowing the events until the designated interval to process and modify.

3. The dashboard and alerting system provides a nearly real-time progress of each passenger’s stages.

4. Quick fixes considerably shorter airport wait times and delays.

5. More accurate reporting.

An elaborate use case for Striim services: Striim-Powered Real-Time Data Integration of Core Banking System with Azure Synapse Analytics

Conclusion

The lengthy part of flying is waiting in line for the boarding process due to the densely populated airport. A more effective tracking system offers a practical way to track individual passengers for a comfortable journey. Using Striim’s windows technique, we can process and change airport authorities at any stage of the boarding process by holding every individual passenger detail in-memory directly from the real-time queuing system. Additionally, Striim aids in the migration of events to alternative target systems for better visual representations.

The Striim experts at Indium have cross-domain experience and can create solutions specifically for each of our clients’ individual needs.

The post Spot Realtime SLA Breaches in Airline On-Boarding Process Using Striim appeared first on Indium.

While businesses have access to lots of data thanks to the growth in IoT-based devices, cloud, enterprise systems, and so on, they face two problems. One is that the data is in raw format and two, it is stored in multiple systems and multiple formats. As a result, businesses need a solution that can pull structured and unstructured data in one place and convert this data into a unified format to act as the single source of truth.

A Gartner survey for the data integration tools market, titled ‘Adopt Stream Data Integration to Meet Your Real-Time Data Integration and Analytics Requirements’ and published in March 2019, indicates that 47% of organizations require streaming data that can help them build a digital business platform. However, only 12% had an integrated streaming data solution for their data and analytics requirements.

Traditionally, businesses depended on the ETL model – Extract, Transform and Load – but this is run as batch jobs periodically, rendering the data outdated and of limited use for some use cases.

In these times when businesses have to take quick decisions and respond to changing external and internal in a timely manner to remain competitive, depending on the ETL can be limiting to growth.

Real-Time Data Movement with In-Flight Processing

The face of data has changed tremendously today. Data is not only that which is stored in tables but also textual and documents across different formats stored in document stores such as MongoDB, Amazon DynamoDB, Couchbase Server and Azure Cosmos DB. An ETL can transfer data from one database to another, but with unstructured documents stored in these data stores, businesses need in-flight processing and built-in delivery validation along with real-time data movement.

MongoDB, for instance, is a document store where many of the sources are relational, flat, or unstructured. It will require a real-time continuous data processing solution such as Striim to create the necessary document structure as required by the target database.

Striim Features for Data Movement

Striim, an end-to-end, in-memory platform, collects, filters, transforms, enriches, aggregates, analyzes, and delivers big data in real-time. Designed especially for stream data integration, it uses low-impact change data capture to extract real-time data from different sources such as IoT devices, document stores, cloud applications, log files, and message queues and deliver it in the format needed and can deliver to or extract from MongoDB (or equivalent) as required.

With CDC, it delivers to MongoDB one collection per table, inserting, deleting, and updating documents based on the CDC operation, the row tuple contents with metadata, and the fields containing data elements.

It facilitates filtering and transforming data using SQL. Data enrichment is made possible by coupling it with external data in caches. Query output or custom determine the JSON document structure.

Custom transformations are also possible for complex cases with custom processors. While it is possible to achieve granular document updates, moving data from master/detail-related tables into a document hierarchy is also possible.

Benefits of Striim

Some of the key features of Striim that enable businesses to improve operational efficiency and deliver from and to document stores in real-time with in-flight processing for data integrity include:

• Low-impact change data capture from enterprise databases that allows for continuous and non-intrusive ingestion of high-volume data. It can support data warehouses such as Oracle Exadata, Amazon Redshift and Teradata; and databases such as MongoDB, Oracle, SQL Server, HPE NonStop, MySQL, PostgreSQL, Amazon RDS for Oracle, and Amazon RDS for MySQL. It enables data collection in real-time a variety of sources such as logs, sensors, Hadoop and message queues to enable real-time analytics.

• Non-stop data processing and delivery are effected through an inline transformation using processes such as denormalization, filtering, aggregation, and enrichment. This facilitates storing only the relevant data in the required format. A hub and spoke architecture is supported using real-time data subsetting and optimized delivery is enabled in both streaming and batch modes.

• Built-in monitoring and validation allow for non-stop verification of the consistency of the source and target databases. In addition to interactive, live dashboards for streaming data pipelines, it also enables real-time alerts via web, text or email.

Striim makes it possible for businesses to upgrade from ETL solutions to streaming data integration at an extreme scale by providing a wide range of supported sources. Any data can be made available in platforms such as MongoDB in real-time, in the required format to leverage scalable document storage and analysis.

Some of the key benefits of Striim include continuous data movement from a variety of sources with sub-second latency in real-time; a non-intrusive collection of real-time data from production systems with least disruption; and in-flight denormalization and other transformations of data.

Indium – A Striim Partner

Indium Software is a strategic partner of Striim, empowering businesses to make data-driven decisions by leveraging the real-time Big Data Analytics platform. Indium offers innovative data pipeline solutions for the continuous ingestion of real-time data from different databases, cloud applications, etc., leveraging Striim’s highly scalable, reliable, and secure end-to-end architecture that enables the seamless integration of a variety of relational databases. Indium’s expertise in Big Data coupled with the capabilities on the Striim platform enables us to offer solutions that meet the transformation and in-flight processing needs of our customers.

To find out how Indium can help you with your efforts to replace your traditional ETL solutions with a next-gen Striim platform for real-time data movement with in-flight processing, contact us now:

The post Ready to Replace Your Traditional ETL Solutions? Indium can help you use Striim for Real-Time Data Movement with In-Flight Processing appeared first on Indium.