The cloud has emerged as a pivotal factor in the growth of digital-native enterprises, furnishing them with the flexibility, scalability, and agility needed to fulfill their customer experience commitments and maintain a competitive edge. Presently, cloud services encompass a diverse array of offerings, encompassing support for software development and testing, bolstered security measures, streamlined governance, automation of compliance processes, AI and ML platforms, as well as tools that facilitate value-adding capabilities like augmented reality/virtual reality (AR/VR) and robotics.

Key Trends for Digital Natives:

Digital natives, born in the cloud era and characterized as data-centric tech companies, heavily rely on SaaS (Software as a Service) solutions built upon cloud-native infrastructure. This robust foundation empowers them with agile, adaptable operations that can effortlessly scale to meet their evolving demands. Furthermore, they leverage AI (Artificial Intelligence) and Machine Learning to optimize their business processes, seamlessly integrating data across their backend systems.

In “The Data-Driven Enterprise in 2023,” McKinsey & Company outlines seven pivotal characteristics shaping the data-driven enterprise landscape:

1. Data Integration: Data seamlessly integrates into every facet of decision-making, interactions, and business processes, serving as the bedrock of operations.

2. Real-Time Processing: Swift, real-time data processing enables rapid decision-making and responsive actions.

3. Flexible Data Stores: Enterprises employ versatile data storage solutions to integrate easily accessible data for diverse purposes.

4. Data as a Product: A data-centric operating model recognizes data’s inherent value, emphasizing its potential to generate substantial value.

5. Chief Data Officer’s Role: The Chief Data Officer’s role expands to focus on extracting value from data, acknowledging its pivotal role in organizational success.

6. Data Ecosystems: Collaboration and data-sharing within industry-specific data ecosystems become standard practices as enterprises realize the advantages of collective participation.

7. Data Management: Prioritized and automated data management ensures privacy, security, and resilience in an increasingly data-driven landscape.

The quote from McKinsey & Company underscores the importance of data streaming, enabling precise data usage in real-time contexts. Below, we showcase successful data-driven approaches.

In the digital landscape, essential components include real-time visibility, feature-rich mobile apps, and seamless integration with cutting-edge technologies like managed cloud services, 5G networks, and augmented reality. Data streaming enhances these capabilities by facilitating real-time data integration and correlation, with Striim as a crucial enabler.

Digital native enterprises, or Digital Native Businesses (DNBs), are defined by IDC as companies leveraging cloud-native tech, data, and AI across all operations. They rely on digital technology for core processes, fully utilizing data streaming for real-time messaging, storage, integration, and correlation.

Case & Point!

Etsy, much like many other digital-native startups, has been heavily reliant on data analytics since its inception in 2005. In its early days, the company faced challenges in truly understanding its customers, which resulted in subpar digital experiences for sellers and a failure to accurately capture customer preferences. To address this, Etsy significantly transformed by establishing a dedicated research department that merged quantitative and qualitative insights. These insights were integrated into every company department, resulting in elevated user satisfaction levels and more informed product decisions. Etsy has witnessed an astounding 400% growth since 2012, a testament to this shift.

What Etsy accomplished was a transition from being merely “data-aware” or data-driven to becoming an “insights-driven” business. While data-aware firms prioritize data collection and mining for insights, insights-driven businesses excel at data analytics, applying quantitative insights to address issues and embedding these insights into their business models, operations, and organizational culture.

Another notable example is Tesla, where vehicles are essentially insights-driven. Tesla continuously streams real-time performance data from each car to its data scientists, who develop models to diagnose driving-related issues and remotely provide software or firmware updates. The result is a seamless enhancement of the driving experience and an insightful system that enables testing, learning, and iterative improvement over time.

Exploring the Practical Applications of AI and Machine Learning Beyond the Buzz!

Indeed, Gartner’s perspective that “ChatGPT, while cool, is just the beginning; enterprise uses for generative AI are far more sophisticated” rings true. It’s essential to recognize that the potential of AI, particularly machine learning, goes beyond the buzz and is already being effectively applied in numerous enterprises.

Amidst the current hype around Generative AI (GenAI), it’s valuable to focus on tangible real-world success stories where analytic models have been utilized for many years. These models have been instrumental in tasks such as fraud detection, upselling to customers, and predicting machine failures. GenAI represents another advanced model that seamlessly integrates into an organization’s IT infrastructure and business processes.

In today’s fast-paced digital landscape, providing and correlating information correctly in the right context is crucial for enterprises seeking to stay competitive. Real-time data streaming, where information is processed in milliseconds, seconds, or minutes, is often superior to delayed data processing, ensuring that insights are harnessed swiftly and effectively.

Data streaming + AI/machine learning = Real-time intelligence

For example, Duolingo, an AI-powered language-learning platform, utilizes the PyTorch framework on AWS to deliver customized algorithms that offer tailored lessons in 32 languages. These algorithms rely on extensive data points, ranging from 100,000 to 30 million, to make 300 million daily predictions, such as the likelihood of a user recalling a word and answering a question correctly.

Duolingo’s system employs deep learning, a subset of AI and ML, to analyze user interactions with words, including correct responses, response modes, and practice intervals. Based on these predictions, the platform presents words in contexts that users need to master them, enhancing the learning experience.

While Duolingo initially used traditional cognitive science algorithms when it started in 2009, these algorithms couldn’t process real-time data to create personalized learning experiences. The adoption of deep learning tools improved prediction accuracy and increased user engagement, with a 12% increase in users returning to the service on the second day after implementing these tools. Duolingo’s success story, with 300 million subscribers, underscores the pivotal role of the AWS cloud in enhancing platform speed, scalability, and predictive capabilities.

As demonstrated by Duolingo, the cloud now offers a wide range of capabilities, delivering three key advantages:

1. Operational Excellence: Empowering companies to prioritize differentiated work over maintenance or commodity tasks, resulting in cost reduction, heightened security, and increased reliability.

2. New Levers and Capabilities: Facilitating organizations in accelerating the development of new products, features, and market expansion.

3. Accelerated Innovation: Combining operational excellence and new capabilities to drive faster, more agile, maintainable, and scalable development processes.

Coinbase, a prominent digital currency wallet and platform provider with 30 million customers, has leveraged AWS Step Functions to automate and enhance the deployment of new software features and updates. This approach has not only resulted in successful deployments 97% of the time but has also significantly accelerated the process of adding new accounts, reducing it from days to mere seconds. Furthermore, Coinbase has significantly reduced the number of customer support tickets, thus enhancing user satisfaction and operational efficiency, while bolstering cybersecurity measures to protect users from cyberattacks.

Personalization driven by AI and ML can indeed yield powerful results. Notable examples include Intuit, a financial software company, which employed the Amazon Personalize service to rapidly create and deploy a recommendation engine for its Mint consumer budget tracking and planning app. Similarly, Keen, a outdoor footwear manufacturer, harnessed the same Amazon service to monitor customers’ browsing and purchase histories, enabling the provision of tailored shopping recommendations. Keen’s implementation of the recommendation feature via test emails resulted in a substantial revenue increase of nearly 13%.

Additionally, Ably, a South Korean startup in the apparel e-commerce sector, has successfully integrated AI to provide personalized recommendations on its app’s front page. Leveraging individual customer browsing and purchasing histories, Ably’s recommendation engine has empowered the company to develop sophisticated AI capabilities, even without prior experience in ML technology. These instances underscore how AI-driven personalization can significantly enhance user experiences and boost business outcomes across various industries.

Natural language processing (NLP) with data streaming for real-time Generative AI (GenAI)

Natural Language Processing (NLP) has proven to be a valuable tool in numerous real-world projects, enhancing service desk automation, enabling customer interactions with chatbots, moderating social network content, and serving many other use cases. Generative AI (GenAI) represents the latest evolution of these analytical models, adding even more capabilities to the mix. Many enterprises have successfully integrated NLP with data streaming for years to power real-time business processes.

Striim has emerged as a central orchestration layer within machine learning platforms, facilitating the integration of diverse data sources, scalable processing, and real-time model inference. Below is an architecture that illustrates how teams can seamlessly incorporate Generative AI and other machine learning models, such as large language models (LLM), into their existing data streaming framework:

This architecture showcases the integration of Generative AI and LLM into the data streaming architecture, allowing organizations to harness the power of these advanced models to further enhance their real-time data-driven processes.

Time to market is undeniably critical in today’s fast-paced business landscape. The beauty of incorporating AI is that it often doesn’t necessitate a complete overhaul of an enterprise’s architecture. A well-designed, truly decoupled system enables organizations to seamlessly introduce new applications and technologies and integrate them into existing business processes. This approach ensures agility and adaptability, allowing businesses to swiftly capitalize on emerging opportunities and stay competitive without undergoing extensive infrastructure changes.

An exemplary example is our project with an airline company employing Striim to enhance operational efficiency by modernizing its legacy data store. (Read more)

The post Real-time Insights-Driven Businesses and the Impact of Cloud on the Digital Native Ecosystem appeared first on Indium.

Problem Statement 

As we all know, going through each stage to board a flight at an airport can be stressful. As shown in the image, there are typically five steps in the processing process: obtaining a boarding pass, bagging, security inspection, immigration inspection, and finally boarding. To prevent boarding or flight delays, all five of these processes should be finished within a SLA established for each phase. Internally, airport authorities use the message queueing system to populate each stage of the event, but the challenge would be to spot SLA breaches in real-time and report them to address the delay.

Process of Boarding at the Airport

The procedure would be monitored using MQ systems, where each step emits an event with the flight and passenger information. Finding the stage that exceeds the SLA required to pass a certain stage of checking is the challenge here. If any of the processes were to miss the SLA cut-off, the following effects would result.

1. Delaying the current flight and any subsequent flights
2. Panic among the passengers
3. Longer wait times.
4. A process could be missing.
5. There would be a security breach as a result.

The illustration below shows how MQ events are produced as passengers go through the airline’s onboarding procedure.

How does Striim contribute to process improvement?

Striim is a real-time replication tool that enables data streaming from a variety of source systems and aids in event migration to the target systems. Windowing technique is one of its key features, and it can hold data based on options like time, count, and fields to process on-the-fly. The best course of action in this situation is to hold/cache the event until the SAL/cut-off for the boarding process is determined. We can review the events that have occurred after the deadline to force the airport authorities to act right away from the cache. Assumedly, the events that are generated at each stage include the passenger’s boarding and flight information so that the specific passenger can be tracked throughout the boarding process.

Also Read: Use Cases for a Unified Data Integration and Streaming Platform like Striim

Striim Windowing Techniques

Real-time data is constrained within a window by time (for instance, five minutes), event count (for instance, 10,000 events), or both. The creation of a window is necessary for a replication flow to aggregate or process data, fill the dashboard, or send alerts when conditions deviate from expected ranges. An application can only evaluate and respond to individual events without a window to bind the data.

The three types of windows that Striim supports are sliding, jumping, and session windows. When a query’s contents change (sliding), expire (jumping), or there has been a lull in use activity, Windows sends data to the queries that follow (session). Jumping windows, which are regularly updated with an entirely new set of events, are the best fit for our use case out of these three types. Data sets for the hours of 8:00 am–8:04:59 am, 8:05 am–8:09 am, and so on would be produced, for instance, by a five-minute jumping window. A new data set would be produced by a 10,000-event jumping window after each 10,000 events. The window would output a new data set each time it accumulated 10,000 events or five minutes had passed since the previous data set was output if both five minutes and 10,000 events were specified. With the help of this Windows feature, we are putting forth an architecture that will both capture events coming from MQ systems and those that are approaching the cut-off time.

Proposed Architecture for the Airline On-Boarding System

With this suggested architecture, the airline onboarding processing can detect an SLA breach during passenger check-in for a flight with speed and accuracy. By storing the data in a designated window, it operates using the caching technique. The Striim partitioning feature enables us to classify every passenger according to their boarding pass number, allowing us to identify anyone having trouble during the flight. Striim’ s SQL-like queries are used to group and aggregate the events from jumping windows for each stage, from checking in to boarding flights.

CREATE OR REPLACE JUMPING WINDOW Boarding_Data_Window OVER admin.Boarding_Data_Win KEEP 2 ROWS WITHIN 90 SECOND PARTITION BY BoardingPassNo;

CREATE OR REPLACE CQ Passenger_Data_Boarding INSERT INTO admin.NotBoarded SELECT p.flightNo as flightNo, p.boardingPassNo as boardingPassNo , case when b.boardingPassNo is null then “Not Boarded” else “Boarded” END as BoardingStatus, b.boardingPassNo as bagBoardingPassNo FROM Boarding_Data_Window b right join PassengerDataWindow p on p.boardingPassNo = b.boardingPassNo;

CREATE OR REPLACE CQ NotBoarded INSERT INTO admin.NotBoardedResult SELECT * FROM NotBoarded n where BoardingStatus=”Not Boarded” group by n.boardingPassNo having count(*) <2 ;

Here are some benefits of using Striim as a replication tool in this scenario to record and modify SLA beaches during the flight boarding:

1. Real-time data collection that aids in processing the event at every stage.

2. Windowing the events until the designated interval to process and modify.

3. The dashboard and alerting system provides a nearly real-time progress of each passenger’s stages.

4. Quick fixes considerably shorter airport wait times and delays.

5. More accurate reporting.

An elaborate use case for Striim services: Striim-Powered Real-Time Data Integration of Core Banking System with Azure Synapse Analytics

Conclusion

The lengthy part of flying is waiting in line for the boarding process due to the densely populated airport. A more effective tracking system offers a practical way to track individual passengers for a comfortable journey. Using Striim’s windows technique, we can process and change airport authorities at any stage of the boarding process by holding every individual passenger detail in-memory directly from the real-time queuing system. Additionally, Striim aids in the migration of events to alternative target systems for better visual representations.

The Striim experts at Indium have cross-domain experience and can create solutions specifically for each of our clients’ individual needs.

The post Spot Realtime SLA Breaches in Airline On-Boarding Process Using Striim appeared first on Indium.