The requests for data are now made in real-time rather than in batches to accommodate all business requirements. The scale adds to the system’s complexity and difficulty in addition to the things mentioned above. Therefore, it is essential to address optimization and cost reduction from the beginning to the end of a project.

Why BI Engine

BI Engine is helpful and acts as an acceleration layer between BigQuery and BI Tools. So overall the vision for BigQuery BI Engine is to democratise BI, enabling business analysts to perform interactive analytics in real-time. Now, three key value propositions of BigQuery BI Engine that is:

1. It provides sub-second queries:
   • Column-orientated. Dynamic in-memory execution engine.
   • Horizontal scaling to support higher concurrency.
   • Naive integration with BigQuery Streaming for real-time data refreshes.
2. It simplifies the architecture:
   • Built on existing BigQuery Storage.
   • Eliminates the need to manage BI Servers, ETL pipelines, or complex extracts.
   • No need to build and manage traditional OLAP cubes.
   • Open API for partner integration
3. It has smart tuning built:
   • In-memory cache optimization avoids querying raw data.
   • Ability to scale capacity up and down inside BigQuery UI.
   • Full visibility into metrics including aggregate refresh time, cache hit ratios, query latency, etc. inside Stackdriver.

Architecture of BI Engine

Before jumping to the BI Engine we need to understand BigQuery Architecture.

Let’s start with the high-level architecture of BigQuery, we see three main pieces here, one on the left is storage, that’s where all of its data is stored. On the right, compute that’s where query processing happened. And between them a shuffle layer, which facilitates distributed processing.

Now let’s see where the BI Engine fits into this picture.

We see BI Engine is just another feature of BigQuery. So, it works on top of the same storage, same network, and same shuffle layer, and it runs alongside BigQuery workers.

Now the question arises: If everything is the same as BigQuery architecture, then what makes it possible for BI Engine to get such high-performance improvements? And this is exactly what we are going to learn today. So if you see in the above picture there are three pieces and all three of them contribute to the performance of the query.  

• Storage is disk I/O time.
• Shuffle is a network I/O time.
• Compute is CPU.

Let’s pick each one of these three components and see how BI Engine reduces time to zero.

1. Memory Management

Starting with Memory Management which is the heart of BI Engine:

As we know BigQuery is a columnar database, so we put only specific columns into memory, not the whole table. Moreover, we are putting on only a chunk of the column, and that works equally well with both partitioned tables and non-partitioned tables. Here Memory Manager not only keeps raw data but also intermediate results of aggregations. It has very high integration with the metadata service, it knows exactly when the transaction commits. Also, memory managers know how to use load balancers, even if you have a small table but with lots of queries, it may choose to implicate the same data over multiple machines, which is easy for it to do because BigQuery is a distributed database. So, this way it will use small memory, but all of the queries will still have a very faster response time.

Now what kind of control users we have, this control comes in form of BI reservations which are very similar to classic BigQuery reservations but the currency is memory, so you are buying memory, not slots or CPU. All the metrics are served through Stackdriver.

The next component is the Compute Component, which uses vectorized processing and puts in the new runtime into BI Engine. So there are a lot of benefits for the CPU by using this concept, first, cache locality is much better as it runs on localized loops and second, code doesn’t jump around the operator to the operator which fills the CPU pipeline much tighter.

It also uses modern CPU instruction, which is SMG instructions that can process many values at the same clock time.

Also read: The Business Value of Data Virtualization with Denodo

2. Network Component

Now let’s move to another component which is the network.

Look at a straightforward query that includes a count condition that internally uses the sum function as a kind of filter. It will compute the interim results and then distribute them across the networks. If you look closely at these images, you will see that they are identical. What makes them different, then? The answer is that a parallel plan is run rather than a distributed plan. As a result, network traffic is decreased.

Enabling BI Engine in GCP

Let’s create BI Engine Reservation:

Step 1: Visit Google Cloud Console. Search BI Engine

Step 2: Enable Big Query Reservation API

Step 3: Click ‘Create a Reservation’ to create Reservation

Step 4: Select Location, Allocate Memory

Step 5: (Optional) Select the Tables on which you want to run BI Engine

Step 6: Next, Create

Acceleration statistics in INFORMATION_SCHEMA 

BI Engine acceleration statistics is queried as a part of INFORMATION_SCHEMA.JOBS_BY_* views through the col bi_engine_statistics. For example, below query will result all the bi-engine statistics.

select creation_time, job_id, bi_engine_statistics 
from `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT 
where creation_time BETWEEN TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY) AND CURRENT_TIMESTAMP() 
  and job_type = “QUERY” 

Conclusion

In this blog, we went through the impact of the BI Engine on cost and execution time. BI Engine is an important service that addresses the above challenges and what we see as the benefit of cost reduction is 50% and execution time reduction is 80%.

The post Cost Optimization using BI ENGINE appeared first on Indium.

Forty-six of the 50 top telecasts of 2017 were the NFL games (National Football League) with an average of 15.8 million viewers through the season. What made it very interesting was the kind of statistics about the game and the players NFL was able to present using an advanced system to collect data and draw insights from it.  

NFL uses Next Gen Stats (NGS), a sophisticated tracking technology to collect data about the players in its 32 teams, playing in different stadiums at any given moment. It uses RFID devices embedded in the shoulder pads of the players to identify the location of the player within inches and calculate the speed and direction in which they are moving. This wealth of information shared through its multiple media partners hooks its more than 180 million fans worldwide. NFL partners with Amazon Web Services (AWS) to leverage its data using QuickSight for generating sophisticated analytics and machine learning models to interpret the gameplay, enthrall viewers, and improve the quality of the game.  

Amazon QuickSight is a cloud-based, machine learning-powered business intelligence tool that uses natural language for exploring data using interactive dashboards. It helps business users identify patterns and outliers through relevant visualizations without the need for IT admins or experts to generate charts and graphs. It breaks silos by accessing data stored in different sources to provide a holistic view.

QuickSight Features

One of the key features of QuickSight is ML Insights, which enables advanced data analytics solutions for deeper insights by leveraging AWS’s machine learning (ML) and natural language capabilities. Using these powerful, out-of-the-box features, users can identify hidden trends and outliers, determine key business opportunities, and forecast trends to accelerate growth. It does not require technical know-how or ML experience to use the tool, thus empowering business users with the tool to generate their insights quickly and easily to make timely decisions. 

Contextual visualization is made possible by QuickSight’s machine learning capabilities that assess the meaning and relationships between the different business data. This helps improve the accuracy of the response to queries, supported by relevant visualizations. 

The tool can be trained for different functions and using the language and terms relevant to your organization. This helps to customize and increase the relevance of the insights. Machine learning also helps to improve the answers to the questions over time. 

Amazon QuickSight allows the seamless embedding of interactive dashboards and visualizations into applications. This helps to personalize the user interface for reports and dashboards. This also allows for a consistent user experience across the application. 

The QuickSight Enterprise Edition allows the embedding of rich, interactive dashboards in wikis, apps, and portals without the need to call embedding APIs. It does not need server deployments or infrastructure licensing, allowing authorized end-users to access the dashboards instantly. With the QuickSight Enterprise Edition, custom-branded reports can be shared over emails to increase credibility.  

Ad-hoc analysis and ML capabilities help with detecting anomalies, forecasting trends, and customizing natural language queries to generate dashboards and reports. This also increases avenues for differentiating and monetizing your applications. 

Being serverless, QuickSight analytics workflows can be managed programmatically, be it setting up a multi-tenant architecture to isolating data between multiple users, automating deployments, shifting dashboards across accounts, or managing access for users with Single Sign-On (SSO).

Benefits of QuickSight

Some of the key benefits of QuickSight include: 

Scalability: Using robust AWS APIs, developers can deploy scalable embedded analytics for thousands of users.  

Collaboration: Being cloud-based, QuckSight allows data visualization and insights to be shared across the organization using the web, email, mobile, or embedded applications. 

Consistency: Admins can ensure consistent, scalable performance. Periodic updates help improve performance without version conflicts, any downtime, or compatibility issues.  

Cost-Effective: QuickSight’s pay-per-session pricing makes it a cost-effective solution even for large-scale deployments.  

Forecast Trends: Interactive, point-and-click what-if analysis using business metrics helps businesses predict future trends and make informed decisions. 

Auto-Narratives: Customizable auto-narratives that can be woven into dashboards provide deeper context for improving actionable insights. 

Self-Service: End-users can generate their own reports with technical know-how to accelerate the speed of decision making 

Simple Data Pipelines: With Amazon SageMaker integration. sophisticated ML can be integrated to generate models without needing complex data pipelines 

Compliance: The solution provides end-to-end data encryption for data and encryption at rest for data in SPICE, ensuring security, governance, and compliance. 

Check out our customizable AI/ML-powered text analytics accelerator

Use Cases of Amazon QuickSight

Some of the use cases of Amazon Quicksight include doing sentiment analysis to understand customer views and customize products and solutions to improve customer delight. It can help create focused campaigns and targeted promotions to increase sales.  

The forecasting ability can help predict demand and manage production schedules effectively to reduce costs, and wastages, and improve profitability. It can help with increasing operational efficiency by identifying areas of improvement and strengths. 

QuickSight can be used across industries and in organizations of any size. It is a next-gen solution that can scale as the company grows, thereby protecting investments and increasing RoI. 

Indium Approach to QuickSight

Indium Software is an Amazon partner that has a team of experts in AWS solutions including QuickSight to help businesses break barriers to innovation. Our team of data engineers have experience working across industries and across functional areas, which helps to develop solutions that incorporate best practices for optimum result. 

The post Using Amazon QuickSight for Cloud BI with ML Capabilities  appeared first on Indium.

Almost every app developer and mobile app development company is looking to develop a strategic solution for deploying their business data on mobile devices . The objective is to fully leverage the power of mobile action and visualisation. With mobile BI on the verge of prevalent adoption, many top app development companies are already planning to launch their business data on mobile devices. This will allow for more convenient and easy access to critical data.

Consider a mobile app that learns about its users, their behaviour patterns, likes/dislikes, preferences, and choices, and then adapts to them. For example, a news app may automatically turn off notifications and activate the vibrate mode when the user is working on his laptop. Or a sports app that shares a live cricket score (rather than a football score) when the user enters his house.

Consider this scenario: a user enters a shopping complex and visits the showroom of her favourite shoe brand. She then receives a notification on her phone about a 50 percent discount that is only available to her. This is no longer a pipe dream, but a reality made possible by business intelligence in mobile apps.

Mobile business intelligence solutions is a strategy that businesses can use to provide employees with data analysis and business intelligence tools via their smartphones. However, this does not imply a static data push, where a number of sales or informative alerts are automatically pushed to your mobile device. In reality, it refers to a robust mobile business intelligence strategy that allows your workforce to manage, filter, sort, and visualise critical business data in real time. This gives you an overall advantage over your competitors. Let us look at how:

• 1. Improved Interactivity: Mobile business intelligence entails more than simply providing access to data from any location. It does, however, have something to do with data manipulation and visualisation. You must allow your employees to work remotely regardless of their time constraints. For example, if a random new idea to improve your company’s marketing strategy pops into your head at any point, you can simply use your mobile BI app such as ‘Board’ and feed important data into it to use it in the future.
• 2. Powerful analytics: Business intelligence provides numerous advantages and opportunities. It streamlines data analysis and interpretation, as well as transforming the same data into well-structured information. BI can entail a number of time-consuming processes, such as prescriptive analysis, statistical analysis, risk analysis, business process modelling, data mining, process mining, and many others. So, BI mobile app development is becoming too common in the digital business world. It is also increasing the demand for mobile app developers and mobile app development company.

You might be interested in: Artificial Intelligence And Its Impact On Mobile Applications

Advantages of BI for developing mobile applications

#1 Intelligent Data Collection

Business owners can customise and streamline information collection with business intelligence in mobile apps based on their specific business needs. Business Analytics is a great example of business intelligence that can be tailored to your specific needs.

#2 Increased Productivity

BI can be a great productivity and efficiency boost for both users and creators of mobile applications. With business intelligence, the application can better understand the user and adapt its functionality and processes to meet that user’s specific demands and needs. Similarly, management can provide better services, offers/discounts, and offerings in less time and at a price that meets the expectations of the users.

#3 Making Sound Decisions

Teams can make educated judgments, frequently in real time, using curated and processed data. For example, if you know that the majority of app users are looking for a specific brand of clothing right now, you can use that information to immediately stock up on that brand’s goods for quick and timely deliveries.

#4 Competitive Advantage

Mobile and digital businesses with the right information at the right time can have a distinct competitive advantage because supported by BI they become aware of their shortcomings, strengths and weaknesses.

#5 Excellent Customer Satisfaction

This advantage is now unparalleled and unprecedented: Customer satisfaction is the foundation of all great businesses, and with business intelligence, you can ensure stunning customer satisfaction on a consistent basis. Users are more satisfied and loyal to the platform when the creator of the mobile application makes better decisions. Users’ queries and questions can be responded to in a timely, even automated manner, ensuring outstanding customer satisfaction.

#6 Increased Revenue

Smart decision making, intelligent data collection, and improved customer satisfaction all lead to increased revenue and profit. Customers will be willing to spend more money and time on the application, management will have less waste, resources can be managed optimally, and revenues will skyrocket, directly impacting the bottom line.

To sum up

Today’s business landscape has become app-dependent, as most business processes require to be supported with apps for smooth execution. Business Intelligence (BI) has already been one of the important components for businesses and with increasing mobile app adoption, is witnessing evolution. Thus, for a successful enterprise mobile apps and business intelligence (BI) are inseparable from each other. To exploit the combined power of these technologies, all you need is a guidance and services of a professional mobile app development company.

The post Mobile App Development Clubbed With Business Intelligence appeared first on Indium.

Many businesses are finding their existing infrastructure that includes servers and racks to be limiting their ability to meet their need for increased storage space and compute power. In the traditional architecture, often businesses went for proprietary end-to-end solutions right from centralized data collection to storage, and analysis, to optimize resources and minimize costs but lost control of their own data.

As technology grows by leaps and bounds, businesses are also facing the problem of the tools being unable to up with the changing times and storage being insufficient for their growing needs. The cost of expanding infrastructure is also formidable, needing viable alternate solutions.

According to a McKinsey report, businesses are trying to simplify their current architectural approaches and accelerate deliveries across data activities such as acquisition, storage, processing, analysis, and exposure, to create a modern infrastructure that can support their future analytics and BI efforts. For this, six foundational shifts are being effected on data-architecture blueprints, leaving the core technology stack untouched. This can result in an increase in RoI due to lower IT costs, improved capabilities productivity, and lower regulatory and operational risk.

The Six Features of a Modern Data Ecosystem

As per the report, the six foundational shifts that facilitate the creation of a modern data ecosystem to support future analytics and BI efforts include:

1. Shifting to Cloud-based Data Platforms: Cloud-based solutions from providers such as Amazon, Azure Google, are disrupting the data architecture approach for sourcing, deployment, and running data infrastructure, platforms, and applications at scale. Two key components of this revolution are serverless data and containerized data solutions.

a. Serverless platforms such as Amazon S3 and Google BigQuery eliminate the need for installing and configuring solutions or managing workloads while enabling businesses to build and manage data-centric applications at scale at almost no operational overhead.

b. With containerized data solutions using Kubernetes, businesses can decouple compute power and data storage while automating the deployment of additional systems.

2. Real-time Data Processing: Real-time data streaming is a cost-effective solution that allows data consumers to receive a constant feed of the data they need by subscribing to relevant categories from a common data lake that is the source and retains all granular transactions. It can be of three types: messaging platforms such as Apache Kafka; streaming processing and analytics solutions such as Apache Spark Streaming, Apache Kafka Streaming, Apache Storm, and Apache Flume; alerting platforms such as Graphite or Splunk

3. Modular Platforms: A modular data architecture that leverages open source and best-of-breed components provides businesses with the flexibility to discard old technologies and embrace new ones with the least disruptions using data pipelines and API-based interfaces and analytics workbenches. They also facilitate the integration of disparate tools and platforms to connect with several underlying databases and services.

4. Decoupled Data Access: For effective reuse of data by different teams, businesses can provide limited and secure views and modify data access by exposing data through APIs. It also enables quick access to up-to-date and common data sets. As a result, analytics teams can collaborate seamlessly and accelerate the development of AI solutions. The two key components that facilitate this are an API management platform or API gateway and a data platform.

5. Domain-based Architecture: Instead of a central enterprise data lake, domain-driven data architecture designs help businesses customize and accelerate time-to-market new data products and services. Data sets can also be organized in a more easily consumable manner for domain users and downstream data consumers. Some of the enabling features of this architecture are its data-infrastructure-as-a-platform model integrated with data virtualization techniques, and data cataloging tools.

6. Flexible, Extensible Data Schemas: The traditional, predefined, and proprietary data models built into highly normalized schemas tend to be rigid, limiting the addition of new data elements or data sources due to the risk to data integrity. Therefore, with schema-light approaches with denormalized data models and fewer physical tables, data can be organized for optimizing performance, agility, and flexibility. This is facilitated by components and concepts such as Data-point modeling, graph databases, dynamic table structures, and JavaScript Object Notation (JSON).

This piece of content might be of your interest: Modern Data Analytics And Modern Data Architecture

Indium Approach

A McKinsey Digital report shows that only 14 percent of companies launching digital transformations are able to see sustained and substantial improvements in performance. This is because, though there are several digital engineering solutions and analytics service options available to draw insights through business intelligence and analytics, , organizations are unable to suit the right architecture for their business. What is popular or obvious may not be the right fit for them.

Indium Software is a data engineering specialist offering cross-domain and cross-functional expertise and experience to understand the unique needs of each business. We use commercial and open source tools based on the cost and business requirements to meet your unique data engineering and analytics needs.

To accelerate your data modernization journey to make your business future-ready, we help you with large-scale data transformation and Cloud Data & Analytics. To ensure the efficiency of your data modernization process, we help you with aligning your data management strategy with your business plan.

Our solution encompasses:

ETL Modernization: The three-step process of extract, transform, and load is designed to help you overcome your business challenges.

Data Governance: User- and role-based data access ensures data security, privacy, and compliance while enabling informed decision-making.

Data Visualization: Our experts use cutting-edge business intelligence solutions to enable data visualizations for actionable insights

Data Management: Data abnormalities are identified as they occur, reducing time and money in rectifying mistakes.

The post Support Your Analytics and BI Efforts for the Next 10 Years with a Modern Data Ecosystem appeared first on Indium.

Power BI dashboards can be shared with users within or outside an organization in the form of

• URLs
• .pbix files or
• Embedded reports in HTML/Javascript portal.

This blog deals with the steps required to embed a Power BI report in Angular JS. The ngx-powerbi library in Angular JS will be used for embedding the dashboards.

Pre-Requisites:

• Power BI Pro Account
• Angular JS
• ngx – powerbi (Library for embedding Power Bi reports in Angular JS)
• Java

This blog deals with the steps required to embed a Power BI report in Angular JS. The ngx-powerbi library in Angular JS will be used for embedding the dashboards.

The entire process can be broken down into the following parts.

1. Create a Power BI Pro account.
2. Grant API and data permissions to your app on Azure Active Directory.
3. Grant requisite permissions to your Apps on Azure Active Directory using Azure Portal.
4. Create a sample Power BI application.
5. Write a Java API to generate Access tokens for login.
6. Create Angular JS UI to embed the dashboard.

The following steps can be followed to embed a Power BI report in Angular JS.

Step 1: Create a Power BI Pro account

A Power BI Pro should be created using your organization’s email address. Users can sign up here. A Pro account costs $9.99 per month.

Step 2: Grant API & data permissions to the App on Azure Active Directory

• Register your Power BI Application on Azure Active Directory.
• Sign In using Power BI credentials.

• Give your application a suitable name.
• Application Type should be Native as we are using this for embedding.

• Give the App permission to access all APIs.
• Register the Application.
• This will create a new App and register it under Azure Active Directory. Note down the Application ID that gets generated. This is used later for access token generation.

Step 3: Grant permissions to the App on Azure Active Directory using Azure Portal

• Sign In to Azure Portal.
• Look up Azure Active Directory -> App Registrations

• Click on the Application and then go to Settings -> Required Permissions
• Select the two API properties listed and click on Grant Permissions button under Required Permissions section. This is required for embedding the Power BI report.

Step 4: Create a Sample Power BI application

Login to Power BI portal. Create a sample Power BI report under a newly created app workspace. The report can also be a .pbix file that is generated from Power BI Desktop Application and published using Power BI Service. Then do File -> Embed.

This will generate an Embed URL that will be used for embedding the report in Angular JS UI. The embed URL will look as follows:

https://app.powerbi.com/reportEmbed?reportId=8d63c6b1-f34d-4de4-9d0a-f5fd79ece1d5&groupId=fbcba1c5-78a5-491a-8eac-4832152b8a97&autoAuth=true&ctid=e983fad9-ddbe-48d7-bb95-eec937e4b59f

The report ID and Group ID are highlighted in bold in the above URL and they are important as they are required for access token generation during login using the Angular JS UI.

Step 5: Write a Java API to generate access token

Write a simple Java API that will generate access tokens using the Application ID that was created in Step 1 when the app was registered. In addition to the application id, Power BI username and password along with the embed URL is also required.

The screenshot above shows a generic implementation of the Java API. In the code snippet below, the application id, Power BI username, password & embed URL are stored in a database and then retrieved from there.

Step 6: Create a simple Angular JS UI to embed the dashboard

An Angular JS UI with single sign on functionality should be created. Every time a user logs in, an access token will get generated from the Java API. The code snippet for embedding the dashboard in Angular JS is as follows.

It uses the ngx-powerbi library from Angular JS for embedding. The embed URL and Access token generated from the Java API are inputs for this component. The login page created in Angular JS looks like the snapshot below:

When you login using the credentials, in the backend an access token gets generated based on the Java API the embedded dashboards can be seen as shown below.

The embedded dashboards can have data connected to persistent storage. In that case, Power BI provides the facility to automatically refresh data using Power BI Gateway Service. More information on how that can be done is provided in this link.

The post Experience the awesomeness of a Power BI dashboard from your web portal appeared first on Indium.