Introduction
Data Engineering comprises of all of the engineering and operational tasks that are required to make data available for analytics including handling tasks related to:
Data Ingestion
Data Synchronization
Schema Synchronization
Data Transformation
Data Models
OLAP Cubes
Platform Governance
Operations Orchestration
Agile data engineering applies the process and automation to perform all of these tasks in a manner that is fast, flexible, and sustainable. Infoworks provides the only end-to-end platform that addresses all of these items in a single, fully integrated solution.
Objectives
In this tutorial, we will build a use-case to see how weather conditions in various cities impact a company’s sales. The sales data is obtained from an Oracle database, while the weather data is available in CSV files.
On completion of this training, you will have a basic understanding on how to:
Crawl and ingest data from Relational Databases and File Sources.
Prepare the data and transform to create a semantic layer for data consumption using a modeling technique of choice, dimensional stars, snowflakes, etc.
Build a cube pre-aggregating the data for high performance analysis and reporting.
Access the data using a query or reporting tool.
Define a workflow, schedule it to ingest new data, and then refresh the data models and cubes daily.
Typically on Hadoop, this would require you to code many of these steps and would take you weeks to complete. With Infoworks, you will be able to complete all these in a few hours and without writing a single line of code.
Product Components Used
Data Ingestion
Data Transformation and Pipeline Build
Data Cube Build
Data Analysis and Reporting
Workflow Build and Execution