Business Intelligence

Data Warehouse drives the corporate information supply chain to support Corporate Business Intelligence process.

Business Intelligence introduced by Howard Dresner of the Gartner Group in 1989 is a set of concepts and methodologies to improve decisionmaking in business through the use of facts and fact-based systems.

Fact-Based Systems
Executive Information Systems Decision Support Systems Enterprise Information Systems Management Support Systems OLAP Data & Text Mining Data Visualization Geographic Information Systems

Data Warehousing
This simple concept has recently become a Multi-Billion dollar industry. New Breeds of vendors are introducing tools and technologies at an alarming rate to deliver data warehouse solutions.

Business Intelligence
-

The Construction of a Data Warehouse requires three key steps:

Extract data from transactions systems. Manipulate extracted data to generate reports. Makes such reports accessible to the decision-makers.

Data Warehouse Categories

Terms Used to define data warehouse objects vary from one vendor to another and can be divided into the following categories.

Data Warehouse. Data Mart.

First Chapter
What is Data Warehousing? Data Warehousing Architecture Components of Data Warehousing Evolution of ERP Data Warehousing A Multi-Dimensional Data model

Data Warehouse.
This is conceptually the same defined my Inmon.
It could be one large physical instance or a collection of several physical data object instances (Detailed & Aggregated), each serving a special purpose conforming to a bigger corporate vision.

Data Mart
Data Marts are stand-alone small data warehouses limited to a subject area (Ex:- Sales Analysis).
We have Dependent and Independent data marts. Dependent Data Marts are extracted views of a corporate data warehouse. Independent Data Marts are those which are built directly against transaction systems.

Data Warehouse Architecture

Building a House is analogous to building a Data Warehouse. The architect draws up a blueprint based on your needs and requirements. During design process, the architect makes sure that your living requirements are met.

If the house does not adhere to an architectural plan, its integrity will always be in question.

SAP BW
The SAP Business Information Warehouse (SAP BW) follows this paradigm. Under one frame work , you can have a huge Extraprise Data warehouse of a hierarchy of enterprise data warehouses or data marts, all conforming to the same architecture, infrastructure and information delivery methods.

What is SAP Business Information Warehouse?

Today's decision makers urgently need accurate information - a complete and up-to-date picture of their business and their business environment.

However, that information is spread over a wide variety of platforms and applications throughout the corporate IT structures.
This can make obtaining vital facts and figures a complex and time-consuming task.

Components of Data Warehouse

Inmon, Ralph Kimball, and Doug Hackney have published several books articulating traditional data warehouse architectures and construction processes. Following are the critical layers that are important in understanding the SAP BW architecture.

Data Provider. Service Provider. Information Consumer. Data Warehouse Management.

Data Provider Layer

The Data provider layer is the primary gateway to the data sources (OLTP Applications). Major tasks performed at this layer provide an environment in which to construct subject-oriented data analysis models. Metadata (data about the data) is pulled into a data warehouse environment from its data sources.

Extraction, Transformation, & Transport services fetch data from data sources , Qualify , perform value- added data manipulation, and push data out to data warehouse data objects.

Data Provider Layer

Key services performed at this level is

Data Transport. Data Transformation. Data cleansing. Data Extraction. Subject Models.

Service Provider Layer

The Service Provider Layer is responsible for managing and distributing data objects across the enterprise to support business intelligence activities in a controlled and secured fashion.

At this layer, data is further transformed for specific data analysis tasks such as drill-down analysis and predefined reports integration with third-party subscribed data. This Layer is very complex within the data warehouse architecture. Key services performed at this layer are the following:

Service Provider Layer

Analytical Applications Integration. Data Distribution. Data Profiling. Data Partitioning. Data Consolidation. Data staging. Data Storage.

Information Consumer Layer

The information Consumer Layer Accesses information Objects from a data warehouse. Information Delivery services , Provided by service Providers, Must be robust enough to handle large data volumes and multimedia objects. Data Access and delivery services must be robust enough to handle al such scenarios. Key services performed at this layer as follows.

Information Consumer Layer

Information Presentation Search Engines End-User Data Synchronization Data Conversions Information Access APIs Information Consumer Profiling Global Catalogs Information Delivery

Data Warehouse Management Layer

The Data Warehouse management layer provides services to manage all data objects in all layers. Additional services at this layer include component installation, monitoring, tuning, scheduling, networking, database operations, and component problem tracking/ analysis that can be performed globally.

Key services performed at this layer are the following.

Key Services Of DW Mgmt Layer

Governing Services Track resource Utilization Audit & Controls Scheduling Client Profile Multi-tiered Models Warehouse Operations Source/Target Mgmt Data Dictionary Development Management Hardware/Software Security Metadata

End Of Intro

Definition Data Warehouse

Defined in many different ways, but not rigorously.

A decision support database that is maintained separately from the organizations operational database Support information processing by providing a solid platform of consolidated, historical data for analysis.

W.H.Inmons Definition
A data warehouse is a timevariant, integrated, nonvolatile , & subject-oriented collection of data in support of managements decision-making process. William.H.Inmon - Father of Data
Warehousing

Time Variant
The time horizon for the data warehouse is significantly longer than that of operational systems.

Operational database: current value data. Data warehouse data: provide information from a historical perspective (e.g., past 5-10 years)

Every key structure in the data warehouse

Contains an element of time, explicitly or implicitly But the key of operational data may or may not contain time element.

Integrated
Constructed by integrating multiple, heterogeneous data sources

Relational databases, flat files, on-line transaction records

Data cleaning and data integration techniques are applied.

Ensure consistency in naming conventions, encoding structures, attribute measures, etc. among different data sources
E.g., Hotel price: currency, tax, breakfast covered, etc.

When data is moved to the warehouse, it is converted.

Non-Volatile
A physically separate store of data transformed from the

operational environment.
Operational update of data does not occur in the data warehouse environment.

Does not require transaction processing, recovery, and concurrency control mechanisms Requires only two operations in data accessing:

initial loading of data and access of data.

Subject-Oriented
Organized around major subjects, such as customer, product, sales. Focusing on the modeling and analysis of data for decision makers, not on daily operations or transaction processing. Provide a simple and concise view around particular subject issues by excluding data that are not useful in the decision support process.

Data Warehouse vs. Operational DBMS

OLTP (on-line transaction processing)

Major task of traditional relational DBMS Day-to-day operations: purchasing, inventory, banking, manufacturing, payroll, registration, accounting, etc.

OLAP (on-line analytical processing)

Major task of data warehouse system

Data analysis and decision making

OLTP Vs. OLAP

Distinct features (OLTP vs. OLAP):

User and system orientation: customer Vs. market

Data contents: Current, Detailed Vs. Historical, Consolidated

Database design: ER + Application Vs. Star + Subject

View: Current, Local Vs. Evolutionary, Integrated Access patterns: Update vs. Read-only but Complex Queries

OLTP vs. OLAP

OLTP OLAP Users Function DB design Data Usage Access Unit of work # Records accessed # Users DB size Metric Clerk, IT Professional Day To Day Operations Application-Oriented Current, Up-To-Date Detailed, Flat Relational Isolated Repetitive Knowledge Worker Decision Support Subject-Oriented Historical, Summarized, Multidimensional Integrated, Consolidated Ad-Hoc

Read / Write Lots Of Scans Index / Hash On Prim. Key Short, Simple Transaction Complex Query Tens Thousands 100mb-Gb Transaction Throughput Millions Hundreds 100GB-TB Query Throughput, Response

Why Separate Data Warehouse?

Different functions and different data:

Missing data: Decision Support requires historical data which operational DBMS do not typically maintain. Data consolidation: Decision Support requires consolidation (Aggregation, Summarization) of data from heterogeneous sources. Data quality: Different Sources typically use inconsistent data representations, codes and formats which have to be reconciled.

A Multi-Dimensional Data Model

A Data Warehouse is based on a Multidimensional data model which views data in the form of a data cube A data cube, such as sales, allows data to be modeled and viewed in multiple dimensions

Dimension tables, such as item (item name, brand, type), or time (day, week, month, quarter, year)

Fact table contains measures (such as dollars) and keys to each of the related dimension tables

Conceptual Modeling of Data Warehouses

Modeling data warehouses: dimensions & measures

Star schema: A fact table in the middle connected to a

set of dimension tables Snowflake schema: A refinement of star schema where some dimensional hierarchy is normalized into a set of smaller dimension tables, forming a shape similar to snowflake Fact constellations: Multiple fact tables share dimension tables, viewed as a collection of stars, therefore called galaxy schema or fact constellation

Example of Star Schema

time
time_key day day_of_the_week month quarter year

item
Sales Fact Table time_key item_key branch_key
item_key item_name brand type supplier_type

branch
branch_key branch_name branch_type

location_key units_sold

location
location_key street city province_or_street country

dollars_sold
avg_sales

Measures

time

Example of Snowflake Schema

item
Sales Fact Table
item_key item_name brand type supplier_key

time_key day day_of_the_week month quarter year

supplier
supplier_key supplier_type

time_key
item_key branch_key

branch
branch_key branch_name branch_type

location
location_key street city_key

location_key
units_sold dollars_sold avg_sales Measures

city

city_key city province_or_street country

time
time_key day day_of_the_week month quarter year

Example of Fact Constellation

item
Sales Fact Table time_key item_key branch_key
item_key item_name brand type supplier_type

Shipping Fact Table time_key

item_key
shipper_key

from_location
location
location_key street city province_or_street country

branch
branch_key branch_name branch_type

location_key units_sold dollars_sold avg_sales Measures

to_location dollars_cost units_shipped shipper

shipper_key shipper_name location_key shipper_type

The Classic Star Schema

Multi-dimensional data models are needed for the creation of data warehousing or OLAP application, in other words, for analytical applications. The classic star schema, is the most frequently used multidimensional model for relational databases. This database schema classifies two groups of data: Facts (sales or quantity, for example) and Dimension attributes (customer, material, time, for example).

Facts are the focus of the analysis of a business activities.

The Fact data (values for the facts) are stored in a highly normalized fact table.

The values of the Dimension attributes are stored in various denormalized Dimension tables (from a semantical point of view: The Dimensions)
Here, logically related dimension attributes are stored as hierarchy (parent-child relationships) within the dimension table. The dimension tables are linked relationally with the central fact table by way of foreign or primary key relationships. The dimensional attribute with the finest level of detail of the corresponding dimension table is a foreign key in the fact table. In this way, all data records in the facts table can be identified uniquely.

 Each classic star schema is made up of exactly one fact table.

The fact table contains the fact data. It contains the facts

sales with the fact data (50,000, 3,000, 100,000) and

quantity with the fact data (100,60,250,)

The central fact table is connected relationally with the surrounding dimension tables via a unique key. The primary key of the fact table is made up of all dimension keys (=foreign keys). In the above graphic, the primary key of the fact table is made up of the dimension keys DAY_ID, CUSTOMER_ID and MATERIAL_ID. The result is that all data records (and so all fact data) in the fact table can be identified uniquely.

Customer Dimension Table

CUSTOMER_ID K100 K200 Customer Name Jogensen Fourier

Material Dimension Table

MATERIAL_ID M1111 M2222 Material Name Monitor Keyboard

Fact Table
DAY_ID 03.01.2002 03.01.2002 CUSTOMER_ID K100 K100 MATERIAL_ID M1111 M2222 Sales 50.000 3.000 Quantity 100 60

Time Dimension Table

CUSTOMER_ID K100 K200 Customer Name Jogensen Fourier

Figure 11: Sales Example: Classic Star Schema

The SAP BW Star Schema

The multi-dimensional model in SAP BW is based on the SAP BW star schema, which was developed as an enhanced (refined) star schema as a response to problems experienced with the classic star schema. The following graphic shows the crossover between the classic star schema and the SAP BW star schema, using the same sales example from the first diagram. For the time being, only components relevant to the modeling view are taken into consideration.

DIM_ID_MATERIAL

Material Group
CUSTOMER_ID Customer Name Customer Dimension Table DIM_ID_CUSTOMER SID_CUSTOMER Fact Table DIM_ID_MATERIAL DIM_ID_CUSTOMER DIM_ID_TIME DIM_ID_MATERIAL SID_CUSTOMER Material Dimension Table

SID_CUSTOMER

External Hierarchy

CUSTOMER_ID City Region

MATERIAL_ID

Sales Volume Quantity

Time Dimension Table DIM-ID_TIME

Material Name

Text Table MATEIAL_ID

Attribute Table

SID_DAY SID_MONTH SID_QUARTER SID_YEAR

Material Group

Attribute Table Figure 14: The SAP BW Star Schema

This graphic shows how the SAP BW-star schema is an enhancement of the classic star schema. The enhancement comes from the fact that the dimension tables do not contain master data information. This master data information is stored in separate tables, called master data tables. This section firstly explains the SAP BW-Star Schema in detail. At the end of the section, both star schemas are compared in terms of their advantages and disadvantages. In the SAP BW-star schema, the distinction is made between two self-contained areas:

InfoCube Master Data Tables/Surrogate ID (SID-) Tables

InfoCube InfoCubes are the central objects of the multi-dimensional model in SAP BW. Reports and analyses are based on these. From a reporting perspective, an InfoCube describes a self-contained data set within a business area, for which you can define queries. An InfoCube (BasicCube) consists of a number of relational tables that are combined on a multi-dimensional basis. In other words, in consists of a central fact table and several surrounding dimension tables. Hint: There are various types of InfoCube in BW. The InfoCube with type BasicCube is the InfoCube relevant for modelling, since only physical objects (objects that contain data) are considered in the modeling within the SAP BW-data model. For this reason, InfoCube always refers to BasicCube in this section. (You can find additional information about other cube types in the Virtual Cubes lesson).

Data Package Dimension Table

Time Dimension Table

DIM_ID_DATENPAKET SID_REQUEST Fact Table DIM_ID_DATAPACKET DIM_ID_TIME DIM_ID_UNIT DIM_ID_CUSTOMER DIM_ID_MATERIAL Revenue Quality

Units Dimension Table

DIM_ID_TIME

DIM_ID_EINHEIT

SID_DAY

SID_CURRENCY

Materiel Dimension Table

Cust Dim Table DIM_ID_CUSTOMER

DIM_ID_,MATERIAL

SID_MATERIAL
SID_CUSTOMER

Figure 15:InfoCube

SAP BW Star Schema

In the SAP BW-star schema, the facts in the fact table are referred to as Key figures and the dimension attributes as characteristics. The dimension table are linked relationally with the central fact table by way of foreign or primary key relationships. In contrast to the classic star schema, characteristics are not components of the dimension tables, in other words, the characteristic values are not stored in the dimension tables. A numerical SID key is generated for each characteristic. This foreign key replaces the characteristic as the component of the dimension table. Here, SID stands for Surrogate ID (replacement key).

Master Data Tables/SID Tables Additional information about characteristics is referred to as master data in the SAP BW. A distinction is made between the following master data types: Attributes Texts (External) Hierarchies

Master Data Tables/SID Tables

Master data information is stored in separate tables, that is independent of InfoCube, in what are called Master data tables (separately for attributes, texts and hierarchies). In the following graphic, for example, the attribute material group is stored in the attribute table, the text description for material name is stored in the text table and the material hierarchy is stored in the Hierarchy table for the characteristic MATERIAL. In this way, the characteristic MATERIAL is the primary key for the master data tables belonging to this characteristic. As was already mentioned in the InfoCubes section above, precisely one numerical SID key is assigned to each characteristic.

This assignment is made in a SID table for the respect characteristic, whereby the characteristic becomes the primary key in the SID table. In the following graphic, the SID key SID_MATERIAL is assigned to the characteristic MATERIAL in the SID table for characteristic MATERIAL. The SID table is connected to the associated master data tables via the characteristic key. Hint: By using the term Hierarchy, we usually mean an arrangement of objects having a 1:N relationship to each other.

Connecting Master Data Tables to an InfoCube

Master data tables are connected to an InfoCube (and thus to the key figures of the fact table) by way of the SID tables. The excavation of master data from the dimension tables using SID technology allows you to use the master data with different InfoCubes. In other words, the master data is InfoCubeindependent, and can be used by several InfoCubes at the same time.

Classic Star Schema in Comparison with the SAP BW-Star Schema Firstly, let us compare terminology for the two schemas.
Classic Star Schema Fact Dimension Attribute Described Attribute Dimension Tables Dimension = Dimension Table BW Star Schema Key Figure Characteristic Attribute Tex External Hierarchies Dimension Tables (contain no master data) Dimension Table, SID Tables (optional)Master Data Tables

Advantages

Advantages and Disadvantages of the Classic Star Schema

Data access runs performantly due to the small number of join operations (there are only join fact tables and the operations between the

involved dimension tables).

Disadvantages
Redundant entries exist in the dimension tables. In contrast to the historization of fact data (the

same

reference is given implicitly via the time

historization of dimensions (

dimension table),

slowly changing dimension) is not easy to model.

 The multi-lingual capability is cumbersome.

Modeling

some

hierarchy

types

(parallel

and

imbalanced hierarchies for example) in a dimension can lead to anomalies. Query performance is also made worse, since aggregates and Basic fact data stored in the same table (fact table).

Advantages of the SAP BW Star Schema

Advantages. The use of automatically generated INT4 keys I (SID keys, DIMID keys) enables faster access to data than via long alpha-numeric keys.

Historizing dimensions Multi-lingual capability Cross-InfoCube use of master data (shared dimensions)

The query performance is improved here as aggregated key figures can be stored in their own fact tables.
Hint: Another enhancement vis--vis the classic star schema is the excavation of aggregated key figures in their own fact tables using the construction of aggregates that were previously not taken into account in the SAP BW-star schema. You can find additional information about aggregates in the Administering InfoCubes & Aggregates unit.

Metadata Repository
Meta data is the data defining warehouse objects. It has the following kinds

Description of the structure of the warehouse

schema, view, dimensions, hierarchies, derived data definition, data mart locations and contents

Operational meta-data
data lineage (history of migrated data and transformation path), currency of data (active, archived, or purged), monitoring information (warehouse usage statistics, error reports, audit trails)

The algorithms used for summarization The mapping from operational environment to the data warehouse Data related to system performance
warehouse schema, view and derived data definitions

Business data
business terms and definitions, ownership of data, charging policies

End of Multi Dimensional Model