DATAOPS – MINDSET PART

Adoption in Dufresne

(JCO. 2022.07.20 Draft version, under construction)

What is DataOps??
“Function within an organization that control the data journey between source and value”
KPIs Real-Time recommendation Route Optimizations Real time - Cross Selling

Increasing Sales Descriptive Analytics KPIs

Optimizing Costs Diagnostic Analytics Math Models Batch
Value Becoming more competitive Near Real-Time
More resilient constant changes Predictive Analytics ML Models
Better relationships w/ customers Prescriptive Analytics IA Models

Control.
this value needs to be delivered consistently and it needs to be delivered FAST!

Data Journey -Agility (DevOps, Agile Development, Lean Manufacturing)

-Best-to-breed is best: architecture. Big Data + Real Time + ML Models
-Governance (Roles, policies), Security (Access control)
-Data Management, Data Dictionary (Gold record, one version of the truth)
-Data Quality

Exponential Growth (Volume)

Structured Near Real-Time Big Data
Sources Growth in Data Variety
Semi Structured Batch Social Networks
Data changing
Non-Structured
Data Journey - Best-to-breed is best?
Sources Data Journey Value

+CDC

+ Bronze, Silver, Gold

+ Catalog (MDM)

Microsoft inc.
Data Journey - Agility

CD + CI + IaC
- Pipelines
- Resources for ML Models

Microsoft inc.
DataOps - Agility

A winning hand is defined as

a small, agile project with a
skilled team and sponsor and
an emotionally mature
environment.

A losing hand is defined as a

large, non-agile project with
an unskilled team and
sponsor, an emotionally
immature environment, and
poor decision latency.

(*) By Standish Group

DataOps - Manifesto

Individuals and interactions over processes and tools

Working analytics over comprehensive documentation

Customer collaboration over contract negotiation

Experimentation, iteration, and feedback over extensive

upfront design

Cross-functional ownership of operations over siloed

responsibilities

https://dataopsmanifesto.org/en/
DataOps - Principles
Value

1. Continually satisfy
your customer

2. Value working analytics 3. Embrace change

16. Monitor quality

18. Improve cycle times 4. It’s a team sport
and performance

17. Reuse
9. Analytics is code
15. Quality is 8. Reflect
paramount 5. Daily interactions
10. Orchestrate
7. Reduce heroism

14. Analytics is 6. Self-organize

11. Make it 12. Disposable 13. Simplicity manufacturing
reproducible environments

https://dataopsmanifesto.org/en/
DataOps - Maturity Models. Sample 1

Where are we today?

SSP, SPG, Margin, Shot on Goals,
TUs, REAC%, PPP%, Financing%,… Where is our goal in 1, 2, or 3 years?

https://www.intel.com/content/www/us/en/analytics/getting-started-advanced-analytics-planning-guide.html
DataOps - Maturity Models. Sample 1

Integration with key Ad-hoc analysis Prebuilt domain ML ML Recommendation

Prescriptive
Diagnostic

Predictive
Descriptive

datasources Drill Down and AI Models for optimal path

Domain-are specific dimensional Metric/KPI
dashboards and hierarchy to root forecasting
reports level Mining of historical
Ad-hoc reporting Drill across multiple data
Basic OLAP sources
Business user define
own metrics
Self-service reporting
Prebuilt process
mining models
Near real-time
reporting

https://www.intel.com/content/www/us/en/analytics/getting-started-advanced-analytics-planning-guide.html
DataOps - Maturity Models.

Which model adopt?

Where are we today?
Where is our goal in 1, 2, or 3 years?

12 Intel Model (descriptive, diagnostic, predictive, prescriptive, cognitive)

13 OECD Model (strategy, governance, Data Management, etc.) Analytics Maturity Models: An Overview. Karol Król, and Dariusz Zdonek
DataOps - Maturity Models.
Level Key areas
i) building analytic models;
ii) deploying analytic models;
iii) managing and operating analytic infrastructure;
iv) protecting analytic assets through appropriate policies and procedures;
v) operating an analytic governance structure; and
vi) identifying analytic opportunities, making decisions, and allocating resources
based upon an analytic strategy.

https://www.sciencedirect.com/science/article/pii/S0268401217300026
DataOps - Maturity Models. Sample 2
DataOps - Maturity Models. Sample 3 – Grossman*

Level 1. Level 2. Level 3. Level 4. Level 5.

Analytic Analytic Modeling Repeatable Strategy driven Strategy driven
Reporting Analytics Repeatable Enterprise
Analytics Analytics
Analyze data Ability to analyze data build and deploy organization uses a Creating a culture of
and built reports. analytic models. Make repeatable process for analytics across the
Summarizing past predictions about building and deploying organization
events future, clustering, analytic models.
networks, etc. measuring the impact
of an analytic model
Strategy KPI driven Be able to measure the selecting the right Organization has an strategy driven
result of a deployed analytic opportunities analytic strategy aligns repeatable analytics
analytic model to pursue and supports the across different
corporate strategy, divisions.
having appropriate analytic governance
analytic infrastructure integrates models
in place from across the
organization to provide
processes and value for the
infrastructure in place organization
to deploy models as a whole
People Business users, + Data scientist, + Data engineer + Senior leader that is Analytic Center of
software developers statisticians responsible analytic Excellence
DevOps for Pipelines. strategy, analytic
IT or product group be IT or product group be the modelers have governance uses commons analytic
able to built reports able to integrate the quick access to all the infrastructure and
and KPIs model into operational data required. Process processes throughout
systems for building, deploying the organization
and refreshing models
DataOps – Next steps

VISION

Q1: Diagnostic. Q2: Maturity Model Level

Current state Desired
Q3: Plan Y1
Today Vision
Year1 Year2 Year3
DataOps – First phase
DataOps’s Team Brian’s Team

DATAOPS MLOPS
- On-Premise jobs to DataLake (or NPs) - Azure synapse (Sql Pool)
- On Premise SSIS - Azure Stream Analytics
- Azure synapse (DataFactory/Pipelines/RealTime) - Azure Machine Learning
- Azure Functions - CosmosDB Abalytics Benefit(value)/Cost$$$
- Azure LogicApps (+PowerAutomate) - PowerBI
- Azure Events (From DevOps Team)
- Azure Spark – DataBricks (high compute consuming processes)
DD
DataOps – First phase

Sources Data Journey Value

+CDC
DataOps – First phase

Q4: Customer Insights will be approved?

DD
New Organization

Scope DevOps Team (Mobile, Desktop, APIs and Web Apps)

Scope DataOps Team (Integrations, ingest and analytics)
MLOPs will be part of same structure? Or another area?

Q5. Transition period??

DD
Thanks!!!!

(JCO. 2022.07.20 Draft version, under construction)