What Is Azure Synapse Data Explorer (Preview) - Azure Synapse Analytics - Microsoft Docs

4/26/22, 2:28 PM What is Azure Synapse Data Explorer (Preview) - Azure Synapse Analytics | Microsoft Docs
Azure / Synapse Analytics /  Ｄ／ 
What is Azure Synapse Data Explorer?

(Preview)
Article • 03/24/2022 • 5 minutes to read • 4 contributors  
In this article
What makes Azure Synapse Data Explorer unique?
When to use Azure Synapse Data Explorer?
Data Explorer pool architecture
Next Steps
Azure Synapse Data Explorer provides customers with an interactive query experience to
unlock insights from log and telemetry data. To complement existing SQL and Apache
Spark analytics runtime engines, the Data Explorer analytics runtime is optimized for
efficient log analytics using powerful indexing technology to automatically index free-
text and semi-structured data commonly found in telemetry data.
To learn more, see the following video:
https://docs.microsoft.com/en-us/azure/synapse-analytics/data-explorer/data-explorer-overview 1/6
What makes Azure Synapse Data Explorer

unique?
Easy ingestion - Data Explorer offers built-in integrations for no-code/low-code,
high-throughput data ingestion, and caching data from real-time sources. Data
can be ingested from sources such as Event Hub, Kafka, Azure Data Lake, open
source agents such as Fluentd/Fluent Bit, and a wide variety of cloud and on-
premises data sources.
No complex data modeling - With Data Explorer, there is no need to build

complex data models and no need for complex scripting to transform data before
it's consumed.
No index maintenance - There is no need for maintenance tasks to optimize data

for query performance and no need for index maintenance. With Data Explorer, all
raw data is available immediately, allowing you to run high-performance and high-
concurrency queries on your streaming and persistent data. You can use these
queries to build near real-time dashboards and alerts, and connect operational
analytics data with the rest of data analytics platform.
Democratizing data analytics - Data Explorer democratizes self-service, big data
analytics with the intuitive Kusto Query Language (KQL) that provides the
expressiveness and power of SQL with the simplicity of Excel. KQL is highly
optimized for exploring raw telemetry and time series data by leveraging Data
Explorer's best-in-class text indexing technology for efficient free-text and regex
search, and comprehensive parsing capabilities for querying traces\text data and
JSON semi-structured data including arrays and nested structures. KQL offers
advanced time series support for creating, manipulating, and analyzing multiple
time series with in-engine Python execution support for model scoring.
Proven technology at petabyte scale - Data Explorer is a distributed system with

compute and storage that can scale independently, enabling analytics on
gigabytes or petabytes of data.
Integrated - Azure Synapse Analytics provides interoperability across data between

Data Explorer, Apache Spark, and SQL engines empowering data engineers, data
scientists, and data analysts to easily, and securely, access and collaborate on the
same data in the data lake.
When to use Azure Synapse Data Explorer?

Use Data Explorer as a data platform for building near real-time log analytics and IoT
analytics solutions to:
Consolidate and correlate your logs and events data across on-premises, cloud,
and third-party data sources.
Accelerate your AI Ops journey (pattern recognition, anomaly detection,
forecasting, and more).
Replace infrastructure-based log search solutions to save cost and increase
productivity.
Build IoT analytics solutions for your IoT data.

Build analytics SaaS solutions to offer services to your internal and external
customers.
Data Explorer pool architecture

Data Explorer pools implement a scale out architecture by separating the compute and
storage resources. This enables you to independently scale each resource and, for
example, run multiple read only computes on the same data. Data Explorer pools consist
of a set of computes running the engine that is responsible for automatically indexing,
compressing, caching, and serving distributed queries. They also have a second set of
computes running the data management service responsible for background system
jobs, and managed and queued data ingestion. All data is persisted on managed blob
storage accounts using a compressed columnar format.
Data Explorer pools support a rich ecosystem for ingesting data using connectors, SDKs,
REST APIs, and other managed capabilities. It offers various ways to consume data for
adhoc queries, reports, dashboards, alerts, REST APIs, and SDKs.
There are many unique capabilities that makes Data Explore the best analytical engine
for log and time series analytics on Azure. If you are interested in learning more about
how Data Explorer works, see Azure Data Explorer white paper .
The following sections highlight the key differentiators.
Free-text and semi-structured data indexing enables near

real time high performance and high concurrent queries
Data Explorer indexes semi-structured data (JSON) and unstructured data (free text) that
makes running queries very performant on this type of data. By default, every field is
indexed during the data ingestion with the option to use a low-level encoding policy to
fine tune or disable the index for specific fields. The scope of the index is a single data
shard.
The implementation of the index depends on the type of the field, as follows:
Field Indexing implementation

type
String The engine builds an inverted term index for string column values. Each string value
is analyzed and split into normalized terms and an ordered list of logical positions,
containing record ordinals, is recorded for each term. The resulting sorted list of
terms and their associated positions is stored as an immutable B-tree.
Field Indexing implementation

type
Numeric The engine builds a simple range-based forward index. The index records the
DateIime min/max values for each block, for a group of blocks and for the entire column within
TimeSpan the data shard.
Dynamic The ingestion process enumerates all "atomic" elements within the dynamic value,
such as property names, values, and array elements, and forwards them to the index
builder. Dynamic fields have the same inverted term index as string fields.
These efficient indexing capabilities enables Data Explore to make the data available in
near-real-time for high-performance and high-concurrency queries. The system
automatically optimizes data shards to further boost performance.
Kusto Query Language

KQL has a large, growing community with the rapid adoption of Azure Monitor Log
Analytics and Application Insights, Microsoft Sentinel, Azure Data Explorer, and other
Microsoft offerings. The language is well designed with an easy-to-read syntax and
provides a smooth transition from simple one-liner to complex data processing queries.
This allows Data Explorer to provide rich Intellisense support and a rich set of language
construct and built-in capabilities for aggregations, time series, and user analytics that
aren't available in SQL for rapid exploration of telemetry data.
Next Steps
Create a Data Explorer pool using Synapse Studio
Create a Data Explorer pool using the Azure portal (Preview)
Recommended content
Quickstart: Create a Data Explorer pool using Synapse Studio (Preview) - Azure
Synapse Analytics
Create a Data Explorer pool using Synapse Studio by following the steps in this guide.
Azure Synapse Data Explorer data ingestion overview (Preview) - Azure

Synapse Analytics
Learn about the different ways you can ingest (load) data in Azure Synapse Data Explorer.
Quickstart: Get started analyzing with Data Explorer pools (Preview) - Azure
Synapse Analytics
In this quickstart, you'll learn to analyze data with Data Explorer.
Azure Synapse Studio notebooks - Azure Synapse Analytics

This tutorial provides an overview on how to create a Power BI dashboard using Apache
Spark and a Serverless SQL pool.
What is Apache Spark - Azure Synapse Analytics

This article provides an introduction to Apache Spark in Azure Synapse Analytics and the
different scenarios in which you can use Spark.
FAQ - Azure Synapse Analytics

FAQ for Azure Synapse Analytics
QuickStart on Azure Synapse lake database and database templates - Azure

Synapse Analytics
Quickstart how to use the synapse lake database and the database templates
Azure Synapse Spark Common Data Model (CDM) connector - Azure Synapse
Analytics
Learn how to use the Azure Synapse Spark CDM connector to read and write CDM entities in
a CDM folder on ADLS.
Show more Ｓ

What Is Azure Synapse Data Explorer (Preview) - Azure Synapse Analytics - Microsoft Docs

Uploaded by

Copyright:

Available Formats

What Is Azure Synapse Data Explorer (Preview) - Azure Synapse Analytics - Microsoft Docs

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

What Is Azure Synapse Data Explorer (Preview) - Azure Synapse Analytics - Microsoft Docs

Uploaded by

Copyright:

Available Formats

What are some of the unique features of Azure Synapse Data Explorer?

What are some of the unique features of Azure Synapse Data Explorer?

How does Azure Synapse Data Explorer handle data ingestion and indexing?

How does Azure Synapse Data Explorer handle data ingestion and indexing?

4/26/22, 2:28 PM What is Azure Synapse Data Explorer (Preview) - Azure Synapse Analytics | Microsoft Docs

Azure ​/ Synapse Analytics ​/  Ｄ ／ 

What is Azure Synapse Data Explorer?

To learn more, see the following video:

What makes Azure Synapse Data Explorer

No complex data modeling - With Data Explorer, there is no need to build

No index maintenance - There is no need for maintenance tasks to optimize data

Proven technology at petabyte scale - Data Explorer is a distributed system with

Integrated - Azure Synapse Analytics provides interoperability across data between

When to use Azure Synapse Data Explorer?

Build IoT analytics solutions for your IoT data.

Data Explorer pool architecture

The following sections highlight the key differentiators.

Free-text and semi-structured data indexing enables near

Field Indexing implementation

Field Indexing implementation

Kusto Query Language

Azure Synapse Data Explorer data ingestion overview (Preview) - Azure

Azure Synapse Studio notebooks - Azure Synapse Analytics

What is Apache Spark - Azure Synapse Analytics

FAQ - Azure Synapse Analytics

QuickStart on Azure Synapse lake database and database templates - Azure

You might also like

Azure / Synapse Analytics /  Ｄ／ 