BigQuery: introducing powerful new enterprise data

warehousing features
Thursday, September 29, 2016

Posted by Fausto Ibarra, Product Management Director, Google Cloud Platform

In todays world, data powers the most amazing applications to make businesses successful.
Data forms the basis for todays business intelligence, as well as the foundation for new
machine learning endeavors.
Google BigQuery, our cloud data warehouse, is a core part of Google Cloud Platform's data
and analytics solution. BigQuerys serverless architecture continues to redefine what it means
to be fully managed. It scales seamlessly, analyzes terabytes of data in seconds and supports
concurrent data queries for large organizations, without requiring capacity provisioning or
systems tuning. As always, BigQuery is highly available, and encrypts all data at rest.
Today, were excited to announce an update to BigQuery that dramatically advances cloud
data analytics for large-scale businesses.Easing your move to the cloud with BigQuerys
latest features enable it to seamlessly integrate with a wide array of popular business
intelligence and enterprise data integration tools. With todays update, BigQuery is easier to
use by the millions of developers and business analysts who are familiar with SQL queries,
paving the way for businesses to migrate their analytical workloads to Google Cloud
Platform (GCP).
Specifically:

BigQuery support for Standard SQL, implementing the SQL 2011 standard, is now
generally available.

New ODBC drivers make it possible to use BigQuery with a number of tools ranging
from Microsoft Excel to traditional business intelligence systems such as
Microstrategy and Qlik.

More control and visibility

When handling critical business data, data management becomes a serious challenge. To
address this, BigQuery now offers:

The ability to update, delete and insert rows and columns in BigQuery datasets using
Standard SQL

Integration with Cloud Identity and Access Management to manage fine-grained

security policies for BigQuery users and resources

Monitoring through Google StackDriver to track workload performance and usage

Query sharing via links, to foster knowledge sharing and collaboration within
organizations

We're also now offering monthly flat-rate-pricing. With flat-rate pricing, you perform all the
queries you want, delivering the power and scale of BigQuery with a simple, predictable bill.
In this model, storage is priced separately, allowing you to economically keep as much data
as you want.

Rich partner ecosystem

Choosing the right technology partners is important for any business. From data management
and development to visualization, partners make it simple for our customers to access a broad
set of products and integrate our data analytics platform into their everyday business
operations. We continue to expand the partner ecosystem for BigQuery, including:

Enterprise data integration partners, such as Informatica and Talend, integrate with
BigQuery to connect it to a diverse set of cloud and on-premises data sources

Business intelligence partners, such as Tableau, iCharts and Looker, offer reporting
and dashboarding tools to help businesses visualize their data in BigQuery for all
types of users

To see the full list of our partners, check out our growing list of integrated services and tools.

How businesses are using BigQuery today

Google has invested almost two decades of engineering in data and analytics, a core part of
which is our internal data warehouse, Dremel. Six years ago, we began to offer Dremel to
Google Cloud customers under the name Google BigQuery. Since then, weve seen
businesses come to rely on the simplicity and power of BigQuery much in the same way
Google has.
Coca-Cola European Partners, with 300 million consumers and 25,000 employees, is
leveraging the power of IoT and BigQuery to drive the sale of Coca-Cola products through its
retail partners, including convenience stores, restaurants and other locations. Using BigQuery,
the organization analyzes and segments data collected from beacons located throughout its
large network of distributors. These insights are then used to retarget opted-in consumers
using mobile media app advertisements.
The New York Times deployed BigQuery to help its analysts determine the impact of
content on digital readers. Compared to their previous cloud and hardware data warehouses,
BigQuery's performance and reliability have changed the nature of the job itself. According
to Justin Stile, Director of Analytics: The speed of BigQuery allows an analyst to really dig
into the data and look beyond the results returned. The focus moves from the rote nature of
setting up and optimizing long running queries to understanding the results returned and
drilling down further. Analysts no longer have to decide if the understanding they are hoping
to gain is worth the time it will take for the query to return a result.

Viant, a leading advertising technology company, uses BigQuery to provide data and
information to its advertising customers, including multinational corporations. On a typical
day, the company tracks hundreds of millions of transactions and helps its customers turn that
raw data into actionable intelligence. A common query is to understand a set of actions from a
unique user that led to a certain event, for example, identifying which page he or she visited
before completing a purchase. According to Viant, before BigQuery, using an on-prem
database, that query took 24 hours to complete; with BigQuery it takes 10 seconds.
Customers need real-time information to make fast decisions and dont want to wait for those
insights.

Other GCP data processing services

BigQuery is complemented by a rich set of data processing services on GCP. The two most
commonly associated with BigQuery are:

Google Cloud Dataflow is a fully-managed service to execute data processing

pipelines. As a complement to partners enterprise data integration tools, Dataflow is
ideally suited for high-throughput data preparation, especially in streaming mode.
Examples include ingesting application logs, mobile app events or IoT data streams,
preparing them on-the-fly using advanced windowing mechanisms, and saving them
in BigQuery. The upcoming 1.8 release of the Cloud Dataflow SDK will support
BigQuery Standard SQL and the new BigQuery data types.

Google Cloud Dataproc pairs the versatility of Spark and Hadoop ecosystem with the
ease-of-use and efficiency of GCP. Cloud Dataproc clusters are an excellent
complement to BigQuery, allowing you to perform advanced analytics using tools
from the Apache big data ecosystem. For example, if youre running graph analysis
on data in BigQuery, you can use your favorite Spark graph library via Cloud
Dataproc. To better support enterprise requirements, Dataproc recently released
Hadoop High Availability deployments, integration with Cloud IAM and audit
logging.

To learn more about BigQuery and how to manage your data across all GCP products, check
out our new Data Lifecycle on GCP solutions paper and our new GCP for Data Science page.
You can also sign up for a free trial to test out BigQuery here

Google Cloud Machine Learning: now open to all with

new professional services and education programs
Thursday, September 29, 2016

Posted by Riku Inoue, Product Manager, Google Cloud Platform

Earlier this year at GCP NEXT, we introduced new Cloud Machine Learning products with
the intention to change the way businesses operate and create new customer experiences,
while deepening the insights derived from data.Today, we want to share how Google aims to

help more businesses benefit from the advancements in machine learning, while making it
easier for them to use it.
Google Cloud Machine Learning is now publicly available in beta and can empower all
businesses to easily train quality machine learning models at a faster rate. With its powerful
distributed training capability, you can train models on terabytes of data within hours, instead
of waiting for days. Integrated with Google Cloud Platform (GCP), Cloud Machine Learning
is a fully-managed service that can scale and creates a rich environment
across TensorFlow and cloud computing tools such as Google Cloud
Dataflow, BigQuery, Cloud Storage and Cloud Datalab.
We're also introducing a new feature, HyperTune, that automatically improves predictive
accuracy. It allows data scientists to build better performing models faster by automatically
tuning hyperparameters, instead of manually discovering values that work for their model.
Machine learning can unlock new possibilities for businesses from improving customer
service to more accurately streamlining operations to creating new applications and
experiences.
One of our customers, Airbus Defense and Space, tested the use of Google Cloud Machine
Learning to automate the process of detecting and correcting satellite images that contain
imperfections such as the presence of cloud formations. Historically, this process was time
consuming, prone-to-error and was unable to scale as satellite technology improved the
quality and amount of images available.Satellite images are often used by industries such as
agriculture and civil engineering. Common use-cases are precision farming for predicting
crop yields and identifying the crops health, environmental groups for monitoring forestry,
and city governments for land management. The ability to detect patterns in satellite images
such as the difference between snow and clouds is critical to Airbus Defense and
Spaces users who depend on highly precise, up-to-date and reliable information.
In our tests, Google Cloud Machine Learning enabled us to improve the accuracy and speed
at which we analyze the images captured from our satellites. It solved a problem that has
existed for decades. - Mathias Ortner, Data Analysis and Image Processing Lead

This first image illustrates how

difficult it is to guess the difference between snow and clouds.

With Cloud Machine Learning,

automatic detection is able to identify that clouds are on the top right side (highlighted in
red).
We're also excited to announce a dedicated machine learning practice within our Professional
Services team. Our practice helps immerse businesses into the full capabilities of machine
learning to determine how machine learning can help to solve individual needs.
Today, we're introducing:

Machine Learning Advanced Solutions Lab that allows businesses to engage with
Google Machine Learning engineers to help them solve their complex problems.

The Cloud Start program that offers educational workshops for businesses to learn
about the fundamentals of the public cloud and how to identify opportunities with
advancements such as machine learning.

For more information on these services and the rest of the Professional Services offered,
please contact your Google Cloud Sales Representative or visit our website.
And last, but certainly not the least, were rolling out a certification program to help bring
machine learning to more businesses than ever. The program is based on Googles internal
educational programs and will be taught by Googlers. It's aimed at partners, businesses and
data scientists who want to design, train and deploy accurate machine learning models to gain
insights previously out of reach.
We recognize that machine learning has traditionally required specialized training and
expertise. By opening these programs, more users can learn to create new machine learning
applications.
Ready to try out Cloud Machine Learning? Tell us what you think, and well do our best to
address your feedback and make data science and analysis even simpler for you.