Meet the first AI ​​data preparation solution in BigQuery!

Source: cloud.google.com/blog

In today’s information-driven world, the ability to effectively transform raw data into actionable insights is paramount. However, preparing and cleaning diverse data is a significant challenge. Gartner clients report that they spend 90% or more of their time preparing data (up to 94% in complex industries) for advanced analytics, data science, and data engineering.

Reducing this time and efficiently transforming raw data into insights is critical to staying competitive. In October, Google Cloud introduced BigQuery data preparation, the first AI-powered solution that optimizes and simplifies the data preparation process.

In the preview version, BigQuery data preparation provides a number of capabilities:

• AI-powered insights: BigQuery data preparation leverages Gemini in BigQuery to analyze your data and schemas and provide intelligent suggestions for cleaning, transforming, and enriching your data. This dramatically reduces the time and effort required for manual data preparation.
• Data cleansing and standardization: Easily identify and fix inconsistencies, missing values, and formatting errors in your data.
• Visual data pipelines: An intuitive, low-code visual interface helps both technical and non-technical users easily develop complex data pipelines and leverage the rich and extensible capabilities of BigQuery SQL.
• Data pipeline orchestration: Automate the execution and monitoring of your data pipelines. SQL generated with BigQuery data preparation can become part of a Dataform data pipeline that you can deploy and orchestrate using CI/CD for a shared development experience.

BigQuery data preparation helps ensure the accuracy and reliability of your data, enabling better informed business decisions. BigQuery data preparation automates data quality checks and integrates with other Google Cloud services, such as Dataform and Cloud Storage, providing a unified, scalable environment for your data needs.

$300 in free credit to try Google Cloud Data Analytics

Build intelligent apps that leverage real-time insights with a free $300 credit for new customers. Learn more about Google Cloud solutions and offerings from Wise IT experts:

How it works

Getting started is easy. When you take a sample BigQuery table from BigQuery data preparation, the solution uses state-of-the-art underlying data evaluation models and schemas to generate data preparation recommendations, such as filters and transformation suggestions. For example, it knows how to determine valid date formats by country and which columns can act as join keys, speeding up data processing.

In the example above (using synthetic data), the column “BirthDate” contains two different date formats and is of type STRING. BigQuery data preparation suggests “Convert column “BirthDate” from type string to date with the following format(s): ‘%Y-%m-%d’,’%m/%d/%Y”. After applying the suggestion card, you can check the converted preview data in the DATE format column.

With BigQuery AI data preparation, you can:

• Significantly reduce the time spent identifying data quality issues and cleaning data using Gemini-powered proposal cards
• Customize your own proposal cards by providing an example in the data grid
• Improve operational efficiency by deploying data preparation with staged processing

What BigQuery customers say

Customers are already solving numerous problems using BigQuery data preparation.

GAF is a major roofing manufacturer in North America and uses BigQuery data preparation to build data pipelines in BigQuery.

“GAF is committed to modernizing its ETL infrastructure and adopting its own low-code BigQuery solution. BigQuery data preparation will help our skilled business users and analytics team with data preparation processes to enable self-service analytics.” – Pooja Panchagnula, Director of Enterprise Data Management and Analytics, GAF

mCloud technologies help companies in industries such as energy, construction, and manufacturing optimize the performance, reliability, and sustainability of their assets.

“We get our data streams from our partners. BigQuery data preparation allows our product managers to prepare and process data stream files themselves with minimal or no help from our data engineering team.” – Jim Christian, Director of Product and Technology, mCloud Technologies

Public Value Technologies is a joint venture between two German public broadcasting organizations (ARD).

“Public Value Technologies receives data streams from our media partners for our data mesh solution and AI applications. BigQuery data preparation allows our analysts and scientists to quickly integrate data streams by standardizing and pre-processing them in a low-code format.” – Corbinian Schwinger, Data Engineering Team Leader, Public Value Technologies.

Getting started

With powerful AI capabilities, an intuitive interface, and tight integration with the Google Cloud ecosystem, BigQuery data preparation is set to transform the way organizations manage and prepare data. By automating tedious tasks, improving data quality, and empowering users, this innovative solution reduces the time you spend on data preparation and increases your productivity.