Google Cloud Dataproc

Dataproc is a fully managed and highly scalable service for running Apache Hadoop, Apache Spark, Apache Flink, Presto, open source tools and frameworks. Use Dataproc for data lake modernization, ETL, and secure data science, at scale, integrated with Google Cloud This allows you to easily integrate Google Dataproc and allow your enterprise workload automation solution to manage the scheduling of these workflows. You are instantly inheriting the advanced capabilities of your enterprise solution to deliver your digital transformation quicker and more successfully. By extending your existing enterprise automation to Google Dataproc you maintain end-to-end visibility, regain centralized command and control, alerting, SLA management, reporting and auditing.

Return to list

Benefits and Use Cases

Provision clusters on demand, run jobs and shut them down automatically.
Schedule and orchestrate Dataproc jobs using time-based, event-based, or dependency-based triggers.
Monitor job execution status and performance in real time.
Centralize control over Dataproc jobs with other business processes and batch workloads.

Example Use Case: End-to-End Big Data Job Orchestration

Optimize resource utilization and streamline data workflows with automation for a Spark-optimized Dataproc cluster to deploy, execute a machine learning data pipeline that processes Cloud Storage datasets, shut down on job completion, and seamlessly chain with data ingestion, post-processing, or reporting tasks.

Automic Features

List, Start and Stop Cluster

Submit Job to cluster -submit and submitAsOperation

Workflow Template instantiate -Normal and Inline
View execution log for problem diagnosis
Alert on failure of Google DataProc
Incorporate Google DataProc processing in existing enterprise automation workflows including dependency enforcement