Google Cloud Dataproc
Dataproc is a fully managed and highly scalable service for running Apache Hadoop, Apache Spark, Apache Flink, Presto, open source tools and frameworks. Use Dataproc for data lake modernization, ETL, and secure data science, at scale, integrated with Google Cloud This allows you to easily integrate Google Dataproc and allow your enterprise workload automation solution to manage the scheduling of these workflows. You are instantly inheriting the advanced capabilities of your enterprise solution to deliver your digital transformation quicker and more successfully. By extending your existing enterprise automation to Google Dataproc you maintain end-to-end visibility, regain centralized command and control, alerting, SLA management, reporting and auditing.
Benefits and Use Cases
- Provision clusters on demand, run jobs and shut them down automatically.
- Schedule and orchestrate Dataproc jobs using time-based, event-based, or dependency-based triggers.
- Monitor job execution status and performance in real time.
- Centralize control over Dataproc jobs with other business processes and batch workloads.
Example Use Case: End-to-End Big Data Job Orchestration
Optimize resource utilization and streamline data workflows with automation for a Spark-optimized Dataproc cluster to deploy, execute a machine learning data pipeline that processes Cloud Storage datasets, shut down on job completion, and seamlessly chain with data ingestion, post-processing, or reporting tasks.
Automic Features
- List, Start and Stop Cluster
- Submit Job to cluster -submit and submitAsOperation
- Workflow Template instantiate -Normal and Inline
- View execution log for problem diagnosis
- Alert on failure of Google DataProc
- Incorporate Google DataProc processing in existing enterprise automation workflows including dependency enforcement