Google Cloud Dataflow
Google Cloud Dataflow is a serverless data processing services for stream and batch data with automated infrastructure provisioning and auto scaling as data grows. Automating the execution of jobs from within Cloud Dataflow creates an additional island of automation for IT Operations to manage. This allows you to easily integrate Google Cloud Dataflow workflows with your existing enterprise workload automation solution. Simply define your jobs within Google Cloud Dataflow and allow your enterprise workload automation solution to manage the scheduling of these workflows. You are instantly inheriting the advanced capabilities of your enterprise solution to deliver your digital transformation quicker and more successfully. By extending your existing enterprise automation to Google Cloud Dataflow you maintain end-to-end visibility, regain centralized command and control, alerting, SLA management, reporting and auditing.
Benefits and Use Cases
- Integrate Dataflow pipelines and template jobs into broader enterprise workflows with time-based or event-driven scheduling
- Monitor pipeline execution status and job logs.
- Automate failure handling by configuring alerts and retries based on execution outcomes.
Example Use Case: Real-Time Data Transformation Pipeline
Automate the ingestion and enrichment of streaming data via Pub/Sub and BigQuery, trigger downstream processing jobs on completion, and ensure seamless workflow orchestration through automated failure notifications and retries.
Automic Features
- Run and Monitor Cloud Dataflow template jobs to completion
- View execution log for problem diagnosis
- Alert on failure of Cloud Dataflow template jobs
- Incorporate Cloud Dataflow processing in existing enterprise automation workflows including dependency enforcement