Google Cloud Composer (GCC)
Google Cloud Composer (GCC) is a managed workflow automation service built on Apache Airflow, used to develop, schedule and monitor batch-oriented processing. Automating execution from within GCC creates an additional island of automation for IT Operations to manage. This allows you to easily integrate directed acyclic graphs (Airflow DAGs) with your existing enterprise workload automation solution. You are instantly inheriting the advanced capabilities of your enterprise solution to deliver your digital transformation quicker and more successfully. By extending your existing enterprise automation to GCC you maintain end-to-end visibility, regain centralized command and control, alerting, SLA management, reporting and auditing.
Benefits and Use Cases
- Trigger Airflow DAGs from to automate end-to-end workflows.
- Monitor DAG execution status and track task progress in real time.
- Integrate Composer workflows with other enterprise scheduling jobs for unified orchestration.
- Reduce manual intervention and improve reliability by centralizing job control and monitoring.
Example Use Case: Multi-Source Data Pipeline Orchestration
Automate multi-step data pipelines by triggering a directed acyclic graph (DAG) that extracts data from multiple cloud storage buckets and APIs, executes transformation logic within Airflow using BigQuery and Python operators, and monitor task progress and failures.
Automic Features
- Schedule execution of Apache Airflow DAGs
- Monitor to completion Airflow DAGs
- Retrieve DAG logs for problem diagnosis
- Restart DAG processing
- Alert on failure of DAG processing
- Create dependencies between DAG executions
- Incorporate DAG processing in existing enterprise automation workflows including dependency enforcement