Google Cloud Composer (GCC) Airflow
Google Cloud Composer (GCC) is a managed workflow automation service built on Apache Airflow, used to develop, schedule and monitor batch-oriented processing. Automating execution from within GCC creates an additional island of automation for IT Operations to manage. It allows you to easily integrate directed acyclic graphs (Airflow DAGs) with your existing enterprise workload automation solution. You are instantly inheriting the advanced capabilities of your enterprise solution to deliver your digital transformation quicker and more successfully. By extending your existing enterprise automation to GCC you maintain end-to-end visibility, regain centralized command and control, alerting, SLA management, reporting and auditing.
Benefits and Use Cases
- Automate complex data workflows, pipelines, and task dependencies for greater control and visibility.
- Manage and monitor Airflow jobs with enterprise workloads in a single automation platform.
- View DAG execution status and task-level progress.
- Access execution logs and trigger retries to reduce manual intervention and downtime.
Example Use Case: Data Pipeline Orchestration and Monitoring
Execute scheduled or on-demand Airflow DAGs to orchestrate ETL workflows that extract, transform, and load data across systems. Monitor these DAG status and individual task progress in real time and automatically recover from failures by triggering retries or fallback DAGs based on scheduler logic.
Automic Features
- Schedule execution of Apache Airflow DAGs
- Monitor to completion Airflow DAGs
- Retrieve DAG logs for problem diagnosis
- Alert on failure of DAG processing
- Create dependencies between DAG executions
- Incorporate DAG processing in existing enterprise automation workflows including dependency enforcement