Azure Airflow (Workflow Orchestration Manager)
Azure Data Factory's Workflow Orchestration Manager leverages Python-based Directed Acyclic Graphs (DAGs), the core of Apache Airflow, to power your data orchestration workflows. Broadcom's Automation solution makes it simple to integrate these Airflow DAGs with your current enterprise workload automation systems. This means you immediately gain the advanced features of your existing enterprise solution, accelerating your digital transformation efforts. By connecting your enterprise automation with Azure Managed Airflow, you'll maintain complete end-to-end visibility, regain centralized command and control, and benefit from robust alerting, SLA management, reporting, and auditing capabilities across all your workflows.
Benefits and Use Cases
- Automate complex data workflows, pipelines, and task dependencies for greater control and visibility.
- Manage and monitor Airflow jobs with enterprise workloads in a single automation platform.
- View DAG execution status and task-level progress.
- Access execution logs and trigger retries to reduce manual intervention and downtime.
Example Use Case: Data Pipeline Orchestration and Monitoring
Execute scheduled or on-demand Airflow DAGs to orchestrate ETL workflows that extract, transform, and load data across systems. Monitor these DAG status and individual task progress in real time and automatically recover from failures by triggering retries or fallback DAGs based on scheduler logic.
Automic Features
- Schedule execution of Azure Airflow DAGs
- Monitor to completion Airflow DAGs
- Retrieve DAG logs for problem diagnosis
- Alert on failure of DAG processing
- Create dependencies between DAG executions
- Incorporate DAG processing in existing enterprise automation workflows including dependency enforcement