Apache Airflow
Apache Airflow is an open-source workflow management platform used to develop, schedule and monitor batch-oriented processing. Automating execution from within Apache Airflow creates an additional island of automation for IT Operations to manage. This allows you to easily integrate directed acyclic graphs (Airflow DAGs) with your existing enterprise workload automation solution. You are instantly inheriting the advanced capabilities of your enterprise solution to deliver your digital transformation quicker and more successfully. By extending your existing enterprise automation to Apache Airflow you maintain end-to-end visibility, regain centralized command and control, alerting, SLA management, reporting and auditing.
Benefits and Use Cases
- Automate complex data workflows, pipelines, and task dependencies for greater control and visibility.
- Manage and monitor Airflow jobs with enterprise workloads in a single automation platform.
- View DAG execution status and task-level progress.
- Access execution logs and trigger retries to reduce manual intervention and downtime.
Example Use Case: Data Pipeline Orchestration and Monitoring
Schedule, execute, and monitor critical data pipelines across cloud and on-prem environments.
Automic Features
- Schedule execution of Apache Airflow DAGs
- Monitor to completion Airflow DAGs
- Retrieve DAG logs for problem diagnosis
- Alert on failure of DAG processing
- Create dependencies between DAG executions
- Incorporate DAG processing in existing enterprise automation workflows including dependency enforcement