Databricks
Databricks is a popular data platform used for ETL processing, analytics and machine learning. Automating execution from within Databricks creates an additional island of automation for IT Operations to manage. This allows you to easily integrate Databricks jobs with your existing enterprise workload automation solution. Simply define your jobs within Databricks allow your enterprise workload automation solution to manage the scheduling of these processes. You are instantly inheriting the advanced capabilities of your enterprise solution to deliver your digital transformation quicker and more successfully. By extending your existing enterprise automation to Databricks you maintain end-to-end visibility, regain centralized command and control, alerting, SLA management, reporting and auditing.
Benefits and Use Cases
- Start and stop Databricks clusters to optimize resource usage and cost.
- Run and monitor Databricks jobs using the Run Now or Runs Submit APIs.
- Centralize control and visibility of your Databricks workloads within your enterprise automation environment.
- Automate complex data and machine learning pipelines across hybrid or multi-cloud environments.
Example Use Case: Scalable Data Pipeline Execution
Automate end-to-end data processing and machine learning workflows across large data volumes by scheduling Databricks clusters, triggering ETL jobs, and submitting training pipelines with complete visibility and control.
Automic Features
- Reduce costs by starting and stopping Databricks Clusters as needed
- Run and Monitor Databricks jobs to completion
- Run jobs using Run Now and Run Submit payloads
- View execution log for problem diagnosis
- Restart Databrick jobs
- Alert on failure of Databrick jobs
- Incorporate Databricks processing in existing enterprise automation workflows including dependency enforcement