GitOps-Native DagBundle Configuration for Team Self-Onboarding

[rpofuk]

Summary

We need file-based DagBundle configuration with automatic reloading to support GitOps workflows and team self-onboarding in multi-tenant Airflow deployments.

The Problem

Right now, Airflow 3.x DagBundles require JSON configurations embedded directly in airflow.cfg or environment variables. This creates several problems for teams trying to automate onboarding and follow GitOps principles.

Current approach looks like this:

[dag_processor]
dag_bundle_config_list = [{"name": "team1", "classpath": "...", "kwargs": {...}}, ...]

This causes real pain:

• We have to manually convert JSON files to embedded strings
• Every configuration change requires restarting Airflow services
• Automated team onboarding workflows break because of manual steps
• Version control diffs are hard to read when JSON is embedded in INI files
• Not friendly for cloud-native deployments with ConfigMaps

Use Case: Multi-Team GitOps Onboarding

Our platform serves more than 30 teams with decentralized DAG ownership. When a new team requests Airflow access, here's what happens:

1. GitHub workflow automatically generates their configuration
2. Writes dag_bundle_config.json to our Git repository
3. Manual step required: Someone has to convert JSON to a string, update the config file, and restart all Airflow services

This manual intervention completely breaks the self-service model we're trying to build. Teams can't truly onboard themselves if platform engineers need to restart services every time.

What We're Proposing

Let Airflow load DagBundle configuration from files using the file:// protocol, with optional automatic reloading:

[dag_processor]
dag_bundle_config_list = file:///etc/airflow/config/dag_bundle_config.json
dag_bundle_config_list_watch = True  # Optional: auto-reload on change

The configuration file would be standard JSON:

[
  {
    "name": "analytics",
    "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
    "kwargs": {
      "repo_url": "https://github.com/org/team-analytics-dags",
      "tracking_ref": "main",
      "refresh_interval": 60
    }
  }
]

Why This Helps GitOps Workflows

Before this feature:

• Team onboarding takes 2-4 hours because of manual steps
• Every configuration change causes service interruption
• Platform team has to be heavily involved in every onboarding

With this feature:

• Team onboarding down to 10-15 minutes through automated PR workflow
• Zero downtime for configuration updates
• Platform team just reviews PRs, no manual operations

The GitOps workflow becomes simple:

1. Team opens a PR adding their config to dag_bundle_config.json
2. Automated tests validate the JSON schema
3. Platform team approves the PR
4. Merge triggers the deployment pipeline
5. Airflow automatically reloads the configuration without restart
6. Team's DAGs become available immediately

Future Extensions

Once file:// is stable and proven, we could extend the same approach to remote protocols:

• https://config-server.example.com/dag_bundles.json
• s3://bucket/path/dag_bundles.json
• git://repo/config/dag_bundles.json

But starting with local files keeps things simple and covers most use cases.

We're willing to contribute to the implementation if there's interest from the community.

[potiuk]

Feel free to propose a PR - that sounds interesting, but details can be discussed in a PR you open.