UI latency pausing DAGs from main /dags page (Airflow 3.1.4, Celery + Docker Compose)

[marcosmartinezfco]

Apache Airflow version

Other Airflow 3 version (please specify below)

If "Other Airflow 3 version" selected, which one?

3.1.4

What happened?

When navigating the main DAGs list page /dags UI, clicking the pause toggle for a DAG result in significant latency before the UI reflects the new state. Users see a long delay between the click and the toggle animation updating.

However, when pausing the same DAG from its detail view (inside the DAG view), the action is much faster and responsive.

What you think should happen instead?

Pausing a DAG from the main DAGs list should be as responsive as pausing from the DAG detail view. Latency should be minimal (<1 second ideally) on both pages.

How to reproduce

1. Deploy Airflow 3.1.4 using Docker Compose + Celery.
2. Navigate to the main UI DAGs list page.
   3.Click the pause toggle on any listed DAG.
3. Observe the long delay before the UI updates.
4. Navigate to the individual DAG view and pause the same DAG — observe the faster response.

Operating System

AlmaLinux 9.5 (Teal Serval)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.18.0
apache-airflow-providers-celery==3.14.0
apache-airflow-providers-cncf-kubernetes==10.11.0
apache-airflow-providers-common-compat==1.10.0
apache-airflow-providers-common-io==1.7.0
apache-airflow-providers-common-messaging==2.0.1
apache-airflow-providers-common-sql==1.30.0
apache-airflow-providers-docker==4.5.0
apache-airflow-providers-elasticsearch==6.4.0
apache-airflow-providers-fab==3.0.3
apache-airflow-providers-ftp==3.14.0
apache-airflow-providers-git==0.1.0
apache-airflow-providers-google==19.1.0
apache-airflow-providers-grpc==3.9.0
apache-airflow-providers-hashicorp==4.4.0
apache-airflow-providers-http==5.6.0
apache-airflow-providers-microsoft-azure==12.9.0
apache-airflow-providers-mysql==6.4.0
apache-airflow-providers-odbc==4.11.0
apache-airflow-providers-openlineage==2.9.0
apache-airflow-providers-postgres==6.5.0
apache-airflow-providers-redis==4.4.0
apache-airflow-providers-sendgrid==4.2.0
apache-airflow-providers-sftp==5.5.0
apache-airflow-providers-slack==9.6.0
apache-airflow-providers-smtp==2.4.0
apache-airflow-providers-snowflake==6.7.0
apache-airflow-providers-ssh==3.14.0
apache-airflow-providers-standard==1.10.0

Deployment

Docker-Compose

Deployment details

No response

Anything else?

Environment

• Deployment: Docker Compose with CeleryExecutor
• Webserver / Scheduler / Workers: AWS r7i.8xlarge (ample hardware)
• Database: PostgreSQL Aurora (RDS) cluster with writer instance big enough to handle it
• Number of DAGs: 35

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[Lohith625]

I’d like to take a closer look at this. I’ll start by reproducing the behavior locally and comparing the API calls and UI flow between pausing a DAG from the main /dags list and from the DAG detail page.
My initial plan is to identify where the additional latency is coming from (for example, extra refetches or blocking UI updates) and propose a focused fix if possible.
Please let me know if there are any pointers I should be aware of before starting. Thanks!

[marcosmartinezfco]

I’d like to take a closer look at this. I’ll start by reproducing the behavior locally and comparing the API calls and UI flow between pausing a DAG from the main /dags list and from the DAG detail page. My initial plan is to identify where the additional latency is coming from (for example, extra refetches or blocking UI updates) and propose a focused fix if possible. Please let me know if there are any pointers I should be aware of before starting. Thanks!

Thank you. Not really, the set up is quite simple and I don't really know what's going on. I have another airflow instance with pretty much the same config running on k8s and it feels way smoother.

[Lohith625]

I’d like to take a closer look at this. I’ll start by reproducing the behavior locally and comparing the API calls and UI flow between pausing a DAG from the main /dags list and from the DAG detail page. My initial plan is to identify where the additional latency is coming from (for example, extra refetches or blocking UI updates) and propose a focused fix if possible. Please let me know if there are any pointers I should be aware of before starting. Thanks!

Thank you. Not really, the set up is quite simple and I don't really know what's going on. I have another airflow instance with pretty much the same config running on k8s and it feels way smoother.

Thanks for the additional context — that’s helpful.
The fact that the same setup feels noticeably smoother on Kubernetes suggests this may not be a pure UI issue, but could be related to how the API request or state refresh behaves under Docker Compose (for example, networking, process model, or request blocking).
I’ll try to reproduce this specifically in a Docker Compose + Celery setup and compare:
the API calls made when pausing from the DAG list vs the DAG detail page
response times and any extra refetches or blocking behavior in the UI
I’ll report back once I have more concrete findings. Thanks again for the clarification.

[potiuk]

Could be just lack of memory for Docker.

[Lohith625]

Could be just lack of memory for Docker.

Thanks for the suggestion — that makes sense.

Before proposing any code changes, I’ll focus on verifying whether this is primarily a Docker Compose resource issue versus a UI/API behavior difference. I’ll compare the API calls and response timings when pausing from the DAG list versus the DAG detail page and report back with concrete findings.

If it turns out to be environment-related rather than a code issue, I’ll close the loop here accordingly.

[marcosmartinezfco]

If it turns out to be environment-related rather than a code issue, I’ll close the loop here accordingly.

What environment related issue could potentially cause it? Seems like memory is fine and docker has plenty of resources 32vCPU and 256gb of memory. Quick look with htop and resources look fine, plus historical data from the box seems also fine in grafana.

It's also worth mentioning that the main dags menu takes an awful time to load. When refreshing/loading the page you see the skeletons as the dags are loading for a solid 5-6 seconds whereas in k8s it's pretty much instant.

[potiuk]

One of the reasons might also be not upgrading to latest version. 3.1.5 had some "N+1" dag fixes (too many sql queries generated on dag list page) . What can also slow down things are huge history in dag files that might slow down queries (airflow db clean might help).

Maybe those are limited resources on your database, not vacuumed/cleaned/rebuilt indexes - the usual database maintenance that should happen regardless of what application is using it. You should take a loook at the monitoring of your database, looking at excessive queries, peaks in resource usage, strange errors - possibly looking at plans of queries generated by your database and looking for full-table scans. Generally when database statistics are not updated and cleaned up regularly, your table stats migh wildy differ from the actual data stored and database engine might use sub-optimal ways of planning the queries, whihc - for example -might result in loading all rows to memory instead of just loading indexes.

Also we are just about (next week latest) to have RC of 3.1.6 which you might want to take for a spin before we release it (watch the devlist for announcement) and it has even more small fixes and optimisations in those UI/DB queries generated and it might be that this will help.

Another thing you might take a look at - Fast API (when run in containers) recommends ONE worker per container and multiple containers - so comparing to flask it's way better to have several running api-servers - each in a separate container rather than several workers in a single api-server (as counter-intuitive it might seem).

You can try it all - ideally one-by-one and see if things are improving after every step - it would be great to have feedback if some of that helped.

[marcosmartinezfco]

Also we are just about (next week latest) to have RC of 3.1.6 which you might want to take for a spin before we release it (watch the devlist for announcement) and it has even more small fixes and optimisations in those UI/DB queries generated and it might be that this will help.

Yeah let's see how it goes, hopefully it solves the issue. I'm just surprised on the diff load times between celery executor and k8s executor. Also worth mentioning that on my k8s setup I'm using git bundles and in the docker compose one I use s3 bundles.

[marcosmartinezfco]

One of the reasons might also be not upgrading to latest version. 3.1.5 had some "N+1" dag fixes (too many sql queries generated on dag list page) . What can also slow down things are huge history in dag files that might slow down queries (airflow db clean might help)

So far 3.1.5 has been the version that better behaves out of the list we tried. Started in 3.1.2 and performance improved significantly after 3.1.3

You should take a loook at the monitoring of your database, looking at excessive queries, peaks in resource usage, strange errors - possibly looking at plans of queries generated by your database and looking for full-table scans. Generally when database statistics are not updated and cleaned up regularly, your table stats migh wildy differ from the actual data stored and database engine might use sub-optimal ways of planning the queries, whihc - for example -might result in loading all rows to memory instead of just loading indexes.

DB is cleaned fairly often that why I doubt it's a db issue. All metrics in the aws console look healthy and what I would expect

[marcosmartinezfco]

You can try it all - ideally one-by-one and see if things are improving after every step - it would be great to have feedback if some of that helped.

Seems like none of them made improvements with latency. @potiuk Is there any more info that I can give so that we can try to pinpoint the issue?

Happy to debug more on my side if you have any other ideas.

[potiuk]

Here is another: try 3.1.6rc1 that has been up for testing yesterday morning.

[potiuk]

Converting to discussion -(should be converted already) - let us know how the new releases work for you.

[marcosmartinezfco]

I'm afraid it still the same, no improvement. What's the latency expectation for having multiple s3 dag bundles in airflow. We have around 6 different bundles pointing to 6 different buckets.

Guess I will have to try to monitor/test the networking. Thanks again for the help, I'll keep you posted