feat: 新增域名级QPS限流功能(重构了重提的PR)#305
Closed
ShellMonster wants to merge 2 commits intoBoris-code:masterfrom
Closed
Conversation
added 2 commits
November 25, 2025 17:49
新增功能: - 支持按域名配置独立的QPS限制 - 支持精确匹配和通配符匹配(*.example.com) - 支持单机模式(LocalTokenBucket)和分布式模式(RedisTokenBucket) - 令牌桶算法实现精确的QPS控制 架构设计: - QPSScheduler:单线程调度器,管理DelayHeap和ReadyQueue - DomainRateLimiter:域名级限流管理器,自动路由到对应令牌桶 - 零侵入性:QPS关闭时代码流程与原始完全一致 支持的爬虫类型: - AirSpider(单机内存队列) - Spider/BatchSpider/TaskSpider(分布式Redis队列) 配置项: - DOMAIN_RATE_LIMIT_ENABLE:是否启用 - DOMAIN_RATE_LIMIT_RULES:域名QPS规则 - DOMAIN_RATE_LIMIT_DEFAULT:默认QPS - DOMAIN_RATE_LIMIT_STORAGE:存储模式(local/redis) 测试验证: - 单机QPS精度测试:误差 < 2% - 分布式多进程共享QPS测试:2进程共享配额,误差 1.7% 文档: - 新增 docs/source_code/域名级QPS限流.md Author: ShellMonster
问题:当就绪队列达到max_prefetch上限时,所有工作线程都卡在 submit()的背压循环中,无法调用get_ready_request()消费队列, 导致死锁。 解决方案: 1. 调整_get_request流程:先尝试从就绪队列获取,再提交新请求 2. submit使用非阻塞模式(block=False),避免背压死锁 3. 新增调度器状态日志,便于监控和调试
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
新增功能:
架构设计:
支持的爬虫类型:
配置项:
测试验证:
文档:
AirSpider 原始架构示例图(详情看PR文档,包含测试结果)
AirSpider 含QPS架构示例图(详情看PR文档,包含测试结果)