Repository navigation
perf(storage): eliminate per-object V2 route SQL with bounded per-library cache - #48
Merged
Merged
Conversation
servicememeai
marked this pull request as ready for review
October 27, 2026 10:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
目标与边界
CloudFile 的长期目标调整为:单库 1000 万文件/目录条目下,仍具有良好的交互、搜索、写入性能;并要求超大库负载不因共用存储路径解析而拖慢其他库。
本 PR 是第一个低风险性能切片:消除 V2 per-library local storage 在 C Commit/FS/Block 和 Go Read/Write/Exists/Stat 中每对象重复查询
RepoStorageId等表的问题。不声称单库 1000 万已经压测通过,不更改数据库表/索引/SQL 语义,不改 Seafile FS/Commit/Block 格式。最小变更
invalidateRoute当前仅是测试 seam,没有生产管理调用方。线下修复/改路由须先停写并重启进程,不提供热迁移或即时跨进程失效保证。SQL 查询、目录 stat 和磁盘 I/O 都不持有缓存全局锁;一个库的慢查询不会串行化其他库。多库影响
不同库的 Branch/head 和物理目录独立,但共享 DB 连接、CPU、IOPS、进程工作池和搜索资源。本 PR 只解决共同热点之一:存储路由请求的重复 SQL 与锁竞争。后续的单目录大对象解析、10M 索引构建/增量追赶、写入事务竞争、GC/搜索后台任务与真实多库压测需独立按证据优化。
定向验证与合入条件
新增/补充 Go 与 C storage fixture:重复对象读取只查一次、两个库映射隔离、单库失效与 TTL 到期、数据库失败不缓存、挂载移除拒绝、慢库路由查询不阻塞其他库的缓存操作。旧的 CE / V2 / VirtualRepo / 历史路由语义不变。
研发验收指标和多库负载矩阵见新增的
doc/target-single-library-10m.md。可运行:
cd fileserver && go test ./objstore -count=1 -race(完整包,覆盖TestV2ObjectStoreSelectsManagedRoute,不能再仅用-run 'TestV2Storage')bash tests/local_storage/run.sh.github/workflows/cloudfile-checks.yml,要求checks和build-c均为 SUCCESS现有验证证据及阻断修复:
0a1f764的 CloudFile checks run 40489186919:build-c通过(102 passed / 9 skipped / 191 subtests);checks未通过,原因是cf_v2_route_wiring_test.go模拟改掉lookup却保留合法缓存命中。lookup后显式invalidateRoute(repoID);保留“坏绑定必须拒绝”的断言。这是测试模型的失效语义修复,不是生产放宽校验。已知边界
V2 路由最多 10 秒、CE/虚拟库路由最多 1 秒缓存已经成功解析过的正向物理路由;常规映射视为不可变;人工离线改映射应先停止写入并重启相关进程,不可依赖 1/10 秒 TTL 当作在线热迁移协议。缓存命中不能替代身份/ACL 授权。GC 或手工删除目录时,
lstat仍检查目录是否存在且为真实目录,但不证明底层挂载设备身份未改变。其他 RepoStorageId 写入点属于单独离线/配置约束,不加入本 PR 的在线失效框架。已知的冷缓存并发 miss 会有重复 SQL、旧查询结果迟到可能短时覆盖较新缓存;目前无生产动态路由改写入口,优先按实际负载观察,不为假设问题增加单飞/版本账本。
索引导入/批量更新期间的搜索延迟按产品决策可降级,本 PR 不建设索引追赶加速。