Repository navigation
fix(files): run the provider listing off the loop in acleanup_provider_files - #8642
Conversation
Strix Security ReviewWarning This pull request has 2 commits after the last Strix review ( ✅ No open security findings. PR summary: Moves the synchronous provider file listing in `acleanup_provider_files(delete_all_from_provider=True)` off the event loop via `asyncio.to_thread`, and adds a test that fails if the listing blocks the loop. Review summaryThe diff wraps the synchronous provider listing in Review 1 on this PR · Reviewed by Strix |
…r_files acleanup_provider_files(delete_all_from_provider=True) called the synchronous uploader.list_files() — a provider HTTP listing (Anthropic/ OpenAI files list, Bedrock paginated list_objects_v2, Gemini files.list) — directly on the running event loop. Deletes were already off-loop via adelete under a semaphore, but the listing froze every other coroutine for the duration of the provider round-trips. Change: files = await asyncio.to_thread(uploader.list_files) in the delete_all_from_provider branch; the sync cleanup_provider_files twin is unchanged. Why: an async API must not block the loop for seconds-scale network I/O. Proof: tests/test_acleanup_listing_nonblocking.py — a stub uploader whose list_files sleeps 0.5s freezes a 0.05s ticker coroutine on main (~3 ticks) and lets it keep ticking with the fix (~12); full lib/crewai-files suite 180 passed, 4 skipped. Risk: low — identical behavior except the listing runs in a worker thread; error handling and result ordering unchanged. 🤖 Generated with ZCode (AI agent) on behalf of Wilson0511 Co-Authored-By: ZCode <[email protected]>
34ecc95 to
cd6a9ae
Compare
Vidit-Ostwal
left a comment
There was a problem hiding this comment.
Reviewed the event-loop-blocking cleanup path and its regression coverage. Moving the synchronous provider listing to asyncio.to_thread is the minimal correct fix, and the ticker-based test verifies that other coroutines keep running.
Summary
acleanup_provider_files(..., delete_all_from_provider=True)calleduploader.list_files()— the sync method that hits the provider's HTTP API — directly on the event loop. Deletes were already off-loop (adeleteunder a semaphore +gather), but the listing step froze every other coroutine for the duration of the provider listing. The listing now runs viaasyncio.to_thread, a one-line fix with no uploader API change (alist_filesdoes not exist on the uploaders).Closes #8641
Change
lib/crewai-files/src/crewai_files/cache/cleanup.py:files = await asyncio.to_thread(uploader.list_files)in thedelete_all_from_providerbranch ofacleanup_provider_files(the synccleanup_provider_filestwin is unchanged).lib/crewai-files/tests/test_acleanup_listing_nonblocking.py: regression test with a stub uploader whoselist_filessleeps 0.5 s; a ticker coroutine must keep ticking while the listing runs. Fails onmain(ticker frozen), passes with the fix.Proof
main:AssertionError: event loop was blocked during listing: 3 ticks(expected ≥ 6).1 passed; fulllib/crewai-filessuite:180 passed, 4 skipped.Risk
Low: behavior is identical except the listing executes in a worker thread; result ordering and error handling (
except Exceptionpath) are unchanged.🤖 Generated with ZCode (AI agent) on behalf of Wilson0511
Co-Authored-By: ZCode [email protected]
Note
Low Risk
Same cleanup semantics and error handling; only moves synchronous listing to a worker thread so the event loop stays responsive.
Overview
Fixes event-loop blocking when
acleanup_provider_filesruns withdelete_all_from_provider=True. Provider file listing used synchronouslist_files()(HTTP) on the asyncio loop while deletes were already async; other coroutines stalled for the whole listing call.Listing now runs with
await asyncio.to_thread(uploader.list_files)so the sync uploader API stays unchanged. The synccleanup_provider_filespath is untouched.Adds a regression test with a stub uploader whose
list_filessleeps 0.5s and a ticker task that must keep firing during cleanup—fails if listing blocks the loop.Reviewed by Cursor Bugbot for commit 34ecc95. Bugbot is set up for automated code reviews on this repo. Configure here.