Skip to content

perf: stream report uploads instead of buffering the whole file in memory - #38946

Merged
ormsbee merged 6 commits into
openedx:masterfrom
mitodl:perf/stream-report-store-upload
Nov 16, 2026
Merged

ormsbee merged 6 commits into
openedx:masterfrom
mitodl:perf/stream-report-store-upload

Conversation

@blarghmatey

Copy link
Copy Markdown
Contributor

What

Finding 4 of #38943, the piece deliberately left out of #38944 because it touches shared
storage code used by every instructor-task report rather than only the grade reports.

Why

DjangoStorageReportStore.store() read its whole buffer into memory, re-encoded it, and
copied it into a ContentFile before handing it to the storage backend:

buff_contents = buff.read()                       # whole file into RAM
if not isinstance(buff_contents, bytes):
    buff_contents = buff_contents.encode('utf-8') # second copy
buff = ContentFile(buff_contents)                 # third
self.storage.save(path, buff)

Peak memory at the final step was therefore roughly 3x the report size — and it largely
defeated TemporaryFileReportMixin, which exists precisely to keep the report out of
memory. That mixin spills rows to a temp file, and then the upload read the whole thing
straight back in. So enabling instructor_task.use_on_disk_grade_reporting bounded row
generation but left a full-size spike at upload.

The ReportStore docstring has anticipated this for a while:

Should probably refactor later to create a ReportFile object that can simply be appended
to for the sake of memory efficiency, rather than passing in the whole dataset.

Changes

A binary buffer is now passed to the backend as-is, so it streams — S3Boto3Storage
uploads in parts, FileSystemStorage writes in chunks, and peak memory is one chunk
rather than the whole report.

Text buffers still work and still have to be read in full, because the backends require
bytes. Both in-tree callers that pass text (instructor/views/api_v2.py, and the existing
StringIO usage in tests) are unchanged in behaviour. The type is probed with a
zero-length read rather than by inspecting mode attributes, which are not consistently
present across the raw file objects, ContentFile and BytesIO instances that callers
pass.

Two callers move to binary so they take the streaming path:

  • TemporaryFileReportMixin opens its temp files 'w+b' and writes through a
    TextIOWrapper. This is what makes the on-disk grade report bounded end to end rather
    than only during row generation.
  • store_rows() builds its CSV in a BytesIO by the same means.

Both wrappers use newline='' per the csv module's contract — the writer emits its own
line terminators and must not have them translated again — and are detached rather than
closed, so the underlying file outlives the wrapper.

Testing

  • lms/djangoapps/instructor_task/tests/ — 252 passed, up from 243, the nine new
    being three tests across the three report-store configurations.
  • Two failures in test_api_helper.py::ScheduledBulkEmailInstructorTaskTests are
    pre-existing: they fail identically on unmodified master at 387ee5d9 (verified
    by stashing this branch's changes and re-running). Unrelated to this change — scheduled
    bulk-email task creation, not report storage.

The new tests assert the streaming behaviourally rather than by inspecting call args. A
streaming backend reads in bounded chunks (File.chunks and upload_fileobj both pass an
explicit size), so an unbounded read() is the signature of the report being slurped into
RAM. They live in ReportStoreTestMixin, so they run against all three configurations
including the S3 stub, plus round-trip assertions for the binary, text and store_rows
paths.

Backward compatibility

CSV bytes are unchanged. store() keeps accepting text buffers, so no caller in or out of
tree needs to change. No schema or data migration.

Notes for reviewers

  • instructor/views/api_v2.py still passes a text StringIO for the issued-certificates
    report. It works and is correct; it just does not get the streaming benefit. Left alone
    to keep this diff to the storage layer, and that report is small.
  • Stacks conceptually on perf: bulk-prefetch persisted grades in ProblemGradeReport #38944 but does not depend on it — the two touch different parts
    of grades.py and can land in either order.

…mory

DjangoStorageReportStore.store() read its entire buffer into memory, re-encoded
it, and copied it into a ContentFile before handing it to the storage backend:

    buff_contents = buff.read()                       # whole file into RAM
    if not isinstance(buff_contents, bytes):
        buff_contents = buff_contents.encode('utf-8') # second copy
    buff = ContentFile(buff_contents)                 # third
    self.storage.save(path, buff)

That put peak memory at roughly 3x the report size at the final step, and it
undid most of the benefit of TemporaryFileReportMixin: that mixin spills rows to
a temp file specifically to avoid holding the report in memory, then the upload
read the whole thing straight back in. The ReportStore docstring has anticipated
this for some time -- "Should probably refactor later to create a ReportFile
object that can simply be appended to for the sake of memory efficiency".

A binary buffer is now handed to the backend as-is, so it streams:
S3Boto3Storage uploads in parts, FileSystemStorage writes in chunks, and peak
memory is one chunk rather than the whole report.

Text buffers still work, and still have to be read in full because the backends
require bytes. Both in-tree callers that pass text are unaffected in behaviour;
the type is probed with a zero-length read rather than by inspecting mode
attributes, which are not consistently present across the raw file objects,
ContentFile and BytesIO instances that callers pass.

Two callers move to binary so they take the streaming path:

  * TemporaryFileReportMixin now opens its temp files 'w+b' and writes through a
    TextIOWrapper, which is what makes the on-disk grade report actually bounded
    end to end rather than only during row generation.
  * store_rows() builds its CSV in a BytesIO by the same means.

Both wrappers use newline='' per the csv module's contract -- the writer emits
its own line terminators and must not have them translated again -- and are
detached rather than closed, so the underlying file outlives the wrapper.

Tests assert the streaming behaviourally rather than by inspecting call args: a
streaming backend reads in bounded chunks, so an unbounded read() is the
signature of the report being slurped into RAM. Added to ReportStoreTestMixin so
they run against all three configurations, including the S3 stub. Round-trip
tests cover the binary, text and store_rows paths; CSV bytes are unchanged.

Completes the last of the five findings in the issue.

Refs: openedx#38943

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ki1NYMjSjVB4uz1gdgsjBh
@openedx-webhooks

Copy link
Copy Markdown

Thanks for the pull request, @blarghmatey!

This repository is currently maintained by @openedx/wg-maintenance-openedx-platform-oncall.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

  • If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
    • This process (including the steps you'll need to take) is documented here.
  • If it doesn't, simply proceed with the next step.
🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

  • Dependencies

    This PR must be merged before / after / at the same time as ...

  • Blockers

    This PR is waiting for OEP-1234 to be accepted.

  • Timeline information

    This PR must be merged by XX date because ...

  • Partner information

    This is for a course on edx.org.

  • Supporting documentation
  • Relevant Open edX discussion forum threads
🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details
Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

  • The size and impact of the changes that it introduces
  • The need for product review
  • Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

@openedx-webhooks openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Jul 31, 2026
@github-project-automation github-project-automation Bot moved this to Needs Triage in Contributions Jul 31, 2026
…t store test

Upstream's pylint config flags C1803 on `unbounded_reads == []`. Switched to a
truthiness check and added the recorded reads to the assertion message, so a
failure names which unbounded read happened rather than just reporting False.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ki1NYMjSjVB4uz1gdgsjBh
@blarghmatey

Copy link
Copy Markdown
Contributor Author

@openedx/wg-maintenance-openedx-platform-oncall — this is ready for engineering review, with one caveat on the build noted below.

Checklist from the bot above:

The coverage failure is an artifact flake, not this PR

coverage (3.12) fails with:

Couldn't combine from non-existent path 'reports/*'

Evidence that this is infrastructure rather than the diff:

  • Every coverage-producing shard in that same run succeeded — shared-with-lms-1, shared-with-lms-2 and shared-with-cms-1 are all green. The combine step simply found no artifacts to combine.
  • coverage (3.12) passes on perf: bulk-prefetch persisted grades in ProblemGradeReport #38944, which runs the identical workflow from the same fork.
  • This diff touches only lms/djangoapps/instructor_task/ and cannot affect artifact upload or retention.

I can't re-run it myself — gh run rerun reports Must have admin rights to Repository. Could someone with rerun rights kick job 91265942653? I'd rather not push a no-op commit just to retrigger the workflow.

Short version of the change

DjangoStorageReportStore.store() read its whole buffer into memory, re-encoded it, and copied it into a ContentFile before handing it to the storage backend — roughly 3x the report size in RAM at the final step. That largely defeated TemporaryFileReportMixin, which spills rows to a temp file precisely to keep the report out of memory, only for the upload to read the whole thing straight back in.

Binary buffers now stream to the backend, and the two callers that can supply binary (the temp-file mixin and store_rows) were switched over. Text buffers still work unchanged. The new tests assert the streaming behaviourally rather than by inspecting call args, and run against all three report-store configurations including the S3 stub.

Related: #38944 covers the other four findings from #38943. Independent of this one — different parts of grades.py, either order works.

@ormsbee

ormsbee commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Just as a sanity check:

  1. How big do these reports get?
  2. Has the memory impact of this fix been verified on a running instance?

@mphilbrick211 mphilbrick211 moved this from Needs Triage to In Eng Review in Contributions Aug 3, 2026
@mphilbrick211

Copy link
Copy Markdown

Just as a sanity check:

  1. How big do these reports get?
  2. Has the memory impact of this fix been verified on a running instance?

@blarghmatey bumping Dave's comment for you.

@mphilbrick211 mphilbrick211 added the waiting on author PR author needs to resolve review requests, answer questions, fix tests, etc. label Nov 4, 2026
@blarghmatey
blarghmatey requested a balanced review from Copilot November 9, 2026 16:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The streaming test still uses local filesystem storage in the S3 test case, and the mocked S3 save method prevents verification of the changed upload path and bounded-chunk streaming behavior.

1 open issue
What changed in this PR

This PR addresses the upload-memory issue from #38943 by allowing shared instructor-task report storage to stream binary buffers, complementing the grade-generation improvements in #38944.

Changes:

  • Pass binary buffers directly to storage while preserving text-buffer support.
  • Generate CSV reports through UTF-8 text wrappers over binary buffers and temporary files.
  • Add streaming and round-trip tests for report storage.
File Description
lms/​djangoapps/​instructor_task/​tests/​test_models.py Adds binary streaming, text compatibility, and CSV round-trip tests.
lms/​djangoapps/​instructor_task/​tasks_helper/​grades.py Uses binary temporary files for on-disk grade reports.
lms/​djangoapps/​instructor_task/​models.py Streams binary uploads and generates CSV output in binary buffers.

Review effort: Balanced

Our largest problem grade report is 85 MB, while ORA submission-file
zips, which upload_ora2_submission_files builds in a TemporaryFile, reach
several GB, so they are the case streaming the upload matters for.

Claude-Session: https://claude.ai/code/session_01FGZebDghNhg4LQEvB7NWni
@blarghmatey

Copy link
Copy Markdown
Contributor Author

@ormsbee I haven't profiled it on a running instance; the 3x figure comes from reading store() (the full read(), the encode(), then the ContentFile copy). From our production report buckets: the largest problem grade report in the last year is 42.5 MB (84.7 MB all-time), so for CSV reports this is headroom; our grade-report workers peaked at 2.09 GB against a 4 GiB limit over 30 days with no OOM restarts (numbers in #38943). The bigger case is ORA submission-file exports: upload_ora2_submission_files builds the zip in a TemporaryFile and store() then reads it whole, and ours reach 4.5 GB. With this change a binary buffer goes to the storage backend as a File, so that zip streams from disk instead. I've updated the store() docstring in d7ea966 to say so rather than "hundreds of megabytes".

The test settings set GRADES_DOWNLOAD STORAGE_TYPE to localfs, and
ReportStore.from_config checks STORAGE_TYPE before STORAGE_CLASS, so the
S3 test case built a FileSystemStorage. MockS3Boto3Mixin also replaced
S3Boto3Storage.save, so the S3 upload path was never exercised.

The fixture now drops STORAGE_TYPE and replaces only the S3 resource,
with an in-memory stub that consumes uploads in bounded chunks, so
S3Boto3Storage.save and _save run for real.

Claude-Session: https://claude.ai/code/session_01FGZebF8pmhpsgn5eqD2G83
@ormsbee
ormsbee enabled auto-merge (squash) November 16, 2026 21:01
@ormsbee
ormsbee merged commit 1a96396 into openedx:master Nov 16, 2026
42 checks passed
@github-project-automation github-project-automation Bot moved this from In Eng Review to Done in Contributions Nov 16, 2026
@openedx-webhooks openedx-webhooks removed the waiting on author PR author needs to resolve review requests, answer questions, fix tests, etc. label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

open-source-contribution PR author is not from Axim or 2U

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants