Skip to content

perf: bulk-prefetch persisted grades in ProblemGradeReport - #38944

Open
blarghmatey wants to merge 4 commits into
openedx:masterfrom
mitodl:perf/problem-grade-report-bulk-prefetch
Open

blarghmatey wants to merge 4 commits into
openedx:masterfrom
mitodl:perf/problem-grade-report-bulk-prefetch

Conversation

@blarghmatey

@blarghmatey blarghmatey commented Jul 31, 2026 •

Copy link
Copy Markdown
Contributor

What

Addresses three of the five findings in #38943. Finding 4 (streaming the report upload through DjangoStorageReportStore.store) is in #38946, because it touches shared storage code used by every instructor-task report, not only grade reports. Finding 3 (per-learner caches) turned out not to apply to the grade reports, so this PR leaves it out; see below.

Why it matters

ProblemGradeReport read persisted grades one learner at a time, unlike CourseGradeReport, whose _rows_for_users opens with a _CourseGradeBulkContext. With nothing prefetched, PersistentSubsectionGrade.bulk_read_grades missed the per-course RequestCache and fell through to a query per learner, and this report walks course_grade.problem_scores, so it touches more subsection data per learner than the course report does.

Measured with assertNumQueries (authz tables ignored, as in the file's other query-count tests) around the report, in a course with a graded problem that every learner has answered, so each one has persisted course and subsection grades:

learners 5 25
before 69 289
after 56 216

Before fits 11N + 14, after fits 8N + 16. The prefetch removes three queries per learner; what still scales is read per learner while computing problem scores (courseware state, submission scores and the course access checks), and is out of scope here. At the report's 100-learner batch that is about 816 queries per batch against 1,114 before.

Changes

Finding 1, bulk prefetch. New _ProblemGradeBulkContext, deliberately narrower than _CourseGradeBulkContext: the problem report emits no cohort, team, certificate or course-tag columns, so prefetching those would be wasted work. It loads persisted course and subsection grades plus enrollment states, which is what the rows read.

Finding 2, double block-structure load. graded_scorable_blocks_header now passes the already-loaded structure to grading_context() instead of calling grading_context_for_course(), which re-enters get_course_in_cache. BlockStructureManager.get_collected() deserializes fresh from the cache backend on every call, so a large course paid that cost, and its peak allocation, twice. CourseGradeReport already does this correctly.

Finding 5, dead constant. USER_BATCH_SIZE was defined on CourseGradeReport but never referenced, so the effective batch size was grouper's hardcoded default and tuning the constant did nothing. Moved to GradeReportBase and wired through, so both reports honour it.

Finding 3 is not addressed. The grade reports read through CourseGradeFactory.read and SubsectionGradeFactory.create(read_only=True), and that path never fills the visible-blocks or subsection-override RequestCache entries; only update(force_update_subsections=True), update_or_create_grade and bulk_create_grades do. With persisted grades for every learner and a batch size of 2, both caches held 0 entries before and after every batch for both ProblemGradeReport and CourseGradeReport, so a clear between batches would remove nothing.

Testing

  • lms/djangoapps/instructor_task/tests/test_tasks_helper.py: 104 passed.
  • lms/djangoapps/grades/tests/: 777 passed.
  • TestProblemGradeReport::test_query_count_per_learner pins 56 queries at 5 learners and 216 at 25. Against the pre-PR grades.py it fails with 69 executed, 56 expected. Asserting two cohort sizes means a change in the per-learner cost fails the test, not only a change in the fixed cost.
  • The pre-existing TestInstructorGradeReport::test_query_counts assertion for CourseGradeReport (48 queries) still passes unmodified, so the shared USER_BATCH_SIZE change does not alter that report.

Backward compatibility

CSV contents and format are unchanged. No schema or data migration, and no new public API.

Context

Found while investigating a 5,154-learner course whose problem grade report took about 80 minutes end to end, of which the CSV upload was 0.17s, so nearly all of it was in the per-learner compile loop. Related: #38911 proposes bringing the same CourseGradeReport bulk-prefetch pattern to the synchronous CCX grade report; same defect class, different code path.

…arner caches

ProblemGradeReport read persisted grades one learner at a time, unlike
CourseGradeReport, whose _rows_for_users opens with a _CourseGradeBulkContext.
With nothing prefetched, PersistentSubsectionGrade.bulk_read_grades missed the
per-course RequestCache and fell through to a query per learner -- and this
report walks course_grade.problem_scores, so it touches more subsection data per
learner than the course report does.

Measured with CaptureQueriesContext over the report, varying cohort size:

    learners     5    10    25    50
    before      20    30    60   110     (fits 2N + 10)
    after       13    13    13    13

Adds a regression test asserting the flat count at two cohort sizes; a
single-size assertion would pass while a per-learner read crept back in.

_ProblemGradeBulkContext is deliberately narrower than _CourseGradeBulkContext:
the problem report emits no cohort, team, certificate or course-tag columns, so
prefetching those would be wasted work. It loads persisted course and subsection
grades plus enrollment states, which is what the rows actually read.

Also in the same report path:

  * Pass the already-loaded block structure to grading_context() instead of
    calling grading_context_for_course(), which re-enters get_course_in_cache.
    BlockStructureManager.get_collected() deserializes fresh from the cache
    backend on every call, so a large course paid that cost, and its peak
    allocation, twice. CourseGradeReport already does this correctly.

  * Clear the visible-blocks and subsection-override caches between batches.
    RequestCache is only flushed when the task ends, and unlike the course-keyed
    grade prefetches -- replaced wholesale each batch -- these are keyed per
    (user, course) and only ever grow, though nothing reads an entry again once
    that learner's row is written. The _clear_caches hook already existed and
    was called per batch; the base implementation was empty. Adds the missing
    clear_prefetched_data() counterparts to VisibleBlocks and
    PersistentSubsectionGradeOverride, and a
    clear_prefetched_grade_overrides_and_visible_blocks() api function pairing
    with the existing prefetch_grade_overrides_and_visible_blocks().

  * Wire USER_BATCH_SIZE through to grouper(). It was defined on
    CourseGradeReport but never referenced, so the effective batch size was
    grouper's hardcoded default and tuning the constant did nothing. Moved to
    GradeReportBase so both reports honour it.

CSV contents and format are unchanged by all of the above.

Refs: openedx#38943

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ki1NYMjSjVB4uz1gdgsjBh
@openedx-webhooks openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Jul 31, 2026
@openedx-webhooks

Copy link
Copy Markdown

Thanks for the pull request, @blarghmatey!

This repository is currently maintained by @openedx/wg-maintenance-openedx-platform-oncall.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

  • If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
    • This process (including the steps you'll need to take) is documented here.
  • If it doesn't, simply proceed with the next step.
🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

  • Dependencies

    This PR must be merged before / after / at the same time as ...

  • Blockers

    This PR is waiting for OEP-1234 to be accepted.

  • Timeline information

    This PR must be merged by XX date because ...

  • Partner information

    This is for a course on edx.org.

  • Supporting documentation
  • Relevant Open edX discussion forum threads
🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details
Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

  • The size and impact of the changes that it introduces
  • The need for product review
  • Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

@blarghmatey

Copy link
Copy Markdown
Contributor Author

@openedx/wg-maintenance-openedx-platform-oncall — this is ready for engineering review.

Checklist from the bot above:

Short version: ProblemGradeReport read persisted grades one learner at a time, where CourseGradeReport already bulk-prefetches per batch. Measured with CaptureQueriesContext while varying cohort size, the query count fit 2N + 10 before and is a flat 13 after — so a 5,000-learner course drops from roughly 10,500 grade-related queries to about 700. A regression test asserts the flat count at two different cohort sizes, since a single-size assertion would keep passing if the per-learner read crept back in.

Also folds in three smaller fixes from the same issue: a double block-structure deserialization, two per-learner RequestCaches that were never evicted across a long task, and a USER_BATCH_SIZE constant that was defined but never wired up.

Related: #38946 does the remaining finding from #38943 (streaming the report upload). Independent of this one — they touch different parts of grades.py and can land in either order. #38911 proposes the same bulk-prefetch pattern for the synchronous CCX grade report.

@mphilbrick211 mphilbrick211 moved this from Needs Triage to Ready for Review in Contributions Aug 3, 2026
@mphilbrick211 mphilbrick211 added the needs reviewer assigned PR needs to be (re-)assigned a new reviewer label Aug 3, 2026
@blarghmatey
blarghmatey requested a balanced review from Copilot November 9, 2026 16:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Tests do not exercise persisted problem grades or verify cache eviction between batches, leaving the intended N+1 prevention and bounded cache retention without adequate regression coverage.

2 open issues
What changed in this PR

This PR improves asynchronous problem grade reporting by bulk-loading learner data and bounding per-learner caches, addressing four performance issues from #38943. Shared report-upload streaming remains out of scope.

Changes:

  • Prefetches grades and enrollment states per batch and reuses the loaded course structure.
  • Clears visible-block and override caches between batches and makes both reports honor USER_BATCH_SIZE.
  • Adds a query-count regression test at two learner population sizes.
File Description
lms/​djangoapps/​instructor_task/​tests/​test_tasks_helper.py Adds query-count coverage across two learner populations.
lms/​djangoapps/​instructor_task/​tasks_helper/​grades.py Adds bulk prefetching, structure reuse, shared cache cleanup, and configurable batching.
lms/​djangoapps/​grades/​models.py Adds cache-clearing methods for visible blocks and subsection overrides.
lms/​djangoapps/​grades/​models_api.py Exposes a helper to clear both per-learner caches.

Review effort: Balanced

The grade reports read grades through CourseGradeFactory.read and
SubsectionGradeFactory.create(read_only=True). That path never fills the
VisibleBlocks or PersistentSubsectionGradeOverride RequestCache entries:
only update(force_update_subsections=True), update_or_create_grade and
bulk_create_grades do. With persisted grades for every learner and a
batch size of 2, both caches held 0 entries before and after every batch
for ProblemGradeReport and CourseGradeReport, so the clear between
batches removed nothing.

Claude-Session: https://claude.ai/code/session_01FGZebF8pmhpsgn5eqD2G83
The query-count test had no graded problems, so the report had no
problem columns and never read a subsection grade. With a graded problem
every learner has answered, the count is 8N + 16 (56 at 5 learners, 216
at 25) against 11N + 14 (69, 289) before the bulk prefetch, so the test
pins both counts instead of asserting a flat one.

Claude-Session: https://claude.ai/code/session_01FGZebF8pmhpsgn5eqD2G83
@blarghmatey blarghmatey changed the title perf: bulk-prefetch grades in ProblemGradeReport and bound its per-learner caches perf: bulk-prefetch persisted grades in ProblemGradeReport Nov 13, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs reviewer assigned PR needs to be (re-)assigned a new reviewer open-source-contribution PR author is not from Axim or 2U

Projects

Status: Ready for Review

Development

Successfully merging this pull request may close these issues.

4 participants