Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ Changed
* **Breaking**: ``GET /api/authz/v1/scopes/`` no longer accepts ``course`` and ``library`` as
``scope_type`` values. Use ``course-v1`` and ``lib`` instead. Requests with the old values now
return a 400 validation error, the same as ``/roles/``.
* **Breaking**: ``CourseScope`` references its course only through the course key in ``external_key`` (ADR 0035). Migration ``0013_remove_coursescope_course_overview`` copies the course key of older course scopes into ``external_key`` and removes the ``course_overview`` foreign key, so ORM filters on ``course_overview``, creating a scope with a ``course_overview`` argument and ``CourseOverview.authz_scopes`` no longer work. ``CourseScope.course_overview`` remains as a read-only property, and deleting a course still deletes its course scope and role assignments.

Removed
=======
Expand Down
1 change: 1 addition & 0 deletions docs/how-tos/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ How-tos
:maxdepth: 1

use-built-in-constants
resolve-course-scope-key-conflicts
39 changes: 39 additions & 0 deletions docs/how-tos/resolve-course-scope-key-conflicts.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
Resolve Course Scopes That ``0013_remove_coursescope_course_overview`` Cannot Update
####################################################################################

Migration ``0013_remove_coursescope_course_overview`` copies the course key of each course scope created before ``0010_scope_external_key`` into ``external_key``, and then removes ``course_overview_id``. ``external_key`` is unique across all scopes, and MySQL compares it with the table collation, which ignores case and accents in ``utf8mb4_unicode_ci`` and ``utf8mb4_0900_ai_ci``, so a course key can collide with a key that another scope already holds. When that happens, the migration copies every other key and stops before it removes the column, with an error like this:

.. code-block:: text

CourseScopeKeyConflictError: Cannot copy the course key of these course scopes into external_key, because another scope already holds an equal key: 12: course-v1:OpenedX+DemoX+2024. Resolve them as the openedx-authz how-to "Resolve Course Scopes That 0013_remove_coursescope_course_overview Cannot Update" describes, then run migrate again.

Each ``<scope id>: <course key>`` pair names a course scope whose key is still only in ``course_overview_id``. Since the database treats both keys as equal, both scopes refer to the same course, so the role assignments of the older scope move to the scope that holds the key, and the older scope is removed. Follow these steps for every pair in the error.

#. Back up the database.

#. Open a MySQL shell on the LMS database and find the scope that holds the key. This query returns its ``id``:

.. code-block:: sql

SELECT id, external_key FROM openedx_authz_scope
WHERE external_key = 'course-v1:OpenedX+DemoX+2024';

#. List the role assignments of both scopes, to know what moves. Replace ``12`` with the scope from the error and ``34`` with the ``id`` from the previous step:

.. code-block:: sql

SELECT r.scope_id, c.v0 AS subject, c.v1 AS role, c.v2 AS scope
FROM openedx_authz_extendedcasbinrule r
JOIN casbin_rule c ON c.id = r.casbin_rule_id
WHERE r.scope_id IN (12, 34);

#. Move the role assignments of the older scope to the scope that holds the key, and remove the older scope:

.. code-block:: sql

UPDATE openedx_authz_extendedcasbinrule SET scope_id = 34 WHERE scope_id = 12;
DELETE FROM openedx_authz_coursescope WHERE scope_ptr_id = 12;
DELETE FROM openedx_authz_scope WHERE id = 12;

#. Run ``./manage.py lms migrate`` again, which copies only the keys that are still missing and then removes ``course_overview_id``.

46 changes: 25 additions & 21 deletions src/openedx_authz/handlers.py
Original file line number Diff line number Diff line change
Expand Up @@ -143,36 +143,40 @@ def handle_org_waffle_flag_change(sender, instance, **kwargs) -> None:
post_save.connect(handle_org_waffle_flag_change, sender=WaffleFlagOrgOverrideModel)


def backfill_course_scope(sender, instance, **kwargs): # pylint: disable=unused-argument
def delete_course_scope_on_course_deletion(sender, instance, **kwargs): # pylint: disable=unused-argument
"""
Link a pending CourseScope to the CourseOverview that was just created for it.
Delete the course scope of a CourseOverview that was just deleted.

get_or_create_for_external_key() (openedx_authz/models/scopes.py) allows assigning a
role for a course key before its CourseOverview exists, leaving course_overview null.
This backfills that FK once the course shows up, e.g. after a course rerun finishes
cloning. See CourseScope.link_pending_scope() for the actual lookup/update.
Course scopes reference their course only by its key, so nothing in the database deletes
the scope with its course. Deleting the scope deletes its ExtendedCasbinRule rows through
their foreign key to the scope, and delete_casbin_rule_on_extended_rule_deletion() then
deletes the matching CasbinRule rows, so the role assignments on the course go away and a
course created later with the same key does not inherit them.

This runs on every CourseOverview save, so errors are logged rather than raised:
a problem backfilling a pending scope must not break the caller's save(). The query
runs in its own savepoint so a failure doesn't poison the caller's transaction.
Args:
sender: The CourseOverview model class.
instance: The CourseOverview that was deleted, for example the one with id
``course-v1:OpenedX+DemoX+2024``.
**kwargs: Other arguments Django sends with ``post_delete``.

Returns:
None
"""
try:
with transaction.atomic():
CourseScope.link_pending_scope(instance)
except Exception as exc: # pylint: disable=broad-exception-caught
logger.exception(
"Error linking pending CourseScope to CourseOverview %s",
instance.pk,
exc_info=exc,
)
CourseScope.objects.filter(external_key=CourseScope.external_key_for_object(instance)).delete()


def backfill_content_library_scope(sender, instance, **kwargs): # pylint: disable=unused-argument
"""
Link a pending ContentLibraryScope to the ContentLibrary that was just created for it.

See backfill_course_scope() above; same idea for content libraries, including the
broad exception handling and the isolating savepoint.
get_or_create_for_external_key() (openedx_authz/models/core.py) allows assigning a role
for a library key before its ContentLibrary exists, leaving content_library null. This
backfills that FK once the library shows up. See Scope.link_pending_scope() for the
actual lookup/update.

This runs on every ContentLibrary save, so errors are logged rather than raised:
a problem backfilling a pending scope must not break the caller's save(). The query
runs in its own savepoint so a failure doesn't poison the caller's transaction.
"""
try:
with transaction.atomic():
Expand All @@ -187,7 +191,7 @@ def backfill_content_library_scope(sender, instance, **kwargs): # pylint: disab

# Only register the handlers if the models are available (i.e., running in Open edX)
if CourseOverview is not None:
post_save.connect(backfill_course_scope, sender=CourseOverview)
post_delete.connect(delete_course_scope_on_course_deletion, sender=CourseOverview)

if ContentLibrary is not None:
post_save.connect(backfill_content_library_scope, sender=ContentLibrary)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
from django.db import IntegrityError, migrations, transaction

KEY_CONFLICT_MESSAGE = (
"Cannot copy the course key of these course scopes into external_key, because another scope "
"already holds an equal key: {conflicts}. Resolve them as the openedx-authz how-to "
'"Resolve Course Scopes That 0013_remove_coursescope_course_overview Cannot Update" describes, '
"then run migrate again."
)


class CourseScopeKeyConflictError(Exception):
"""Raised when a course key cannot be copied into external_key because another scope holds it."""


def copy_course_keys_to_external_key(apps, schema_editor):
"""Copy the course key of each course scope without an external_key into external_key.

Course scopes reference their course only by external_key once this migration removes
course_overview. Scopes created before migration 0010_scope_external_key have no
external_key, so their course key exists only in course_overview_id until it is copied.
Each scope is updated in its own transaction, because external_key is unique and the
database decides which keys are equal under its own collation. When any copy is rejected,
the other scopes keep their copied key and the migration stops before the column is
removed, so no course key is lost and running migrate again copies only the keys still
missing. Nothing is updated when every course scope already has an external_key.

Args:
apps: The historical app registry of this migration.
schema_editor: The schema editor running this migration.

Returns:
None

Raises:
CourseScopeKeyConflictError: When one or more course keys collide with an existing key,
for example ``12: course-v1:OpenedX+DemoX+2024``.
"""
course_scope_model = apps.get_model("openedx_authz", "CourseScope")
scopes_without_key = course_scope_model.objects.filter(
external_key__isnull=True,
course_overview__isnull=False,
).values_list("pk", "course_overview_id")
conflicts = []
for scope_id, course_key in scopes_without_key:
try:
with transaction.atomic(using=schema_editor.connection.alias):
course_scope_model.objects.filter(pk=scope_id).update(external_key=str(course_key))
except IntegrityError:
conflicts.append(f"{scope_id}: {course_key}")
if conflicts:
raise CourseScopeKeyConflictError(KEY_CONFLICT_MESSAGE.format(conflicts=", ".join(conflicts)))


class Migration(migrations.Migration):
# Not atomic, so the keys copied before a conflict stay copied when the migration stops.
atomic = False

dependencies = [
("openedx_authz", "0012_move_builtin_schema_sources_to_applets"),
]

operations = [
migrations.RunPython(copy_course_keys_to_external_key, migrations.RunPython.noop),
migrations.RemoveField(
model_name="coursescope",
name="course_overview",
),
]
6 changes: 3 additions & 3 deletions src/openedx_authz/models/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -118,9 +118,9 @@ class Scope(BaseRegistryModel):

objects = ScopeManager()

# Name of the subclass's FK field pointing at its backing object (e.g. "content_library",
# "course_overview"), used by get_or_create_for_external_key()/link_pending_scope() below
# to work generically across scope types.
# Name of the subclass's FK field pointing at its backing object (e.g. "content_library"),
# used by get_or_create_for_external_key()/link_pending_scope() below to work generically
# across scope types that keep such a FK.
LINKED_OBJECT_FIELD: ClassVar[str] = None

# Canonical string form of the scope's key (e.g. a course-v1 course id), set on creation
Expand Down
46 changes: 28 additions & 18 deletions src/openedx_authz/models/scopes.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,25 +92,35 @@ class CourseScope(Scope):
"""

NAMESPACE = "course-v1"
LINKED_OBJECT_FIELD = "course_overview"

# Link to the actual course, if applicable. In other cases, this could be null.
# Piggybacking on the existing CourseOverview model to keep the ExtendedCasbinRule up to date
# by deleting the Scope, and thus the ExtendedCasbinRule, when the CourseOverview is deleted.
#
# When course_overviews IS available, the on_delete=CASCADE will still work at the
# application level through Django's signal handlers.
# Use a string reference to the external app's model so Django won't try
# to import it at model import time. The migration already records the
# dependency on `course_overviews` when the app is present.
course_overview = models.ForeignKey(
settings.OPENEDX_AUTHZ_COURSE_OVERVIEW_MODEL,
on_delete=models.CASCADE,
null=True,
blank=True,
related_name="authz_scopes",
swappable=True,
)
@classmethod
def get_or_create_for_external_key(cls, scope) -> "CourseScope":
"""Get or create the course scope for a course key.

Course scopes reference their course only by ``external_key``, so the scope is the same
whether the course exists yet or not, for example during a course rerun.

Args:
scope: ScopeData object with an external_key attribute, for example
``CourseOverviewData(external_key="course-v1:OpenedX+DemoX+2024")``.

Returns:
CourseScope: The (possibly newly created) course scope for that key.
"""
course_scope, _ = cls.objects.get_or_create(external_key=scope.external_key)
return course_scope

@property
def course_overview(self):
"""Return the course this scope references through its course key.

Returns:
CourseOverview | None: The CourseOverview whose id is ``external_key``, or ``None``
when that course does not exist or the CourseOverview model is not installed.
"""
if CourseOverview is None:
return None
return CourseOverview.objects.filter(id=self.external_key).first()

@classmethod
def external_key_for_object(cls, linked_object) -> str:
Expand Down
18 changes: 0 additions & 18 deletions src/openedx_authz/tests/integration/test_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -1454,21 +1454,3 @@ def test_content_library_scope_backfills_when_library_created(self):

scope.refresh_from_db()
self.assertEqual(scope.content_library, content_library)

def test_scope_created_before_external_key_field_is_reused_not_duplicated(self):
"""A Scope row created before this fix (FK set, no external_key) isn't duplicated on reassignment.

Simulates a pre-existing row the way get_or_create_for_external_key used to create it:
keyed only by the FK, with no external_key.
"""
course_key = CourseLocator("LegacyOrg", "LegacyCourse", "2024_T1")
course_overview = CourseOverviewFactory(id=course_key)
legacy_scope = CourseScope.objects.create(course_overview=course_overview)
self.assertIsNone(legacy_scope.external_key)

scope_data = CourseOverviewData(external_key=str(course_key))
scope = Scope.objects.get_or_create_for_external_key(scope_data)

self.assertEqual(scope.id, legacy_scope.id)
self.assertEqual(scope.external_key, str(course_key))
self.assertEqual(CourseScope.objects.filter(course_overview=course_overview).count(), 1)
Loading
Loading