Skip to content

docs: add a how-to to recover failed course scope migrations - #628

Draft
mariajgrimaldi wants to merge 1 commit into
MJG/fix-coursescope-fkfrom
MJG/coursescope-recovery-docs
Draft

mariajgrimaldi wants to merge 1 commit into
MJG/fix-coursescope-fkfrom
MJG/coursescope-recovery-docs

Conversation

@mariajgrimaldi

@mariajgrimaldi mariajgrimaldi commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Description

MySQL doesn't run schema changes in a transaction, so a failed 0007 leaves an empty openedx_authz_coursescope table behind without recording the migration, and every later migrate fails with error 1050. The how-to "Recover From Failed Course Scope Migrations" checks that the table is empty and not recorded and drops it, so the fixed 0007 from #626 can run.

Related to #593.

How to Test

Save the plugin and the script below in one folder and run chmod +x test-issue-593.sh. The plugin runs MySQL 8.0 with its default collation utf8mb4_0900_ai_ci, as AWS RDS does, and the script prints each command and check it runs. Each init takes a few minutes.

Steps 1 to 3 of #626 follow this how-to on a failed 0007, and step 4 shows the migrations finish afterwards.

Tutor plugin issue593_mysql.yml
# Runs the platform on a MySQL 8.0 server that keeps its default collation (utf8mb4_0900_ai_ci),
# as AWS RDS and MySQL installs outside Tutor do, instead of Tutor's MySQL with utf8mb4_unicode_ci.
name: issue593_mysql
version: 0.1.0
config:
  set:
    RUN_MYSQL: false
    MYSQL_HOST: mysql80
    MYSQL_PORT: 3306
patches:
  local-docker-compose-services: |
    mysql80:
      image: docker.io/mysql:8.0
      environment:
        MYSQL_ROOT_PASSWORD: "{{ MYSQL_ROOT_PASSWORD }}"
      volumes:
        - ../../data/mysql80:/var/lib/mysql:z
      restart: unless-stopped
  local-docker-compose-lms-dependencies: |
    - mysql80
  local-docker-compose-cms-dependencies: |
    - mysql80
Helper script test-issue-593.sh
#!/usr/bin/env bash
# Reproduces openedx-authz issue #593 on an Ulmo-to-Verawood upgrade and checks the fix.
#
# Usage: ./test-issue-593.sh <ulmo|upgrade|recover|fix|main|check|role>
#
#   ulmo      install Ulmo (Tutor 21.0.8) with an admin user and one course
#   upgrade   upgrade to Verawood (Tutor 22.0.4, openedx-authz 1.23.0) and run init
#   recover   run the checks of the recovery how-to and drop the table a failed 0007 leaves
#   fix       install openedx-authz from AUTHZ_REF, run its migrations, then init (works right after ulmo too)
#   main      upgrade to Tutor main (the next release), install openedx-authz from MAIN_AUTHZ_REF, migrate, then init
#   check     print the collations, the foreign key and the openedx_authz migrations
#   role      assign course_staff on the test course and print its course scope
#
# Environment:
#   MYSQL_MODE  external (default): MySQL 8.0 with its default collation, through issue593_mysql.yml
#               tutor: Tutor's own MySQL, with utf8mb4_unicode_ci
#   WORK        working directory (default: ~/issue593-$MYSQL_MODE)
#   AUTHZ_REF   branch, tag or commit to install in "fix" (default: MJG/fix-coursescope-fk-verawood)
#   MAIN_AUTHZ_REF  branch, tag or commit to install in "main" (default: MJG/fix-coursescope-fk)
#
# Each mode gets its own Tutor root, plugins root and Docker Compose project names, so it does not touch
# other Tutor environments on the machine. Tutor stops the dev project named by DEV_PROJECT_NAME whenever
# a local command starts containers, so that name is set too, and a running "tutor_dev" keeps running.
set -euo pipefail

MYSQL_MODE="${MYSQL_MODE:-external}"
WORK="${WORK:-$HOME/issue593-$MYSQL_MODE}"
AUTHZ_REF="${AUTHZ_REF:-MJG/fix-coursescope-fk-verawood}"
MAIN_AUTHZ_REF="${MAIN_AUTHZ_REF:-MJG/fix-coursescope-fk}"
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
COURSE_KEY="course-v1:DemoX+D101+2026"

export TUTOR_ROOT="$WORK/root"
export TUTOR_PLUGINS_ROOT="$WORK/plugins"

case "$MYSQL_MODE" in
    external) MYSQL_SERVICE=mysql80 ;;
    tutor) MYSQL_SERVICE=mysql ;;
    *) echo "MYSQL_MODE must be external or tutor, not '$MYSQL_MODE'" >&2; exit 1 ;;
esac

log() { printf '\n\033[1;34m==> %s\033[0m\n' "$*"; }
run() { printf '\033[2m$ %s\033[0m\n' "$*"; "$@"; }

# Installs a Tutor version ("main" for its main branch) in its own virtualenv under WORK, once,
# and records it as the version that "check" and "role" use.
tutor_cli() {
    local version="$1" venv="$WORK/tutor-$1" requirement="tutor==$1"
    [[ "$version" == main ]] && requirement="git+https://github.com/overhangio/tutor@main"
    if [[ ! -x "$venv/bin/tutor" ]]; then
        log "Installing Tutor $version in $venv" >&2
        python3 -m venv "$venv" >&2
        "$venv/bin/pip" install -q "$requirement" >&2
    fi
    echo "$version" > "$WORK/current-tutor"
    echo "$venv/bin/tutor"
}

# The Tutor version that saved the Tutor root last.
current_tutor() {
    tutor_cli "$(cat "$WORK/current-tutor" 2>/dev/null || echo 22.0.4)"
}

# Runs SQL as root in the openedx database through the MySQL service of this mode.
sql() {
    local password
    password="$("$TUTOR" config printvalue MYSQL_ROOT_PASSWORD)"
    "$TUTOR" local dc exec -T "$MYSQL_SERVICE" mysql -uroot --password="$password" openedx -e "$1" </dev/null 2>&1 | grep -v "Using a password"
}

wait_for_mysql() {
    log "Waiting for the $MYSQL_SERVICE service"
    # The permissions job sets the owner of the data folders, which Tutor's MySQL needs on a new Tutor root.
    run "$TUTOR" local dc run --rm permissions </dev/null
    run "$TUTOR" local start -d "$MYSQL_SERVICE"
    local password
    password="$("$TUTOR" config printvalue MYSQL_ROOT_PASSWORD)"
    for attempt in $(seq 1 30); do
        if "$TUTOR" local dc exec -T "$MYSQL_SERVICE" mysqladmin ping -uroot --password="$password" --silent </dev/null >/dev/null 2>&1; then
            echo "$MYSQL_SERVICE is up"
            return
        fi
        echo "  [$attempt/30] not ready yet"
        sleep 5
    done
    echo "$MYSQL_SERVICE did not start; run: $TUTOR local logs $MYSQL_SERVICE" >&2
    exit 1
}

print_context() {
    log "Context"
    echo "MYSQL_MODE=$MYSQL_MODE WORK=$WORK"
    echo "Tutor: $("$TUTOR" --version)"
    echo "TUTOR_ROOT=$TUTOR_ROOT"
    echo "LOCAL_PROJECT_NAME=$("$TUTOR" config printvalue LOCAL_PROJECT_NAME) DEV_PROJECT_NAME=$("$TUTOR" config printvalue DEV_PROJECT_NAME)"
    echo "RUN_MYSQL=$("$TUTOR" config printvalue RUN_MYSQL) MYSQL_HOST=$("$TUTOR" config printvalue MYSQL_HOST)"
    echo "Image: $("$TUTOR" config printvalue DOCKER_IMAGE_OPENEDX)"
}

cmd_ulmo() {
    TUTOR="$(tutor_cli 21.0.8)"
    mkdir -p "$TUTOR_PLUGINS_ROOT"
    run "$TUTOR" config save --set "LOCAL_PROJECT_NAME=issue593_$MYSQL_MODE" --set "DEV_PROJECT_NAME=issue593_${MYSQL_MODE}_dev"
    if [[ "$MYSQL_MODE" == external ]]; then
        run cp "$SCRIPT_DIR/issue593_mysql.yml" "$TUTOR_PLUGINS_ROOT/"
        run "$TUTOR" plugins enable issue593_mysql
    fi
    print_context
    wait_for_mysql
    log "Initializing Ulmo (this takes several minutes)"
    run "$TUTOR" local do init
    log "Creating an admin user and the course $COURSE_KEY"
    run "$TUTOR" local do createuser --staff --superuser admin [email protected] --password admin
    run "$TUTOR" local run cms ./manage.py cms create_course split [email protected] DemoX D101 2026 Demo
    # No Celery worker runs here, so the course overview is generated directly.
    run "$TUTOR" local run lms ./manage.py lms generate_course_overview --all-courses
    cmd_check
}

# Switches the Tutor root to Verawood, or to Tutor main with "main". Running it again changes nothing.
prepare_verawood() {
    TUTOR="$(tutor_cli "${1:-22.0.4}")"
    run "$TUTOR" config save
    print_context
    # Meilisearch 1.36 in Tutor 22 cannot read the index Ulmo created, and init creates it again.
    if [[ ! -f "$TUTOR_ROOT/data/.meilisearch-reset" ]]; then
        run docker run --rm -v "$TUTOR_ROOT/data:/data" alpine sh -c "rm -rf /data/meilisearch && touch /data/.meilisearch-reset"
    fi
    wait_for_mysql
}

cmd_upgrade() {
    prepare_verawood
    log "Installed openedx-authz and edx-opaque-keys in the Verawood image"
    run "$TUTOR" local run lms sh -c "pip list 2>/dev/null | grep -E '^(openedx-authz|edx-opaque-keys) '"
    log "Running Verawood init"
    if run "$TUTOR" local do init; then
        echo "init finished"
    else
        echo "init failed; the error above is the one to compare with the issue"
    fi
    cmd_check
}

cmd_recover() {
    TUTOR="$(current_tutor)"
    log "0007 must not be recorded (expect no rows)"
    sql "SELECT name FROM django_migrations WHERE app = 'openedx_authz' AND name = '0007_coursescope';"
    log "The table must be empty (expect 0)"
    sql "SELECT COUNT(*) FROM openedx_authz_coursescope;"
    read -r -p "Drop openedx_authz_coursescope? [y/N] " answer
    if [[ "$answer" == y ]]; then
        sql "DROP TABLE openedx_authz_coursescope;"
        echo "Table dropped"
    fi
}

cmd_fix() {
    prepare_verawood
    log "Installing openedx-authz from $AUTHZ_REF and running its migrations"
    run "$TUTOR" local run lms sh -c "pip install -q git+https://github.com/openedx/openedx-authz@$AUTHZ_REF && pip freeze | grep ^openedx-authz && ./manage.py lms migrate openedx_authz"
    log "Running Verawood init"
    run "$TUTOR" local do init
    cmd_check
}

cmd_main() {
    prepare_verawood main
    log "Installing openedx-authz from $MAIN_AUTHZ_REF and running its migrations"
    run "$TUTOR" local run lms sh -c "pip install -q git+https://github.com/openedx/openedx-authz@$MAIN_AUTHZ_REF && pip freeze | grep ^openedx-authz && ./manage.py lms migrate openedx_authz"
    log "Running init on Tutor main"
    run "$TUTOR" local do init
    cmd_check
}

cmd_check() {
    TUTOR="${TUTOR:-$(current_tutor)}"
    log "Server version and default collation"
    sql "SELECT @@version, @@collation_server;"
    log "Collation of the referenced and the referencing column"
    sql "SELECT table_name, column_name, collation_name FROM information_schema.columns WHERE table_schema = 'openedx' AND (table_name, column_name) IN (('course_overviews_courseoverview', 'id'), ('openedx_authz_coursescope', 'course_overview_id'));"
    log "Foreign key from openedx_authz_coursescope to course_overviews_courseoverview"
    sql "SELECT constraint_name FROM information_schema.referential_constraints WHERE constraint_schema = 'openedx' AND table_name = 'openedx_authz_coursescope' AND referenced_table_name = 'course_overviews_courseoverview';"
    log "Applied openedx_authz migrations"
    sql "SELECT name FROM django_migrations WHERE app = 'openedx_authz' ORDER BY id;"
}

cmd_role() {
    TUTOR="$(current_tutor)"
    log "Assigning course_staff to admin on $COURSE_KEY (expect True)"
    run "$TUTOR" local run lms ./manage.py lms shell -c "from openedx_authz.api.users import assign_role_to_user_in_scope; print(assign_role_to_user_in_scope('admin', 'course_staff', '$COURSE_KEY'))"
    log "Course scopes"
    sql "SELECT s.id, s.external_key, c.course_overview_id FROM openedx_authz_scope s JOIN openedx_authz_coursescope c ON c.scope_ptr_id = s.id;"
}

case "${1:-}" in
    ulmo | upgrade | recover | fix | main | check | role) "cmd_$1" ;;
    *) sed -n '2,20p' "$0"; exit 1 ;;
esac

Merge checklist:
Check off if complete or not applicable:

  • Documentation updated (not only docstrings)
  • Fixup commits are squashed away
  • Unit tests added/updated
  • Manual testing instructions provided
  • Noted any: Concerns, dependencies, migration issues, deadlines, tickets

@openedx-webhooks openedx-webhooks added open-source-contribution PR author is not from Axim or 2U core contributor PR author is a Core Contributor (who may or may not have write access to this repo). labels Nov 16, 2026
@openedx-webhooks

Copy link
Copy Markdown

Thanks for the pull request, @mariajgrimaldi!

This repository is currently maintained by @openedx/committers-openedx-authz.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

  • If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
    • This process (including the steps you'll need to take) is documented here.
  • If it doesn't, simply proceed with the next step.
🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

  • Dependencies

    This PR must be merged before / after / at the same time as ...

  • Blockers

    This PR is waiting for OEP-1234 to be accepted.

  • Timeline information

    This PR must be merged by XX date because ...

  • Partner information

    This is for a course on edx.org.

  • Supporting documentation
  • Relevant Open edX discussion forum threads
🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

🔘 Update the status of your PR

Your PR is currently marked as a draft. After completing the steps above, update its status by clicking "Ready for Review", or removing "WIP" from the title, as appropriate.


Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

  • The size and impact of the changes that it introduces
  • The need for product review
  • Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

@github-project-automation github-project-automation Bot moved this to Needs Triage in Contributions Nov 16, 2026
@mphilbrick211 mphilbrick211 moved this from Needs Triage to Waiting on Author in Contributions Nov 16, 2026
@mariajgrimaldi
mariajgrimaldi force-pushed the MJG/coursescope-recovery-docs branch from a1c9c44 to 834cb29 Compare November 18, 2026 12:45
@mariajgrimaldi
mariajgrimaldi removed this pull request from stack #629 November 18, 2026 13:12
@mariajgrimaldi
mariajgrimaldi force-pushed the MJG/coursescope-recovery-docs branch from 834cb29 to b0d5b15 Compare November 18, 2026 13:12
@mariajgrimaldi
mariajgrimaldi changed the base branch from MJG/coursescope-drop-fk to MJG/fix-coursescope-fk November 18, 2026 13:13
@mariajgrimaldi
mariajgrimaldi force-pushed the MJG/coursescope-recovery-docs branch from b0d5b15 to 3eb9ccd Compare November 18, 2026 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core contributor PR author is a Core Contributor (who may or may not have write access to this repo). open-source-contribution PR author is not from Axim or 2U

Projects

Status: Waiting on Author

Development

Successfully merging this pull request may close these issues.

2 participants