Skip to content

fix: align the course_overview_id collation in migration 0007 - #626

Draft
mariajgrimaldi wants to merge 2 commits into
mainfrom
MJG/fix-coursescope-fk
Draft

mariajgrimaldi wants to merge 2 commits into
mainfrom
MJG/fix-coursescope-fk

Conversation

@mariajgrimaldi

@mariajgrimaldi mariajgrimaldi commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Description

Migration 0007_coursescope fails with MySQL error 3780 on databases whose default collation is utf8mb4_0900_ai_ci, such as MySQL 8 and AWS RDS, which blocks the upgrade to Verawood. MySQL needs the same collation on both columns of a foreign key between strings (MySQL foreign key constraints), while course_overview_id gets utf8mb4_unicode_ci from opaque-keys (opaque-keys#426) and the platform's course_overviews_courseoverview.id keeps the database default.

On MySQL, 0007 now gives course_overview_id the collation of the column it references before Django creates the foreign key, and it does nothing on other databases or when the collations already match. The migration state doesn't change, so instances that already applied 0007 see no new migration, and the same fix goes to Verawood as 1.23.1 in #639.

Closes #593.

How to Test

Save the plugin and the script below in one folder and run chmod +x test-issue-593.sh. The plugin runs MySQL 8.0 with its default collation utf8mb4_0900_ai_ci, as AWS RDS does, and the script prints each command and check it runs. Each init takes a few minutes.

  1. ./test-issue-593.sh ulmo installs Ulmo with one course.
  2. ./test-issue-593.sh upgrade moves to Verawood (openedx-authz==1.23.0), and init stops at 0007_coursescope with error 3780, as in openedx_authz.0007_coursescope fails with MySQL error 3780 on databases whose default collation is not utf8mb4_unicode_ci (e.g. MySQL 8 default utf8mb4_0900_ai_ci) #593.
  3. ./test-issue-593.sh recover runs the checks of the how-to and, after you answer y, drops the table.
  4. ./test-issue-593.sh main moves to Tutor main, installs this branch, applies 0007 to 0012 and runs init, which finishes. Both columns are utf8mb4_0900_ai_ci and the foreign key exists.
  5. ./test-issue-593.sh role assigns a course role and prints True.
Tutor plugin issue593_mysql.yml
# Runs the platform on a MySQL 8.0 server that keeps its default collation (utf8mb4_0900_ai_ci),
# as AWS RDS and MySQL installs outside Tutor do, instead of Tutor's MySQL with utf8mb4_unicode_ci.
name: issue593_mysql
version: 0.1.0
config:
  set:
    RUN_MYSQL: false
    MYSQL_HOST: mysql80
    MYSQL_PORT: 3306
patches:
  local-docker-compose-services: |
    mysql80:
      image: docker.io/mysql:8.0
      environment:
        MYSQL_ROOT_PASSWORD: "{{ MYSQL_ROOT_PASSWORD }}"
      volumes:
        - ../../data/mysql80:/var/lib/mysql:z
      restart: unless-stopped
  local-docker-compose-lms-dependencies: |
    - mysql80
  local-docker-compose-cms-dependencies: |
    - mysql80
Helper script test-issue-593.sh
#!/usr/bin/env bash
# Reproduces openedx-authz issue #593 on an Ulmo-to-Verawood upgrade and checks the fix.
#
# Usage: ./test-issue-593.sh <ulmo|upgrade|recover|fix|main|check|role>
#
#   ulmo      install Ulmo (Tutor 21.0.8) with an admin user and one course
#   upgrade   upgrade to Verawood (Tutor 22.0.4, openedx-authz 1.23.0) and run init
#   recover   run the checks of the recovery how-to and drop the table a failed 0007 leaves
#   fix       install openedx-authz from AUTHZ_REF, run its migrations, then init (works right after ulmo too)
#   main      upgrade to Tutor main (the next release), install openedx-authz from MAIN_AUTHZ_REF, migrate, then init
#   check     print the collations, the foreign key and the openedx_authz migrations
#   role      assign course_staff on the test course and print its course scope
#
# Environment:
#   MYSQL_MODE  external (default): MySQL 8.0 with its default collation, through issue593_mysql.yml
#               tutor: Tutor's own MySQL, with utf8mb4_unicode_ci
#   WORK        working directory (default: ~/issue593-$MYSQL_MODE)
#   AUTHZ_REF   branch, tag or commit to install in "fix" (default: MJG/fix-coursescope-fk-verawood)
#   MAIN_AUTHZ_REF  branch, tag or commit to install in "main" (default: MJG/fix-coursescope-fk)
#
# Each mode gets its own Tutor root, plugins root and Docker Compose project names, so it does not touch
# other Tutor environments on the machine. Tutor stops the dev project named by DEV_PROJECT_NAME whenever
# a local command starts containers, so that name is set too, and a running "tutor_dev" keeps running.
set -euo pipefail

MYSQL_MODE="${MYSQL_MODE:-external}"
WORK="${WORK:-$HOME/issue593-$MYSQL_MODE}"
AUTHZ_REF="${AUTHZ_REF:-MJG/fix-coursescope-fk-verawood}"
MAIN_AUTHZ_REF="${MAIN_AUTHZ_REF:-MJG/fix-coursescope-fk}"
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
COURSE_KEY="course-v1:DemoX+D101+2026"

export TUTOR_ROOT="$WORK/root"
export TUTOR_PLUGINS_ROOT="$WORK/plugins"

case "$MYSQL_MODE" in
    external) MYSQL_SERVICE=mysql80 ;;
    tutor) MYSQL_SERVICE=mysql ;;
    *) echo "MYSQL_MODE must be external or tutor, not '$MYSQL_MODE'" >&2; exit 1 ;;
esac

log() { printf '\n\033[1;34m==> %s\033[0m\n' "$*"; }
run() { printf '\033[2m$ %s\033[0m\n' "$*"; "$@"; }

# Installs a Tutor version ("main" for its main branch) in its own virtualenv under WORK, once,
# and records it as the version that "check" and "role" use.
tutor_cli() {
    local version="$1" venv="$WORK/tutor-$1" requirement="tutor==$1"
    [[ "$version" == main ]] && requirement="git+https://github.com/overhangio/tutor@main"
    if [[ ! -x "$venv/bin/tutor" ]]; then
        log "Installing Tutor $version in $venv" >&2
        python3 -m venv "$venv" >&2
        "$venv/bin/pip" install -q "$requirement" >&2
    fi
    echo "$version" > "$WORK/current-tutor"
    echo "$venv/bin/tutor"
}

# The Tutor version that saved the Tutor root last.
current_tutor() {
    tutor_cli "$(cat "$WORK/current-tutor" 2>/dev/null || echo 22.0.4)"
}

# Runs SQL as root in the openedx database through the MySQL service of this mode.
sql() {
    local password
    password="$("$TUTOR" config printvalue MYSQL_ROOT_PASSWORD)"
    "$TUTOR" local dc exec -T "$MYSQL_SERVICE" mysql -uroot --password="$password" openedx -e "$1" </dev/null 2>&1 | grep -v "Using a password"
}

wait_for_mysql() {
    log "Waiting for the $MYSQL_SERVICE service"
    # The permissions job sets the owner of the data folders, which Tutor's MySQL needs on a new Tutor root.
    run "$TUTOR" local dc run --rm permissions </dev/null
    run "$TUTOR" local start -d "$MYSQL_SERVICE"
    local password
    password="$("$TUTOR" config printvalue MYSQL_ROOT_PASSWORD)"
    for attempt in $(seq 1 30); do
        if "$TUTOR" local dc exec -T "$MYSQL_SERVICE" mysqladmin ping -uroot --password="$password" --silent </dev/null >/dev/null 2>&1; then
            echo "$MYSQL_SERVICE is up"
            return
        fi
        echo "  [$attempt/30] not ready yet"
        sleep 5
    done
    echo "$MYSQL_SERVICE did not start; run: $TUTOR local logs $MYSQL_SERVICE" >&2
    exit 1
}

print_context() {
    log "Context"
    echo "MYSQL_MODE=$MYSQL_MODE WORK=$WORK"
    echo "Tutor: $("$TUTOR" --version)"
    echo "TUTOR_ROOT=$TUTOR_ROOT"
    echo "LOCAL_PROJECT_NAME=$("$TUTOR" config printvalue LOCAL_PROJECT_NAME) DEV_PROJECT_NAME=$("$TUTOR" config printvalue DEV_PROJECT_NAME)"
    echo "RUN_MYSQL=$("$TUTOR" config printvalue RUN_MYSQL) MYSQL_HOST=$("$TUTOR" config printvalue MYSQL_HOST)"
    echo "Image: $("$TUTOR" config printvalue DOCKER_IMAGE_OPENEDX)"
}

cmd_ulmo() {
    TUTOR="$(tutor_cli 21.0.8)"
    mkdir -p "$TUTOR_PLUGINS_ROOT"
    run "$TUTOR" config save --set "LOCAL_PROJECT_NAME=issue593_$MYSQL_MODE" --set "DEV_PROJECT_NAME=issue593_${MYSQL_MODE}_dev"
    if [[ "$MYSQL_MODE" == external ]]; then
        run cp "$SCRIPT_DIR/issue593_mysql.yml" "$TUTOR_PLUGINS_ROOT/"
        run "$TUTOR" plugins enable issue593_mysql
    fi
    print_context
    wait_for_mysql
    log "Initializing Ulmo (this takes several minutes)"
    run "$TUTOR" local do init
    log "Creating an admin user and the course $COURSE_KEY"
    run "$TUTOR" local do createuser --staff --superuser admin [email protected] --password admin
    run "$TUTOR" local run cms ./manage.py cms create_course split [email protected] DemoX D101 2026 Demo
    # No Celery worker runs here, so the course overview is generated directly.
    run "$TUTOR" local run lms ./manage.py lms generate_course_overview --all-courses
    cmd_check
}

# Switches the Tutor root to Verawood, or to Tutor main with "main". Running it again changes nothing.
prepare_verawood() {
    TUTOR="$(tutor_cli "${1:-22.0.4}")"
    run "$TUTOR" config save
    print_context
    # Meilisearch 1.36 in Tutor 22 cannot read the index Ulmo created, and init creates it again.
    if [[ ! -f "$TUTOR_ROOT/data/.meilisearch-reset" ]]; then
        run docker run --rm -v "$TUTOR_ROOT/data:/data" alpine sh -c "rm -rf /data/meilisearch && touch /data/.meilisearch-reset"
    fi
    wait_for_mysql
}

cmd_upgrade() {
    prepare_verawood
    log "Installed openedx-authz and edx-opaque-keys in the Verawood image"
    run "$TUTOR" local run lms sh -c "pip list 2>/dev/null | grep -E '^(openedx-authz|edx-opaque-keys) '"
    log "Running Verawood init"
    if run "$TUTOR" local do init; then
        echo "init finished"
    else
        echo "init failed; the error above is the one to compare with the issue"
    fi
    cmd_check
}

cmd_recover() {
    TUTOR="$(current_tutor)"
    log "0007 must not be recorded (expect no rows)"
    sql "SELECT name FROM django_migrations WHERE app = 'openedx_authz' AND name = '0007_coursescope';"
    log "The table must be empty (expect 0)"
    sql "SELECT COUNT(*) FROM openedx_authz_coursescope;"
    read -r -p "Drop openedx_authz_coursescope? [y/N] " answer
    if [[ "$answer" == y ]]; then
        sql "DROP TABLE openedx_authz_coursescope;"
        echo "Table dropped"
    fi
}

cmd_fix() {
    prepare_verawood
    log "Installing openedx-authz from $AUTHZ_REF and running its migrations"
    run "$TUTOR" local run lms sh -c "pip install -q git+https://github.com/openedx/openedx-authz@$AUTHZ_REF && pip freeze | grep ^openedx-authz && ./manage.py lms migrate openedx_authz"
    log "Running Verawood init"
    run "$TUTOR" local do init
    cmd_check
}

cmd_main() {
    prepare_verawood main
    log "Installing openedx-authz from $MAIN_AUTHZ_REF and running its migrations"
    run "$TUTOR" local run lms sh -c "pip install -q git+https://github.com/openedx/openedx-authz@$MAIN_AUTHZ_REF && pip freeze | grep ^openedx-authz && ./manage.py lms migrate openedx_authz"
    log "Running init on Tutor main"
    run "$TUTOR" local do init
    cmd_check
}

cmd_check() {
    TUTOR="${TUTOR:-$(current_tutor)}"
    log "Server version and default collation"
    sql "SELECT @@version, @@collation_server;"
    log "Collation of the referenced and the referencing column"
    sql "SELECT table_name, column_name, collation_name FROM information_schema.columns WHERE table_schema = 'openedx' AND (table_name, column_name) IN (('course_overviews_courseoverview', 'id'), ('openedx_authz_coursescope', 'course_overview_id'));"
    log "Foreign key from openedx_authz_coursescope to course_overviews_courseoverview"
    sql "SELECT constraint_name FROM information_schema.referential_constraints WHERE constraint_schema = 'openedx' AND table_name = 'openedx_authz_coursescope' AND referenced_table_name = 'course_overviews_courseoverview';"
    log "Applied openedx_authz migrations"
    sql "SELECT name FROM django_migrations WHERE app = 'openedx_authz' ORDER BY id;"
}

cmd_role() {
    TUTOR="$(current_tutor)"
    log "Assigning course_staff to admin on $COURSE_KEY (expect True)"
    run "$TUTOR" local run lms ./manage.py lms shell -c "from openedx_authz.api.users import assign_role_to_user_in_scope; print(assign_role_to_user_in_scope('admin', 'course_staff', '$COURSE_KEY'))"
    log "Course scopes"
    sql "SELECT s.id, s.external_key, c.course_overview_id FROM openedx_authz_scope s JOIN openedx_authz_coursescope c ON c.scope_ptr_id = s.id;"
}

case "${1:-}" in
    ulmo | upgrade | recover | fix | main | check | role) "cmd_$1" ;;
    *) sed -n '2,20p' "$0"; exit 1 ;;
esac

Merge checklist:
Check off if complete or not applicable:

  • Documentation updated (not only docstrings)
  • Fixup commits are squashed away
  • Unit tests added/updated
  • Manual testing instructions provided
  • Noted any: Concerns, dependencies, migration issues, deadlines, tickets

@openedx-webhooks openedx-webhooks added open-source-contribution PR author is not from Axim or 2U core contributor PR author is a Core Contributor (who may or may not have write access to this repo). labels Nov 16, 2026
@openedx-webhooks

Copy link
Copy Markdown

Thanks for the pull request, @mariajgrimaldi!

This repository is currently maintained by @openedx/committers-openedx-authz.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

  • If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
    • This process (including the steps you'll need to take) is documented here.
  • If it doesn't, simply proceed with the next step.
🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

  • Dependencies

    This PR must be merged before / after / at the same time as ...

  • Blockers

    This PR is waiting for OEP-1234 to be accepted.

  • Timeline information

    This PR must be merged by XX date because ...

  • Partner information

    This is for a course on edx.org.

  • Supporting documentation
  • Relevant Open edX discussion forum threads
🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

🔘 Update the status of your PR

Your PR is currently marked as a draft. After completing the steps above, update its status by clicking "Ready for Review", or removing "WIP" from the title, as appropriate.


Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

  • The size and impact of the changes that it introduces
  • The need for product review
  • Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core contributor PR author is a Core Contributor (who may or may not have write access to this repo). open-source-contribution PR author is not from Axim or 2U

Projects

Status: Waiting on Author

2 participants