Skip to content

feat: add ADR 0002, use-case registration API and adapter contract [FC-0144] - #20

Open
rpenido wants to merge 55 commits into
openedx:mainfrom
open-craft:search-schema-adr
Open

rpenido wants to merge 55 commits into
openedx:mainfrom
open-craft:search-schema-adr

Conversation

@rpenido

@rpenido rpenido commented Oct 21, 2026 •

Copy link
Copy Markdown

Description

This PR adds ADR 0002 for openedx-search and the first implementation of the contract it defines: the engine-neutral schema model, the use-case registration layer, and the adapter contract with the consumer API shape.

The ADR settles how a use case declares its document schema, how those declarations stay engine-neutral, how a deployment selects an engine, and how adapters render, reconcile, rebuild and validate the declaration. Index settings are derived from the schema. It also records the alternatives considered with pros and cons, including the narrow flat-string adapter contract and its benchmark in #3 and #4.

The implementation provides IndexSchema, Field and Capabilities, with required fields and a validated id format. The shared model rejects declarations the two engines cannot both represent: searchable is limited to strings and objects, sortable rejects arrays, fields inside object arrays must be arrays, required fields need required parents, and the primary key is the required, non-array string field id without indexing capabilities. The distinct attribute is not modelled; Studio's distinctAttribute setting is being dropped, not carried over.

Sort precedence is fixed rather than declared: an explicit sort always outranks keyword relevance, and relevance applies only when the caller asks for it. Meilisearch applies it in the index's rankingRules with sort first, which reconcile sets; Typesense composes it per request in sort_by, so the scoped credential embeds query_by and query_by_weights but not the sort.

The ADR defines document validation so both engines accept and reject the same documents: required fields present, declared types and shapes, datetimes accepted as aware values and rendered as POSIX seconds, no unknown or null fields, and string-only free-form objects. The validator and DocumentError land with upsert, so this diff documents the rules rather than implementing them.

It also provides a thread-safe in-memory use-case registry with validation; per-use-case engine resolution from deployment settings; and the Adapter ABC covering schema validation, reconcile, writes, deletes, write confirmation, rebuilds and queries. openedx_search.api exposes the consumer-side signatures for search, mint_credential, upsert, delete, check_write and rebuild; the bodies land with the engine adapters. Tests cover the schema, including the cross-engine constraints, the registry and engine selection.

Registrations are consumer-owned: a consumer calls register_use_case(..., replace=True) from its own AppConfig.ready(). The library does not scan packages or auto-discover registrations. A registration declares the use case, the logical index and the schema; query terms stay per request and scoped credentials carry tenant filters. The ADR places schema compatibility checks in a Django system check during deployment, not at registration; the check itself lands with the adapters.

The ADR specifies that a rebuild records its target in a shared Redis cache with a lease, so every writer process dual-writes during a reindex and a crashed worker cannot block the next rebuild. A schema version bump is a migration with writers stopped. The cache and rebuild code land with the adapters.

Testing instructions

  • For now, the main review point is the ADR002. The implementation is a draft, so we can have a more tangible discussion
  • Additionally, you can run:
    • uv run --group test --with djangorestframework --with django-filter python -m pytest

Other information

Until #2 lands, main's test settings import rest_framework and django_filters, so the command above passes the two packages with --with. After #2, uv sync --group test and uv run --group test pytest are enough.

AI disclosure: I used Claude to research the upstream libraries, create the IndexSchema and its accessory types, and help to write the ADR. I reviewed everything and also did multiple manual edits.


Private ref:

@openedx-webhooks openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Oct 21, 2026
@openedx-webhooks

openedx-webhooks commented Oct 21, 2026 •

Copy link
Copy Markdown

Thanks for the pull request, @rpenido!

This repository is currently maintained by @ormsbee.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

  • If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
    • This process (including the steps you'll need to take) is documented here.
  • If it doesn't, simply proceed with the next step.
🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

  • Dependencies

    This PR must be merged before / after / at the same time as ...

  • Blockers

    This PR is waiting for OEP-1234 to be accepted.

  • Timeline information

    This PR must be merged by XX date because ...

  • Partner information

    This is for a course on edx.org.

  • Supporting documentation
  • Relevant Open edX discussion forum threads
🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details
Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

  • The size and impact of the changes that it introduces
  • The need for product review
  • Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

@github-project-automation github-project-automation Bot moved this to Needs Triage in Contributions Oct 21, 2026
@rpenido
rpenido force-pushed the search-schema-adr branch 2 times, most recently from 2a680a4 to d0f19b1 Compare October 21, 2026 19:43
@rpenido rpenido changed the title feat: add ADR 0002 and use-case registration API feat: add ADR 0002 and use-case registration API [FC-0144] Oct 21, 2026
@rpenido
rpenido force-pushed the search-schema-adr branch 14 times, most recently from 08e52f3 to a8ecc2b Compare October 21, 2026 22:21
Comment thread docs/how-tos/registering-use-cases.rst Resolved
@mphilbrick211 mphilbrick211 moved this from Needs Triage to Waiting on Author in Contributions Oct 22, 2026
``validate`` runs at registration and before settings are applied; a use case
that needs an unsupported capability fails there, not silently. The library
validates documents against ``IndexSchema`` before handing them to a backend, so
every engine rejects the same bad document for the same reason.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it is necessary to specify what is meant by a "valid document." This is because neither the code nor the ADR mentions fields that might be missing from the document or that could be null. In Meilisearch, a missing field causes no issues, but in Typesense, it throws an error if the field is not optional.

@rpenido rpenido Oct 22, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous wording and methods were a bit confusing, so I rewrote them here: c9910de

In the context of the IndexSchema, the adapter implements validate_schema (previous validate) to check if the current proposed schema match is supported. I put an example in the ADR.

In the context for document validation (now validate_document), we will match the document with the schema provided. I added the required to Field here: 7f37eb4

If we have a "required" field, like id in STUDIO_COURSE_SCHEMA, we register it with Field("id", FieldType.STRING, required=True).

When creating the index, for meilisearch we will just ignore it. For typesense, we will pass "optional": not field.required accordingly.

While populating the collection with documents, the adapter will validate the document against its schema. So passing a document with a missing required (by schema) field will throw. Even if we are using meilisearch. Our goal here is to have the same behavior for the consumer.

Does that make sense?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds good. And another question: what should be done in the opposite case: when a document arrives with an extra field that is not in the schema?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my opinion, we should throw an error in the case. If not thrown, we should at least drop it to have all engines with the same behavior (Typensense silent drops it).

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the ADR here: 5f579a2

The validator and DocumentError land with upsert, which is still a stub.

Comment thread tests/fixtures.py Resolved
use_case str
index str logical index name
schema IndexSchema
query_shape QueryShape filters, facets, sort, defaults

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the code, Field capabilities already declare what is filterable, facetable and sortable. What would QueryShape add beyond query defaults?

@rpenido rpenido Oct 22, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. We will probably need something for the query defaults, but I don't think they belong to the registration.
We can visit that later in the implementation. Removed as part of b141421

Comment on lines +71 to +73
raise BackendNotConfigured(
"No search backend is configured. Set OPENEDX_SEARCH_DEFAULT_BACKEND "
"and declare it in OPENEDX_SEARCH_BACKENDS."

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick question: does this mean a backend must always be configured, even if it’s just the default? Is there no option for an "I do not wish to configure search" state?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there no option for an "I do not wish to configure search" state?
In the openedx-search library level, I don't think so.

We will probably do it on openedx-platform, or in tutor, though. Since the search engine should also be provisioned, I think the default doesn't belong to the library itself.

Any thoughts on that?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't necessarily need a dummy backend/adapter, but I think we should have a way to turn off search and/or turn off individual use cases. That could either be an enabled flag or be achieved by configuring a dummy backend for that use case, but I think an enabled: false style is more efficient than a null adapter, because it won't spend time preparing documents for insert that then get ignored.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't necessarily need a dummy backend/adapter, but I think we should have a way to turn off search and/or turn off individual use cases.

Do you mind elaborating more on that? In which case would we use it?
And should we add an api.is_enabled(use_case) -> bool so the consumer is aware of it? And make the api (write, delete, rebuild, etc..) no-op while disabled?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I forgot to ping you here @bradenmacdonald!

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And should we add an api.is_enabled(use_case) -> bool so the consumer is aware of it? And make the api (write, delete, rebuild, etc..) no-op while disabled?

Yeah I was thinking of something like that. But I don't have any strong opinion on this, so I'm open to your ideas.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As implementation, I think no-op methods + api.is_enable(use_case) should do it.

I'm more curious about "where/when do we need this"? Studio will just break without the backend. Are there some apps that we can check if we have the backend and, if not, have a fallback? Is it for a maintenance kill-switch so we can isolate the search for some reason?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well courseware search is an optional feature that can be turned on and off. But maybe it already has its own flag, so perhaps you're right that we don't need this in general.

Libraries v2 (with Meilisearch) was an optional feature for a while but I guess it's pretty much alway enabled now.

Yeah, I guess we don't need the enabled/disabled at this layer after all.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I guess we don't need the enabled/disabled at this layer after all.

If you think it is a "nice to have", please let me know! It's not complex to implement.

@rpenido

rpenido commented Nov 5, 2026

Copy link
Copy Markdown
Author

Thanks for the second pass @blarghmatey! I think I addressed all your comments.

One more Typesense detail for the adapter contract: the import endpoint returns HTTP 200 when a document is rejected, with the failure only in that document's result line ("success": false), so upsert has to read every line before it can report a write as complete.

Agreed, and it is in the ADR in 2d07758: a 200 from the Typesense import endpoint never marks a batch complete, the adapter reads every result line, and any "success": false raises WriteFailed.

The description says documents are validated and raise DocumentError, but neither exists in this diff, and neither do the system check or the Redis lease. It should say that those land later.

Agreed. The description now presents the validation rules, DocumentError, the system check and the rebuild cache as contracts that land with the adapters, and the implementation summary lists only what is in the diff.

The uv run ... pytest command in the description also fails for me with ModuleNotFoundError: No module named 'rest_framework'. uv run --group test --with djangorestframework --with django-filter python -m pytest passes all 57 tests.

Fixed. The testing instructions use that command until #2 lands, and uv sync --group test plus uv run --group test pytest after it.

Status
******

**Draft** (for upstream discussion)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the correct status?

@rpenido

rpenido commented Nov 5, 2026

Copy link
Copy Markdown
Author

While re-checking the contract I found four gaps the review had not covered, all fixed now:

  • e306e3d rejects bytes and mappings in FilterTerm values; they would otherwise be split into integers or keys.
  • 0737a41 validates the selected backend name, so a non-string override or default is a configuration error instead of a TypeError.
  • 45a9e1c caps derived document ids at 468 bytes, the Meilisearch filterable-value limit, since id may be filterable.
  • 1f4ecfe validates RebuildTarget parts before they render an index name.

The ADR records the id limit. I am listing these here because they are small hardening fixes rather than replies to a thread, and the diff alone makes them easy to miss.

@rpenido

rpenido commented Nov 16, 2026

Copy link
Copy Markdown
Author

Thanks for the approval, @blarghmatey!
@bradenmacdonald @ormsbee Can we get this merged?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

open-source-contribution PR author is not from Axim or 2U

Projects

Status: In Eng Review

5 participants