Repository navigation
Conversation
|
Thanks for the pull request, @rpenido! This repository is currently maintained by Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review. 🔘 Get product approvalIf you haven't already, check this list to see if your contribution needs to go through the product review process.
🔘 Provide contextTo help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:
🔘 Get a green buildIf one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green. DetailsWhere can I find more information?If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources: When can I expect my changes to be merged?Our goal is to get community contributions seen and reviewed as efficiently as possible. However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:
💡 As a result it may take up to several weeks or months to complete a review and merge your PR. |
2a680a4 to
d0f19b1
Compare
08e52f3 to
a8ecc2b
Compare
a8ecc2b to
3562a56
Compare
| ``validate`` runs at registration and before settings are applied; a use case | ||
| that needs an unsupported capability fails there, not silently. The library | ||
| validates documents against ``IndexSchema`` before handing them to a backend, so | ||
| every engine rejects the same bad document for the same reason. |
There was a problem hiding this comment.
I believe it is necessary to specify what is meant by a "valid document." This is because neither the code nor the ADR mentions fields that might be missing from the document or that could be null. In Meilisearch, a missing field causes no issues, but in Typesense, it throws an error if the field is not optional.
There was a problem hiding this comment.
The previous wording and methods were a bit confusing, so I rewrote them here: c9910de
In the context of the IndexSchema, the adapter implements validate_schema (previous validate) to check if the current proposed schema match is supported. I put an example in the ADR.
In the context for document validation (now validate_document), we will match the document with the schema provided. I added the required to Field here: 7f37eb4
If we have a "required" field, like id in STUDIO_COURSE_SCHEMA, we register it with Field("id", FieldType.STRING, required=True).
When creating the index, for meilisearch we will just ignore it. For typesense, we will pass "optional": not field.required accordingly.
While populating the collection with documents, the adapter will validate the document against its schema. So passing a document with a missing required (by schema) field will throw. Even if we are using meilisearch. Our goal here is to have the same behavior for the consumer.
Does that make sense?
There was a problem hiding this comment.
That sounds good. And another question: what should be done in the opposite case: when a document arrives with an extra field that is not in the schema?
There was a problem hiding this comment.
In my opinion, we should throw an error in the case. If not thrown, we should at least drop it to have all engines with the same behavior (Typensense silent drops it).
There was a problem hiding this comment.
Updated the ADR here: 5f579a2
The validator and DocumentError land with upsert, which is still a stub.
| use_case str | ||
| index str logical index name | ||
| schema IndexSchema | ||
| query_shape QueryShape filters, facets, sort, defaults |
There was a problem hiding this comment.
In the code, Field capabilities already declare what is filterable, facetable and sortable. What would QueryShape add beyond query defaults?
There was a problem hiding this comment.
You are right. We will probably need something for the query defaults, but I don't think they belong to the registration.
We can visit that later in the implementation. Removed as part of b141421
| raise BackendNotConfigured( | ||
| "No search backend is configured. Set OPENEDX_SEARCH_DEFAULT_BACKEND " | ||
| "and declare it in OPENEDX_SEARCH_BACKENDS." |
There was a problem hiding this comment.
Quick question: does this mean a backend must always be configured, even if it’s just the default? Is there no option for an "I do not wish to configure search" state?
There was a problem hiding this comment.
Is there no option for an "I do not wish to configure search" state?
In theopenedx-searchlibrary level, I don't think so.
We will probably do it on openedx-platform, or in tutor, though. Since the search engine should also be provisioned, I think the default doesn't belong to the library itself.
Any thoughts on that?
There was a problem hiding this comment.
We don't necessarily need a dummy backend/adapter, but I think we should have a way to turn off search and/or turn off individual use cases. That could either be an enabled flag or be achieved by configuring a dummy backend for that use case, but I think an enabled: false style is more efficient than a null adapter, because it won't spend time preparing documents for insert that then get ignored.
There was a problem hiding this comment.
We don't necessarily need a dummy backend/adapter, but I think we should have a way to turn off search and/or turn off individual use cases.
Do you mind elaborating more on that? In which case would we use it?
And should we add an api.is_enabled(use_case) -> bool so the consumer is aware of it? And make the api (write, delete, rebuild, etc..) no-op while disabled?
There was a problem hiding this comment.
And should we add an api.is_enabled(use_case) -> bool so the consumer is aware of it? And make the api (write, delete, rebuild, etc..) no-op while disabled?
Yeah I was thinking of something like that. But I don't have any strong opinion on this, so I'm open to your ideas.
There was a problem hiding this comment.
As implementation, I think no-op methods + api.is_enable(use_case) should do it.
I'm more curious about "where/when do we need this"? Studio will just break without the backend. Are there some apps that we can check if we have the backend and, if not, have a fallback? Is it for a maintenance kill-switch so we can isolate the search for some reason?
There was a problem hiding this comment.
Well courseware search is an optional feature that can be turned on and off. But maybe it already has its own flag, so perhaps you're right that we don't need this in general.
Libraries v2 (with Meilisearch) was an optional feature for a while but I guess it's pretty much alway enabled now.
Yeah, I guess we don't need the enabled/disabled at this layer after all.
There was a problem hiding this comment.
Yeah, I guess we don't need the enabled/disabled at this layer after all.
If you think it is a "nice to have", please let me know! It's not complex to implement.
3562a56 to
c171d32
Compare
5f353db to
534c95f
Compare
534c95f to
3d5e981
Compare
8aa694a to
9cdab91
Compare
|
Thanks for the second pass @blarghmatey! I think I addressed all your comments.
Agreed, and it is in the ADR in 2d07758: a 200 from the Typesense import endpoint never marks a batch complete, the adapter reads every result line, and any
Agreed. The description now presents the validation rules,
Fixed. The testing instructions use that command until #2 lands, and |
| Status | ||
| ****** | ||
|
|
||
| **Draft** (for upstream discussion) |
|
While re-checking the contract I found four gaps the review had not covered, all fixed now:
The ADR records the id limit. I am listing these here because they are small hardening fixes rather than replies to a thread, and the diff alone makes them easy to miss. |
|
Thanks for the approval, @blarghmatey! |
Description
This PR adds ADR 0002 for
openedx-searchand the first implementation of the contract it defines: the engine-neutral schema model, the use-case registration layer, and the adapter contract with the consumer API shape.The ADR settles how a use case declares its document schema, how those declarations stay engine-neutral, how a deployment selects an engine, and how adapters render, reconcile, rebuild and validate the declaration. Index settings are derived from the schema. It also records the alternatives considered with pros and cons, including the narrow flat-string adapter contract and its benchmark in #3 and #4.
The implementation provides
IndexSchema,FieldandCapabilities, with required fields and a validatedidformat. The shared model rejects declarations the two engines cannot both represent:searchableis limited to strings and objects,sortablerejects arrays, fields inside object arrays must be arrays, required fields need required parents, and the primary key is the required, non-array string fieldidwithout indexing capabilities. The distinct attribute is not modelled; Studio'sdistinctAttributesetting is being dropped, not carried over.Sort precedence is fixed rather than declared: an explicit sort always outranks keyword relevance, and relevance applies only when the caller asks for it. Meilisearch applies it in the index's
rankingRuleswithsortfirst, whichreconcilesets; Typesense composes it per request insort_by, so the scoped credential embedsquery_byandquery_by_weightsbut not the sort.The ADR defines document validation so both engines accept and reject the same documents: required fields present, declared types and shapes, datetimes accepted as aware values and rendered as POSIX seconds, no unknown or null fields, and string-only free-form objects. The validator and
DocumentErrorland withupsert, so this diff documents the rules rather than implementing them.It also provides a thread-safe in-memory use-case registry with validation; per-use-case engine resolution from deployment settings; and the
AdapterABC covering schema validation, reconcile, writes, deletes, write confirmation, rebuilds and queries.openedx_search.apiexposes the consumer-side signatures forsearch,mint_credential,upsert,delete,check_writeandrebuild; the bodies land with the engine adapters. Tests cover the schema, including the cross-engine constraints, the registry and engine selection.Registrations are consumer-owned: a consumer calls
register_use_case(..., replace=True)from its ownAppConfig.ready(). The library does not scan packages or auto-discover registrations. A registration declares the use case, the logical index and the schema; query terms stay per request and scoped credentials carry tenant filters. The ADR places schema compatibility checks in a Django system check during deployment, not at registration; the check itself lands with the adapters.The ADR specifies that a rebuild records its target in a shared Redis cache with a lease, so every writer process dual-writes during a reindex and a crashed worker cannot block the next rebuild. A schema version bump is a migration with writers stopped. The cache and rebuild code land with the adapters.
Testing instructions
uv run --group test --with djangorestframework --with django-filter python -m pytestOther information
Until #2 lands, main's test settings import
rest_frameworkanddjango_filters, so the command above passes the two packages with--with. After #2,uv sync --group testanduv run --group test pytestare enough.AI disclosure: I used Claude to research the upstream libraries, create the IndexSchema and its accessory types, and help to write the ADR. I reviewed everything and also did multiple manual edits.
Private ref: