feat: update data contract fields to use nested nouns and verbs (#714) by prince-0408 · Pull Request #715 · scribe-org/Scribe-Android

prince-0408 · 2026-10-06T22:32:31Z

Contributor checklist

This pull request is on a separate branch and not the main branch
I have tested my code with the ./gradlew lintKotlin detekt test command as directed in the testing section of the contributing guide

Description

In upstream Scribe-Data (PRs #717 and #721), data contracts were restructured to group linguistic columns under nouns and verbs top-level keys to allow query generation.

This PR updates Scribe-Android to align with the new contract specification:

Updated Asset Data Contracts (app/src/main/assets/data-contracts/):
- Synchronized all 8 language contract YAML files (de.yaml, en.yaml, es.yaml, fr.yaml, it.yaml, pt.yaml, ru.yaml, sv.yaml) with upstream Scribe-Data.
- Restructured root keys: numbers and genders are now nested under nouns, and conjugations is nested under verbs. numbers maps indices to NumberPair(singular, plural).
Model Layer (DataContract.kt):
- Added @Serializable data class Nouns(val genders: Genders, val numbers: Map<Int, NumberPair>).
- Added @Serializable data class NumberPair(val singular: String, val plural: String).
- Added @Serializable data class Verbs(val conjugations: Map<Int, TenseGroup>).
- Updated DataContract to contain nouns, verbs, and translations.
Consumers & Managers:
- GenderDataManager: Updated noun column name and gender lookup paths to contract.nouns.numbers.values.firstOrNull()?.singular and contract.nouns.genders.*.
- PluralFormsManager: Updated getAllPluralForms and findPluralOfWord to read from contract.nouns.numbers.
- ConjugateDataManager: Updated conjugation lookups and label extraction to yamlData?.verbs?.conjugations.
- KeyboardDataHandler: Updated numbersColumns resolution to flat-map it.singular and it.plural from dataContract?.nouns?.numbers?.values.
- ConjugationSelectionScreen: Updated conjugation grouping to iterate over contract.verbs.conjugations.values.
Testing & Documentation:
- Updated mock setup in PluralFormsManagerTest.kt to match the new Nouns and NumberPair structure.
- Added ContractDataLoaderTest.kt unit test suite covering valid YAML parsing with nested nouns and verbs, file-not-found handling, malformed YAML fallback, and verifying that all 8 language contracts in assets parse successfully.
- Added entry to CHANGELOG.md under ### 🗃️ Data.

How Was This Tested

Ran ./gradlew formatKotlin to ensure ktlint formatting.
Ran ./gradlew lintKotlin detekt test passing all 143 Gradle tasks across debug and release flavors with 0 errors.

Related issue

…be-org#714) - Update data contracts in assets with new nouns and verbs structure from Scribe-Data - Update DataContract data classes to support nouns, number pairs, and verbs - Update GenderDataManager, PluralFormsManager, ConjugateDataManager, KeyboardDataHandler, and ConjugationSelectionScreen - Update existing plural forms test and add ContractDataLoaderTest - Update CHANGELOG.md

andrewtavis · 2026-10-07T07:45:27Z

@@ -1,103 +1,109 @@
 # Database column mappings that are read by Scribe applications to structure how data is displayed to the user.
 # Includes interface titles and the order that values should be presented (chronological or alphabetical).
+# Contracts are also how Scribe-Data generates queries. All data type end values are query lexeme forms.


Thanks for the PR, @prince-0408: One thing is that ideally we'd be able to have this work without the contracts in the repo as they should be being sent by Scribe-Server with the data. Could we remove these files and write it so that it works with the contracts that are being sent by the server?

CC @DeleMike, @angrezichatterbox and @Roniscend for this point :)

So let's remove all contracts in data-contracts and just have this directory saved with the .gitkeep that's in there. We can then verify that new contracts from the server are always being used 😊

Note @Roniscend that your work on the conjugate verb UI depends on this. We can try to bring this in via reviews in the coming days!

Thank you, @andrewtavis! It makes perfect sense to make the APK independent of bundled contracts and load them on the fly from the server.

The following changes have been committed to resolve the issue:

Removed all 8 contract YAML files from app/src/main/assets/data-contracts/ except for the .gitkeep.

Modified ContractDataLoader to check for and load the contract files from local application storage (filesDir/data-contracts/) falling back to assets.

Added methods saveContract and parseContract to ContractDataLoader to save contracts downloaded from the server.

Modified unit tests of ContractDataLoader to check the contract downloading and saving from the local storage.

@DeleMike @andrewtavis - For the wiring of the downloading of the contract, please let me know the following:

Does the contract file come inside the response of the GET /api/v1/data/{lang} request?

Or does the client need to make a separate GET /api/v1/contracts?lang={lang} request when downloading language packs?

Based on your answer, it will not be a problem to wire the saving functionality to DataDownloadViewModel.

…undled assets (scribe-org#714) - Remove bundled YAML contracts from assets/data-contracts preserving .gitkeep - Update ContractDataLoader to load from local app storage with assets fallback - Add saveContract helper in ContractDataLoader to persist downloaded contracts - Update ContractDataLoaderTest and CHANGELOG.md

Roniscend · 2026-10-14T08:47:57Z

-        Log.d("ContractDataLoader", "Attempting to load contract: $contractName")
+        Log.d(TAG, "Attempting to load contract: $contractName")
+
+        val localFile = File(File(context.filesDir, CONTRACTS_DIR), contractName)


@prince-0408 Connect the contract download to the app or keep the bundled contracts until thats done. Right now nothing saves contracts, so after merge nobody will have one and the Conjugate screen and keyboard features will stop working.
CC @andrewtavis you can give more clarity on this

Thanks @Roniscend! Yes we should be connecting the contract download to the apps in this PR. This means that we won't have contracts until we get them from Scribe-Server, but also we'll know that things are working with the contracts we're getting from the server rather than maybe working with the ones we have saved to the repo and then things break if we switch to the ones from the server.

Appreciate the point here!

andrewtavis · 2026-10-17T13:35:19Z

Checking in here with @angrezichatterbox right now, who confirmed that the contracts are not being saved. Let's plan on getting this done today or tomorrow at the latest, and then we can move on to the following two issues:

Conjugate command returns "Not in Wikidata" for valid verbs #598 (ideally would also be closed by this PR, as if we're not returning verbs, then there's something wrong 😊)
Populate verb conjugations in UI from the database #571 (@Roniscend)
Remove profanity from autosuggestions and autocompletions #665 (@angrezichatterbox)

andrewtavis · 2026-10-17T13:36:21Z

Quick note that I've updated the commit message to also close #598 when this is merged, as @angrezichatterbox and I agreed that we need to verify that all verbs are working before this is brought in :)

andrewtavis · 2026-10-17T13:37:54Z

@prince-0408, do let us know what your availability is to finalize this PR, as it'd be great if we could finish it tonight or tomorrow. @angrezichatterbox has offered to finish it up, in case you're a bit busy (no stress, if so!) 😊

andrewtavis · 2026-10-17T14:16:25Z

@angrezichatterbox, FYI @prince-0408 is working on this today, so we should be good for a review soon and then we can move on to other issues 😊

andrewtavis · 2026-10-20T20:17:35Z

@DeleMike, can you let everyone here know when the changes from scribe-org/Scribe-Server#114 are live so that we can look into finalizing this?

CC @angrezichatterbox, @prince-0408 and @Roniscend 👋

andrewtavis-wmde · 2026-10-21T08:15:14Z

Writing in from my work account quickly. FYI @DeleMike, @angrezichatterbox, @Roniscend and @prince-0408: There's currently a security incident with PHP that is affecting WMF services. See: https://wikitech.wikimedia.org/wiki/Incidents/2026-10-21_PHP_security_update/Community. Scribe-Server is thus down, with the quote from the link that relates to this being:

All of Cloud VPS and Toolforge. These services host a large number of PHP applications, and may remain offline for a more extended period of time. This includes all *.wmcloud.org, *.wmflabs.org, and *.toolforge.org sites.

Let's follow this and see when we can get this up and running!

DeleMike · 2026-10-21T09:56:32Z

Thanks @andrewtavis for the update! This is sad. I was looking forward to updating the live server today, but I will be watching, as stated on their updates page, that things might be up by 14:00 UTC. Let's be hopeful.

andrewtavis · 2026-10-21T18:26:58Z

Note that what we're hearing is that Toolforge will be down for a bit. I assume/hope that it will be up and running by the end of the week.

…ribe-org#714) - Add getContract endpoint in ApiService for Scribe-Server /contracts route - Add downloadAndSaveContract and hasContract helpers in ContractDataLoader - Connect contract download to DataDownloadViewModel and ConjugateDataDownloadViewModel - Add on-demand contract download fallback in ConjugationSelectionScreen - Support JSON contracts and add unit tests in ContractDataLoaderTest - Update CHANGELOG.md

- Add kotlinx-serialization-json dependency to app/build.gradle.kts - Refine exception handling in ContractDataLoader for specific parsing and network exceptions - Fix ktlint formatting issues in ApiService, ConjugateDataDownloadViewModel, and tests

andrewtavis · 2026-10-22T18:29:23Z

Quick note all that there's a hope that parts of Toolforge will be up and running again in the next 24 hours. This isn't to say that we'll be able to get the server back on line, but progress is being made :)

andrewtavis · 2026-10-23T15:33:16Z

Scribe-Server is back up and running after some amazing work by @DeleMike! @Roniscend and @angrezichatterbox, would you two have a moment to check this out?

andrewtavis · 2026-10-24T09:12:29Z

Quick note on the usability of the changes here :) It looks like this PR so far solves #714, but I'm not able to get conjugations for valid verbs, meaning so far #598 isn't finalized. I think it would be great if we could get to that before bringing this in! 😊

CC @angrezichatterbox, as you'd said you'd looked into potential fixes 👋

Roniscend · 2026-10-24T15:30:47Z

+        val extension = if (isJson) "json" else "yaml"
+        val file = File(dir, "$lang.$extension")
+        return try {
+            file.writeText(yamlContent)


Parse check the content before writeText, and write to a temp file then rename, so a bad 200 response can’t overwrite a working contract , reject column names that fail SqlIdentifier.isValid in the same check, since they now come from the network and reach sql

Roniscend · 2026-10-24T15:31:26Z

                        dbHelper.syncDatabase(response)

+                        val contractLoader = ContractDataLoader(getApplication())
+                        contractLoader.downloadAndSaveContract(langCode)


The Boolean result is ignored, so a failed contract download still shows success and saves the timestamp. Existing users also never get a contract until they tap download, so please handle the failure and add a one-time backfill

Roniscend · 2026-10-24T15:32:22Z

                        dbHelper.syncConjugateDatabase(response)

+                        val contractLoader = ContractDataLoader(getApplication())
+                        contractLoader.downloadAndSaveContract(langCode)


Again the result of downloadAndSaveContract is ignored, so a failed download still reports success. Handle the false case here too

…oad failures (scribe-org#714) - Add SqlIdentifier utility to validate column names and prevent SQL injection - Parse-check and SQL-validate contract before writing, and use temporary file with atomic rename in ContractDataLoader - Handle contract download failures in DataDownloadViewModel and ConjugateDataDownloadViewModel to prevent false success and timestamps - Add backfillMissingContracts in download view models for existing users with downloaded databases - Add unit tests for SqlIdentifier, atomic save, and invalid contract rejection

… management

…date validation in ContractDataLoader

angrezichatterbox · 2026-10-27T18:02:00Z

@prince-0408 Did some quick changes here so that we could finalise the PR a bit quicker. Really thanks for your help.

@andrewtavis this PR is ready for review.

prince-0408 · 2026-10-27T18:38:39Z

@prince-0408 Did some quick changes here so that we could finalise the PR a bit quicker. Really thanks for your help.

@andrewtavis this PR is ready for review.

Thanks @angrezichatterbox for pushing those final touches for complex verb forms! Everything looks good on my end as well, and all requested changes are in place. Looking forward to your review @andrewtavis!

andrewtavis

praise: All's looking and working great well here, @prince-0408! Thanks for getting this done, and thanks also @Roniscend and @angrezichatterbox for the detailed reviews/improvements! 🚀 Really getting close! 😊

prince-0408 requested a review from andrewtavis October 6, 2026 22:33

andrewtavis reviewed Oct 7, 2026

View reviewed changes

prince-0408 requested a review from andrewtavis October 8, 2026 12:49

andrewtavis mentioned this pull request Oct 12, 2026

[Declension] Allow for declined pronoun selection from prepositions case annotations #276

Open

2 tasks done

Roniscend self-requested a review October 14, 2026 08:47

Roniscend suggested changes Oct 14, 2026

View reviewed changes

andrewtavis added the -next release- Included in the next release label Oct 17, 2026

andrewtavis mentioned this pull request Oct 17, 2026

Conjugate command returns "Not in Wikidata" for valid verbs #598

Open

2 tasks done

Roniscend mentioned this pull request Oct 17, 2026

Populate verb conjugations in UI from the database #571

Open

2 tasks done

andrewtavis requested a review from angrezichatterbox October 17, 2026 14:15

angrezichatterbox force-pushed the feat/data-contract-nouns-verbs-714 branch from c820d7e to 9c57c45 Compare October 18, 2026 17:16

This was referenced Oct 18, 2026

feat: highlight autocomplete suggestion when it matches typed word #648

Open

Redundant sequential SQLite file opens and identical SELECT queries #738

Open

Fix/return data contracts scribe-org/Scribe-Server#114

Merged

prince-0408 added 2 commits October 22, 2026 06:37

prince-0408 requested a review from Roniscend October 22, 2026 20:45

Roniscend suggested changes Oct 24, 2026

View reviewed changes

prince-0408 added 2 commits October 25, 2026 19:08

chore: merge upstream/main into feat/data-contract-nouns-verbs-714

4647de1

prince-0408 requested a review from Roniscend October 25, 2026 15:41

angrezichatterbox added 2 commits October 27, 2026 23:23

feat: enhance contract loading and downloading logic, streamline data…

8b9947d

… management

feat: enhance parsing of complex forms in ConjugateDataManager and up…

b021dd8

…date validation in ContractDataLoader

Merge branch 'main' into feat/data-contract-nouns-verbs-714

c871583

andrewtavis approved these changes Oct 27, 2026

View reviewed changes

andrewtavis merged commit 9a7dfa0 into scribe-org:main Oct 27, 2026
7 checks passed

andrewtavis mentioned this pull request Oct 27, 2026

Remove profanity from autosuggestions and autocompletions #665

Open

2 tasks done

Conversation

prince-0408 commented Oct 6, 2026 • edited by andrewtavis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contributor checklist

Description

How Was This Tested

Related issue

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewtavis commented Oct 17, 2026

Uh oh!

andrewtavis commented Oct 17, 2026

Uh oh!

andrewtavis commented Oct 17, 2026

Uh oh!

andrewtavis commented Oct 17, 2026

Uh oh!

andrewtavis commented Oct 20, 2026

Uh oh!

andrewtavis-wmde commented Oct 21, 2026

Uh oh!

DeleMike commented Oct 21, 2026

Uh oh!

andrewtavis commented Oct 21, 2026

Uh oh!

andrewtavis commented Oct 22, 2026

Uh oh!

andrewtavis commented Oct 23, 2026

Uh oh!

andrewtavis commented Oct 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

angrezichatterbox commented Oct 27, 2026

Uh oh!

prince-0408 commented Oct 27, 2026

Uh oh!

andrewtavis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

prince-0408 commented Oct 6, 2026 •

edited by andrewtavis

Loading