From 1e1d6c347876e7b562a8b7ed4e82a0c0c44575b8 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Sat, 17 Oct 2026 14:23:14 -0700 Subject: [PATCH 01/13] ADFA-6597: Find a tooltip through Tooltips' (categoryId, tag) index Written as a join of Tooltips and TooltipCategories, the tooltip query made SQLite scan all of Tooltips - 46,128 rows, 13 MB - for every tooltip shown, because documentation.db carries no ANALYZE statistics (EXPLAIN QUERY PLAN: "SCAN T"). Looking the category's id up first lets the UNIQUE (categoryId, tag) index find the row: 6.2 ms -> 0.004 ms per lookup on a desktop against the real database, same rows. IN rather than =, so two categories sharing a name still come back as duplicates, which getTooltip reports as before. The buttons query still scans TooltipButtons (5.4 MB); that needs an index in the database, which this change cannot add. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3 --- docs/documentation-database.md | 2 +- idetooltips/build.gradle.kts | 1 + .../androidide/idetooltips/ToolTipManager.kt | 13 +-- .../idetooltips/TooltipQueryTest.kt | 85 +++++++++++++++++++ 4 files changed, 95 insertions(+), 6 deletions(-) create mode 100644 idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt diff --git a/docs/documentation-database.md b/docs/documentation-database.md index 8fb2303bcc..1bd58218cd 100644 --- a/docs/documentation-database.md +++ b/docs/documentation-database.md @@ -76,7 +76,7 @@ Of the five sites below, only `DocumentationContentSource` and `ToolTipManager` - **`common/.../documentation/DocumentationContentSource.kt`** — the one Tier 3 pipeline that reads this database: row lookup (a request target is matched raw first, since stored `Content.path` values are percent-encoded, with a percent-decoded fallback on a miss — shared by both transports so they agree on which pages exist), chunked-row reassembly, dictionary-aware Brotli decode gated on the declared documentation version, and the database swaps — to a newer sdcard debug database, and reopening the installed file when an asset install rewrites it in place under the cached handle — under a read/write lock so several threads can read while a swap cannot close the handle under them. It also renders the rows that are Pebble template contexts (`templateId > 0`, the Kotlin doc set's pages) — resolving each template's references to other templates against `Templates.name` as it goes — so both transports below serve finished pages and neither needs the template engine itself. All of that logic exists once (ADFA-5176). - **`common/.../documentation/DocumentationRequestInterceptor.kt`** — serves Tier 3 *in-process* for the app's WebViews (`HelpActivity`, the tooltip fragment, `FAQActivity`), through `WebViewClient.shouldInterceptRequest`, so a page's assets cost a database read instead of a TCP connection each (ADFA-5176). It matches the same `http://localhost:6174/...` URL space, so the strings.xml entries, `ToolTipManager`'s link builder and the `DocumentationExtension` contract need no changes; anything it declines — a `/pr/` endpoint, an unknown path, a failed read — falls through to `WebServer` unchanged. Both transports get their `Content-Type` charset from the same place, `ContentTypeHeaders` (ADFA-5241), so a row does not describe itself differently depending on which one served it. Set `/sdcard/Download/CodeOnTheGo.nointercept` to force documentation back onto the server. - **`app/.../localWebServer/WebServer.kt`** — serves Tier 3 over HTTP on port 6174, for WebViews that are not wired to the interceptor above and for the `/pr/` developer endpoints. It reads through `DocumentationContentSource`, so it holds no database, template engine or decode logic of its own; what remains here is HTTP: request parsing, the `/pr/` pages, error responses, and the CSS/asset shortcuts. Also serves a Dynamic Bookshelf JSON payload (joining `Content`/`Bookshelf`/`BookCategories`, rendered through the `bookshelf` template) and debug-only HTML dumps at `/pr/db` (`LastChange`, last 20 rows) and `/pr/pr` (recent projects, from a *different* database). -- **`idetooltips/.../ToolTipManager.kt`** — serves Tier 1/2. Looks up `Tooltips` joined to `TooltipCategories` by `(category, tag)`, then `TooltipButtons` for the Tier 3 links shown at the bottom. +- **`idetooltips/.../ToolTipManager.kt`** — serves Tier 1/2. Looks up a tooltip by `(category, tag)`: the category's id from `TooltipCategories` first, so the row is found through `Tooltips`' `UNIQUE (categoryId, tag)` index rather than a scan of the whole table (ADFA-6597). Then `TooltipButtons` for the Tier 3 links shown at the bottom, which still scans that table: it has no index on `tooltipId`. - **`plugin-manager/.../documentation/PluginDocumentationManager.kt`** (with `Tier3AssetWalker.kt`, and the `DocumentationExtension` contract in `plugin-api`) — lets plugins contribute their own help content into the same lookup paths, under the reserved `plugin//...` prefix. Brotli assets are compressed against the same `CompressionDictionary` the rest of the table uses, read from the database being written. When that dictionary can't be read the install is abandoned rather than written plain, and retried on the plugin's next activation. ## Editing the database diff --git a/idetooltips/build.gradle.kts b/idetooltips/build.gradle.kts index 442393178a..dd544aa7fc 100644 --- a/idetooltips/build.gradle.kts +++ b/idetooltips/build.gradle.kts @@ -28,4 +28,5 @@ dependencies { implementation(projects.common) testImplementation(libs.tests.junit) + testImplementation(libs.tests.robolectric) } diff --git a/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt index 6e2e4de7b3..2abc0cc6c5 100644 --- a/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt +++ b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt @@ -53,12 +53,15 @@ object TooltipManager { private val debugDatabaseFile: File = File(android.os.Environment.getExternalStorageDirectory().toString() + "/Download/documentation.db") - private const val QUERY_TOOLTIP = """ + // Looks the category up first, so the tag is found through Tooltips' UNIQUE (categoryId, tag) + // index. Written as a join, it made SQLite's planner scan all of Tooltips (46,128 rows, 13 MB) + // for every tooltip shown, because documentation.db carries no ANALYZE statistics (ADFA-6597). + // IN rather than =, so two categories sharing a name still come back as duplicates, as before. + internal const val QUERY_TOOLTIP = """ SELECT T.rowid, T.id, T.summary, T.detail - FROM Tooltips AS T, TooltipCategories AS TC - WHERE T.categoryId = TC.id + FROM Tooltips AS T + WHERE T.categoryId IN (SELECT id FROM TooltipCategories WHERE category = ?) AND T.tag = ? - AND TC.category = ? """ @@ -93,7 +96,7 @@ object TooltipManager { "n/a" } - database.rawQuery(QUERY_TOOLTIP, arrayOf(tag, category)).use { c -> + database.rawQuery(QUERY_TOOLTIP, arrayOf(category, tag)).use { c -> when (c.count) { // A missing tag is a normal outcome (e.g. a Gradle task with no // authored tooltip). Honor the nullable contract so the caller diff --git a/idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt b/idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt new file mode 100644 index 0000000000..0c6ad30180 --- /dev/null +++ b/idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt @@ -0,0 +1,85 @@ +package com.itsaky.androidide.idetooltips + +import android.database.sqlite.SQLiteDatabase +import org.junit.After +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertTrue +import org.junit.Before +import org.junit.Test +import org.junit.runner.RunWith +import org.robolectric.RobolectricTestRunner +import org.robolectric.annotation.Config +import org.robolectric.annotation.ConscryptMode + +/** + * Runs [TooltipManager.QUERY_TOOLTIP] against documentation.db's own Tooltips schema. The query + * plan matters as much as the rows: the join it replaced scanned the whole 13 MB table for every + * tooltip shown (ADFA-6597). + */ +@RunWith(RobolectricTestRunner::class) +@Config(manifest = Config.NONE, sdk = [28]) +@ConscryptMode(ConscryptMode.Mode.OFF) +class TooltipQueryTest { + private lateinit var db: SQLiteDatabase + + @Before + fun setUp() { + db = SQLiteDatabase.create(null) + db.execSQL("CREATE TABLE TooltipCategories (id INTEGER PRIMARY KEY, category TEXT NOT NULL)") + db.execSQL( + "CREATE TABLE Tooltips (id INTEGER PRIMARY KEY AUTOINCREMENT, categoryId INTEGER NOT NULL, " + + "tag TEXT NOT NULL, summary TEXT NOT NULL, detail TEXT NOT NULL, UNIQUE (categoryId, tag), " + + "FOREIGN KEY(categoryId) REFERENCES TooltipCategories(id))", + ) + listOf("ide", "xml", "java", "kotlin").forEachIndexed { i, category -> + db.execSQL("INSERT INTO TooltipCategories VALUES (?, ?)", arrayOf(i + 1, category)) + } + db.beginTransaction() + for (categoryId in 1..4) { + for (n in 0 until 500) { + db.execSQL( + "INSERT INTO Tooltips (categoryId, tag, summary, detail) VALUES (?, ?, ?, ?)", + arrayOf(categoryId, "tag$n", "summary $categoryId/$n", "detail"), + ) + } + } + db.setTransactionSuccessful() + db.endTransaction() + } + + @After + fun tearDown() { + db.close() + } + + @Test + fun `finds the one tooltip for a category and tag`() { + db.rawQuery(TooltipManager.QUERY_TOOLTIP, arrayOf("java", "tag7")).use { c -> + assertEquals(1, c.count) + c.moveToFirst() + assertEquals("summary 3/7", c.getString(2)) + } + db.rawQuery(TooltipManager.QUERY_TOOLTIP, arrayOf("java", "no-such-tag")).use { assertEquals(0, it.count) } + db.rawQuery(TooltipManager.QUERY_TOOLTIP, arrayOf("no-such-category", "tag7")).use { assertEquals(0, it.count) } + } + + @Test + fun `two categories with the same name still come back as two rows`() { + db.execSQL("INSERT INTO TooltipCategories VALUES (5, 'java')") + db.execSQL("INSERT INTO Tooltips (categoryId, tag, summary, detail) VALUES (5, 'tag7', 'duplicate', 'detail')") + + db.rawQuery(TooltipManager.QUERY_TOOLTIP, arrayOf("java", "tag7")).use { assertEquals(2, it.count) } + } + + @Test + fun `finds the tag through the (categoryId, tag) index, not a scan of Tooltips`() { + val plan = + db.rawQuery("EXPLAIN QUERY PLAN " + TooltipManager.QUERY_TOOLTIP, arrayOf("java", "tag7")).use { c -> + buildList { while (c.moveToNext()) add(c.getString(3)) } + } + + assertTrue(plan.toString(), plan.any { "sqlite_autoindex_Tooltips_1" in it }) + assertFalse(plan.toString(), plan.any { Regex("""\bSCAN (TABLE )?(T|Tooltips)\b""").containsMatchIn(it) }) + } +} From e5f44a844be365d3c6552ce815cea5f06ea4b10b Mon Sep 17 00:00:00 2001 From: David Schachter Date: Sat, 17 Oct 2026 15:14:03 -0700 Subject: [PATCH 02/13] ADFA-6597: Test the tooltip query's arguments in the order the app binds them From review of PR #2160: - TooltipQueryTest bound the query's arguments itself, so it would pass whatever order getTooltip used, and this PR reversed that order. The SQL and its binding now live in queryTooltip, in TooltipQuery.kt, which getTooltip calls with named arguments and the test calls too. Swapping the binding fails two of the three tests. - QUERY_TOOLTIP is marked @VisibleForTesting: only the plan test reads it. It is still a public static field in bytecode, as it was inside TooltipManager, and so still in the plugin-api jar. - Drops @ConscryptMode(OFF). The Conscrypt leak it guarded against broke a TLS test in common; idetooltips has none. A file of its own rather than top-level code in ToolTipManager.kt: ktlint leaves ToolTipManager.kt unformatted while its only top-level declaration, object TooltipManager, does not match the file name. A second declaration there makes Spotless reformat all of the file's ~380 changed lines, burying this change. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3 --- docs/documentation-database.md | 2 +- .../androidide/idetooltips/ToolTipManager.kt | 14 +---------- .../androidide/idetooltips/TooltipQuery.kt | 25 +++++++++++++++++++ .../idetooltips/TooltipQueryTest.kt | 18 ++++++------- 4 files changed, 35 insertions(+), 24 deletions(-) create mode 100644 idetooltips/src/main/java/com/itsaky/androidide/idetooltips/TooltipQuery.kt diff --git a/docs/documentation-database.md b/docs/documentation-database.md index 1bd58218cd..d85a0d0b20 100644 --- a/docs/documentation-database.md +++ b/docs/documentation-database.md @@ -76,7 +76,7 @@ Of the five sites below, only `DocumentationContentSource` and `ToolTipManager` - **`common/.../documentation/DocumentationContentSource.kt`** — the one Tier 3 pipeline that reads this database: row lookup (a request target is matched raw first, since stored `Content.path` values are percent-encoded, with a percent-decoded fallback on a miss — shared by both transports so they agree on which pages exist), chunked-row reassembly, dictionary-aware Brotli decode gated on the declared documentation version, and the database swaps — to a newer sdcard debug database, and reopening the installed file when an asset install rewrites it in place under the cached handle — under a read/write lock so several threads can read while a swap cannot close the handle under them. It also renders the rows that are Pebble template contexts (`templateId > 0`, the Kotlin doc set's pages) — resolving each template's references to other templates against `Templates.name` as it goes — so both transports below serve finished pages and neither needs the template engine itself. All of that logic exists once (ADFA-5176). - **`common/.../documentation/DocumentationRequestInterceptor.kt`** — serves Tier 3 *in-process* for the app's WebViews (`HelpActivity`, the tooltip fragment, `FAQActivity`), through `WebViewClient.shouldInterceptRequest`, so a page's assets cost a database read instead of a TCP connection each (ADFA-5176). It matches the same `http://localhost:6174/...` URL space, so the strings.xml entries, `ToolTipManager`'s link builder and the `DocumentationExtension` contract need no changes; anything it declines — a `/pr/` endpoint, an unknown path, a failed read — falls through to `WebServer` unchanged. Both transports get their `Content-Type` charset from the same place, `ContentTypeHeaders` (ADFA-5241), so a row does not describe itself differently depending on which one served it. Set `/sdcard/Download/CodeOnTheGo.nointercept` to force documentation back onto the server. - **`app/.../localWebServer/WebServer.kt`** — serves Tier 3 over HTTP on port 6174, for WebViews that are not wired to the interceptor above and for the `/pr/` developer endpoints. It reads through `DocumentationContentSource`, so it holds no database, template engine or decode logic of its own; what remains here is HTTP: request parsing, the `/pr/` pages, error responses, and the CSS/asset shortcuts. Also serves a Dynamic Bookshelf JSON payload (joining `Content`/`Bookshelf`/`BookCategories`, rendered through the `bookshelf` template) and debug-only HTML dumps at `/pr/db` (`LastChange`, last 20 rows) and `/pr/pr` (recent projects, from a *different* database). -- **`idetooltips/.../ToolTipManager.kt`** — serves Tier 1/2. Looks up a tooltip by `(category, tag)`: the category's id from `TooltipCategories` first, so the row is found through `Tooltips`' `UNIQUE (categoryId, tag)` index rather than a scan of the whole table (ADFA-6597). Then `TooltipButtons` for the Tier 3 links shown at the bottom, which still scans that table: it has no index on `tooltipId`. +- **`idetooltips/.../ToolTipManager.kt`** — serves Tier 1/2. Looks up a tooltip by `(category, tag)` through `queryTooltip` in `TooltipQuery.kt`: the category's id from `TooltipCategories` first, so the row is found through `Tooltips`' `UNIQUE (categoryId, tag)` index rather than a scan of the whole table (ADFA-6597). Then `TooltipButtons` for the Tier 3 links shown at the bottom, which still scans that table: it has no index on `tooltipId`. - **`plugin-manager/.../documentation/PluginDocumentationManager.kt`** (with `Tier3AssetWalker.kt`, and the `DocumentationExtension` contract in `plugin-api`) — lets plugins contribute their own help content into the same lookup paths, under the reserved `plugin//...` prefix. Brotli assets are compressed against the same `CompressionDictionary` the rest of the table uses, read from the database being written. When that dictionary can't be read the install is abandoned rather than written plain, and retried on the plugin's next activation. ## Editing the database diff --git a/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt index 2abc0cc6c5..9bf8dd6259 100644 --- a/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt +++ b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt @@ -53,18 +53,6 @@ object TooltipManager { private val debugDatabaseFile: File = File(android.os.Environment.getExternalStorageDirectory().toString() + "/Download/documentation.db") - // Looks the category up first, so the tag is found through Tooltips' UNIQUE (categoryId, tag) - // index. Written as a join, it made SQLite's planner scan all of Tooltips (46,128 rows, 13 MB) - // for every tooltip shown, because documentation.db carries no ANALYZE statistics (ADFA-6597). - // IN rather than =, so two categories sharing a name still come back as duplicates, as before. - internal const val QUERY_TOOLTIP = """ - SELECT T.rowid, T.id, T.summary, T.detail - FROM Tooltips AS T - WHERE T.categoryId IN (SELECT id FROM TooltipCategories WHERE category = ?) - AND T.tag = ? - """ - - private const val QUERY_TOOLTIP_BUTTONS = """ SELECT description, uri FROM TooltipButtons @@ -96,7 +84,7 @@ object TooltipManager { "n/a" } - database.rawQuery(QUERY_TOOLTIP, arrayOf(category, tag)).use { c -> + queryTooltip(database, category = category, tag = tag).use { c -> when (c.count) { // A missing tag is a normal outcome (e.g. a Gradle task with no // authored tooltip). Honor the nullable contract so the caller diff --git a/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/TooltipQuery.kt b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/TooltipQuery.kt new file mode 100644 index 0000000000..71354eedc0 --- /dev/null +++ b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/TooltipQuery.kt @@ -0,0 +1,25 @@ +package com.itsaky.androidide.idetooltips + +import android.database.Cursor +import android.database.sqlite.SQLiteDatabase +import androidx.annotation.VisibleForTesting + +// Looks the category up first, so the tag is found through Tooltips' UNIQUE (categoryId, tag) +// index. Written as a join, it made SQLite's planner scan all of Tooltips (46,128 rows, 13 MB) +// for every tooltip shown, because documentation.db carries no ANALYZE statistics (ADFA-6597). +// IN rather than =, so two categories sharing a name still come back as duplicates, as before. +@VisibleForTesting +internal const val QUERY_TOOLTIP = """ + SELECT T.rowid, T.id, T.summary, T.detail + FROM Tooltips AS T + WHERE T.categoryId IN (SELECT id FROM TooltipCategories WHERE category = ?) + AND T.tag = ? +""" + +// Outside TooltipManager, so tests can run it without initializing the object, which reads the +// installed app's Environment. Binds the arguments in the order QUERY_TOOLTIP expects them. +internal fun queryTooltip( + database: SQLiteDatabase, + category: String, + tag: String, +): Cursor = database.rawQuery(QUERY_TOOLTIP, arrayOf(category, tag)) diff --git a/idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt b/idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt index 0c6ad30180..6af7b78ba1 100644 --- a/idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt +++ b/idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt @@ -10,16 +10,14 @@ import org.junit.Test import org.junit.runner.RunWith import org.robolectric.RobolectricTestRunner import org.robolectric.annotation.Config -import org.robolectric.annotation.ConscryptMode /** - * Runs [TooltipManager.QUERY_TOOLTIP] against documentation.db's own Tooltips schema. The query - * plan matters as much as the rows: the join it replaced scanned the whole 13 MB table for every - * tooltip shown (ADFA-6597). + * Runs [queryTooltip], the lookup TooltipManager.getTooltip makes, against documentation.db's own + * Tooltips schema. The query plan matters as much as the rows: the join it replaced scanned the + * whole 13 MB table for every tooltip shown (ADFA-6597). */ @RunWith(RobolectricTestRunner::class) @Config(manifest = Config.NONE, sdk = [28]) -@ConscryptMode(ConscryptMode.Mode.OFF) class TooltipQueryTest { private lateinit var db: SQLiteDatabase @@ -55,13 +53,13 @@ class TooltipQueryTest { @Test fun `finds the one tooltip for a category and tag`() { - db.rawQuery(TooltipManager.QUERY_TOOLTIP, arrayOf("java", "tag7")).use { c -> + queryTooltip(db, category = "java", tag = "tag7").use { c -> assertEquals(1, c.count) c.moveToFirst() assertEquals("summary 3/7", c.getString(2)) } - db.rawQuery(TooltipManager.QUERY_TOOLTIP, arrayOf("java", "no-such-tag")).use { assertEquals(0, it.count) } - db.rawQuery(TooltipManager.QUERY_TOOLTIP, arrayOf("no-such-category", "tag7")).use { assertEquals(0, it.count) } + queryTooltip(db, category = "java", tag = "no-such-tag").use { assertEquals(0, it.count) } + queryTooltip(db, category = "no-such-category", tag = "tag7").use { assertEquals(0, it.count) } } @Test @@ -69,13 +67,13 @@ class TooltipQueryTest { db.execSQL("INSERT INTO TooltipCategories VALUES (5, 'java')") db.execSQL("INSERT INTO Tooltips (categoryId, tag, summary, detail) VALUES (5, 'tag7', 'duplicate', 'detail')") - db.rawQuery(TooltipManager.QUERY_TOOLTIP, arrayOf("java", "tag7")).use { assertEquals(2, it.count) } + queryTooltip(db, category = "java", tag = "tag7").use { assertEquals(2, it.count) } } @Test fun `finds the tag through the (categoryId, tag) index, not a scan of Tooltips`() { val plan = - db.rawQuery("EXPLAIN QUERY PLAN " + TooltipManager.QUERY_TOOLTIP, arrayOf("java", "tag7")).use { c -> + db.rawQuery("EXPLAIN QUERY PLAN " + QUERY_TOOLTIP, arrayOf("java", "tag7")).use { c -> buildList { while (c.moveToNext()) add(c.getString(3)) } } From b900e2ed79ee038a02dd2501a698f1668d3d5467 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Sun, 18 Oct 2026 13:10:30 -0700 Subject: [PATCH 03/13] ADFA-3305: Move plugin tooltip SQL out of PluginDocumentationManager No behavior change. The four helpers that write and delete a plugin's tooltips move, unchanged, to PluginTooltipTables.kt behind two internal functions, writePluginTooltips and removePluginTooltips, so tests can run them against an in-memory database without a Context. The manager now maps each PluginTooltipEntry to its final text and resolved button URIs before handing them over, inside the same transaction as before. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3 --- .../PluginDocumentationManager.kt | 151 ++--------------- .../documentation/PluginTooltipTables.kt | 158 ++++++++++++++++++ 2 files changed, 176 insertions(+), 133 deletions(-) create mode 100644 plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginTooltipTables.kt diff --git a/plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginDocumentationManager.kt b/plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginDocumentationManager.kt index b60f15ed32..f13adaa3ee 100644 --- a/plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginDocumentationManager.kt +++ b/plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginDocumentationManager.kt @@ -110,17 +110,8 @@ class PluginDocumentationManager( db.beginTransaction() try { - removePluginDocumentationInternal(db, pluginId) - - val categoryId = insertOrGetCategoryId(db, pluginCategory(pluginId)) - - for (entry in entries) { - val tooltipId = insertTooltip(db, categoryId, entry) - entry.buttons.sortedBy { it.order }.forEachIndexed { index, button -> - val resolvedUri = resolvePluginButtonUri(pluginId, button.uri, button.directPath) - insertTooltipButton(db, tooltipId, button.description, resolvedUri, index) - } - } + val disclaimer = context.getString(R.string.plugin_documentation_third_party_disclaimer) + writePluginTooltips(db, pluginCategory(pluginId), entries.map { tooltipRow(pluginId, it, disclaimer) }) db.setTransactionSuccessful() Log.d(TAG, "Successfully installed documentation for plugin $pluginId") @@ -158,7 +149,7 @@ class PluginDocumentationManager( db.beginTransaction() try { - removePluginDocumentationInternal(db, pluginId) + removePluginTooltips(db, pluginCategory(pluginId)) db.setTransactionSuccessful() Log.d(TAG, "Successfully removed documentation for plugin $pluginId") true @@ -566,110 +557,6 @@ class PluginDocumentationManager( db.insertOrThrow("Content", null, values) } - private fun removePluginDocumentationInternal( - db: SQLiteDatabase, - pluginId: String, - ) { - val category = pluginCategory(pluginId) - - val cursor = - db.rawQuery( - """ - SELECT T.id FROM Tooltips AS T - INNER JOIN TooltipCategories AS TC ON T.categoryId = TC.id - WHERE TC.category = ? - """.trimIndent(), - arrayOf(category), - ) - - val tooltipIds = mutableListOf() - while (cursor.moveToNext()) { - tooltipIds.add(cursor.getLong(0)) - } - cursor.close() - - if (tooltipIds.isNotEmpty()) { - val placeholders = tooltipIds.joinToString(",") { "?" } - val args = tooltipIds.map { it.toString() }.toTypedArray() - db.delete("TooltipButtons", "tooltipId IN ($placeholders)", args) - db.delete("Tooltips", "id IN ($placeholders)", args) - } - - db.delete("TooltipCategories", "category = ?", arrayOf(category)) - } - - private fun insertOrGetCategoryId( - db: SQLiteDatabase, - category: String, - ): Long { - val cursor = - db.query( - "TooltipCategories", - arrayOf("id"), - "category = ?", - arrayOf(category), - null, - null, - null, - ) - - if (cursor.moveToFirst()) { - val id = cursor.getLong(0) - cursor.close() - return id - } - cursor.close() - - val values = - ContentValues().apply { - put("category", category) - } - return db.insert("TooltipCategories", null, values) - } - - private fun insertTooltip( - db: SQLiteDatabase, - categoryId: Long, - entry: PluginTooltipEntry, - ): Long { - val disclaimer = context.getString(R.string.plugin_documentation_third_party_disclaimer) - - val existingCursor = - db.query( - "Tooltips", - arrayOf("id"), - "categoryId = ? AND tag = ?", - arrayOf(categoryId.toString(), entry.tag), - null, - null, - null, - ) - - if (existingCursor.moveToFirst()) { - val existingId = existingCursor.getLong(0) - existingCursor.close() - - val updateValues = - ContentValues().apply { - put("summary", entry.summary + disclaimer) - put("detail", if (entry.detail.isNotBlank()) entry.detail + disclaimer else "") - } - db.update("Tooltips", updateValues, "id = ?", arrayOf(existingId.toString())) - db.delete("TooltipButtons", "tooltipId = ?", arrayOf(existingId.toString())) - return existingId - } - existingCursor.close() - - val values = - ContentValues().apply { - put("categoryId", categoryId) - put("tag", entry.tag) - put("summary", entry.summary + disclaimer) - put("detail", if (entry.detail.isNotBlank()) entry.detail + disclaimer else "") - } - return db.insert("Tooltips", null, values) - } - private fun escapeLike(value: String): String = value .replace("\\", "\\\\") @@ -684,6 +571,21 @@ class PluginDocumentationManager( return segments.joinToString("/") } + private fun tooltipRow( + pluginId: String, + entry: PluginTooltipEntry, + disclaimer: String, + ): PluginTooltipRow = + PluginTooltipRow( + tag = entry.tag, + summary = entry.summary + disclaimer, + detail = if (entry.detail.isNotBlank()) entry.detail + disclaimer else "", + buttons = + entry.buttons.sortedBy { it.order }.map { button -> + PluginTooltipButtonRow(button.description, resolvePluginButtonUri(pluginId, button.uri, button.directPath)) + }, + ) + private fun resolvePluginButtonUri( pluginId: String, rawUri: String, @@ -696,23 +598,6 @@ class PluginDocumentationManager( return if (absolute) normalized else "plugin/$pluginId/$normalized" } - private fun insertTooltipButton( - db: SQLiteDatabase, - tooltipId: Long, - description: String, - uri: String, - order: Int, - ) { - val values = - ContentValues().apply { - put("tooltipId", tooltipId) - put("description", description) - put("uri", uri) - put("buttonNumberId", order) - } - db.insert("TooltipButtons", null, values) - } - /** * Check if the plugin documentation database is accessible. */ diff --git a/plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginTooltipTables.kt b/plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginTooltipTables.kt new file mode 100644 index 0000000000..5b9e3b2164 --- /dev/null +++ b/plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginTooltipTables.kt @@ -0,0 +1,158 @@ +package com.itsaky.androidide.plugins.manager.documentation + +import android.content.ContentValues +import android.database.sqlite.SQLiteDatabase + +/** One plugin tooltip as it is written: summary and detail final, buttons in display order. */ +internal class PluginTooltipRow( + val tag: String, + val summary: String, + val detail: String, + val buttons: List, +) + +/** One button of a [PluginTooltipRow], its uri already resolved. */ +internal class PluginTooltipButtonRow( + val description: String, + val uri: String, +) + +/** + * Replaces every tooltip in [category] with [tooltips], inside the caller's transaction. + */ +internal fun writePluginTooltips( + db: SQLiteDatabase, + category: String, + tooltips: List, +) { + removePluginTooltips(db, category) + + val categoryId = insertOrGetCategoryId(db, category) + + for (tooltip in tooltips) { + val tooltipId = insertTooltip(db, categoryId, tooltip) + tooltip.buttons.forEachIndexed { index, button -> + insertTooltipButton(db, tooltipId, button.description, button.uri, index) + } + } +} + +/** + * Removes [category] and every tooltip in it, inside the caller's transaction. + */ +internal fun removePluginTooltips( + db: SQLiteDatabase, + category: String, +) { + val cursor = + db.rawQuery( + """ + SELECT T.id FROM Tooltips AS T + INNER JOIN TooltipCategories AS TC ON T.categoryId = TC.id + WHERE TC.category = ? + """.trimIndent(), + arrayOf(category), + ) + + val tooltipIds = mutableListOf() + while (cursor.moveToNext()) { + tooltipIds.add(cursor.getLong(0)) + } + cursor.close() + + if (tooltipIds.isNotEmpty()) { + val placeholders = tooltipIds.joinToString(",") { "?" } + val args = tooltipIds.map { it.toString() }.toTypedArray() + db.delete("TooltipButtons", "tooltipId IN ($placeholders)", args) + db.delete("Tooltips", "id IN ($placeholders)", args) + } + + db.delete("TooltipCategories", "category = ?", arrayOf(category)) +} + +private fun insertOrGetCategoryId( + db: SQLiteDatabase, + category: String, +): Long { + val cursor = + db.query( + "TooltipCategories", + arrayOf("id"), + "category = ?", + arrayOf(category), + null, + null, + null, + ) + + if (cursor.moveToFirst()) { + val id = cursor.getLong(0) + cursor.close() + return id + } + cursor.close() + + val values = + ContentValues().apply { + put("category", category) + } + return db.insert("TooltipCategories", null, values) +} + +private fun insertTooltip( + db: SQLiteDatabase, + categoryId: Long, + tooltip: PluginTooltipRow, +): Long { + val existingCursor = + db.query( + "Tooltips", + arrayOf("id"), + "categoryId = ? AND tag = ?", + arrayOf(categoryId.toString(), tooltip.tag), + null, + null, + null, + ) + + if (existingCursor.moveToFirst()) { + val existingId = existingCursor.getLong(0) + existingCursor.close() + + val updateValues = + ContentValues().apply { + put("summary", tooltip.summary) + put("detail", tooltip.detail) + } + db.update("Tooltips", updateValues, "id = ?", arrayOf(existingId.toString())) + db.delete("TooltipButtons", "tooltipId = ?", arrayOf(existingId.toString())) + return existingId + } + existingCursor.close() + + val values = + ContentValues().apply { + put("categoryId", categoryId) + put("tag", tooltip.tag) + put("summary", tooltip.summary) + put("detail", tooltip.detail) + } + return db.insert("Tooltips", null, values) +} + +private fun insertTooltipButton( + db: SQLiteDatabase, + tooltipId: Long, + description: String, + uri: String, + order: Int, +) { + val values = + ContentValues().apply { + put("tooltipId", tooltipId) + put("description", description) + put("uri", uri) + put("buttonNumberId", order) + } + db.insert("TooltipButtons", null, values) +} From e480d2ac43bf174478b4b0f87ffa119aeda412b2 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Sun, 18 Oct 2026 13:21:04 -0700 Subject: [PATCH 04/13] ADFA-3305: Read tooltips from documentation version 3's keyed tables documentation.db version 3 (OfflineDocumentationTools' migrate_tooltip_schema.py) stores the same tooltips in 5.2 MB less: - Tooltips is a WITHOUT ROWID table keyed on (categoryId, tag) holding only the id, so the separate UNIQUE (categoryId, tag) index, a second copy of every tag, is gone. The text is in TooltipTexts, by id. - TooltipButtons is keyed on (tooltipId, buttonNumberId), so a tooltip's buttons are one key range in display order instead of a scan and sort of the whole table. Each distinct description and uri is stored once, in TooltipButtonDescriptions and TooltipButtonUris. queryTooltip joins TooltipTexts by id, and the new queryTooltipButtons joins the two lookup tables. Tooltips has no rowid any more; it always equalled id, so IDETooltipItem.rowId, which the debug dialog and the feedback email show, now carries the id. Both readers refuse a database declaring a version below 3, through DatabaseVersionResolver.requireKeyedTooltips, with a message naming the version, rather than fail on whichever old column a query meets. Tests run both queries against the version 3 DDL, including their query plans (key searches, no scan, no sort), and the version check for versions 2, 4 and none. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3 --- .../utils/DatabaseVersionResolver.kt | 20 +++ docs/documentation-database.md | 39 ++++-- .../androidide/idetooltips/ToolTipManager.kt | 30 ++-- .../androidide/idetooltips/TooltipQuery.kt | 39 +++++- .../idetooltips/TooltipQueryTest.kt | 129 +++++++++++++++--- 5 files changed, 199 insertions(+), 58 deletions(-) diff --git a/common/src/main/java/com/itsaky/androidide/utils/DatabaseVersionResolver.kt b/common/src/main/java/com/itsaky/androidide/utils/DatabaseVersionResolver.kt index 6dc0439354..46f7d908ab 100644 --- a/common/src/main/java/com/itsaky/androidide/utils/DatabaseVersionResolver.kt +++ b/common/src/main/java/com/itsaky/androidide/utils/DatabaseVersionResolver.kt @@ -21,6 +21,12 @@ object DatabaseVersionResolver { // rows are plain Brotli. const val MAJOR_VERSION_WITH_COMPRESSION_DICTIONARY = 2 + // From this MAJOR version on (ADFA-3305), Tooltips is a WITHOUT ROWID table keyed by + // (categoryId, tag) whose text is in TooltipTexts, and TooltipButtons is keyed by (tooltipId, + // buttonNumberId) with its text in TooltipButtonDescriptions and TooltipButtonUris. Below it the + // tooltip tables have the earlier shape, which this build neither reads nor writes. + const val MAJOR_VERSION_WITH_KEYED_TOOLTIPS = 3 + private const val QUERY_VERSION_TABLE_EXISTS = """ SELECT 1 FROM sqlite_master @@ -135,6 +141,20 @@ object DatabaseVersionResolver { } } + /** + * Throws [IllegalStateException] unless [db] declares at least [MAJOR_VERSION_WITH_KEYED_TOOLTIPS], + * naming the version it does declare. Tooltip readers and writers call it first, so a database + * with the earlier tooltip tables fails with that reason rather than on a missing table or + * column partway through. + */ + fun requireKeyedTooltips(db: SQLiteDatabase) { + val major = resolveMajorVersion(db) + check(major != null && major >= MAJOR_VERSION_WITH_KEYED_TOOLTIPS) { + "documentation.db declares documentation version ${major ?: "none"}; its tooltip tables need " + + "$MAJOR_VERSION_WITH_KEYED_TOOLTIPS or later" + } + } + /** * Formats database change metadata into a readable version string. * diff --git a/docs/documentation-database.md b/docs/documentation-database.md index d85a0d0b20..56d7b4410b 100644 --- a/docs/documentation-database.md +++ b/docs/documentation-database.md @@ -42,27 +42,46 @@ Dimensions: `Languages(id, value)` (4-letter codes, e.g. `EN-us`); `ContentTypes ### Tier 1/2: `Tooltips` (the other fact table) +The shape below is documentation version 3 (ADFA-3305), written by `OfflineDocumentationTools`' `migrate_tooltip_schema.py`. Version 2 kept the text in `Tooltips` (an `AUTOINCREMENT` table with a separate `UNIQUE(categoryId, tag)` index) and `description`/`uri` in every `TooltipButtons` row; this build reads and writes neither, and refuses a database declaring less than 3 (`DatabaseVersionResolver.requireKeyedTooltips`). + ```sql -CREATE TABLE Tooltips ( +CREATE TABLE TooltipTexts ( id INTEGER PRIMARY KEY AUTOINCREMENT, - categoryId INTEGER NOT NULL, - tag TEXT NOT NULL, summary TEXT NOT NULL, - detail TEXT NOT NULL, - UNIQUE(categoryId, tag), - FOREIGN KEY(categoryId) REFERENCES TooltipCategories(id) + detail TEXT NOT NULL ); +CREATE TABLE Tooltips ( + id INTEGER NOT NULL, + categoryId INTEGER NOT NULL, + tag TEXT NOT NULL, + PRIMARY KEY (categoryId, tag), + FOREIGN KEY(categoryId) REFERENCES TooltipCategories(id), + FOREIGN KEY(id) REFERENCES TooltipTexts(id) +) WITHOUT ROWID; +CREATE TABLE TooltipButtons ( + tooltipId INTEGER NOT NULL, + buttonNumberId INTEGER NOT NULL, + descriptionId INTEGER NOT NULL, + uriId INTEGER NOT NULL, + PRIMARY KEY (tooltipId, buttonNumberId), + FOREIGN KEY(tooltipId) REFERENCES TooltipTexts(id), + FOREIGN KEY(buttonNumberId) REFERENCES TooltipButtonNumbers(id), + FOREIGN KEY(descriptionId) REFERENCES TooltipButtonDescriptions(id), + FOREIGN KEY(uriId) REFERENCES TooltipButtonUris(id) +) WITHOUT ROWID; +-- plus TooltipButtonDescriptions(id INTEGER PRIMARY KEY, description) and TooltipButtonUris(id INTEGER PRIMARY KEY, uri) ``` - `summary` is Tier 1 (the initial popup), `detail` is Tier 2 (after "See more"); both may contain HTML. -- Looked up by `(categoryId, tag)`, which the IDE stamps on the UI widget that owns the tooltip. The `UNIQUE` constraint gives this lookup an index, so it's fast. +- Looked up by `(categoryId, tag)`, which the IDE stamps on the UI widget that owns the tooltip. `Tooltips` is keyed on exactly that, so the lookup is a search of the table itself, and the text is one more lookup by id in `TooltipTexts`. The text lives apart because a `WITHOUT ROWID` table is an index B-tree: whole rows in its interior pages and any row over 1,002 bytes (at 4 KB pages) spilled to a mostly empty overflow page, which cost 1.18 MB on the 2026-10-18 database. +- A new tooltip takes its id from `TooltipTexts` (`AUTOINCREMENT`, so never one a deleted tooltip had); `Tooltips` has no rowid, so an insert into it returns none. - `TooltipCategories(id, category)` is a tiny dimension table (four categories at the time of writing). -- `TooltipButtons(tooltipId, buttonNumberId, description, uri)` holds the Tier 3 links shown at the bottom of a Tier 2 tooltip — `tooltipId` -> `Tooltips.id`, `buttonNumberId` -> `TooltipButtonNumbers.id`. `uri` should resolve to a `Content.path` (after stripping `?query`/`#fragment`). +- `TooltipButtons` holds the Tier 3 links shown at the bottom of a Tier 2 tooltip, keyed on `(tooltipId, buttonNumberId)`, so a tooltip's buttons are one range of the key, already in display order. Each distinct description and uri is stored once, in `TooltipButtonDescriptions` and `TooltipButtonUris` ("View full documentation" alone used to be stored 43,703 times). A uri should resolve to a `Content.path` (after stripping `?query`/`#fragment`). Neither lookup table has a `UNIQUE` index, which would cost about as much again; writers look a string up before inserting it. - `TooltipButtonNumbers(id)` exists only to pin a fixed, manually-assigned display order when a tooltip has multiple Tier 3 links. (Flagged in the source design doc as something worth redoing without a whole extra table.) ### Supporting tables -- **`DocumentationDatabaseVersion(major, minor, patch, who, comment, changeTime)`** — the database's own semver (ADFA-5220), replacing the heuristics that used to infer the format from which tables happened to exist. **Exactly one row**: the version the file *is*, which `populate_db.py` replaces rather than appends. Nothing in the DDL enforces that, so both readers (`DatabaseVersionResolver.resolveMajorVersion` here, `database_major_version` in docdb-studio) take the newest row rather than the one with the highest `major` (the app orders by `changeTime DESC, rowid DESC`, so the greatest `changeTime` wins and `rowid` only breaks ties) — a rebuild from an older content set is a downgrade and has to read as one, which `MAX(major)` would get wrong. The app's reader additionally logs a warning when it finds more than one row; docdb-studio's does not. `populate_db.py` collapses a file that somehow accumulated several back to one, and `resolveMajorVersion` returns null for a database predating the table. `MAJOR >= 2` is what tells the app its brotli `Content` rows are dictionary-compressed; below that, `DocumentationContentSource` neither reads nor attaches `CompressionDictionary`. Gating on the declared version rather than on the table's presence matters in both directions: a database can carry the dictionary table while its content is still plain brotli (every brotli row would then be decoded with a dictionary attached and fail outright, since ADFA-5240 removed the plain-decode retry that used to mask this), and a migrated database that lost the table fails loudly instead of quietly. +- **`DocumentationDatabaseVersion(major, minor, patch, who, comment, changeTime)`** — the database's own semver (ADFA-5220), replacing the heuristics that used to infer the format from which tables happened to exist. **Exactly one row**: the version the file *is*, which `populate_db.py` replaces rather than appends. Nothing in the DDL enforces that, so both readers (`DatabaseVersionResolver.resolveMajorVersion` here, `database_major_version` in docdb-studio) take the newest row rather than the one with the highest `major` (the app orders by `changeTime DESC, rowid DESC`, so the greatest `changeTime` wins and `rowid` only breaks ties) — a rebuild from an older content set is a downgrade and has to read as one, which `MAX(major)` would get wrong. The app's reader additionally logs a warning when it finds more than one row; docdb-studio's does not. `populate_db.py` collapses a file that somehow accumulated several back to one, and `resolveMajorVersion` returns null for a database predating the table. `MAJOR >= 3` says the tooltip tables have the shape above. `MAJOR >= 2` is what tells the app its brotli `Content` rows are dictionary-compressed; below that, `DocumentationContentSource` neither reads nor attaches `CompressionDictionary`. Gating on the declared version rather than on the table's presence matters in both directions: a database can carry the dictionary table while its content is still plain brotli (every brotli row would then be decoded with a dictionary attached and fail outright, since ADFA-5240 removed the plain-decode retry that used to mask this), and a migrated database that lost the table fails loudly instead of quietly. - **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. Both the reader and the writer load it through `loadCompressionDictionary` in `:common`, so they cannot disagree about whether a given database's brotli rows carry a dictionary — a row compressed against one the reader won't attach is undecodable, and vice versa. `DocumentationContentSource` loads it lazily -- not at database open or swap time, but on the first content lookup after the active database changes, whatever that row's own compression, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Each row is then decoded exactly once, with the dictionary attached or not according to that version; there is no retry, because there is no longer a second way a row might have been written. - **`Templates(id, name, content)`** — Pebble template source. The DDL is `id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT NOT NULL, content BLOB NOT NULL, UNIQUE('name')` — worth stating, because the single-quoted `UNIQUE('name')` reads like a constant rather than a column and has twice been taken for an absent constraint. SQLite resolves the quoted string to the column, so a duplicate name, a null name and a null body are all rejected by the database, and readers here do not re-check them. `Content.templateId` names a page's outermost template by id; every reference *between* templates is by `name` (ADFA-5405), which is also how `WebServer` reaches well-known templates like `bookshelf`. So a template can `extends`, `include`, `import` or `embed` another one by writing its `Templates.name` verbatim (`{% include "nav.peb" %}`) — the table is a flat namespace, with no directories, prefixes or suffixes applied. A reference to a name with no row fails the request rather than rendering the name; before ADFA-5405 the latter is exactly what happened, silently, because the engine's loader treated the name it was handed as the template body. - **`Bookshelf(contentID, bookCategoryID, title, description)`** / **`BookCategories(id, category, description)`** — the Dynamic Bookshelf: one row per "book" (PDF or similar), linked to its Tier 3 page via `contentID` -> `Content.id`. Two DB triggers keep `Bookshelf` in sync when a PDF row is inserted/deleted from `Content`; `title`/`description` don't come from those triggers and must be set by hand. Non-PDF books need a separate ingestion path (plugin-provided, e.g. via `PluginDocumentationManager`). @@ -76,7 +95,7 @@ Of the five sites below, only `DocumentationContentSource` and `ToolTipManager` - **`common/.../documentation/DocumentationContentSource.kt`** — the one Tier 3 pipeline that reads this database: row lookup (a request target is matched raw first, since stored `Content.path` values are percent-encoded, with a percent-decoded fallback on a miss — shared by both transports so they agree on which pages exist), chunked-row reassembly, dictionary-aware Brotli decode gated on the declared documentation version, and the database swaps — to a newer sdcard debug database, and reopening the installed file when an asset install rewrites it in place under the cached handle — under a read/write lock so several threads can read while a swap cannot close the handle under them. It also renders the rows that are Pebble template contexts (`templateId > 0`, the Kotlin doc set's pages) — resolving each template's references to other templates against `Templates.name` as it goes — so both transports below serve finished pages and neither needs the template engine itself. All of that logic exists once (ADFA-5176). - **`common/.../documentation/DocumentationRequestInterceptor.kt`** — serves Tier 3 *in-process* for the app's WebViews (`HelpActivity`, the tooltip fragment, `FAQActivity`), through `WebViewClient.shouldInterceptRequest`, so a page's assets cost a database read instead of a TCP connection each (ADFA-5176). It matches the same `http://localhost:6174/...` URL space, so the strings.xml entries, `ToolTipManager`'s link builder and the `DocumentationExtension` contract need no changes; anything it declines — a `/pr/` endpoint, an unknown path, a failed read — falls through to `WebServer` unchanged. Both transports get their `Content-Type` charset from the same place, `ContentTypeHeaders` (ADFA-5241), so a row does not describe itself differently depending on which one served it. Set `/sdcard/Download/CodeOnTheGo.nointercept` to force documentation back onto the server. - **`app/.../localWebServer/WebServer.kt`** — serves Tier 3 over HTTP on port 6174, for WebViews that are not wired to the interceptor above and for the `/pr/` developer endpoints. It reads through `DocumentationContentSource`, so it holds no database, template engine or decode logic of its own; what remains here is HTTP: request parsing, the `/pr/` pages, error responses, and the CSS/asset shortcuts. Also serves a Dynamic Bookshelf JSON payload (joining `Content`/`Bookshelf`/`BookCategories`, rendered through the `bookshelf` template) and debug-only HTML dumps at `/pr/db` (`LastChange`, last 20 rows) and `/pr/pr` (recent projects, from a *different* database). -- **`idetooltips/.../ToolTipManager.kt`** — serves Tier 1/2. Looks up a tooltip by `(category, tag)` through `queryTooltip` in `TooltipQuery.kt`: the category's id from `TooltipCategories` first, so the row is found through `Tooltips`' `UNIQUE (categoryId, tag)` index rather than a scan of the whole table (ADFA-6597). Then `TooltipButtons` for the Tier 3 links shown at the bottom, which still scans that table: it has no index on `tooltipId`. +- **`idetooltips/.../ToolTipManager.kt`** — serves Tier 1/2, through `queryTooltip` and `queryTooltipButtons` in `TooltipQuery.kt`. Looks up a tooltip by `(category, tag)`: the category's id from `TooltipCategories` first, so the row is found by `Tooltips`' key rather than a scan of the whole table (ADFA-6597), then its text by id. Then its buttons for the Tier 3 links shown at the bottom, by `TooltipButtons`' key, with their text by id. Refuses a database declaring a version below 3. - **`plugin-manager/.../documentation/PluginDocumentationManager.kt`** (with `Tier3AssetWalker.kt`, and the `DocumentationExtension` contract in `plugin-api`) — lets plugins contribute their own help content into the same lookup paths, under the reserved `plugin//...` prefix. Brotli assets are compressed against the same `CompressionDictionary` the rest of the table uses, read from the database being written. When that dictionary can't be read the install is abandoned rather than written plain, and retried on the plugin's next activation. ## Editing the database diff --git a/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt index 9bf8dd6259..173b3ba82f 100644 --- a/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt +++ b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt @@ -53,13 +53,6 @@ object TooltipManager { private val debugDatabaseFile: File = File(android.os.Environment.getExternalStorageDirectory().toString() + "/Download/documentation.db") - private const val QUERY_TOOLTIP_BUTTONS = """ - SELECT description, uri - FROM TooltipButtons - WHERE tooltipId = ? - ORDER BY buttonNumberId - """ - suspend fun getTooltip(context: Context, category: String, tag: String): IDETooltipItem? { Log.d(TAG, "In getTooltip() for category='$category', tag='$tag'.") @@ -100,24 +93,19 @@ object TooltipManager { } c.moveToFirst() - val rowId = c.getInt(0) - val tooltipId = c.getInt(1) - val summary = c.getString(2) - val detail = c.getString(3) + val tooltipId = c.getInt(0) + val summary = c.getString(1) + val detail = c.getString(2) val buttons = ArrayList>() - database.rawQuery(QUERY_TOOLTIP_BUTTONS, arrayOf(tooltipId.toString())).use { bc -> - while (bc.moveToNext()) { - buttons.add( - Pair( - bc.getString(0), - "http://localhost:6174/" + bc.getString(1) - ) - ) - } + for ((description, uri) in queryTooltipButtons(database, tooltipId)) { + buttons.add(Pair(description, "http://localhost:6174/$uri")) } - IDETooltipItem(rowId, tooltipId, category, tag, summary, detail, buttons, lastChange) + // Tooltips has no rowid from documentation version 3 on. It always equalled + // id before (id was the INTEGER PRIMARY KEY), so the debug dialog's Row is + // still the id. + IDETooltipItem(tooltipId, tooltipId, category, tag, summary, detail, buttons, lastChange) } } } catch (e: NoTooltipFoundException) { diff --git a/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/TooltipQuery.kt b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/TooltipQuery.kt index 71354eedc0..99a1542cf2 100644 --- a/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/TooltipQuery.kt +++ b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/TooltipQuery.kt @@ -3,23 +3,52 @@ package com.itsaky.androidide.idetooltips import android.database.Cursor import android.database.sqlite.SQLiteDatabase import androidx.annotation.VisibleForTesting +import com.itsaky.androidide.utils.DatabaseVersionResolver -// Looks the category up first, so the tag is found through Tooltips' UNIQUE (categoryId, tag) -// index. Written as a join, it made SQLite's planner scan all of Tooltips (46,128 rows, 13 MB) -// for every tooltip shown, because documentation.db carries no ANALYZE statistics (ADFA-6597). +// From documentation version 3 on (ADFA-3305), Tooltips is keyed by (categoryId, tag) and holds +// only the id; the text is in TooltipTexts, one row per id. Looks the category up first, so the +// tag is found by that key. Written as a join, it made SQLite's planner scan all of Tooltips for +// every tooltip shown, because documentation.db carries no ANALYZE statistics (ADFA-6597). // IN rather than =, so two categories sharing a name still come back as duplicates, as before. @VisibleForTesting internal const val QUERY_TOOLTIP = """ - SELECT T.rowid, T.id, T.summary, T.detail + SELECT T.id, X.summary, X.detail FROM Tooltips AS T + JOIN TooltipTexts AS X ON X.id = T.id WHERE T.categoryId IN (SELECT id FROM TooltipCategories WHERE category = ?) AND T.tag = ? """ +// TooltipButtons is keyed by (tooltipId, buttonNumberId), so a tooltip's buttons are one range of +// that key, already in display order. Each distinct description and uri is stored once. +@VisibleForTesting +internal const val QUERY_TOOLTIP_BUTTONS = """ + SELECT D.description, U.uri + FROM TooltipButtons AS B + JOIN TooltipButtonDescriptions AS D ON D.id = B.descriptionId + JOIN TooltipButtonUris AS U ON U.id = B.uriId + WHERE B.tooltipId = ? + ORDER BY B.buttonNumberId +""" + // Outside TooltipManager, so tests can run it without initializing the object, which reads the // installed app's Environment. Binds the arguments in the order QUERY_TOOLTIP expects them. +// Refuses a database older than version 3 first, rather than fail on whichever old table or +// column the query meets. internal fun queryTooltip( database: SQLiteDatabase, category: String, tag: String, -): Cursor = database.rawQuery(QUERY_TOOLTIP, arrayOf(category, tag)) +): Cursor { + DatabaseVersionResolver.requireKeyedTooltips(database) + return database.rawQuery(QUERY_TOOLTIP, arrayOf(category, tag)) +} + +/** The (description, uri) of each of [tooltipId]'s buttons, in display order. */ +internal fun queryTooltipButtons( + database: SQLiteDatabase, + tooltipId: Int, +): List> = + database.rawQuery(QUERY_TOOLTIP_BUTTONS, arrayOf(tooltipId.toString())).use { c -> + buildList { while (c.moveToNext()) add(Pair(c.getString(0), c.getString(1))) } + } diff --git a/idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt b/idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt index 6af7b78ba1..fb15914b09 100644 --- a/idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt +++ b/idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt @@ -4,6 +4,7 @@ import android.database.sqlite.SQLiteDatabase import org.junit.After import org.junit.Assert.assertEquals import org.junit.Assert.assertFalse +import org.junit.Assert.assertThrows import org.junit.Assert.assertTrue import org.junit.Before import org.junit.Test @@ -12,9 +13,11 @@ import org.robolectric.RobolectricTestRunner import org.robolectric.annotation.Config /** - * Runs [queryTooltip], the lookup TooltipManager.getTooltip makes, against documentation.db's own - * Tooltips schema. The query plan matters as much as the rows: the join it replaced scanned the - * whole 13 MB table for every tooltip shown (ADFA-6597). + * Runs [queryTooltip] and [queryTooltipButtons], the lookups TooltipManager.getTooltip makes, + * against documentation.db's version 3 tooltip schema (ADFA-3305), as migrate_tooltip_schema.py in + * OfflineDocumentationTools writes it. The query plans matter as much as the rows: the join + * QUERY_TOOLTIP replaced scanned the whole 13 MB table for every tooltip shown (ADFA-6597), and the + * buttons query used to scan and sort all of TooltipButtons. */ @RunWith(RobolectricTestRunner::class) @Config(manifest = Config.NONE, sdk = [28]) @@ -24,22 +27,23 @@ class TooltipQueryTest { @Before fun setUp() { db = SQLiteDatabase.create(null) - db.execSQL("CREATE TABLE TooltipCategories (id INTEGER PRIMARY KEY, category TEXT NOT NULL)") - db.execSQL( - "CREATE TABLE Tooltips (id INTEGER PRIMARY KEY AUTOINCREMENT, categoryId INTEGER NOT NULL, " + - "tag TEXT NOT NULL, summary TEXT NOT NULL, detail TEXT NOT NULL, UNIQUE (categoryId, tag), " + - "FOREIGN KEY(categoryId) REFERENCES TooltipCategories(id))", - ) + for (statement in SCHEMA) db.execSQL(statement) + setVersion(3) listOf("ide", "xml", "java", "kotlin").forEachIndexed { i, category -> db.execSQL("INSERT INTO TooltipCategories VALUES (?, ?)", arrayOf(i + 1, category)) } + db.execSQL("INSERT INTO TooltipButtonDescriptions VALUES (1, 'View full documentation'), (2, 'Examples')") db.beginTransaction() + var id = 1000 for (categoryId in 1..4) { for (n in 0 until 500) { - db.execSQL( - "INSERT INTO Tooltips (categoryId, tag, summary, detail) VALUES (?, ?, ?, ?)", - arrayOf(categoryId, "tag$n", "summary $categoryId/$n", "detail"), - ) + id++ + db.execSQL("INSERT INTO TooltipTexts VALUES (?, ?, ?)", arrayOf(id, "summary $categoryId/$n", "detail")) + db.execSQL("INSERT INTO Tooltips VALUES (?, ?, ?)", arrayOf(id, categoryId, "tag$n")) + db.execSQL("INSERT INTO TooltipButtonUris VALUES (?, ?)", arrayOf(id, "page$id.html")) + // Button 2 first, so the order has to come from the key, not from insertion. + db.execSQL("INSERT INTO TooltipButtons VALUES (?, 2, 2, ?)", arrayOf(id, id)) + db.execSQL("INSERT INTO TooltipButtons VALUES (?, 1, 1, ?)", arrayOf(id, id)) } } db.setTransactionSuccessful() @@ -51,12 +55,30 @@ class TooltipQueryTest { db.close() } + private fun setVersion(major: Int) { + db.execSQL("DELETE FROM DocumentationDatabaseVersion") + db.execSQL( + "INSERT INTO DocumentationDatabaseVersion (major, minor, patch, who, comment) VALUES (?, 0, 0, 'test', 'test')", + arrayOf(major), + ) + } + + private fun plan( + query: String, + vararg args: String, + ): List = + db.rawQuery("EXPLAIN QUERY PLAN $query", args).use { c -> + buildList { while (c.moveToNext()) add(c.getString(3)) } + } + @Test fun `finds the one tooltip for a category and tag`() { queryTooltip(db, category = "java", tag = "tag7").use { c -> assertEquals(1, c.count) c.moveToFirst() - assertEquals("summary 3/7", c.getString(2)) + assertEquals(1000 + 2 * 500 + 8, c.getInt(0)) + assertEquals("summary 3/7", c.getString(1)) + assertEquals("detail", c.getString(2)) } queryTooltip(db, category = "java", tag = "no-such-tag").use { assertEquals(0, it.count) } queryTooltip(db, category = "no-such-category", tag = "tag7").use { assertEquals(0, it.count) } @@ -65,19 +87,82 @@ class TooltipQueryTest { @Test fun `two categories with the same name still come back as two rows`() { db.execSQL("INSERT INTO TooltipCategories VALUES (5, 'java')") - db.execSQL("INSERT INTO Tooltips (categoryId, tag, summary, detail) VALUES (5, 'tag7', 'duplicate', 'detail')") + db.execSQL("INSERT INTO TooltipTexts VALUES (9999, 'duplicate', 'detail')") + db.execSQL("INSERT INTO Tooltips VALUES (9999, 5, 'tag7')") queryTooltip(db, category = "java", tag = "tag7").use { assertEquals(2, it.count) } } @Test - fun `finds the tag through the (categoryId, tag) index, not a scan of Tooltips`() { - val plan = - db.rawQuery("EXPLAIN QUERY PLAN " + QUERY_TOOLTIP, arrayOf("java", "tag7")).use { c -> - buildList { while (c.moveToNext()) add(c.getString(3)) } - } + fun `finds the tag by Tooltips' key and the text by TooltipTexts' id, scanning neither`() { + val plan = plan(QUERY_TOOLTIP, "java", "tag7") + + assertTrue(plan.toString(), plan.any { Regex("""\bUSING PRIMARY KEY \(categoryId=\? AND tag=\?\)""").containsMatchIn(it) }) + assertTrue(plan.toString(), plan.any { Regex("""\b(X|TooltipTexts)\b.*USING INTEGER PRIMARY KEY""").containsMatchIn(it) }) + assertFalse(plan.toString(), plan.any { Regex("""\bSCAN (TABLE )?(T|X|Tooltips|TooltipTexts)\b""").containsMatchIn(it) }) + } + + @Test + fun `returns a tooltip's buttons in button order with their text`() { + val id = 1000 + 2 * 500 + 8 + + assertEquals( + listOf("View full documentation" to "page$id.html", "Examples" to "page$id.html"), + queryTooltipButtons(db, id), + ) + assertEquals(emptyList>(), queryTooltipButtons(db, 1)) + } + + @Test + fun `finds the buttons by TooltipButtons' key, already in order, and their text by id`() { + val plan = plan(QUERY_TOOLTIP_BUTTONS, "1009") + + assertTrue(plan.toString(), plan.any { Regex("""\bUSING PRIMARY KEY \(tooltipId=\?\)""").containsMatchIn(it) }) + assertFalse(plan.toString(), plan.any { "TEMP B-TREE" in it }) + assertFalse(plan.toString(), plan.any { Regex("""\bSCAN (TABLE )?\w+""").containsMatchIn(it) }) + } + + @Test + fun `refuses a database that declares a version before 3`() { + setVersion(2) + + val e = assertThrows(IllegalStateException::class.java) { queryTooltip(db, category = "java", tag = "tag7") } + assertTrue(e.message, "version 2" in e.message.orEmpty()) + } + + @Test + fun `refuses a database that declares no version`() { + db.execSQL("DROP TABLE DocumentationDatabaseVersion") + + val e = assertThrows(IllegalStateException::class.java) { queryTooltip(db, category = "java", tag = "tag7") } + assertTrue(e.message, "version none" in e.message.orEmpty()) + } + + @Test + fun `reads a database that declares a later version`() { + setVersion(4) + + queryTooltip(db, category = "java", tag = "tag7").use { assertEquals(1, it.count) } + } - assertTrue(plan.toString(), plan.any { "sqlite_autoindex_Tooltips_1" in it }) - assertFalse(plan.toString(), plan.any { Regex("""\bSCAN (TABLE )?(T|Tooltips)\b""").containsMatchIn(it) }) + private companion object { + // documentation.db's version 3 tooltip tables, as migrate_tooltip_schema.py creates them. + val SCHEMA = + listOf( + "CREATE TABLE TooltipCategories (id INTEGER PRIMARY KEY, category TEXT NOT NULL)", + "CREATE TABLE TooltipTexts (id INTEGER PRIMARY KEY AUTOINCREMENT, summary TEXT NOT NULL, detail TEXT NOT NULL)", + "CREATE TABLE Tooltips (id INTEGER NOT NULL, categoryId INTEGER NOT NULL, tag TEXT NOT NULL, " + + "PRIMARY KEY (categoryId, tag), FOREIGN KEY(categoryId) REFERENCES TooltipCategories(id), " + + "FOREIGN KEY(id) REFERENCES TooltipTexts(id)) WITHOUT ROWID", + "CREATE TABLE TooltipButtonDescriptions (id INTEGER PRIMARY KEY, description TEXT NOT NULL)", + "CREATE TABLE TooltipButtonUris (id INTEGER PRIMARY KEY, uri TEXT NOT NULL)", + "CREATE TABLE TooltipButtons (tooltipId INTEGER NOT NULL, buttonNumberId INTEGER NOT NULL, " + + "descriptionId INTEGER NOT NULL, uriId INTEGER NOT NULL, PRIMARY KEY (tooltipId, buttonNumberId), " + + "FOREIGN KEY(tooltipId) REFERENCES TooltipTexts(id), " + + "FOREIGN KEY(descriptionId) REFERENCES TooltipButtonDescriptions(id), " + + "FOREIGN KEY(uriId) REFERENCES TooltipButtonUris(id)) WITHOUT ROWID", + "CREATE TABLE DocumentationDatabaseVersion (major INT NOT NULL, minor INT NOT NULL, patch INT NOT NULL, " + + "who TEXT NOT NULL, comment TEXT NOT NULL, changeTime TIMESTAMP DEFAULT CURRENT_TIMESTAMP)", + ) } } From 5cc8d2bc94f9c7ba6c06feffb4dbfa8160f7b98a Mon Sep 17 00:00:00 2001 From: David Schachter Date: Sun, 18 Oct 2026 13:21:18 -0700 Subject: [PATCH 05/13] ADFA-3305: Write plugin tooltips into documentation version 3's tables PluginTooltipTables.kt now writes the version 3 tooltip tables: - A tooltip's text goes into TooltipTexts first, whose AUTOINCREMENT id then keys the Tooltips row. The id is the one that insert returns: Tooltips is WITHOUT ROWID, and SQLite does not record its inserts as a rowid, so an insert into it returns the previous insert's rowid, not anything of its own. - Each button's description and uri are looked up in, or added to, TooltipButtonDescriptions and TooltipButtonUris, 500 strings per query, so text a built-in button already uses is shared. - A plugin's rows are deleted by category through subqueries, which also removes the old one-variable-per-tooltip IN list, and text no button names any more is deleted after every write and removal. - Every insert throws on failure. db.insert returned -1 instead, so a failed insert left a tooltip without its buttons, or buttons on tooltip -1, and the transaction committed anyway. - Both entry points refuse a database declaring a version below 3 before writing anything. Tests run against the version 3 DDL with one built-in tooltip: what a plugin writes reads back through the app's queries; new ids are never reused; a repeated tag keeps the later entry; text is shared and cleaned up; removing every plugin leaves the tables exactly as they were; 1,200 tooltips write and remove; a version 2 database is refused untouched; and a failed insert into any of the six tables throws and rolls back. plugin-manager's tests now use Robolectric, already in the version catalog, for SQLiteDatabase. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3 --- docs/documentation-database.md | 2 +- plugin-manager/build.gradle.kts | 1 + .../PluginDocumentationManager.kt | 2 +- .../documentation/PluginTooltipTables.kt | 165 ++++++---- .../documentation/PluginTooltipTablesTest.kt | 306 ++++++++++++++++++ 5 files changed, 407 insertions(+), 69 deletions(-) create mode 100644 plugin-manager/src/test/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginTooltipTablesTest.kt diff --git a/docs/documentation-database.md b/docs/documentation-database.md index 56d7b4410b..844c0fdb39 100644 --- a/docs/documentation-database.md +++ b/docs/documentation-database.md @@ -96,7 +96,7 @@ Of the five sites below, only `DocumentationContentSource` and `ToolTipManager` - **`common/.../documentation/DocumentationRequestInterceptor.kt`** — serves Tier 3 *in-process* for the app's WebViews (`HelpActivity`, the tooltip fragment, `FAQActivity`), through `WebViewClient.shouldInterceptRequest`, so a page's assets cost a database read instead of a TCP connection each (ADFA-5176). It matches the same `http://localhost:6174/...` URL space, so the strings.xml entries, `ToolTipManager`'s link builder and the `DocumentationExtension` contract need no changes; anything it declines — a `/pr/` endpoint, an unknown path, a failed read — falls through to `WebServer` unchanged. Both transports get their `Content-Type` charset from the same place, `ContentTypeHeaders` (ADFA-5241), so a row does not describe itself differently depending on which one served it. Set `/sdcard/Download/CodeOnTheGo.nointercept` to force documentation back onto the server. - **`app/.../localWebServer/WebServer.kt`** — serves Tier 3 over HTTP on port 6174, for WebViews that are not wired to the interceptor above and for the `/pr/` developer endpoints. It reads through `DocumentationContentSource`, so it holds no database, template engine or decode logic of its own; what remains here is HTTP: request parsing, the `/pr/` pages, error responses, and the CSS/asset shortcuts. Also serves a Dynamic Bookshelf JSON payload (joining `Content`/`Bookshelf`/`BookCategories`, rendered through the `bookshelf` template) and debug-only HTML dumps at `/pr/db` (`LastChange`, last 20 rows) and `/pr/pr` (recent projects, from a *different* database). - **`idetooltips/.../ToolTipManager.kt`** — serves Tier 1/2, through `queryTooltip` and `queryTooltipButtons` in `TooltipQuery.kt`. Looks up a tooltip by `(category, tag)`: the category's id from `TooltipCategories` first, so the row is found by `Tooltips`' key rather than a scan of the whole table (ADFA-6597), then its text by id. Then its buttons for the Tier 3 links shown at the bottom, by `TooltipButtons`' key, with their text by id. Refuses a database declaring a version below 3. -- **`plugin-manager/.../documentation/PluginDocumentationManager.kt`** (with `Tier3AssetWalker.kt`, and the `DocumentationExtension` contract in `plugin-api`) — lets plugins contribute their own help content into the same lookup paths, under the reserved `plugin//...` prefix. Brotli assets are compressed against the same `CompressionDictionary` the rest of the table uses, read from the database being written. When that dictionary can't be read the install is abandoned rather than written plain, and retried on the plugin's next activation. +- **`plugin-manager/.../documentation/PluginDocumentationManager.kt`** (with `Tier3AssetWalker.kt`, and the `DocumentationExtension` contract in `plugin-api`) — lets plugins contribute their own help content into the same lookup paths, under the reserved `plugin//...` prefix. Its tooltip SQL is in `PluginTooltipTables.kt`: a plugin's tooltips are replaced or removed by category inside one transaction, each button's text is looked up in or added to the lookup tables, and text no button uses any more is deleted afterwards. Brotli assets are compressed against the same `CompressionDictionary` the rest of the table uses, read from the database being written. When that dictionary can't be read the install is abandoned rather than written plain, and retried on the plugin's next activation. ## Editing the database diff --git a/plugin-manager/build.gradle.kts b/plugin-manager/build.gradle.kts index e75526fe2d..5e5bf01abb 100644 --- a/plugin-manager/build.gradle.kts +++ b/plugin-manager/build.gradle.kts @@ -39,4 +39,5 @@ dependencies { testImplementation(libs.tests.junit) testImplementation(libs.tests.google.truth) + testImplementation(libs.tests.robolectric) } diff --git a/plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginDocumentationManager.kt b/plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginDocumentationManager.kt index f13adaa3ee..4b42ae1a39 100644 --- a/plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginDocumentationManager.kt +++ b/plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginDocumentationManager.kt @@ -20,7 +20,7 @@ import kotlin.coroutines.cancellation.CancellationException /** * Manages plugin documentation by writing into the main documentation.db. - * Plugin entries are stored in the existing Tooltips/TooltipCategories/TooltipButtons tables, + * Plugin entries are stored in the existing tooltip tables (see PluginTooltipTables.kt), * differentiated by a "plugin_" category prefix so they never conflict with * built-in documentation. */ diff --git a/plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginTooltipTables.kt b/plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginTooltipTables.kt index 5b9e3b2164..877bbc696b 100644 --- a/plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginTooltipTables.kt +++ b/plugin-manager/src/main/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginTooltipTables.kt @@ -2,6 +2,7 @@ package com.itsaky.androidide.plugins.manager.documentation import android.content.ContentValues import android.database.sqlite.SQLiteDatabase +import com.itsaky.androidide.utils.DatabaseVersionResolver /** One plugin tooltip as it is written: summary and detail final, buttons in display order. */ internal class PluginTooltipRow( @@ -17,57 +18,106 @@ internal class PluginTooltipButtonRow( val uri: String, ) +// A subquery rather than a join, so Tooltips is searched by its (categoryId, tag) key. +private const val CATEGORY_IDS = "SELECT id FROM TooltipCategories WHERE category = ?" +private const val CATEGORY_TOOLTIP_IDS = "SELECT id FROM Tooltips WHERE categoryId IN ($CATEGORY_IDS)" + +// IN takes one variable per string; 500 stays under SQLite's pre-3.32 default limit of 999. +private const val LOOKUP_CHUNK = 500 + /** * Replaces every tooltip in [category] with [tooltips], inside the caller's transaction. + * + * Writes documentation version 3's tooltip tables (ADFA-3305): the key in Tooltips, the text in + * TooltipTexts, and each button's text as ids into TooltipButtonDescriptions and TooltipButtonUris. + * Refuses an older database before writing anything. Every insert throws on failure, so a failed + * one rolls the transaction back rather than commit a tooltip without its buttons. */ internal fun writePluginTooltips( db: SQLiteDatabase, category: String, tooltips: List, ) { - removePluginTooltips(db, category) + DatabaseVersionResolver.requireKeyedTooltips(db) + deleteCategoryTooltips(db, category) val categoryId = insertOrGetCategoryId(db, category) + val buttons = tooltips.flatMap { it.buttons } + val descriptionIds = textIds(db, "TooltipButtonDescriptions", "description", buttons.map { it.description }) + val uriIds = textIds(db, "TooltipButtonUris", "uri", buttons.map { it.uri }) for (tooltip in tooltips) { val tooltipId = insertTooltip(db, categoryId, tooltip) tooltip.buttons.forEachIndexed { index, button -> - insertTooltipButton(db, tooltipId, button.description, button.uri, index) + val values = + ContentValues().apply { + put("tooltipId", tooltipId) + put("buttonNumberId", index) + put("descriptionId", descriptionIds.getValue(button.description)) + put("uriId", uriIds.getValue(button.uri)) + } + db.insertOrThrow("TooltipButtons", null, values) } } + deleteUnusedButtonText(db) } /** - * Removes [category] and every tooltip in it, inside the caller's transaction. + * Removes [category] and every tooltip in it, inside the caller's transaction, with the button + * text no other button uses. Refuses a database older than documentation version 3. */ internal fun removePluginTooltips( db: SQLiteDatabase, category: String, ) { - val cursor = - db.rawQuery( - """ - SELECT T.id FROM Tooltips AS T - INNER JOIN TooltipCategories AS TC ON T.categoryId = TC.id - WHERE TC.category = ? - """.trimIndent(), - arrayOf(category), - ) + DatabaseVersionResolver.requireKeyedTooltips(db) + deleteCategoryTooltips(db, category) + deleteUnusedButtonText(db) +} - val tooltipIds = mutableListOf() - while (cursor.moveToNext()) { - tooltipIds.add(cursor.getLong(0)) - } - cursor.close() +private fun deleteCategoryTooltips( + db: SQLiteDatabase, + category: String, +) { + // Buttons and text first: both are found through the Tooltips rows deleted after them. + db.execSQL("DELETE FROM TooltipButtons WHERE tooltipId IN ($CATEGORY_TOOLTIP_IDS)", arrayOf(category)) + db.execSQL("DELETE FROM TooltipTexts WHERE id IN ($CATEGORY_TOOLTIP_IDS)", arrayOf(category)) + db.execSQL("DELETE FROM Tooltips WHERE categoryId IN ($CATEGORY_IDS)", arrayOf(category)) + db.delete("TooltipCategories", "category = ?", arrayOf(category)) +} - if (tooltipIds.isNotEmpty()) { - val placeholders = tooltipIds.joinToString(",") { "?" } - val args = tooltipIds.map { it.toString() }.toTypedArray() - db.delete("TooltipButtons", "tooltipId IN ($placeholders)", args) - db.delete("Tooltips", "id IN ($placeholders)", args) - } +// A description or uri no button names any more: one only a removed tooltip used. Text shared with +// a built-in button, or another plugin's, stays. +private fun deleteUnusedButtonText(db: SQLiteDatabase) { + db.execSQL("DELETE FROM TooltipButtonDescriptions WHERE id NOT IN (SELECT descriptionId FROM TooltipButtons)") + db.execSQL("DELETE FROM TooltipButtonUris WHERE id NOT IN (SELECT uriId FROM TooltipButtons)") +} - db.delete("TooltipCategories", "category = ?", arrayOf(category)) +/** + * The id of each of [values] in [table], inserting the ones it lacks. [column] has no index -- a + * UNIQUE one would cost about as much as the table -- so the strings are looked up in chunks, each + * one read of the table, rather than one read per string. + */ +private fun textIds( + db: SQLiteDatabase, + table: String, + column: String, + values: List, +): Map { + val distinct = values.distinct() + val ids = HashMap() + for (chunk in distinct.chunked(LOOKUP_CHUNK)) { + val placeholders = chunk.joinToString(",") { "?" } + db.rawQuery("SELECT $column, id FROM $table WHERE $column IN ($placeholders)", chunk.toTypedArray()).use { c -> + while (c.moveToNext()) ids.putIfAbsent(c.getString(0), c.getLong(1)) + } + } + for (value in distinct) { + if (value !in ids) { + ids[value] = db.insertOrThrow(table, null, ContentValues().apply { put(column, value) }) + } + } + return ids } private fun insertOrGetCategoryId( @@ -96,7 +146,7 @@ private fun insertOrGetCategoryId( ContentValues().apply { put("category", category) } - return db.insert("TooltipCategories", null, values) + return db.insertOrThrow("TooltipCategories", null, values) } private fun insertTooltip( @@ -104,55 +154,36 @@ private fun insertTooltip( categoryId: Long, tooltip: PluginTooltipRow, ): Long { - val existingCursor = - db.query( - "Tooltips", - arrayOf("id"), - "categoryId = ? AND tag = ?", - arrayOf(categoryId.toString(), tooltip.tag), - null, - null, - null, - ) - - if (existingCursor.moveToFirst()) { - val existingId = existingCursor.getLong(0) - existingCursor.close() + val existingId = + db + .rawQuery( + "SELECT id FROM Tooltips WHERE categoryId = ? AND tag = ?", + arrayOf(categoryId.toString(), tooltip.tag), + ).use { if (it.moveToFirst()) it.getLong(0) else null } + + val text = + ContentValues().apply { + put("summary", tooltip.summary) + put("detail", tooltip.detail) + } - val updateValues = - ContentValues().apply { - put("summary", tooltip.summary) - put("detail", tooltip.detail) - } - db.update("Tooltips", updateValues, "id = ?", arrayOf(existingId.toString())) + if (existingId != null) { + // Only a tag the plugin lists twice, since the category was just emptied; the later wins. + db.update("TooltipTexts", text, "id = ?", arrayOf(existingId.toString())) db.delete("TooltipButtons", "tooltipId = ?", arrayOf(existingId.toString())) return existingId } - existingCursor.close() - val values = + // TooltipTexts hands out the id: AUTOINCREMENT, so never one a deleted tooltip had. Tooltips is + // WITHOUT ROWID, and SQLite does not record its inserts as a rowid, so what insertOrThrow + // returns for it is some earlier insert's. + val id = db.insertOrThrow("TooltipTexts", null, text) + val key = ContentValues().apply { + put("id", id) put("categoryId", categoryId) put("tag", tooltip.tag) - put("summary", tooltip.summary) - put("detail", tooltip.detail) - } - return db.insert("Tooltips", null, values) -} - -private fun insertTooltipButton( - db: SQLiteDatabase, - tooltipId: Long, - description: String, - uri: String, - order: Int, -) { - val values = - ContentValues().apply { - put("tooltipId", tooltipId) - put("description", description) - put("uri", uri) - put("buttonNumberId", order) } - db.insert("TooltipButtons", null, values) + db.insertOrThrow("Tooltips", null, key) + return id } diff --git a/plugin-manager/src/test/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginTooltipTablesTest.kt b/plugin-manager/src/test/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginTooltipTablesTest.kt new file mode 100644 index 0000000000..07575e8d39 --- /dev/null +++ b/plugin-manager/src/test/kotlin/com/itsaky/androidide/plugins/manager/documentation/PluginTooltipTablesTest.kt @@ -0,0 +1,306 @@ +package com.itsaky.androidide.plugins.manager.documentation + +import android.database.SQLException +import android.database.sqlite.SQLiteDatabase +import org.junit.After +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNotEquals +import org.junit.Assert.assertThrows +import org.junit.Assert.assertTrue +import org.junit.Before +import org.junit.Test +import org.junit.runner.RunWith +import org.robolectric.RobolectricTestRunner +import org.robolectric.annotation.Config + +/** + * Runs [writePluginTooltips] and [removePluginTooltips] against documentation.db's version 3 + * tooltip schema (ADFA-3305), as migrate_tooltip_schema.py in OfflineDocumentationTools writes it, + * holding one built-in tooltip that plugins must never disturb. What a plugin writes is read back + * with the queries the app runs (idetooltips' TooltipQuery.kt). + */ +@RunWith(RobolectricTestRunner::class) +@Config(manifest = Config.NONE, sdk = [28]) +class PluginTooltipTablesTest { + private lateinit var db: SQLiteDatabase + + @Before + fun setUp() { + db = SQLiteDatabase.create(null) + for (statement in SCHEMA) db.execSQL(statement) + setVersion(3) + db.execSQL("INSERT INTO TooltipCategories VALUES (1, 'ide')") + db.execSQL("INSERT INTO TooltipTexts VALUES ($BUILT_IN_ID, 'built-in summary', 'built-in detail')") + db.execSQL("INSERT INTO Tooltips VALUES ($BUILT_IN_ID, 1, 'editor')") + db.execSQL("INSERT INTO TooltipButtonDescriptions VALUES (1, '$SHARED_DESCRIPTION')") + db.execSQL("INSERT INTO TooltipButtonUris VALUES (1, 'a/editor.html')") + db.execSQL("INSERT INTO TooltipButtons VALUES ($BUILT_IN_ID, 1, 1, 1)") + } + + @After + fun tearDown() { + db.close() + } + + private fun setVersion(major: Int) { + db.execSQL("DELETE FROM DocumentationDatabaseVersion") + db.execSQL( + "INSERT INTO DocumentationDatabaseVersion (major, minor, patch, who, comment) VALUES (?, 0, 0, 'test', 'test')", + arrayOf(major), + ) + } + + private fun inTransaction(block: () -> Unit) { + db.beginTransaction() + try { + block() + db.setTransactionSuccessful() + } finally { + db.endTransaction() + } + } + + private fun write( + category: String, + vararg tooltips: PluginTooltipRow, + ) = inTransaction { writePluginTooltips(db, category, tooltips.toList()) } + + private fun remove(category: String) = inTransaction { removePluginTooltips(db, category) } + + /** What the app shows for (category, tag): id, summary, detail and buttons, or null if nothing. */ + private fun shown( + category: String, + tag: String, + ): Shown? { + val tooltip = + db.rawQuery(APP_TOOLTIP_QUERY, arrayOf(category, tag)).use { c -> + assertTrue("more than one row for $category/$tag", c.count <= 1) + if (c.moveToFirst()) Triple(c.getLong(0), c.getString(1), c.getString(2)) else null + } ?: return null + val buttons = + db.rawQuery(APP_BUTTONS_QUERY, arrayOf(tooltip.first.toString())).use { c -> + buildList { while (c.moveToNext()) add(c.getString(0) to c.getString(1)) } + } + return Shown(tooltip.first, tooltip.second, tooltip.third, buttons) + } + + private data class Shown( + val id: Long, + val summary: String, + val detail: String, + val buttons: List>, + ) + + private fun count(table: String): Long = + db.rawQuery("SELECT COUNT(*) FROM $table", null).use { + it.moveToFirst() + it.getLong(0) + } + + private fun contents(): Map>> = + TABLES.associateWith { table -> + db.rawQuery("SELECT * FROM $table ORDER BY 1, 2", null).use { c -> + buildList { while (c.moveToNext()) add((0 until c.columnCount).map { c.getString(it) }) } + } + } + + private val builtIn = Shown(BUILT_IN_ID, "built-in summary", "built-in detail", listOf(SHARED_DESCRIPTION to "a/editor.html")) + + private fun row( + tag: String, + summary: String = "summary of $tag", + detail: String = "detail of $tag", + vararg buttons: Pair, + ) = PluginTooltipRow(tag, summary, detail, buttons.map { PluginTooltipButtonRow(it.first, it.second) }) + + @Test + fun `writes tooltips the app reads back with their buttons in order`() { + write( + "plugin_x", + row("x.one", buttons = arrayOf("Read more" to "plugin/x/index.html", "Overview" to "i/plugins-adfa.html")), + row("x.two", detail = ""), + ) + + val one = shown("plugin_x", "x.one")!! + assertEquals("summary of x.one", one.summary) + assertEquals("detail of x.one", one.detail) + assertEquals(listOf("Read more" to "plugin/x/index.html", "Overview" to "i/plugins-adfa.html"), one.buttons) + val two = shown("plugin_x", "x.two")!! + assertEquals("", two.detail) + assertEquals(emptyList>(), two.buttons) + assertNotEquals(one.id, two.id) + assertEquals(builtIn, shown("ide", "editor")) + } + + @Test + fun `gives each new tooltip an id no tooltip has had`() { + write("plugin_x", row("x.one"), row("x.two")) + val first = listOf(shown("plugin_x", "x.one")!!.id, shown("plugin_x", "x.two")!!.id) + remove("plugin_x") + write("plugin_x", row("x.one")) + + val again = shown("plugin_x", "x.one")!!.id + assertTrue("$first, then $again", first.all { it > BUILT_IN_ID } && again > first.max()) + } + + @Test + fun `keeps the later entry for a tag listed twice`() { + write( + "plugin_x", + row("x.one", summary = "first", buttons = arrayOf("First" to "plugin/x/1.html")), + row("x.one", summary = "second", buttons = arrayOf("Second" to "plugin/x/2.html")), + ) + + val one = shown("plugin_x", "x.one")!! + assertEquals("second", one.summary) + assertEquals(listOf("Second" to "plugin/x/2.html"), one.buttons) + assertEquals(2L, count("Tooltips")) + assertEquals(2L, count("TooltipTexts")) + // "First" and plugin/x/1.html were looked up for the first entry; nothing uses them now. + assertEquals(listOf("Second", SHARED_DESCRIPTION), column("TooltipButtonDescriptions", "description")) + assertEquals(listOf("a/editor.html", "plugin/x/2.html"), column("TooltipButtonUris", "uri")) + } + + private fun column( + table: String, + column: String, + ): List = + db.rawQuery("SELECT $column FROM $table ORDER BY $column", null).use { c -> + buildList { while (c.moveToNext()) add(c.getString(0)) } + } + + @Test + fun `stores each button text once, sharing what built-in buttons use`() { + write( + "plugin_x", + row("x.one", buttons = arrayOf(SHARED_DESCRIPTION to "a/editor.html", "Read more" to "plugin/x/a.html")), + row("x.two", buttons = arrayOf("Read more" to "plugin/x/a.html")), + ) + + assertEquals(listOf("Read more", SHARED_DESCRIPTION), column("TooltipButtonDescriptions", "description")) + assertEquals(listOf("a/editor.html", "plugin/x/a.html"), column("TooltipButtonUris", "uri")) + assertEquals(listOf(SHARED_DESCRIPTION to "a/editor.html", "Read more" to "plugin/x/a.html"), shown("plugin_x", "x.one")!!.buttons) + } + + @Test + fun `removing a plugin leaves the database as it was before the plugin wrote`() { + val before = contents() + write( + "plugin_x", + row("x.one", buttons = arrayOf(SHARED_DESCRIPTION to "plugin/x/a.html", "Read more" to "a/editor.html")), + ) + write("plugin_y", row("y.one", buttons = arrayOf("Read more" to "plugin/y/a.html"))) + remove("plugin_x") + + assertEquals(null, shown("plugin_x", "x.one")) + assertEquals(listOf("Read more" to "plugin/y/a.html"), shown("plugin_y", "y.one")!!.buttons) + remove("plugin_y") + + assertEquals(before, contents()) + assertEquals(builtIn, shown("ide", "editor")) + } + + @Test + fun `writing again replaces the plugin's tooltips rather than adding to them`() { + write("plugin_x", row("x.one", buttons = arrayOf("Read more" to "plugin/x/a.html")), row("x.two")) + write("plugin_x", row("x.one", summary = "new", buttons = arrayOf("Overview" to "plugin/x/b.html"))) + + assertEquals(null, shown("plugin_x", "x.two")) + assertEquals(listOf("Overview" to "plugin/x/b.html"), shown("plugin_x", "x.one")!!.buttons) + assertEquals(2L, count("Tooltips")) + assertEquals(2L, count("TooltipTexts")) + assertEquals(2L, count("TooltipButtons")) + assertEquals(listOf("Overview", SHARED_DESCRIPTION), column("TooltipButtonDescriptions", "description")) + assertEquals(listOf("a/editor.html", "plugin/x/b.html"), column("TooltipButtonUris", "uri")) + } + + // Three lookup chunks. Robolectric's SQLite binds more than 33,000 variables, so this cannot show + // that the chunks keep under SQLite's pre-3.32 default limit of 999; only a device's SQLite can. + @Test + fun `writes and removes 1,200 tooltips at once`() { + val many = (1..1200).map { row("x.$it", buttons = arrayOf("Read $it" to "plugin/x/$it.html")) } + write("plugin_x", *many.toTypedArray()) + + assertEquals(listOf("Read 1100" to "plugin/x/1100.html"), shown("plugin_x", "x.1100")!!.buttons) + assertEquals(1201L, count("TooltipButtonUris")) + remove("plugin_x") + assertEquals(1L, count("Tooltips")) + assertEquals(1L, count("TooltipButtonUris")) + } + + @Test + fun `refuses a database before version 3 and writes nothing`() { + setVersion(2) + val before = contents() + + val written = assertThrows(IllegalStateException::class.java) { write("plugin_x", row("x.one")) } + assertTrue(written.message, "version 2" in written.message.orEmpty()) + val removed = assertThrows(IllegalStateException::class.java) { remove("ide") } + assertTrue(removed.message, "version 2" in removed.message.orEmpty()) + assertEquals(before, contents()) + } + + @Test + fun `throws when any insert fails rather than carry on without the row`() { + for (table in listOf( + "TooltipCategories", + "TooltipTexts", + "Tooltips", + "TooltipButtonDescriptions", + "TooltipButtonUris", + "TooltipButtons", + )) { + db.execSQL("CREATE TRIGGER refuse BEFORE INSERT ON $table BEGIN SELECT RAISE(ABORT, 'refused'); END") + val before = contents() + + assertThrows(table, SQLException::class.java) { + write("plugin_x", row("x.one", buttons = arrayOf("Read more" to "plugin/x/a.html"))) + } + assertEquals(table, before, contents()) + db.execSQL("DROP TRIGGER refuse") + } + } + + private companion object { + const val BUILT_IN_ID = 500L + const val SHARED_DESCRIPTION = "View full documentation" + + val TABLES = + listOf( + "TooltipCategories", + "TooltipTexts", + "Tooltips", + "TooltipButtonDescriptions", + "TooltipButtonUris", + "TooltipButtons", + ) + + // What idetooltips' TooltipQuery.kt runs. + const val APP_TOOLTIP_QUERY = + "SELECT T.id, X.summary, X.detail FROM Tooltips AS T JOIN TooltipTexts AS X ON X.id = T.id " + + "WHERE T.categoryId IN (SELECT id FROM TooltipCategories WHERE category = ?) AND T.tag = ?" + const val APP_BUTTONS_QUERY = + "SELECT D.description, U.uri FROM TooltipButtons AS B " + + "JOIN TooltipButtonDescriptions AS D ON D.id = B.descriptionId " + + "JOIN TooltipButtonUris AS U ON U.id = B.uriId WHERE B.tooltipId = ? ORDER BY B.buttonNumberId" + + // documentation.db's version 3 tooltip tables, as migrate_tooltip_schema.py creates them. + val SCHEMA = + listOf( + "CREATE TABLE TooltipCategories (id INTEGER PRIMARY KEY, category TEXT NOT NULL)", + "CREATE TABLE TooltipTexts (id INTEGER PRIMARY KEY AUTOINCREMENT, summary TEXT NOT NULL, detail TEXT NOT NULL)", + "CREATE TABLE Tooltips (id INTEGER NOT NULL, categoryId INTEGER NOT NULL, tag TEXT NOT NULL, " + + "PRIMARY KEY (categoryId, tag), FOREIGN KEY(categoryId) REFERENCES TooltipCategories(id), " + + "FOREIGN KEY(id) REFERENCES TooltipTexts(id)) WITHOUT ROWID", + "CREATE TABLE TooltipButtonDescriptions (id INTEGER PRIMARY KEY, description TEXT NOT NULL)", + "CREATE TABLE TooltipButtonUris (id INTEGER PRIMARY KEY, uri TEXT NOT NULL)", + "CREATE TABLE TooltipButtons (tooltipId INTEGER NOT NULL, buttonNumberId INTEGER NOT NULL, " + + "descriptionId INTEGER NOT NULL, uriId INTEGER NOT NULL, PRIMARY KEY (tooltipId, buttonNumberId), " + + "FOREIGN KEY(tooltipId) REFERENCES TooltipTexts(id), " + + "FOREIGN KEY(descriptionId) REFERENCES TooltipButtonDescriptions(id), " + + "FOREIGN KEY(uriId) REFERENCES TooltipButtonUris(id)) WITHOUT ROWID", + "CREATE TABLE DocumentationDatabaseVersion (major INT NOT NULL, minor INT NOT NULL, patch INT NOT NULL, " + + "who TEXT NOT NULL, comment TEXT NOT NULL, changeTime TIMESTAMP DEFAULT CURRENT_TIMESTAMP)", + ) + } +} From e22ade6cfb7b8e15cd5370d9af4d992d81aa024f Mon Sep 17 00:00:00 2001 From: David Schachter Date: Sun, 18 Oct 2026 23:47:28 -0700 Subject: [PATCH 06/13] ADFA-3305: Move getTooltip's row mapping into readTooltip No behaviour change. The query, the row-count check and the mapping of the row and its buttons to an IDETooltipItem move out of TooltipManager.getTooltip into readTooltip beside queryTooltip, so tests can run them without initializing TooltipManager, which reads the installed app's Environment. Spotless reformatted the moved `when`. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3 --- .../androidide/idetooltips/ToolTipManager.kt | 31 +----------- .../androidide/idetooltips/TooltipQuery.kt | 47 +++++++++++++++++++ 2 files changed, 48 insertions(+), 30 deletions(-) diff --git a/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt index 173b3ba82f..ca280f43b7 100644 --- a/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt +++ b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt @@ -77,36 +77,7 @@ object TooltipManager { "n/a" } - queryTooltip(database, category = category, tag = tag).use { c -> - when (c.count) { - // A missing tag is a normal outcome (e.g. a Gradle task with no - // authored tooltip). Honor the nullable contract so the caller - // shows nothing rather than an "n/a" popup. - 0 -> throw NoTooltipFoundException(category, tag) - 1 -> { /* Expected case, continue processing */ - } - - else -> throw DatabaseCorruptionException( - "Multiple tooltips found for category='$category', tag='$tag' (found ${c.count} rows). " + - "Each category/tag combination should be unique." - ) - } - - c.moveToFirst() - val tooltipId = c.getInt(0) - val summary = c.getString(1) - val detail = c.getString(2) - - val buttons = ArrayList>() - for ((description, uri) in queryTooltipButtons(database, tooltipId)) { - buttons.add(Pair(description, "http://localhost:6174/$uri")) - } - - // Tooltips has no rowid from documentation version 3 on. It always equalled - // id before (id was the INTEGER PRIMARY KEY), so the debug dialog's Row is - // still the id. - IDETooltipItem(tooltipId, tooltipId, category, tag, summary, detail, buttons, lastChange) - } + readTooltip(database, category = category, tag = tag, lastChange = lastChange) } } catch (e: NoTooltipFoundException) { Log.d(TAG, "No tooltip found for category='$category', tag='$tag'") diff --git a/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/TooltipQuery.kt b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/TooltipQuery.kt index 99a1542cf2..6e407e87aa 100644 --- a/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/TooltipQuery.kt +++ b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/TooltipQuery.kt @@ -44,6 +44,53 @@ internal fun queryTooltip( return database.rawQuery(QUERY_TOOLTIP, arrayOf(category, tag)) } +/** + * The tooltip for [category] and [tag] as TooltipManager shows it, its buttons pointed at the local + * documentation server, stamped with [lastChange]. Throws [NoTooltipFoundException] when the + * database has none and [DatabaseCorruptionException] when it has more than one. + */ +internal fun readTooltip( + database: SQLiteDatabase, + category: String, + tag: String, + lastChange: String, +): IDETooltipItem = + queryTooltip(database, category = category, tag = tag).use { c -> + when (c.count) { + // A missing tag is a normal outcome (e.g. a Gradle task with no + // authored tooltip). Honor the nullable contract so the caller + // shows nothing rather than an "n/a" popup. + 0 -> { + throw NoTooltipFoundException(category, tag) + } + + 1 -> { // Expected case, continue processing + } + + else -> { + throw DatabaseCorruptionException( + "Multiple tooltips found for category='$category', tag='$tag' (found ${c.count} rows). " + + "Each category/tag combination should be unique.", + ) + } + } + + c.moveToFirst() + val tooltipId = c.getInt(0) + val summary = c.getString(1) + val detail = c.getString(2) + + val buttons = ArrayList>() + for ((description, uri) in queryTooltipButtons(database, tooltipId)) { + buttons.add(Pair(description, "http://localhost:6174/$uri")) + } + + // Tooltips has no rowid from documentation version 3 on. It always equalled + // id before (id was the INTEGER PRIMARY KEY), so the debug dialog's Row is + // still the id. + IDETooltipItem(tooltipId, tooltipId, category, tag, summary, detail, buttons, lastChange) + } + /** The (description, uri) of each of [tooltipId]'s buttons, in display order. */ internal fun queryTooltipButtons( database: SQLiteDatabase, From f9bd830cfa94267ae137729292d11479322d4da2 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Sun, 18 Oct 2026 23:51:22 -0700 Subject: [PATCH 07/13] ADFA-3305: Name the refused database file in the tooltip error log getTooltip reads the installed documentation.db or, when it is newer, the /sdcard/Download debug override, and the log said neither: a "declares documentation version 2" line left a developer guessing which file to replace. The log now names the source and its path. Review finding R2163-03. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3 --- .../java/com/itsaky/androidide/idetooltips/ToolTipManager.kt | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt index ca280f43b7..86ca589fa1 100644 --- a/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt +++ b/idetooltips/src/main/java/com/itsaky/androidide/idetooltips/ToolTipManager.kt @@ -58,6 +58,8 @@ object TooltipManager { return withContext(Dispatchers.IO) { var dbPath = Environment.DOC_DB.absolutePath + // Named in the error log: a refused database could be either file, and only the log says which. + var dbSource = "installed database" // TODO: The debug database code should only exist in a debug build. --DS, 30-Jul-2025 val debugDatabaseTimestamp = @@ -66,6 +68,7 @@ object TooltipManager { if (debugDatabaseTimestamp > databaseTimestamp) { // Switch to the debug database. dbPath = debugDatabaseFile.absolutePath + dbSource = "debug database" } try { @@ -88,7 +91,7 @@ object TooltipManager { } catch (e: Exception) { Log.e( TAG, - "Error getting tooltip for category='$category', tag='$tag': ${e.message}" + "Error getting tooltip for category='$category', tag='$tag' from the $dbSource '$dbPath': ${e.message}" ) null } From 49257e05630140db58913edecd786a497a1fffc8 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Sun, 18 Oct 2026 23:51:22 -0700 Subject: [PATCH 08/13] ADFA-3305: Test the tooltip item readTooltip builds from a row No test ran getTooltip's row mapping, though this branch renumbered its columns and made rowId the id: with the summary read from the detail column, all 12 idetooltips tests still passed. A tooltip whose id, summary, detail and buttons all differ is now read back into the exact item, so that mutant, a swapped detail and a wrong rowId each fail it. Also covers the missing-tag and duplicate-category exceptions. Review finding R2163-04. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3 --- .../idetooltips/TooltipQueryTest.kt | 50 ++++++++++++++++++- 1 file changed, 49 insertions(+), 1 deletion(-) diff --git a/idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt b/idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt index fb15914b09..2f669cf444 100644 --- a/idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt +++ b/idetooltips/src/test/java/com/itsaky/androidide/idetooltips/TooltipQueryTest.kt @@ -13,7 +13,8 @@ import org.robolectric.RobolectricTestRunner import org.robolectric.annotation.Config /** - * Runs [queryTooltip] and [queryTooltipButtons], the lookups TooltipManager.getTooltip makes, + * Runs [queryTooltip] and [queryTooltipButtons], the lookups TooltipManager.getTooltip makes, and + * [readTooltip], which turns their rows into the item it shows, * against documentation.db's version 3 tooltip schema (ADFA-3305), as migrate_tooltip_schema.py in * OfflineDocumentationTools writes it. The query plans matter as much as the rows: the join * QUERY_TOOLTIP replaced scanned the whole 13 MB table for every tooltip shown (ADFA-6597), and the @@ -122,6 +123,53 @@ class TooltipQueryTest { assertFalse(plan.toString(), plan.any { Regex("""\bSCAN (TABLE )?\w+""").containsMatchIn(it) }) } + @Test + fun `reads a tooltip's id, text and buttons into the item TooltipManager shows`() { + db.execSQL("INSERT INTO TooltipTexts VALUES (4242, 'Declares a read-only property.', 'Longer detail.')") + db.execSQL("INSERT INTO Tooltips VALUES (4242, 4, 'val')") + db.execSQL("INSERT INTO TooltipButtonDescriptions VALUES (3, 'Learn more about properties')") + db.execSQL("INSERT INTO TooltipButtonUris VALUES (9001, 'k/html/properties.html'), (9002, 'k/html/basic-syntax.html')") + db.execSQL("INSERT INTO TooltipButtons VALUES (4242, 2, 2, 9002), (4242, 1, 3, 9001)") + + val item = readTooltip(db, category = "kotlin", tag = "val", lastChange = "2026-10-18 (test)") + + assertEquals( + IDETooltipItem( + rowId = 4242, + id = 4242, + category = "kotlin", + tag = "val", + summary = "Declares a read-only property.", + detail = "Longer detail.", + buttons = + arrayListOf( + "Learn more about properties" to "http://localhost:6174/k/html/properties.html", + "Examples" to "http://localhost:6174/k/html/basic-syntax.html", + ), + lastChange = "2026-10-18 (test)", + ), + item, + ) + } + + @Test + fun `reading a tag with no tooltip throws NoTooltipFoundException`() { + assertThrows(NoTooltipFoundException::class.java) { + readTooltip(db, category = "java", tag = "no-such-tag", lastChange = "test") + } + } + + @Test + fun `reading a tag two same-named categories hold throws DatabaseCorruptionException`() { + db.execSQL("INSERT INTO TooltipCategories VALUES (5, 'java')") + db.execSQL("INSERT INTO TooltipTexts VALUES (9999, 'duplicate', 'detail')") + db.execSQL("INSERT INTO Tooltips VALUES (9999, 5, 'tag7')") + + assertThrows(DatabaseCorruptionException::class.java) { + readTooltip(db, category = "java", tag = "tag7", lastChange = "test") + } + } + @Test fun `refuses a database that declares a version before 3`() { setVersion(2) From ae7b08f5e33614e9cac0c1fb5d61de7098b26d82 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Sun, 18 Oct 2026 23:52:11 -0700 Subject: [PATCH 09/13] ADFA-3305: Say what an update keeps and where version 2 still lives docs/documentation-database.md said the database is extracted on install and update; InstallationViewModel extracts it at first-time setup only, so an update keeps the installed file, and a version 2 one then shows no tooltips (accepted for ADFA-3305). It also said docdb-studio edits "this same documentation.db" and validates TooltipButtons.uri: the master it edits stays in version 2, and migrate_tooltip_schema.py converts a copy for shipping, so docdb-studio and the docs/docdb scripts use the version 2 columns. The debug-override notes now say a version 2 file shows no tooltips, and the ToolTipManager entry names readTooltip. idetooltips/README.md cited a Room IDETooltipDatabase that does not exist; the module reads documentation.db with raw SQLite. Review finding R2163-05. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3 --- docs/documentation-database.md | 12 ++++++------ idetooltips/README.md | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/documentation-database.md b/docs/documentation-database.md index 844c0fdb39..dab0804000 100644 --- a/docs/documentation-database.md +++ b/docs/documentation-database.md @@ -7,8 +7,8 @@ This is a **prebuilt** database — CoGo never creates or migrates its schema at ## Where it lives - Installed path: `context.getDatabasePath("documentation.db")` (`Environment.DOC_DB` in `common/.../utils/Environment.java`), i.e. the app's private `databases/` dir. -- Bundled as an asset and extracted on install/update by `BundledAssetsInstaller` / `SplitAssetsInstaller`. -- **Debug override:** if `/sdcard/Download/documentation.db` exists and is newer than the installed copy, `DocumentationContentSource` (behind both Tier 3 transports) and `ToolTipManager` swap to it at request time (timestamp-compared per request, not just at startup) — a fast way to test a new database on-device without reinstalling. **The comparison is on modification time, and `adb push` preserves the *source* file's mtime** -- so pushing a database saved earlier than the one already on the device silently does not swap, and the app keeps serving the old one with no error anywhere. Follow a push with `adb shell touch /sdcard/Download/documentation.db` (this cost real debugging time on ADFA-5153). `WebServer`'s debug logging and experiment flags are also file-flag-gated under `/sdcard/Download/` (`CodeOnTheGo.webserver.debug`, `CodeOnTheGo.exp`, `CodeOnTheGo.webserver.cs0`). +- Bundled as an asset and extracted by `BundledAssetsInstaller` / `SplitAssetsInstaller` at first-time setup only: `InstallationViewModel` runs the asset install while no JDK or Android SDK is installed, and returns early once both are. An app update keeps the installed file whatever the new APK bundles, so an install first set up with a version 2 database keeps it, and from documentation version 3 on (ADFA-3305) shows no tooltips; clearing the app's data is what gets it the bundled file. That is accepted for ADFA-3305. +- **Debug override:** if `/sdcard/Download/documentation.db` exists and is newer than the installed copy, `DocumentationContentSource` (behind both Tier 3 transports) and `ToolTipManager` swap to it at request time (timestamp-compared per request, not just at startup) — a fast way to test a new database on-device without reinstalling. **The comparison is on modification time, and `adb push` preserves the *source* file's mtime** -- so pushing a database saved earlier than the one already on the device silently does not swap, and the app keeps serving the old one with no error anywhere. Follow a push with `adb shell touch /sdcard/Download/documentation.db` (this cost real debugging time on ADFA-5153). **A version 2 file pushed there now shows no tooltips:** this build refuses tooltip tables below version 3, and the log line names the debug database and its path. Convert a copy with `OfflineDocumentationTools`' `migrate_tooltip_schema.py` before pushing it. `WebServer`'s debug logging and experiment flags are also file-flag-gated under `/sdcard/Download/` (`CodeOnTheGo.webserver.debug`, `CodeOnTheGo.exp`, `CodeOnTheGo.webserver.cs0`). - **Don't trust a local copy's on-disk schema or row content as ground truth without checking freshness first.** Any manually downloaded or debug-override copy is independent of git history — a stale one can have a different schema (e.g. missing `UNIQUE(path)` or `templateId`) or be missing rows that already exist in the current, maintained database. A stale copy caused a real near-miss in ADFA-5088: a SQL script validated against it would have silently overwritten curated production tooltip content for several tags. Diff or re-download before authoring SQL against a local copy's state, not just before shipping it. ## Schema @@ -95,20 +95,20 @@ Of the five sites below, only `DocumentationContentSource` and `ToolTipManager` - **`common/.../documentation/DocumentationContentSource.kt`** — the one Tier 3 pipeline that reads this database: row lookup (a request target is matched raw first, since stored `Content.path` values are percent-encoded, with a percent-decoded fallback on a miss — shared by both transports so they agree on which pages exist), chunked-row reassembly, dictionary-aware Brotli decode gated on the declared documentation version, and the database swaps — to a newer sdcard debug database, and reopening the installed file when an asset install rewrites it in place under the cached handle — under a read/write lock so several threads can read while a swap cannot close the handle under them. It also renders the rows that are Pebble template contexts (`templateId > 0`, the Kotlin doc set's pages) — resolving each template's references to other templates against `Templates.name` as it goes — so both transports below serve finished pages and neither needs the template engine itself. All of that logic exists once (ADFA-5176). - **`common/.../documentation/DocumentationRequestInterceptor.kt`** — serves Tier 3 *in-process* for the app's WebViews (`HelpActivity`, the tooltip fragment, `FAQActivity`), through `WebViewClient.shouldInterceptRequest`, so a page's assets cost a database read instead of a TCP connection each (ADFA-5176). It matches the same `http://localhost:6174/...` URL space, so the strings.xml entries, `ToolTipManager`'s link builder and the `DocumentationExtension` contract need no changes; anything it declines — a `/pr/` endpoint, an unknown path, a failed read — falls through to `WebServer` unchanged. Both transports get their `Content-Type` charset from the same place, `ContentTypeHeaders` (ADFA-5241), so a row does not describe itself differently depending on which one served it. Set `/sdcard/Download/CodeOnTheGo.nointercept` to force documentation back onto the server. - **`app/.../localWebServer/WebServer.kt`** — serves Tier 3 over HTTP on port 6174, for WebViews that are not wired to the interceptor above and for the `/pr/` developer endpoints. It reads through `DocumentationContentSource`, so it holds no database, template engine or decode logic of its own; what remains here is HTTP: request parsing, the `/pr/` pages, error responses, and the CSS/asset shortcuts. Also serves a Dynamic Bookshelf JSON payload (joining `Content`/`Bookshelf`/`BookCategories`, rendered through the `bookshelf` template) and debug-only HTML dumps at `/pr/db` (`LastChange`, last 20 rows) and `/pr/pr` (recent projects, from a *different* database). -- **`idetooltips/.../ToolTipManager.kt`** — serves Tier 1/2, through `queryTooltip` and `queryTooltipButtons` in `TooltipQuery.kt`. Looks up a tooltip by `(category, tag)`: the category's id from `TooltipCategories` first, so the row is found by `Tooltips`' key rather than a scan of the whole table (ADFA-6597), then its text by id. Then its buttons for the Tier 3 links shown at the bottom, by `TooltipButtons`' key, with their text by id. Refuses a database declaring a version below 3. +- **`idetooltips/.../ToolTipManager.kt`** — serves Tier 1/2, through `readTooltip` in `TooltipQuery.kt`, which runs `queryTooltip` and `queryTooltipButtons` and builds the item shown. Looks up a tooltip by `(category, tag)`: the category's id from `TooltipCategories` first, so the row is found by `Tooltips`' key rather than a scan of the whole table (ADFA-6597), then its text by id. Then its buttons for the Tier 3 links shown at the bottom, by `TooltipButtons`' key, with their text by id. Refuses a database declaring a version below 3. - **`plugin-manager/.../documentation/PluginDocumentationManager.kt`** (with `Tier3AssetWalker.kt`, and the `DocumentationExtension` contract in `plugin-api`) — lets plugins contribute their own help content into the same lookup paths, under the reserved `plugin//...` prefix. Its tooltip SQL is in `PluginTooltipTables.kt`: a plugin's tooltips are replaced or removed by category inside one transaction, each button's text is looked up in or added to the lookup tables, and text no button uses any more is deleted afterwards. Brotli assets are compressed against the same `CompressionDictionary` the rest of the table uses, read from the database being written. When that dictionary can't be read the install is abandoned rather than written plain, and retried on the plugin's next activation. ## Editing the database -Schema changes and data edits happen **outside this repo**, in `OfflineDocumentationTools/docdb-studio` (a Flet GUI over this same `documentation.db`). Conventions enforced there that matter if you're reasoning about data correctness here: +Schema changes and data edits happen **outside this repo**, in `OfflineDocumentationTools/docdb-studio` (a Flet GUI over the master `documentation.db`). The master stays in documentation version 2: `docdb-studio` and the other `OfflineDocumentationTools` scripts read and write the version 2 tooltip tables, and `migrate_tooltip_schema.py` converts a copy of the master to version 3 for shipping. Conventions enforced there that matter if you're reasoning about data correctness here: - The schema is locked — `docdb-studio`'s own `AGENTS.md` says never change it. If a new column/table is genuinely needed, it's a cross-repo change coordinated with that project, not something to route around in CoGo. -- Tooltip uniqueness is `(categoryId, tag)`; `TooltipButtons.uri` values are validated there against `Content.path` (post `?query`/`#fragment` stripping) before being allowed into the database. +- Tooltip uniqueness is `(categoryId, tag)`; button uris (`TooltipButtons.uri` in the version 2 master, `TooltipButtonUris.uri` once converted) are validated there against `Content.path` (post `?query`/`#fragment` stripping) before being allowed into the database. - Every edit made through the tool updates `LastChange` for the affected documentation set, which is how `DatabaseVersionResolver`'s debug logging can say what build of the docs is loaded. ### Writing one-off SQL scripts against this database -Some tickets (e.g. ADFA-5088) ship a one-off `.sql` script under `docs/docdb/` for a `docdb-studio` maintainer to run against the real database, rather than editing it directly through the tool. Gotchas found writing those scripts: +Some tickets (e.g. ADFA-5088) ship a one-off `.sql` script under `docs/docdb/` for a `docdb-studio` maintainer to run against the real database, rather than editing it directly through the tool. They run against the version 2 master, so they use its tooltip columns (`Tooltips.summary`, `TooltipButtons.uri`); never run one against a converted version 3 file. Gotchas found writing those scripts: - **The system SQLite has no JSON1 on some devices, and your desktop does.** `JSON_OBJECT`, `JSON_GROUP_ARRAY` and friends compile fine under the `sqlite3` CLI (3.44 ships JSON1 built in) and then fail at runtime with `no such function: JSON_OBJECT` on real hardware — reproduced on a Galaxy Note 20 Ultra, where the Dynamic Bookshelf was an HTTP 500 until the query was rewritten (ADFA-5179). Nothing on the desktop side of the fence will warn you. Write plain relational SQL and assemble the JSON in Kotlin, and treat any device-only 500 from a new query as this until proven otherwise. Applies to every reader of this database, not just `WebServer`: a `docdb-studio` script, a tooltip query in `ToolTipManager`, and `PluginDocumentationManager` are all equally exposed. - **Keep each `.system` line simple.** The sqlite3 CLI's `.system` dot-command can hit a content-dependent shell-parsing failure when a line chains multiple operators (`;`, `&&`, `||`, parentheses) — it reproduces for some input strings and not others, so it won't necessarily show up in a quick test. Stick to one plain `command | pipe > file` per `.system` line. diff --git a/idetooltips/README.md b/idetooltips/README.md index 5f9ac508ce..7f45aa5ad7 100644 --- a/idetooltips/README.md +++ b/idetooltips/README.md @@ -13,7 +13,7 @@ This module provides functionalities such as: * **Dynamic Tooltip Display:** Show tooltips anchored to specific UI elements or coordinates. * **Rich Content:** Supports HTML content within tooltips for flexible formatting, including links. -* **Database Integration:** Fetches tooltip content and metadata from the Room database - `IDETooltipDatabase`. +* **Database Integration:** Reads tooltip content read-only from `documentation.db` with raw SQLite (`TooltipQuery.kt`), the database described in [docs/documentation-database.md](../docs/documentation-database.md). * **"See More" Functionality:** Handles expanding content or navigating to detailed documentation. * **Custom callbacks:** Customizable action callbacks for link clicks. From 9ce9b2d33e82068b130c91a30378e56745c97a3d Mon Sep 17 00:00:00 2001 From: David Schachter Date: Mon, 19 Oct 2026 00:03:51 -0700 Subject: [PATCH 10/13] ADFA-3305: Decode compressed templates from documentation version 3 From documentation version 3 every Templates row is Brotli-compressed against the database's CompressionDictionary, like a brotli Content row. No column says so: the release build compresses every template, and nothing on the device writes Templates. - DatabaseTemplateLoader passes each row's bytes through a decode before Pebble sees them, so included templates are covered too. A decode failure is logged with the template's name and the database's path, and fails the page with a message that names the template but no file, since that message reaches HTTP responses. - DocumentationContentSource decodes with the active database's codec when the database declares major >= 3 (expectsCompressedTemplates). Below 3, and with no version, templates are read as plain UTF-8 as before: an app update never replaces the installed database, so a 2.x file keeps rendering its templated pages. The answer is read once per database, before the engine runs, and cleared on every swap. - An internal constructor takes the decode, so :common's tests, which have no brotli4j desktop native, can stand one in; CompressedTemplateTest in :app covers the real decode. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3 --- .../localWebServer/CompressedTemplateTest.kt | 247 ++++++++++++++++++ .../documentation/DatabaseTemplateLoader.kt | 23 +- .../DocumentationContentSource.kt | 59 ++++- .../utils/DatabaseVersionResolver.kt | 7 + .../utils/DocumentationCompression.kt | 10 + .../DatabaseTemplateLoaderTest.kt | 53 +++- .../DocumentationContentSourceTest.kt | 146 ++++++++++- 7 files changed, 521 insertions(+), 24 deletions(-) create mode 100644 app/src/test/java/com/itsaky/androidide/localWebServer/CompressedTemplateTest.kt diff --git a/app/src/test/java/com/itsaky/androidide/localWebServer/CompressedTemplateTest.kt b/app/src/test/java/com/itsaky/androidide/localWebServer/CompressedTemplateTest.kt new file mode 100644 index 0000000000..cfd3dcd130 --- /dev/null +++ b/app/src/test/java/com/itsaky/androidide/localWebServer/CompressedTemplateTest.kt @@ -0,0 +1,247 @@ +package com.itsaky.androidide.localWebServer + +import android.database.Cursor +import android.database.sqlite.SQLiteDatabase +import com.itsaky.androidide.documentation.DocumentationContentSource +import com.itsaky.androidide.documentation.DocumentationLookup +import com.itsaky.androidide.logging.provider.IdeLogRouter +import com.itsaky.androidide.utils.BrotliDictionaryCodec +import com.itsaky.androidide.utils.toDirectByteBuffer +import io.mockk.every +import io.mockk.mockk +import io.mockk.mockkStatic +import io.mockk.unmockkAll +import org.junit.After +import org.junit.Assert.assertEquals +import org.junit.Assert.assertThrows +import org.junit.Assert.assertTrue +import org.junit.Before +import org.junit.Rule +import org.junit.Test +import org.junit.rules.TemporaryFolder +import org.slf4j.event.Level +import java.io.ByteArrayInputStream +import java.io.File +import java.io.IOException + +/** + * Templates read through [DocumentationContentSource]'s own decode (ADFA-3305): from documentation + * version 3 every `Templates` row is Brotli-compressed against the database's dictionary, and below + * it they are plain UTF-8. `:common`'s tests cannot run that decode, since they lack brotli4j's + * desktop native (see common/build.gradle.kts), so they render through a stand-in decode; these + * cover the real one. + */ +class CompressedTemplateTest { + @get:Rule + val folder = TemporaryFolder() + + private lateinit var installedFile: File + private lateinit var debugFile: File + + @Before + fun setUp() { + installedFile = folder.newFile("documentation.db") + debugFile = File(folder.root, "debug-documentation.db") + mockkStatic(SQLiteDatabase::class) + } + + @After + fun tearDown() { + unmockkAll() + } + + @Test + fun `a template compressed against the database's dictionary renders`() { + val dictionary = dictionary("kotlin") + every { SQLiteDatabase.openDatabase(any(), isNull(), any()) } returns + database(dictionary, "page.peb" to stored(page("kotlin"), dictionary)) + + assertEquals(page("kotlin").replace("{{ who }}", "Kotlin"), render(source())) + } + + @Test + fun `an included template is decoded too`() { + val dictionary = dictionary("kotlin") + every { SQLiteDatabase.openDatabase(any(), isNull(), any()) } returns + database( + dictionary, + "page.peb" to stored("""{% include "nav.peb" %}|${page("kotlin")}""", dictionary), + "nav.peb" to stored(nav("kotlin"), dictionary), + ) + + assertEquals(nav("kotlin") + "|" + page("kotlin").replace("{{ who }}", "Kotlin"), render(source())) + } + + // An app update never replaces the installed database, so a 2.x file -- plain templates, though it + // carries a dictionary for its rows -- has to keep rendering its templated pages. + @Test + fun `a version 2 database's plain template renders`() { + val dictionary = dictionary("kotlin") + every { SQLiteDatabase.openDatabase(any(), isNull(), any()) } returns + database(dictionary, "page.peb" to page("kotlin").toByteArray(), declaredMajorVersion = 2) + + assertEquals(page("kotlin").replace("{{ who }}", "Kotlin"), render(source())) + } + + // A version 3 file whose template was left plain, as a release step that skipped compression + // would leave it. The page fails, naming the template, and the log names the database too -- + // never a page rendered from whatever the Brotli decoder made of plain text. + @Test + fun `a version 3 template that does not decode fails its page and is logged`() { + val dictionary = dictionary("kotlin") + every { SQLiteDatabase.openDatabase(any(), isNull(), any()) } returns + database(dictionary, "page.peb" to page("kotlin").toByteArray()) + val errors = mutableListOf() + val sink = IdeLogRouter.ExternalSink { level, _, message, _ -> if (level == Level.ERROR) errors += message } + IdeLogRouter.addSink(sink) + val lookup = + try { + source().lookup("k/html/basic-syntax.html") + } finally { + IdeLogRouter.removeSink(sink) + } + + val failed = lookup as DocumentationLookup.Failed + assertEquals("Cannot decode template 'page.peb'", failed.cause.message) + assertTrue(errors.toString(), errors.any { it.contains("'page.peb'") && it.contains(DATABASE_PATH) }) + } + + // The two dictionaries are the same length and differ only in bytes the templates reference, so + // decoding the new database's template with the old database's dictionary does not fail: it + // returns different text (see BrotliDictionaryDecodeTest). Only a codec rebuilt for the new + // database, and a compiled-template cache dropped with the old one, renders the new page. + @Test + fun `after a swap, templates are decoded with the new database's dictionary`() { + val installedDictionary = dictionary("kotlin") + val debugDictionary = dictionary("scalax") + every { SQLiteDatabase.openDatabase(installedFile.absolutePath, isNull(), any()) } returns + database(installedDictionary, "page.peb" to stored(page("kotlin"), installedDictionary)) + every { SQLiteDatabase.openDatabase(debugFile.absolutePath, isNull(), any()) } returns + database(debugDictionary, "page.peb" to stored(page("scalax"), debugDictionary)) + val source = source() + + assertEquals(page("kotlin").replace("{{ who }}", "Kotlin"), render(source)) + + swapToDebugDatabase() + + assertEquals(page("scalax").replace("{{ who }}", "Kotlin"), render(source)) + } + + @Test + fun `a swap from a version 2 file to a version 3 file renders both`() { + val dictionary = dictionary("kotlin") + every { SQLiteDatabase.openDatabase(installedFile.absolutePath, isNull(), any()) } returns + database(dictionary, "page.peb" to page("kotlin").toByteArray(), declaredMajorVersion = 2) + every { SQLiteDatabase.openDatabase(debugFile.absolutePath, isNull(), any()) } returns + database(dictionary, "page.peb" to stored(page("scalax"), dictionary)) + val source = source() + + assertEquals(page("kotlin").replace("{{ who }}", "Kotlin"), render(source)) + + swapToDebugDatabase() + + assertEquals(page("scalax").replace("{{ who }}", "Kotlin"), render(source)) + } + + @Test + fun `a swap from a version 3 file to a version 2 file renders both`() { + val dictionary = dictionary("kotlin") + every { SQLiteDatabase.openDatabase(installedFile.absolutePath, isNull(), any()) } returns + database(dictionary, "page.peb" to stored(page("kotlin"), dictionary)) + every { SQLiteDatabase.openDatabase(debugFile.absolutePath, isNull(), any()) } returns + database(dictionary, "page.peb" to page("scalax").toByteArray(), declaredMajorVersion = 2) + val source = source() + + assertEquals(page("kotlin").replace("{{ who }}", "Kotlin"), render(source)) + + swapToDebugDatabase() + + assertEquals(page("scalax").replace("{{ who }}", "Kotlin"), render(source)) + } + + private fun source() = DocumentationContentSource(installedFile, debugFile, 0) + + private fun swapToDebugDatabase() { + debugFile.writeText("newer") + debugFile.setLastModified(installedFile.lastModified() + 60_000) + } + + private fun render(source: DocumentationContentSource): String = + (source.lookup("k/html/basic-syntax.html") as DocumentationLookup.Found).content.bytes.toString(Charsets.UTF_8) + + private fun dictionary(word: String) = "$SHARED $word interface companion object ".repeat(400) + + private fun page(word: String) = "$SHARED $word {{ who }} interface companion object ".repeat(6) + + private fun nav(word: String) = "".repeat(4) + + /** + * [template] as the pipeline stores it: Brotli against [dictionary]. Checked to reference the + * dictionary, failing without it, since a stream that never matched into the dictionary decodes + * the same either way and would not show that the dictionary is attached. + */ + private fun stored( + template: String, + dictionary: String, + ): ByteArray { + val bytes = BrotliDictionaryCodec(toDirectByteBuffer(dictionary.toByteArray())).compress(template.toByteArray()) + assertThrows(IOException::class.java) { BrotliDictionaryCodec(null).decompress(ByteArrayInputStream(bytes)) } + return bytes + } + + /** + * A database declaring [declaredMajorVersion] with [dictionary] as its CompressionDictionary, one + * Content row -- k/html/basic-syntax.html, templated with id 7, the first of [templates] -- and + * [templates] as its `Templates` rows, by name. + */ + private fun database( + dictionary: String, + vararg templates: Pair, + declaredMajorVersion: Int = 3, + ): SQLiteDatabase = + mockk(relaxed = true) { + every { path } returns DATABASE_PATH + every { rawQuery(match { it.contains("FROM Content") }, any()) } returns + cursor(rows = 1) { + every { getBlob(0) } returns """{"who": "Kotlin"}""".toByteArray() + every { getString(1) } returns "text/html" + every { getString(2) } returns "none" + every { getInt(3) } returns 7 + } + every { rawQuery(match { it.contains("WHERE id = ?") }, arrayOf("7")) } returns + cursor(rows = 1) { every { getString(0) } returns templates.first().first } + every { rawQuery(match { it.contains("FROM Templates WHERE name = ?") }, any()) } answers { + val body = templates.toMap()[secondArg>()[0]] + cursor(rows = if (body == null) 0 else 1) { every { getBlob(0) } returns body } + } + every { + rawQuery(match { it.contains("FROM sqlite_master") && it.contains("DocumentationDatabaseVersion") }, any()) + } returns cursor(rows = 1) + every { rawQuery(match { it.contains("FROM DocumentationDatabaseVersion") }, any()) } returns + cursor(rows = 1) { + every { isNull(0) } returns false + every { getInt(0) } returns declaredMajorVersion + every { getInt(1) } returns 1 + } + every { rawQuery(match { it.contains("FROM sqlite_master") && it.contains("CompressionDictionary") }, null) } returns + cursor(rows = 1) + every { rawQuery(match { it.contains("SELECT data FROM CompressionDictionary") }, null) } returns + cursor(rows = 1) { every { getBlob(0) } returns dictionary.toByteArray() } + } + + private fun cursor( + rows: Int, + columns: Cursor.() -> Unit = {}, + ) = mockk(relaxed = true) { + every { count } returns rows + every { moveToFirst() } returns (rows > 0) + columns() + } + + private companion object { + // Text the dictionaries and templates share, so a stored template matches into its dictionary. + const val SHARED = """

docs-sidebar toc-element""" + + const val DATABASE_PATH = "/sdcard/Download/documentation.db" + } +} diff --git a/common/src/main/java/com/itsaky/androidide/documentation/DatabaseTemplateLoader.kt b/common/src/main/java/com/itsaky/androidide/documentation/DatabaseTemplateLoader.kt index 19e80caa5d..266004245a 100644 --- a/common/src/main/java/com/itsaky/androidide/documentation/DatabaseTemplateLoader.kt +++ b/common/src/main/java/com/itsaky/androidide/documentation/DatabaseTemplateLoader.kt @@ -37,13 +37,21 @@ import java.io.StringReader * Names are `Templates.name` values, matched exactly: the table is a flat namespace with no * directories, so there is no prefix, suffix or relative path to apply. * + * A row's bytes pass through [decode] before the engine sees them: from documentation version 3 + * (ADFA-3305) every template is Brotli-compressed against the database's dictionary, while an + * earlier database stores them as plain UTF-8. Every template the engine loads comes through here, + * `include`d ones too. + * * @param database Supplies the database to read, or null when none is open. Called on every * resolution rather than captured, because the source swaps the handle when a newer database * appears; callers resolve under the read lock that a swap excludes, so the handle cannot change * mid-render. + * @param decode Turns a row's stored bytes, read from the given database, into the template's + * UTF-8 text. */ internal class DatabaseTemplateLoader( private val database: () -> SQLiteDatabase?, + private val decode: (SQLiteDatabase, ByteArray) -> ByteArray, ) : Loader { override fun getReader(name: String): Reader { val database = database() ?: throw LoaderException(null, "No documentation database is open, for template '$name'") @@ -83,7 +91,18 @@ internal class DatabaseTemplateLoader( throw LoaderException(e, "Cannot read template '$name' from the database") } - return StringReader(body.toString(Charsets.UTF_8)) + // Exception, not RuntimeException: a Brotli decode failure is an IOException. Rewrapped for + // the same reason as above, and so the failure names the template it belongs to. Logged here + // with the database, which the exception's message leaves out: it reaches HTTP responses. + val text = + try { + decode(database, body) + } catch (e: Exception) { + log.error("Template '{}' in '{}' does not decode; the page using it fails", name, database.path, e) + throw LoaderException(e, "Cannot decode template '$name'") + } + + return StringReader(text.toString(Charsets.UTF_8)) } override fun resourceExists(name: String): Boolean { @@ -119,7 +138,7 @@ internal class DatabaseTemplateLoader( anchorPath: String, ): String = relativePath - /** Template bodies are stored as UTF-8 blobs, so the engine's charset setting does not apply. */ + /** Template bodies decode to UTF-8, so the engine's charset setting does not apply. */ override fun setCharset(charset: String) = Unit /** Names are exact `Templates.name` values; decorating them would stop them matching. */ diff --git a/common/src/main/java/com/itsaky/androidide/documentation/DocumentationContentSource.kt b/common/src/main/java/com/itsaky/androidide/documentation/DocumentationContentSource.kt index 173d95a84e..5d4235ad3c 100644 --- a/common/src/main/java/com/itsaky/androidide/documentation/DocumentationContentSource.kt +++ b/common/src/main/java/com/itsaky/androidide/documentation/DocumentationContentSource.kt @@ -24,6 +24,7 @@ import com.google.gson.GsonBuilder import com.google.gson.ToNumberPolicy import com.google.gson.reflect.TypeToken import com.itsaky.androidide.utils.BrotliDictionaryCodec +import com.itsaky.androidide.utils.expectsCompressedTemplates import com.itsaky.androidide.utils.loadCompressionDictionary import io.pebbletemplates.pebble.PebbleEngine import io.pebbletemplates.pebble.error.PebbleException @@ -71,6 +72,12 @@ fun joinChunks(chunks: List): ByteArray { return joined } +/** A `Templates` row's text, Brotli-decoded with [codec], the active database's. */ +private fun decompressTemplate( + codec: BrotliDictionaryCodec, + body: ByteArray, +): ByteArray = codec.decompress(ByteArrayInputStream(body)) + /** * One row of documentation content, decoded and rendered, ready to send. * @@ -141,17 +148,30 @@ data class RequestLookup( * One pipeline with two callers (ADFA-5176): `WebServer`, which wraps it in HTTP, and * [DocumentationRequestInterceptor], which answers a WebView in-process with no socket at all. A row * that is a Pebble template context is rendered here too, so both transports serve a finished page - * and neither needs the template engine itself. + * and neither needs the template engine itself. From documentation version 3 (ADFA-3305) templates + * are Brotli-compressed against the same dictionary as the rows; an earlier database's are plain + * UTF-8 and are read as they are, since an app update never replaces the installed database. * * Thread-safe: [lookup] and [withDatabase] hold a read lock for the whole read, and the swap takes * the write lock, since swapping closes the handle a reader could be using. Readers never block * each other. + * + * @param decodeTemplate Turns a `Templates` row's bytes into the template's text, given the codec + * for the database it came from. A seam for `:common`'s tests only, which have no brotli4j desktop + * native (see common/build.gradle.kts); the public constructor always Brotli-decodes. */ -class DocumentationContentSource( +class DocumentationContentSource internal constructor( private val databaseFile: File, private val debugDatabaseFile: File, - debugCheckIntervalMs: Long = 1_000, + debugCheckIntervalMs: Long, + private val decodeTemplate: (BrotliDictionaryCodec, ByteArray) -> ByteArray, ) : Closeable { + constructor( + databaseFile: File, + debugDatabaseFile: File, + debugCheckIntervalMs: Long = 1_000, + ) : this(databaseFile, debugDatabaseFile, debugCheckIntervalMs, ::decompressTemplate) + private val log = LoggerFactory.getLogger(DocumentationContentSource::class.java) private val databaseLock = ReentrantReadWriteLock() @@ -206,14 +226,23 @@ class DocumentationContentSource( private var codec: BrotliDictionaryCodec? = null private var codecStale = true + // Whether the active database's templates are Brotli-compressed (documentation version 3 on), or + // null until its first render asks. Cleared on every swap: the next database may be either kind. + @Volatile + private var templatesCompressed: Boolean? = null + // The loader reads Templates rows, so a template can reference another one (ADFA-5405). It also // makes the engine's own cache the compiled-template cache, keyed by name: a partial pulled in // by several pages is compiled once, and dropping a database means invalidating that cache too. + // A compressed row is decoded with the active database's codec, the one its Content rows use. private val pebbleEngine = PebbleEngine .Builder() - .loader(DatabaseTemplateLoader { database }) - .maxRenderedSize(MAX_RENDERED_CHARS) + .loader( + DatabaseTemplateLoader({ database }) { database, body -> + if (templatesCompressed(database)) decodeTemplate(codec(database), body) else body + }, + ).maxRenderedSize(MAX_RENDERED_CHARS) .build() // Template names by id, for the active database. Content rows reference a template by id; every @@ -356,7 +385,7 @@ class DocumentationContentSource( name: String, path: String, contextJson: (SQLiteDatabase) -> ByteArray, - ): ByteArray = withDatabase { database -> renderNamed(name, contextJson(database), path) } + ): ByteArray = withDatabase { database -> renderNamed(database, name, contextJson(database), path) } /** * Clears all cached templates, compiled and by name. @@ -483,7 +512,7 @@ class DocumentationContentSource( templateName(database, templateId, path) } - return renderNamed(name, contextJson, path) + return renderNamed(database, name, contextJson, path) } /** @@ -492,16 +521,22 @@ class DocumentationContentSource( * Callers hold the read lock, since the loader the engine resolves through reads the active * database -- for this template and for every one it references. * + * @param database The active database, which every template is loaded from. * @param name The template's `Templates.name`. * @param contextJson The JSON-encoded context supplied to the template. * @param path The content path associated with the rendering request. * @return The rendered template content encoded as UTF-8 bytes. */ private fun renderNamed( + database: SQLiteDatabase, name: String, contextJson: ByteArray, path: String, ): ByteArray { + // Resolved here, before the engine runs, so a failure to read the version arrives as itself + // rather than inside the loader, which would report it as a template that does not decode. + templatesCompressed(database) + val contextString = contextJson.toString(Charsets.UTF_8) if (contextString.isBlank() || contextString.trim() == "null") { throw TemplateRenderException("Template '$name' has empty or null JSON context, for path '$path'") @@ -529,6 +564,13 @@ class DocumentationContentSource( } } + /** + * Whether [database]'s templates are Brotli-compressed, from the version it declares; asked once + * per database. A failure to read the version is not cached, so the next render asks again. + */ + private fun templatesCompressed(database: SQLiteDatabase): Boolean = + templatesCompressed ?: expectsCompressedTemplates(database).also { templatesCompressed = it } + /** * Resolves a template id to the name the engine loads it by. * @@ -750,6 +792,9 @@ class DocumentationContentSource( // reused. codec = null codecStale = true + // The new database may store its templates the other way, and against its own dictionary: + // the compiled templates go with the cache clear below. + templatesCompressed = null clearTemplateCache() generation++ diff --git a/common/src/main/java/com/itsaky/androidide/utils/DatabaseVersionResolver.kt b/common/src/main/java/com/itsaky/androidide/utils/DatabaseVersionResolver.kt index 46f7d908ab..7ad62718e8 100644 --- a/common/src/main/java/com/itsaky/androidide/utils/DatabaseVersionResolver.kt +++ b/common/src/main/java/com/itsaky/androidide/utils/DatabaseVersionResolver.kt @@ -27,6 +27,13 @@ object DatabaseVersionResolver { // tooltip tables have the earlier shape, which this build neither reads nor writes. const val MAJOR_VERSION_WITH_KEYED_TOOLTIPS = 3 + // From this MAJOR version on (ADFA-3305), every Templates row is Brotli-compressed against + // CompressionDictionary, as brotli Content rows are. No row says so: the pipeline compresses + // every template when it builds the database, and nothing on the device writes Templates. + // Below it templates are plain UTF-8, and are still read that way: an app update never replaces + // the installed database, so a 2.x file has to keep rendering its templated pages. + const val MAJOR_VERSION_WITH_COMPRESSED_TEMPLATES = 3 + private const val QUERY_VERSION_TABLE_EXISTS = """ SELECT 1 FROM sqlite_master diff --git a/common/src/main/java/com/itsaky/androidide/utils/DocumentationCompression.kt b/common/src/main/java/com/itsaky/androidide/utils/DocumentationCompression.kt index 8eb43c5880..486c510716 100644 --- a/common/src/main/java/com/itsaky/androidide/utils/DocumentationCompression.kt +++ b/common/src/main/java/com/itsaky/androidide/utils/DocumentationCompression.kt @@ -126,6 +126,16 @@ fun expectsCompressionDictionary(db: SQLiteDatabase): Boolean { return majorVersion != null && majorVersion >= DatabaseVersionResolver.MAJOR_VERSION_WITH_COMPRESSION_DICTIONARY } +/** + * Whether [db] declares a version whose `Templates` rows are Brotli-compressed against its + * dictionary (see [DatabaseVersionResolver.MAJOR_VERSION_WITH_COMPRESSED_TEMPLATES]); below it, and + * for a database declaring none, they are plain UTF-8. + */ +fun expectsCompressedTemplates(db: SQLiteDatabase): Boolean { + val majorVersion = DatabaseVersionResolver.resolveMajorVersion(db) + return majorVersion != null && majorVersion >= DatabaseVersionResolver.MAJOR_VERSION_WITH_COMPRESSED_TEMPLATES +} + /** * Compresses and decompresses documentation `Content` blobs against [dictionary], the shared Brotli * dictionary from the database those blobs live in (see [loadCompressionDictionary]). A null diff --git a/common/src/test/java/com/itsaky/androidide/documentation/DatabaseTemplateLoaderTest.kt b/common/src/test/java/com/itsaky/androidide/documentation/DatabaseTemplateLoaderTest.kt index a44445357c..b78725b945 100644 --- a/common/src/test/java/com/itsaky/androidide/documentation/DatabaseTemplateLoaderTest.kt +++ b/common/src/test/java/com/itsaky/androidide/documentation/DatabaseTemplateLoaderTest.kt @@ -4,11 +4,14 @@ import android.database.Cursor import android.database.sqlite.SQLiteDatabase import android.database.sqlite.SQLiteException import com.google.common.truth.Truth.assertThat +import com.itsaky.androidide.logging.provider.IdeLogRouter import io.mockk.every import io.mockk.mockk import io.pebbletemplates.pebble.error.LoaderException import org.junit.Assert.assertThrows import org.junit.Test +import org.slf4j.event.Level +import java.io.IOException /** * Covers the loader that lets one template reference another (ADFA-5405): a name resolves to the @@ -34,7 +37,11 @@ class DatabaseTemplateLoaderTest { } } - private fun loader(database: SQLiteDatabase?) = DatabaseTemplateLoader { database } + // Rows here are stored as their text, so the decode is the identity unless a test says otherwise. + private fun loader( + database: SQLiteDatabase?, + decode: (SQLiteDatabase, ByteArray) -> ByteArray = { _, body -> body }, + ) = DatabaseTemplateLoader({ database }, decode) @Test fun `a name resolves to its template row`() { @@ -43,6 +50,50 @@ class DatabaseTemplateLoaderTest { assertThat(reader.readText()).isEqualTo("[nav]") } + // From documentation version 3 a row holds the template Brotli-compressed, so Pebble has to be + // handed what the decode returns, never the stored bytes. + @Test + fun `a row's bytes reach the engine through the decode, not as stored`() { + val database = database("nav.peb" to "]van[") + + val reader = loader(database) { _, body -> body.reversedArray() }.getReader("nav.peb") + + assertThat(reader.readText()).isEqualTo("[nav]") + } + + @Test + fun `a decode failure fails by name, as a loader failure`() { + // A Brotli decode failure is an IOException: not a RuntimeException, so it would escape a + // catch written for the database's own failures, and Pebble does not wrap it either. + val loader = loader(database("nav.peb" to "[nav]")) { _, _ -> throw IOException("Brotli decoder failed") } + + val thrown = assertThrows(LoaderException::class.java) { loader.getReader("nav.peb") } + + // pebbleMessage, not message: the content source passes that on, without the "(?:?)" suffix. + assertThat(thrown.pebbleMessage).isEqualTo("Cannot decode template 'nav.peb'") + } + + // The exception's message reaches HTTP responses, so it names no file; the log line is where the + // database is named, so a broken release file can be found from the log alone. + @Test + fun `a decode failure is logged with the template and the database`() { + val database = database("nav.peb" to "[nav]") + every { database.path } returns "/sdcard/Download/documentation.db" + val errors = mutableListOf() + val sink = IdeLogRouter.ExternalSink { level, _, message, _ -> if (level == Level.ERROR) errors += message } + IdeLogRouter.addSink(sink) + try { + val loader = loader(database) { _, _ -> throw IOException("Brotli decoder failed") } + assertThrows(LoaderException::class.java) { loader.getReader("nav.peb") } + } finally { + IdeLogRouter.removeSink(sink) + } + + assertThat(errors).hasSize(1) + assertThat(errors.single()).contains("'nav.peb'") + assertThat(errors.single()).contains("/sdcard/Download/documentation.db") + } + @Test fun `a name with no row fails, rather than resolving to itself`() { val loader = loader(database("nav.peb" to "[nav]")) diff --git a/common/src/test/java/com/itsaky/androidide/documentation/DocumentationContentSourceTest.kt b/common/src/test/java/com/itsaky/androidide/documentation/DocumentationContentSourceTest.kt index f5cad65a29..a90750ebd6 100644 --- a/common/src/test/java/com/itsaky/androidide/documentation/DocumentationContentSourceTest.kt +++ b/common/src/test/java/com/itsaky/androidide/documentation/DocumentationContentSourceTest.kt @@ -2,7 +2,9 @@ package com.itsaky.androidide.documentation import android.database.Cursor import android.database.sqlite.SQLiteDatabase +import android.database.sqlite.SQLiteException import com.google.common.truth.Truth.assertThat +import com.itsaky.androidide.utils.BrotliDictionaryCodec import io.mockk.every import io.mockk.mockk import io.mockk.mockkStatic @@ -15,6 +17,7 @@ import org.junit.Rule import org.junit.Test import org.junit.rules.TemporaryFolder import java.io.File +import java.io.IOException /** * Covers the pipeline both documentation transports read through (ADFA-5176): what a lookup reports, @@ -46,7 +49,15 @@ class DocumentationContentSourceTest { unmockkAll() } - private fun source(debugCheckIntervalMs: Long = 1_000) = DocumentationContentSource(installedFile, debugFile, debugCheckIntervalMs) + // The template rows these tests store are plain text, and :common's tests have no brotli4j native + // to compress them with, so the decode is the identity unless a test says otherwise. The Brotli + // decode itself is covered in :app, by CompressedTemplateTest. + private fun source( + debugCheckIntervalMs: Long = 1_000, + decode: (BrotliDictionaryCodec, ByteArray) -> ByteArray = { _, body -> body }, + ) = DocumentationContentSource(installedFile, debugFile, debugCheckIntervalMs, decode) + + private val reversing: (BrotliDictionaryCodec, ByteArray) -> ByteArray = { _, body -> body.reversedArray() } /** A Content row as the source's query sees it. */ private fun contentCursor( @@ -77,17 +88,7 @@ class DocumentationContentSourceTest { ): SQLiteDatabase = mockk(relaxed = true) { every { rawQuery(match { it.contains("FROM Content") }, any()) } returns contentCursor - every { - rawQuery(match { it.contains("FROM sqlite_master") && it.contains("DocumentationDatabaseVersion") }, any()) - } returns mockk(relaxed = true) { every { moveToFirst() } returns (declaredMajorVersion != null) } - if (declaredMajorVersion != null) { - every { rawQuery(match { it.contains("FROM DocumentationDatabaseVersion") }, any()) } returns - mockk(relaxed = true) { - every { moveToFirst() } returns true - every { isNull(0) } returns false - every { getInt(0) } returns declaredMajorVersion - } - } + declaresMajorVersion(declaredMajorVersion) every { rawQuery(match { it.contains("FROM sqlite_master") && it.contains("CompressionDictionary") }, null) } returns mockk(relaxed = true) { every { moveToFirst() } returns (dictionary != null) } @@ -98,6 +99,21 @@ class DocumentationContentSourceTest { } } + /** Stubs ADFA-5220's version table on this mock: declaring [major], or no table when null. */ + private fun SQLiteDatabase.declaresMajorVersion(major: Int?) { + every { + rawQuery(match { it.contains("FROM sqlite_master") && it.contains("DocumentationDatabaseVersion") }, any()) + } returns mockk(relaxed = true) { every { moveToFirst() } returns (major != null) } + if (major != null) { + every { rawQuery(match { it.contains("FROM DocumentationDatabaseVersion") }, any()) } returns + mockk(relaxed = true) { + every { moveToFirst() } returns true + every { isNull(0) } returns false + every { getInt(0) } returns major + } + } + } + // ADFA-5220: the dictionary is gated on the version the database declares, not on whether a // CompressionDictionary table happens to exist. These three cover the gate that WebServerTest // used to own before the decode moved into this class. @@ -470,13 +486,115 @@ class DocumentationContentSourceTest { .isEqualTo("Goodbye Kotlin!") } + // ADFA-3305: from documentation version 3 every template is Brotli-compressed; below it they are + // plain UTF-8, and still read that way, since an app update never replaces the installed file. + // The decode here reverses the bytes, so a test can tell whether a template went through it. + @Test + fun `a database below version 3 has its templates read as plain text, without the decode`() { + val database = templatedDatabase("page.peb" to "Hello {{ who }}!", declaredMajorVersion = 2) + every { SQLiteDatabase.openDatabase(any(), isNull(), any()) } returns database + + val lookup = source(decode = reversing).lookup("k/html/basic-syntax.html") + + assertThat((lookup as DocumentationLookup.Found).content.bytes.toString(Charsets.UTF_8)).isEqualTo("Hello Kotlin!") + } + + @Test + fun `a database declaring no version is read the same way`() { + val database = templatedDatabase("page.peb" to "Hello {{ who }}!", declaredMajorVersion = null) + every { SQLiteDatabase.openDatabase(any(), isNull(), any()) } returns database + + val lookup = source(decode = reversing).lookup("k/html/basic-syntax.html") + + assertThat((lookup as DocumentationLookup.Found).content.bytes.toString(Charsets.UTF_8)).isEqualTo("Hello Kotlin!") + } + + @Test + fun `a version 3 database has its templates decoded`() { + val database = templatedDatabase("page.peb" to "Hello {{ who }}!".reversed()) + every { SQLiteDatabase.openDatabase(any(), isNull(), any()) } returns database + + val lookup = source(decode = reversing).lookup("k/html/basic-syntax.html") + + assertThat((lookup as DocumentationLookup.Found).content.bytes.toString(Charsets.UTF_8)).isEqualTo("Hello Kotlin!") + } + + // Never rendered as whatever the bytes happen to be: the page fails, naming the template. + @Test + fun `a version 3 template that does not decode fails its page, naming the template`() { + val database = templatedDatabase("page.peb" to "Hello {{ who }}!") + every { SQLiteDatabase.openDatabase(any(), isNull(), any()) } returns database + + val lookup = source(decode = { _, _ -> throw IOException("Brotli decoder failed") }).lookup("p") + + assertThat(lookup).isInstanceOf(DocumentationLookup.Failed::class.java) + assertThat((lookup as DocumentationLookup.Failed).cause).isInstanceOf(TemplateRenderException::class.java) + assertThat(lookup.cause).hasMessageThat().isEqualTo("Cannot decode template 'page.peb'") + } + + // The version is read before the engine runs. Read inside the loader instead, a database error + // would arrive as a template that does not decode -- the wrong diagnosis, and a template failure, + // whose message both transports send. + @Test + fun `a failure to read the version is reported as itself, not as a template that does not decode`() { + val database = templatedDatabase("page.peb" to "Hello {{ who }}!") + every { database.rawQuery(match { it.contains("FROM DocumentationDatabaseVersion") }, any()) } throws + SQLiteException("disk I/O error (code 10)") + every { SQLiteDatabase.openDatabase(any(), isNull(), any()) } returns database + + val lookup = source(decode = reversing).lookup("p") as DocumentationLookup.Failed + + assertThat(lookup.cause).isInstanceOf(SQLiteException::class.java) + } + + @Test + fun `a swap from a version 2 file to a version 3 file decodes the new one's templates`() { + val installed = templatedDatabase("page.peb" to "Hello {{ who }}!", declaredMajorVersion = 2) + val debug = templatedDatabase("page.peb" to "Goodbye {{ who }}!".reversed()) + every { SQLiteDatabase.openDatabase(installedFile.absolutePath, isNull(), any()) } returns installed + every { SQLiteDatabase.openDatabase(debugFile.absolutePath, isNull(), any()) } returns debug + val source = source(debugCheckIntervalMs = 0, decode = reversing) + + assertThat((source.lookup("p") as DocumentationLookup.Found).content.bytes.toString(Charsets.UTF_8)) + .isEqualTo("Hello Kotlin!") + + debugFile.writeText("newer") + debugFile.setLastModified(installedFile.lastModified() + 60_000) + + assertThat((source.lookup("p") as DocumentationLookup.Found).content.bytes.toString(Charsets.UTF_8)) + .isEqualTo("Goodbye Kotlin!") + } + + @Test + fun `a swap from a version 3 file to a version 2 file reads the new one's templates as plain text`() { + val installed = templatedDatabase("page.peb" to "Hello {{ who }}!".reversed()) + val debug = templatedDatabase("page.peb" to "Goodbye {{ who }}!", declaredMajorVersion = 2) + every { SQLiteDatabase.openDatabase(installedFile.absolutePath, isNull(), any()) } returns installed + every { SQLiteDatabase.openDatabase(debugFile.absolutePath, isNull(), any()) } returns debug + val source = source(debugCheckIntervalMs = 0, decode = reversing) + + assertThat((source.lookup("p") as DocumentationLookup.Found).content.bytes.toString(Charsets.UTF_8)) + .isEqualTo("Hello Kotlin!") + + debugFile.writeText("newer") + debugFile.setLastModified(installedFile.lastModified() + 60_000) + + assertThat((source.lookup("p") as DocumentationLookup.Found).content.bytes.toString(Charsets.UTF_8)) + .isEqualTo("Goodbye Kotlin!") + } + /** * A database whose single Content row is templated with id 7, and whose `Templates` rows are * [templates] as name-to-body pairs. Template id 7 is the first pair; the rest are reachable - * only by name, which is how one template references another. + * only by name, which is how one template references another. It declares + * [declaredMajorVersion], by default 3, the first whose templates this build reads. */ - private fun templatedDatabase(vararg templates: Pair): SQLiteDatabase = + private fun templatedDatabase( + vararg templates: Pair, + declaredMajorVersion: Int? = 3, + ): SQLiteDatabase = mockk(relaxed = true) { + declaresMajorVersion(declaredMajorVersion) every { rawQuery(match { it.contains("FROM Content") }, any()) } returns contentCursor(bytes = """{"who": "Kotlin"}""".toByteArray(), templateId = 7) every { rawQuery(match { it.contains("WHERE id = ?") }, arrayOf("7")) } returns From 2bb16a676d6f0c3e7c87b051faa859d0fcbfbd2c Mon Sep 17 00:00:00 2001 From: David Schachter Date: Mon, 19 Oct 2026 00:03:51 -0700 Subject: [PATCH 11/13] ADFA-3305: Describe compressed templates in documentation-database.md Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3 --- docs/documentation-database.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/documentation-database.md b/docs/documentation-database.md index dab0804000..20e955387e 100644 --- a/docs/documentation-database.md +++ b/docs/documentation-database.md @@ -81,9 +81,9 @@ CREATE TABLE TooltipButtons ( ### Supporting tables -- **`DocumentationDatabaseVersion(major, minor, patch, who, comment, changeTime)`** — the database's own semver (ADFA-5220), replacing the heuristics that used to infer the format from which tables happened to exist. **Exactly one row**: the version the file *is*, which `populate_db.py` replaces rather than appends. Nothing in the DDL enforces that, so both readers (`DatabaseVersionResolver.resolveMajorVersion` here, `database_major_version` in docdb-studio) take the newest row rather than the one with the highest `major` (the app orders by `changeTime DESC, rowid DESC`, so the greatest `changeTime` wins and `rowid` only breaks ties) — a rebuild from an older content set is a downgrade and has to read as one, which `MAX(major)` would get wrong. The app's reader additionally logs a warning when it finds more than one row; docdb-studio's does not. `populate_db.py` collapses a file that somehow accumulated several back to one, and `resolveMajorVersion` returns null for a database predating the table. `MAJOR >= 3` says the tooltip tables have the shape above. `MAJOR >= 2` is what tells the app its brotli `Content` rows are dictionary-compressed; below that, `DocumentationContentSource` neither reads nor attaches `CompressionDictionary`. Gating on the declared version rather than on the table's presence matters in both directions: a database can carry the dictionary table while its content is still plain brotli (every brotli row would then be decoded with a dictionary attached and fail outright, since ADFA-5240 removed the plain-decode retry that used to mask this), and a migrated database that lost the table fails loudly instead of quietly. +- **`DocumentationDatabaseVersion(major, minor, patch, who, comment, changeTime)`** — the database's own semver (ADFA-5220), replacing the heuristics that used to infer the format from which tables happened to exist. **Exactly one row**: the version the file *is*, which `populate_db.py` replaces rather than appends. Nothing in the DDL enforces that, so both readers (`DatabaseVersionResolver.resolveMajorVersion` here, `database_major_version` in docdb-studio) take the newest row rather than the one with the highest `major` (the app orders by `changeTime DESC, rowid DESC`, so the greatest `changeTime` wins and `rowid` only breaks ties) — a rebuild from an older content set is a downgrade and has to read as one, which `MAX(major)` would get wrong. The app's reader additionally logs a warning when it finds more than one row; docdb-studio's does not. `populate_db.py` collapses a file that somehow accumulated several back to one, and `resolveMajorVersion` returns null for a database predating the table. `MAJOR >= 3` says the tooltip tables have the shape above and every template is compressed (see `Templates` below). `MAJOR >= 2` is what tells the app its brotli `Content` rows are dictionary-compressed; below that, `DocumentationContentSource` neither reads nor attaches `CompressionDictionary`. Gating on the declared version rather than on the table's presence matters in both directions: a database can carry the dictionary table while its content is still plain brotli (every brotli row would then be decoded with a dictionary attached and fail outright, since ADFA-5240 removed the plain-decode retry that used to mask this), and a migrated database that lost the table fails loudly instead of quietly. - **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. Both the reader and the writer load it through `loadCompressionDictionary` in `:common`, so they cannot disagree about whether a given database's brotli rows carry a dictionary — a row compressed against one the reader won't attach is undecodable, and vice versa. `DocumentationContentSource` loads it lazily -- not at database open or swap time, but on the first content lookup after the active database changes, whatever that row's own compression, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Each row is then decoded exactly once, with the dictionary attached or not according to that version; there is no retry, because there is no longer a second way a row might have been written. -- **`Templates(id, name, content)`** — Pebble template source. The DDL is `id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT NOT NULL, content BLOB NOT NULL, UNIQUE('name')` — worth stating, because the single-quoted `UNIQUE('name')` reads like a constant rather than a column and has twice been taken for an absent constraint. SQLite resolves the quoted string to the column, so a duplicate name, a null name and a null body are all rejected by the database, and readers here do not re-check them. `Content.templateId` names a page's outermost template by id; every reference *between* templates is by `name` (ADFA-5405), which is also how `WebServer` reaches well-known templates like `bookshelf`. So a template can `extends`, `include`, `import` or `embed` another one by writing its `Templates.name` verbatim (`{% include "nav.peb" %}`) — the table is a flat namespace, with no directories, prefixes or suffixes applied. A reference to a name with no row fails the request rather than rendering the name; before ADFA-5405 the latter is exactly what happened, silently, because the engine's loader treated the name it was handed as the template body. +- **`Templates(id, name, content)`** — Pebble template source. The DDL is `id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT NOT NULL, content BLOB NOT NULL, UNIQUE('name')` — worth stating, because the single-quoted `UNIQUE('name')` reads like a constant rather than a column and has twice been taken for an absent constraint. SQLite resolves the quoted string to the column, so a duplicate name, a null name and a null body are all rejected by the database, and readers here do not re-check them. `Content.templateId` names a page's outermost template by id; every reference *between* templates is by `name` (ADFA-5405), which is also how `WebServer` reaches well-known templates like `bookshelf`. So a template can `extends`, `include`, `import` or `embed` another one by writing its `Templates.name` verbatim (`{% include "nav.peb" %}`) — the table is a flat namespace, with no directories, prefixes or suffixes applied. A reference to a name with no row fails the request rather than rendering the name; before ADFA-5405 the latter is exactly what happened, silently, because the engine's loader treated the name it was handed as the template body. From documentation version 3 (ADFA-3305), every `content` is the template's UTF-8 text Brotli-compressed against `CompressionDictionary`, like a brotli `Content` row. No column says so: `OfflineDocumentationTools` compresses every template when it builds the database for release, and nothing on the device writes `Templates`. Below 3, and in a database declaring no version, `content` is plain UTF-8 and is read as it is (`expectsCompressedTemplates`): an app update never replaces the installed database, so a 2.x file has to keep rendering its templated pages. In a 3.x file a template that does not decode fails the page that uses it, logged with the template's name and the database's path. - **`Bookshelf(contentID, bookCategoryID, title, description)`** / **`BookCategories(id, category, description)`** — the Dynamic Bookshelf: one row per "book" (PDF or similar), linked to its Tier 3 page via `contentID` -> `Content.id`. Two DB triggers keep `Bookshelf` in sync when a PDF row is inserted/deleted from `Content`; `title`/`description` don't come from those triggers and must be set by hand. Non-PDF books need a separate ingestion path (plugin-provided, e.g. via `PluginDocumentationManager`). - **`LastChange(documentationSet, changeTime, who)`** — audit trail for edits made through `docdb-studio`; not shown to end users. `DatabaseVersionResolver` reads the `documentationSet = 'wholedb'` row to report the DB's build/edit stamp in debug logging, falling back to the most recent row of any set if `'wholedb'` is missing. - Misc `ide_tooltip_table` and `PUCC` tables are historical/example artifacts — not part of the live lookup paths above. @@ -92,7 +92,7 @@ CREATE TABLE TooltipButtons ( Of the five sites below, only `DocumentationContentSource` and `ToolTipManager` open the file, with `SQLiteDatabase.openDatabase(..., OPEN_READONLY)` — `WebServer` and the interceptor read through the content source, and `PluginDocumentationManager` is the one write path, opening `OPEN_READWRITE` to install plugin content (see ADR 0001 for why raw SQLite is justified here instead of Room). -- **`common/.../documentation/DocumentationContentSource.kt`** — the one Tier 3 pipeline that reads this database: row lookup (a request target is matched raw first, since stored `Content.path` values are percent-encoded, with a percent-decoded fallback on a miss — shared by both transports so they agree on which pages exist), chunked-row reassembly, dictionary-aware Brotli decode gated on the declared documentation version, and the database swaps — to a newer sdcard debug database, and reopening the installed file when an asset install rewrites it in place under the cached handle — under a read/write lock so several threads can read while a swap cannot close the handle under them. It also renders the rows that are Pebble template contexts (`templateId > 0`, the Kotlin doc set's pages) — resolving each template's references to other templates against `Templates.name` as it goes — so both transports below serve finished pages and neither needs the template engine itself. All of that logic exists once (ADFA-5176). +- **`common/.../documentation/DocumentationContentSource.kt`** — the one Tier 3 pipeline that reads this database: row lookup (a request target is matched raw first, since stored `Content.path` values are percent-encoded, with a percent-decoded fallback on a miss — shared by both transports so they agree on which pages exist), chunked-row reassembly, dictionary-aware Brotli decode gated on the declared documentation version, and the database swaps — to a newer sdcard debug database, and reopening the installed file when an asset install rewrites it in place under the cached handle — under a read/write lock so several threads can read while a swap cannot close the handle under them. It also renders the rows that are Pebble template contexts (`templateId > 0`, the Kotlin doc set's pages) — resolving each template's references to other templates against `Templates.name` as it goes, and decoding each template with the same dictionary as the rows — so both transports below serve finished pages and neither needs the template engine itself. All of that logic exists once (ADFA-5176). - **`common/.../documentation/DocumentationRequestInterceptor.kt`** — serves Tier 3 *in-process* for the app's WebViews (`HelpActivity`, the tooltip fragment, `FAQActivity`), through `WebViewClient.shouldInterceptRequest`, so a page's assets cost a database read instead of a TCP connection each (ADFA-5176). It matches the same `http://localhost:6174/...` URL space, so the strings.xml entries, `ToolTipManager`'s link builder and the `DocumentationExtension` contract need no changes; anything it declines — a `/pr/` endpoint, an unknown path, a failed read — falls through to `WebServer` unchanged. Both transports get their `Content-Type` charset from the same place, `ContentTypeHeaders` (ADFA-5241), so a row does not describe itself differently depending on which one served it. Set `/sdcard/Download/CodeOnTheGo.nointercept` to force documentation back onto the server. - **`app/.../localWebServer/WebServer.kt`** — serves Tier 3 over HTTP on port 6174, for WebViews that are not wired to the interceptor above and for the `/pr/` developer endpoints. It reads through `DocumentationContentSource`, so it holds no database, template engine or decode logic of its own; what remains here is HTTP: request parsing, the `/pr/` pages, error responses, and the CSS/asset shortcuts. Also serves a Dynamic Bookshelf JSON payload (joining `Content`/`Bookshelf`/`BookCategories`, rendered through the `bookshelf` template) and debug-only HTML dumps at `/pr/db` (`LastChange`, last 20 rows) and `/pr/pr` (recent projects, from a *different* database). - **`idetooltips/.../ToolTipManager.kt`** — serves Tier 1/2, through `readTooltip` in `TooltipQuery.kt`, which runs `queryTooltip` and `queryTooltipButtons` and builds the item shown. Looks up a tooltip by `(category, tag)`: the category's id from `TooltipCategories` first, so the row is found by `Tooltips`' key rather than a scan of the whole table (ADFA-6597), then its text by id. Then its buttons for the Tier 3 links shown at the bottom, by `TooltipButtons`' key, with their text by id. Refuses a database declaring a version below 3. From 119f5ed99a9a470f4bd01e92c3a3c742712880c2 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Mon, 19 Oct 2026 09:23:15 -0700 Subject: [PATCH 12/13] ADFA-3305: Test that templates' non-ASCII text renders as UTF-8 640 of the 3.0.0 build's 1,126 templates hold non-ASCII text, most of them U+00A0, but every template in the tests was ASCII: reading the decoded bytes as ISO-8859-1 at DatabaseTemplateLoader.kt:105 left all 66 :common documentation tests and 62 :app localWebServer tests passing, while a real page would show "A-circumflex" before each no-break space. Six tests now render text with U+00A0, U+2014 and U+2026, plain and decoded: two in the loader's tests, two through DocumentationContentSource with a stand-in decode, and two in :app through the real Brotli decode. The page tests compare bytes. All six fail with ISO_8859_1 at line 105 (the one line both paths share), and the four page tests also fail with the rendered output encoded as ISO_8859_1. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3 --- .../localWebServer/CompressedTemplateTest.kt | 32 ++++++++++++++++-- .../DatabaseTemplateLoaderTest.kt | 21 ++++++++++++ .../DocumentationContentSourceTest.kt | 33 +++++++++++++++++++ 3 files changed, 84 insertions(+), 2 deletions(-) diff --git a/app/src/test/java/com/itsaky/androidide/localWebServer/CompressedTemplateTest.kt b/app/src/test/java/com/itsaky/androidide/localWebServer/CompressedTemplateTest.kt index cfd3dcd130..9c866c4d90 100644 --- a/app/src/test/java/com/itsaky/androidide/localWebServer/CompressedTemplateTest.kt +++ b/app/src/test/java/com/itsaky/androidide/localWebServer/CompressedTemplateTest.kt @@ -12,6 +12,7 @@ import io.mockk.mockk import io.mockk.mockkStatic import io.mockk.unmockkAll import org.junit.After +import org.junit.Assert.assertArrayEquals import org.junit.Assert.assertEquals import org.junit.Assert.assertThrows import org.junit.Assert.assertTrue @@ -59,6 +60,28 @@ class CompressedTemplateTest { assertEquals(page("kotlin").replace("{{ who }}", "Kotlin"), render(source())) } + // 640 of the 3.0.0 build's 1,126 templates hold non-ASCII text, most of them U+00A0. Compared as + // the page's bytes, so a template decoded to text in any charset but UTF-8 fails here. + @Test + fun `a compressed template's non-ASCII text renders as UTF-8`() { + val dictionary = dictionary("kotlin") + val template = "$NON_ASCII ${page("kotlin")}" + every { SQLiteDatabase.openDatabase(any(), isNull(), any()) } returns + database(dictionary, "page.peb" to stored(template, dictionary)) + + assertArrayEquals(template.replace("{{ who }}", "Kotlin").toByteArray(Charsets.UTF_8), renderedBytes(source())) + } + + @Test + fun `a version 2 database's plain non-ASCII template renders as UTF-8`() { + val dictionary = dictionary("kotlin") + val template = "$NON_ASCII ${page("kotlin")}" + every { SQLiteDatabase.openDatabase(any(), isNull(), any()) } returns + database(dictionary, "page.peb" to template.toByteArray(Charsets.UTF_8), declaredMajorVersion = 2) + + assertArrayEquals(template.replace("{{ who }}", "Kotlin").toByteArray(Charsets.UTF_8), renderedBytes(source())) + } + @Test fun `an included template is decoded too`() { val dictionary = dictionary("kotlin") @@ -166,8 +189,10 @@ class CompressedTemplateTest { debugFile.setLastModified(installedFile.lastModified() + 60_000) } - private fun render(source: DocumentationContentSource): String = - (source.lookup("k/html/basic-syntax.html") as DocumentationLookup.Found).content.bytes.toString(Charsets.UTF_8) + private fun render(source: DocumentationContentSource): String = renderedBytes(source).toString(Charsets.UTF_8) + + private fun renderedBytes(source: DocumentationContentSource): ByteArray = + (source.lookup("k/html/basic-syntax.html") as DocumentationLookup.Found).content.bytes private fun dictionary(word: String) = "$SHARED $word interface companion object ".repeat(400) @@ -243,5 +268,8 @@ class CompressedTemplateTest { const val SHARED = """

docs-sidebar toc-element""" const val DATABASE_PATH = "/sdcard/Download/documentation.db" + + // A no-break space (in Latin-1) and an em dash and an ellipsis (outside it), as real templates hold. + const val NON_ASCII = "a b — c…" } } diff --git a/common/src/test/java/com/itsaky/androidide/documentation/DatabaseTemplateLoaderTest.kt b/common/src/test/java/com/itsaky/androidide/documentation/DatabaseTemplateLoaderTest.kt index b78725b945..b6bc6a7b0b 100644 --- a/common/src/test/java/com/itsaky/androidide/documentation/DatabaseTemplateLoaderTest.kt +++ b/common/src/test/java/com/itsaky/androidide/documentation/DatabaseTemplateLoaderTest.kt @@ -61,6 +61,22 @@ class DatabaseTemplateLoaderTest { assertThat(reader.readText()).isEqualTo("[nav]") } + // Templates hold non-ASCII text: 640 of the 3.0.0 build's 1,126 do, most of them U+00A0. Read in + // any charset but UTF-8, a no-break space reaches the page as "Â" plus a no-break space. + @Test + fun `a plain row's non-ASCII text reaches the engine as UTF-8`() { + val reader = loader(database("nav.peb" to NON_ASCII)).getReader("nav.peb") + + assertThat(reader.readText()).isEqualTo(NON_ASCII) + } + + @Test + fun `a decoded row's non-ASCII text reaches the engine as UTF-8`() { + val loader = loader(database("nav.peb" to "[nav]")) { _, _ -> NON_ASCII.toByteArray(Charsets.UTF_8) } + + assertThat(loader.getReader("nav.peb").readText()).isEqualTo(NON_ASCII) + } + @Test fun `a decode failure fails by name, as a loader failure`() { // A Brotli decode failure is an IOException: not a RuntimeException, so it would escape a @@ -218,4 +234,9 @@ class DatabaseTemplateLoaderTest { assertThat(loader(database).resourceExists("nav.peb")).isFalse() } + + private companion object { + // A no-break space (in Latin-1) and an em dash and an ellipsis (outside it), as real templates hold. + const val NON_ASCII = "a b — c…" + } } diff --git a/common/src/test/java/com/itsaky/androidide/documentation/DocumentationContentSourceTest.kt b/common/src/test/java/com/itsaky/androidide/documentation/DocumentationContentSourceTest.kt index a90750ebd6..edb0b25bd5 100644 --- a/common/src/test/java/com/itsaky/androidide/documentation/DocumentationContentSourceTest.kt +++ b/common/src/test/java/com/itsaky/androidide/documentation/DocumentationContentSourceTest.kt @@ -59,6 +59,10 @@ class DocumentationContentSourceTest { private val reversing: (BrotliDictionaryCodec, ByteArray) -> ByteArray = { _, body -> body.reversedArray() } + // For non-ASCII text, whose UTF-8 bytes reversed are not UTF-8: reverses the characters instead. + private val reversingChars: (BrotliDictionaryCodec, ByteArray) -> ByteArray = + { _, body -> body.toString(Charsets.UTF_8).reversed().toByteArray(Charsets.UTF_8) } + /** A Content row as the source's query sees it. */ private fun contentCursor( bytes: ByteArray = "hello".toByteArray(), @@ -519,6 +523,30 @@ class DocumentationContentSourceTest { assertThat((lookup as DocumentationLookup.Found).content.bytes.toString(Charsets.UTF_8)).isEqualTo("Hello Kotlin!") } + // Most real templates hold non-ASCII text, U+00A0 above all. Compared as the page's bytes, so a + // template read in any charset but UTF-8, plain or decoded, fails here. + @Test + fun `a version 2 template's non-ASCII text renders as UTF-8`() { + val database = templatedDatabase("page.peb" to "$NON_ASCII {{ who }}", declaredMajorVersion = 2) + every { SQLiteDatabase.openDatabase(any(), isNull(), any()) } returns database + + val lookup = source(decode = reversing).lookup("k/html/basic-syntax.html") + + assertThat((lookup as DocumentationLookup.Found).content.bytes) + .isEqualTo("$NON_ASCII Kotlin".toByteArray(Charsets.UTF_8)) + } + + @Test + fun `a version 3 template's non-ASCII text renders as UTF-8`() { + val database = templatedDatabase("page.peb" to "$NON_ASCII {{ who }}".reversed()) + every { SQLiteDatabase.openDatabase(any(), isNull(), any()) } returns database + + val lookup = source(decode = reversingChars).lookup("k/html/basic-syntax.html") + + assertThat((lookup as DocumentationLookup.Found).content.bytes) + .isEqualTo("$NON_ASCII Kotlin".toByteArray(Charsets.UTF_8)) + } + // Never rendered as whatever the bytes happen to be: the page fails, naming the template. @Test fun `a version 3 template that does not decode fails its page, naming the template`() { @@ -789,4 +817,9 @@ class DocumentationContentSourceTest { assertThat(source.lookup("p")).isEqualTo(DocumentationLookup.NotFound) verify(exactly = 1) { SQLiteDatabase.openDatabase(any(), isNull(), any()) } } + + private companion object { + // A no-break space (in Latin-1) and an em dash and an ellipsis (outside it), as real templates hold. + const val NON_ASCII = "a b — c…" + } } From f2178d7b39272438c0a7a442e9ea97e355b7b710 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Mon, 19 Oct 2026 09:24:23 -0700 Subject: [PATCH 13/13] ADFA-3305: Say the dictionary is re-minted, and which pages are templated documentation-database.md said CompressionDictionary was "trained once ... never retrained", but OfflineDocumentationTools' remint_dictionary.py (on its main since ADFA-5153) retrains it and recompresses every brotli Content row, and from version 3 every template too (ODT #48). Now that templates depend on the dictionary, the rule a reader needs is that a re-mint is safe only when it carries every row and template across. The content source's entry called the templated rows "the Kotlin doc set's pages"; a 2.x database also templates the i/ pages, and the 3.0.0 build the a/ pages. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3 --- docs/documentation-database.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/documentation-database.md b/docs/documentation-database.md index 20e955387e..aa642e13d3 100644 --- a/docs/documentation-database.md +++ b/docs/documentation-database.md @@ -82,7 +82,7 @@ CREATE TABLE TooltipButtons ( ### Supporting tables - **`DocumentationDatabaseVersion(major, minor, patch, who, comment, changeTime)`** — the database's own semver (ADFA-5220), replacing the heuristics that used to infer the format from which tables happened to exist. **Exactly one row**: the version the file *is*, which `populate_db.py` replaces rather than appends. Nothing in the DDL enforces that, so both readers (`DatabaseVersionResolver.resolveMajorVersion` here, `database_major_version` in docdb-studio) take the newest row rather than the one with the highest `major` (the app orders by `changeTime DESC, rowid DESC`, so the greatest `changeTime` wins and `rowid` only breaks ties) — a rebuild from an older content set is a downgrade and has to read as one, which `MAX(major)` would get wrong. The app's reader additionally logs a warning when it finds more than one row; docdb-studio's does not. `populate_db.py` collapses a file that somehow accumulated several back to one, and `resolveMajorVersion` returns null for a database predating the table. `MAJOR >= 3` says the tooltip tables have the shape above and every template is compressed (see `Templates` below). `MAJOR >= 2` is what tells the app its brotli `Content` rows are dictionary-compressed; below that, `DocumentationContentSource` neither reads nor attaches `CompressionDictionary`. Gating on the declared version rather than on the table's presence matters in both directions: a database can carry the dictionary table while its content is still plain brotli (every brotli row would then be decoded with a dictionary attached and fail outright, since ADFA-5240 removed the plain-decode retry that used to mask this), and a migrated database that lost the table fails loudly instead of quietly. -- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. Both the reader and the writer load it through `loadCompressionDictionary` in `:common`, so they cannot disagree about whether a given database's brotli rows carry a dictionary — a row compressed against one the reader won't attach is undecodable, and vice versa. `DocumentationContentSource` loads it lazily -- not at database open or swap time, but on the first content lookup after the active database changes, whatever that row's own compression, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Each row is then decoded exactly once, with the dictionary attached or not according to that version; there is no retry, because there is no longer a second way a row might have been written. +- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. First trained from a representative sample across the whole `Content` table by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py`, and retrained since by its `remint_dictionary.py`, which recompresses every brotli `Content` row against the new dictionary in the same transaction (from documentation version 3, every template as well: OfflineDocumentationTools #48). A dictionary-compressed row or template is decodable only against the exact dictionary it was compressed with, so a re-mint is safe only when it carries every one of them across; replacing the dictionary alone would silently orphan them all. Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. Both the reader and the writer load it through `loadCompressionDictionary` in `:common`, so they cannot disagree about whether a given database's brotli rows carry a dictionary — a row compressed against one the reader won't attach is undecodable, and vice versa. `DocumentationContentSource` loads it lazily -- not at database open or swap time, but on the first content lookup after the active database changes, whatever that row's own compression, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Each row is then decoded exactly once, with the dictionary attached or not according to that version; there is no retry, because there is no longer a second way a row might have been written. - **`Templates(id, name, content)`** — Pebble template source. The DDL is `id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT NOT NULL, content BLOB NOT NULL, UNIQUE('name')` — worth stating, because the single-quoted `UNIQUE('name')` reads like a constant rather than a column and has twice been taken for an absent constraint. SQLite resolves the quoted string to the column, so a duplicate name, a null name and a null body are all rejected by the database, and readers here do not re-check them. `Content.templateId` names a page's outermost template by id; every reference *between* templates is by `name` (ADFA-5405), which is also how `WebServer` reaches well-known templates like `bookshelf`. So a template can `extends`, `include`, `import` or `embed` another one by writing its `Templates.name` verbatim (`{% include "nav.peb" %}`) — the table is a flat namespace, with no directories, prefixes or suffixes applied. A reference to a name with no row fails the request rather than rendering the name; before ADFA-5405 the latter is exactly what happened, silently, because the engine's loader treated the name it was handed as the template body. From documentation version 3 (ADFA-3305), every `content` is the template's UTF-8 text Brotli-compressed against `CompressionDictionary`, like a brotli `Content` row. No column says so: `OfflineDocumentationTools` compresses every template when it builds the database for release, and nothing on the device writes `Templates`. Below 3, and in a database declaring no version, `content` is plain UTF-8 and is read as it is (`expectsCompressedTemplates`): an app update never replaces the installed database, so a 2.x file has to keep rendering its templated pages. In a 3.x file a template that does not decode fails the page that uses it, logged with the template's name and the database's path. - **`Bookshelf(contentID, bookCategoryID, title, description)`** / **`BookCategories(id, category, description)`** — the Dynamic Bookshelf: one row per "book" (PDF or similar), linked to its Tier 3 page via `contentID` -> `Content.id`. Two DB triggers keep `Bookshelf` in sync when a PDF row is inserted/deleted from `Content`; `title`/`description` don't come from those triggers and must be set by hand. Non-PDF books need a separate ingestion path (plugin-provided, e.g. via `PluginDocumentationManager`). - **`LastChange(documentationSet, changeTime, who)`** — audit trail for edits made through `docdb-studio`; not shown to end users. `DatabaseVersionResolver` reads the `documentationSet = 'wholedb'` row to report the DB's build/edit stamp in debug logging, falling back to the most recent row of any set if `'wholedb'` is missing. @@ -92,7 +92,7 @@ CREATE TABLE TooltipButtons ( Of the five sites below, only `DocumentationContentSource` and `ToolTipManager` open the file, with `SQLiteDatabase.openDatabase(..., OPEN_READONLY)` — `WebServer` and the interceptor read through the content source, and `PluginDocumentationManager` is the one write path, opening `OPEN_READWRITE` to install plugin content (see ADR 0001 for why raw SQLite is justified here instead of Room). -- **`common/.../documentation/DocumentationContentSource.kt`** — the one Tier 3 pipeline that reads this database: row lookup (a request target is matched raw first, since stored `Content.path` values are percent-encoded, with a percent-decoded fallback on a miss — shared by both transports so they agree on which pages exist), chunked-row reassembly, dictionary-aware Brotli decode gated on the declared documentation version, and the database swaps — to a newer sdcard debug database, and reopening the installed file when an asset install rewrites it in place under the cached handle — under a read/write lock so several threads can read while a swap cannot close the handle under them. It also renders the rows that are Pebble template contexts (`templateId > 0`, the Kotlin doc set's pages) — resolving each template's references to other templates against `Templates.name` as it goes, and decoding each template with the same dictionary as the rows — so both transports below serve finished pages and neither needs the template engine itself. All of that logic exists once (ADFA-5176). +- **`common/.../documentation/DocumentationContentSource.kt`** — the one Tier 3 pipeline that reads this database: row lookup (a request target is matched raw first, since stored `Content.path` values are percent-encoded, with a percent-decoded fallback on a miss — shared by both transports so they agree on which pages exist), chunked-row reassembly, dictionary-aware Brotli decode gated on the declared documentation version, and the database swaps — to a newer sdcard debug database, and reopening the installed file when an asset install rewrites it in place under the cached handle — under a read/write lock so several threads can read while a swap cannot close the handle under them. It also renders the rows that are Pebble template contexts (`templateId > 0`: the `k/` and `i/` pages in a 2.x database, and the `a/` pages too in a 3.x one) — resolving each template's references to other templates against `Templates.name` as it goes, and decoding each template with the same dictionary as the rows — so both transports below serve finished pages and neither needs the template engine itself. All of that logic exists once (ADFA-5176). - **`common/.../documentation/DocumentationRequestInterceptor.kt`** — serves Tier 3 *in-process* for the app's WebViews (`HelpActivity`, the tooltip fragment, `FAQActivity`), through `WebViewClient.shouldInterceptRequest`, so a page's assets cost a database read instead of a TCP connection each (ADFA-5176). It matches the same `http://localhost:6174/...` URL space, so the strings.xml entries, `ToolTipManager`'s link builder and the `DocumentationExtension` contract need no changes; anything it declines — a `/pr/` endpoint, an unknown path, a failed read — falls through to `WebServer` unchanged. Both transports get their `Content-Type` charset from the same place, `ContentTypeHeaders` (ADFA-5241), so a row does not describe itself differently depending on which one served it. Set `/sdcard/Download/CodeOnTheGo.nointercept` to force documentation back onto the server. - **`app/.../localWebServer/WebServer.kt`** — serves Tier 3 over HTTP on port 6174, for WebViews that are not wired to the interceptor above and for the `/pr/` developer endpoints. It reads through `DocumentationContentSource`, so it holds no database, template engine or decode logic of its own; what remains here is HTTP: request parsing, the `/pr/` pages, error responses, and the CSS/asset shortcuts. Also serves a Dynamic Bookshelf JSON payload (joining `Content`/`Bookshelf`/`BookCategories`, rendered through the `bookshelf` template) and debug-only HTML dumps at `/pr/db` (`LastChange`, last 20 rows) and `/pr/pr` (recent projects, from a *different* database). - **`idetooltips/.../ToolTipManager.kt`** — serves Tier 1/2, through `readTooltip` in `TooltipQuery.kt`, which runs `queryTooltip` and `queryTooltipButtons` and builds the item shown. Looks up a tooltip by `(category, tag)`: the category's id from `TooltipCategories` first, so the row is found by `Tooltips`' key rather than a scan of the whole table (ADFA-6597), then its text by id. Then its buttons for the Tier 3 links shown at the bottom, by `TooltipButtons`' key, with their text by id. Refuses a database declaring a version below 3.