Repository navigation
perf(rows): index cell values by column for number, selection, datetime and usergroup - #59
rjzondervan wants to merge 5 commits into
Conversation
…me and usergroup Filters compared every value of a column, because the cell tables only had a (column_id, row_id) index. The number, selection, datetime and usergroup cell tables now get a (column_id, value) index: through occ db:add-missing-indices on existing installations, and through the migrations that create these tables on new ones. - selection, datetime and usergroup values are text: MySQL and MariaDB index their first 64 characters, which covers option ids, dates and user, group and team ids; Oracle stores them as CLOB and cannot index them, so they are left out there - text cells stay unindexed, as long texts cannot be indexed on every database - the existing (column_id, row_id) indices are left as they are Refs nextcloud#3092 Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Robert Zondervan <robert@conduction.nl>
…indices Signed-off-by: Robert Zondervan <robert@conduction.nl>
…indices Signed-off-by: Robert Zondervan <robert@conduction.nl>
rjzondervan
left a comment
There was a problem hiding this comment.
Inline findings for this round; the verdict follows as a separate comment.
There was a problem hiding this comment.
Verdict: APPROVE (Thorough) — self-review posted as COMMENT (GitHub blocks self-APPROVE)
No blockers: the indices are correct on every install and upgrade path. All four findings were fixed on this branch in 9fe3687 and 73014e0; nothing is left open from this review.
🟡 Concerns (2)
- ✅ fixed in
9fe3687— "Covers selection option ids entirely" is false for multi-selection —lib/Listener/AddMissingIndicesListener.php:30. Multi-selection storesjson_encode()of the id array (RowCellSelectionMapper.php:50), which passes 64 characters at about 15–20 options; the same claim is in the commit message of fd4dca6. - ✅ fixed in
9fe3687— PostgreSQL rejects long multi-selection values —lib/Migration/Version000700Date20230916000000.php:116. Reproduced on PostgreSQL 16: with the index, inserting a 1,000-id multi-selection fails withindex row size 3288 exceeds btree version 4 maximum 2704; skipping the selection index on PostgreSQL costs nothing measured.
🟢 Minor (2)
- ✅ fixed in
9fe3687— Test does not pin the number index's table —tests/unit/Listener/AddMissingIndicesListenerTest.php:35. Changing the number index's table name keeps the test green. - ✅ fixed in
73014e0— Nothing ties migration index definitions to listener —lib/Migration/Version000700Date20230916000000.php:106. Three copies agree today, but a drifted name would give fresh installs a duplicate index without any test going red.
✅ Verified clean
- No duplicate index on any path:
db:add-missing-indiceschecks by name, the listener and both migrations use the same four names, and a fresh SQLite install at this head has all four_cvindices with nothing left to add (a dry run on PostgreSQL also finds nothing). - Editing the shipped migrations is Nextcloud's standard pattern here: fresh installs get the index from the creating migration, existing installs from the
AddMissingIndicesEventlistener, exactly once each. - API compatibility for NC 33–36:
Table|ITableinaddValueIndex(),addMissingIndex(..., $options)andgetDatabaseProvider()exist on every supported version (stable33 OCP stubs and server 36 checked; 34/35 not checked separately). - MySQL/MariaDB key size: 64 characters × 4 bytes + the int column stays well under the 767/3072-byte limits; Oracle skips only the three CLOB columns.
- The existing
(column_id, row_id)indices are untouched, as the maintainer asked on nextcloud#3092. - phpunit 426/426 at this head (MariaDB).
Gates: not run — no Hydra gate runner on this machine and no gate job in this repository; the repository's own checks (cs, psalm, rector, lint, phpunit) ran green locally on this tree.
CI: the four Playwright jobs failed on GitHub API 504s during composer install (infrastructure, not this change); no check is required on this base.
Fixes: 9fe3687 leaves the selection value index out on PostgreSQL (listener and migration), corrects the prefix docblock and pins the number index's table in the test; 73014e0 adds CellValueIndicesTest, which compares the migrations' value indices with the listener's on MySQL, PostgreSQL, SQLite and Oracle. Checked after the last edit: cs, psalm, rector and lint clean; phpunit 431/431 (SQLite); four control mutations each turned a test red; a fresh PostgreSQL 16 install has no selection value index and saves a 1,000-id multi-selection. A pre-push adversarial check found no new issue in the fix; its wording note on the new test's docblock is applied (the message of 73014e0 keeps the earlier wording).
Out of scope, not fixed here: UsergroupBusiness::validateValue() accepts any integer type without an existence check, so on PostgreSQL an oversized fake id now fails with a 500 instead of being stored. That code predates this PR; it is a candidate for a follow-up issue.
CI on 73014e0: started at the push; not waited for.
Connected issues
- nextcloud#3092 (open) — implemented by this PR. The f2 fix narrows it: the selection index will not be added on PostgreSQL, like the text indices on Oracle, so the issue's acceptance criterion should say so. No state change.
🤖 Generated with Claude Code
A multi-selection keeps all its option ids in one value, which has no length limit. MySQL and MariaDB index only its first 64 characters, but PostgreSQL indexes the whole value and rejects index entries above 2704 bytes: with the index, saving a multi-selection of about 1,000 options failed with "index row size exceeds btree version 4 maximum". The selection value index is therefore not added on PostgreSQL, the same way the text value indices are not added on Oracle; in the benchmark it gave no measurable gain on PostgreSQL. - the listener and Version000700Date20230916000000 both leave it out - the prefix length docblock now says a multi-selection is indexed by its first option ids only - the listener test pins the table of the number index and covers the PostgreSQL case Refs nextcloud#3092 Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Robert Zondervan <robert@conduction.nl>
…r adds Fresh installations get the cell value indices from the migrations that create the cell tables, existing installations from AddMissingIndicesListener. A test now runs both migrations against a mocked schema on MySQL, PostgreSQL, SQLite and Oracle and compares the value indices they create with the ones the listener adds, so a name or definition that drifts in one place fails instead of giving fresh installations a second index. Refs nextcloud#3092 Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Robert Zondervan <robert@conduction.nl>
Internal review draft. This PR exists only to run the review skill on the fork before the upstream PR is opened. The base
review-base/mainequals nextcloud/tablesmain(8309f95). The upstream PR description will be written separately.Refs nextcloud#3092
Changes
AddMissingIndicesListeneradds(column_id, value)indices to the number, selection, datetime and usergroup cell tables, so existing installations get them throughocc db:add-missing-indices.Checks run locally
db:add-missing-indicesfinds nothing missing afterwards.AI assistance: Claude Code (claude-opus-5-5) was used for implementation and benchmarking; commits carry
Assisted-bytrailers.🤖 Generated with Claude Code