Query: Optimize schema traversal for non-leading tag IN predicates

[Caideyipi]

Description

Problem

Table-model tag predicates are converted to OR-concatenated precise-filter branches. Both an IN predicate and its equivalent OR expression therefore produce multiple device patterns.

When the predicate is on a non-leading tag and an earlier tag level is not fixed, every expanded pattern traverses the same wildcard prefix independently. With N values, the schema tree prefix is visited N times. Rewriting IN as OR does not avoid this behavior.

Solution

• Keep predicate expansion in the relational planner so complete device IDs can still benefit from the existing device cache and partition lookup paths.
• Compact branches in DeviceFilterUtil only when they differ by a precise value at a non-leading tag behind an unfixed earlier tag level.
• Represent the merged values as an InFilter attached to ExtendedPartialPath.
• Add multi-exact transitions to SimpleNFA so the shared wildcard prefix is traversed only once.
• At each in-memory MTree parent, compare the effective multi-exact value count with the current child key-set size. Iterate child keys when their count is less than or equal to the IN count; otherwise perform precise child lookups. When the complete child count is unavailable, retain precise child lookup.

The comparison intentionally uses a stable 1:1 candidate-count heuristic rather than a wall-clock threshold. An opt-in performance UT confirmed that direct memory-tree child iteration wins the equal-count tie because it avoids one direct child lookup per transition.

Leading-tag predicates and fully fixed prefixes remain as separate precise paths. Branches with null precise values, additional filters on the same tag, or correlated conditions such as (card = a AND device = x) OR (card = b AND device = y) are not merged.

For the motivating case, six values on the second tag are reduced from six schema readers over the same first-tag wildcard subtree to one adaptive reader.

Manual performance UT

SchemaRegionTableDevicePerformanceTest is disabled by default with a JUnit assumption. It compares the legacy N-reader expansion with the merged adaptive reader, verifies identical results, and reports median CPU time, thread allocation, peak-heap delta, and speedup. There are no wall-clock performance assertions.

The values property accepts a comma-separated sweep, and hit.values=0 models the important empty-result case:

mvn -pl iotdb-core/datanode \
  -Dtest=SchemaRegionTableDevicePerformanceTest \
  -Diotdb.schema.non.leading.in.perf.enabled=true \
  -Diotdb.schema.non.leading.in.perf.meters=50000 \
  -Diotdb.schema.non.leading.in.perf.children.per.meter=8 \
  -Diotdb.schema.non.leading.in.perf.values=7,8,9 \
  -Diotdb.schema.non.leading.in.perf.hit.values=0 \
  -Diotdb.schema.non.leading.in.perf.warmups=2 \
  -Diotdb.schema.non.leading.in.perf.iterations=3 \
  -Diotdb.schema.non.leading.in.perf.rounds=5 \
  test

Representative local Windows results with 400,000 devices and zero matches:

IN values	Selected side	Legacy CPU/op	Optimized CPU/op	Speedup
7	precise IN lookups	98.958 ms	41.667 ms	2.38x
8	child-key iteration	98.958 ms	31.250 ms	3.17x
9	child-key iteration	125.000 ms	31.250 ms	4.00x

For the equal-count case, switching from precise lookup to child-key iteration reduced the optimized path from 46.875 ms/op to 31.250 ms/op in the same 50,000-by-8 empty-result workload (about 1.5x faster).

Tests

• DeviceFilterUtilTest: 10 tests passed, including value-smaller, child-smaller, and equal-count adaptive directions
• SchemaPredicateUtilTest: 2 tests passed, covering both expanded IN and equivalent OR
• SchemaRegionTableDeviceTest: 20 tests passed
• SchemaRegionTableDevicePerformanceTest: enabled benchmark passed; disabled mode skips all 4 parameterized instances
• Spotless, Checkstyle, RAT, and staged diff checks passed

---

This PR has:

• been self-reviewed.
• added comments explaining the intent of the optimization and heuristic.
• added unit tests covering the new code paths and safety guards.

---

Key changed/added classes

• DeviceFilterUtil
• ExtendedPartialPath
• SimpleNFA
• AbstractTreeVisitor
• Traverser
• DeviceFilterUtilTest
• SchemaPredicateUtilTest
• SchemaRegionTableDevicePerformanceTest