Skip to content

Support strict tree table pipe isolation#17948

Open
Caideyipi wants to merge 7 commits into
apache:masterfrom
Caideyipi:pipe-tree-table-isolation
Open

Support strict tree table pipe isolation#17948
Caideyipi wants to merge 7 commits into
apache:masterfrom
Caideyipi:pipe-tree-table-isolation

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR updates Pipe tree/table isolation from capture.tree / capture.table based visibility to SQL dialect based visibility.

The short rule is:

A user-created Pipe belongs to exactly one SQL dialect. A tree-dialect Pipe only sees tree-model data and metadata. A table-dialect Pipe only sees table-model data and metadata. mode.double-living=true is implemented by creating two same-name Pipes, one under each dialect, instead of creating one Pipe visible to both dialects.

Complete Feature Definition

1. The source of truth for Pipe visibility

The decisive attribute is __system.sql-dialect in the Pipe source/extractor attributes.

__system.sql-dialect value Pipe visibility Visible from tree dialect Visible from table dialect
tree tree-only yes no
table table-only no yes
absent tree-only, for backward compatibility yes no

A normal user-created Pipe is no longer visible to both tree and table models at the same time.

A Pipe is visible to a dialect if and only if its calculated visibility is compatible with that dialect:

  • tree-only Pipe: visible only to tree dialect operations.
  • table-only Pipe: visible only to table dialect operations.
  • old or malformed user input using capture.tree / capture.table: ignored for visibility; visibility still follows __system.sql-dialect.

2. capture.tree and capture.table no longer define visibility

capture.tree and capture.table, including the source.* and extractor.* aliases, are kept only as accepted compatibility parameters.

They do not decide:

  • whether a Pipe is visible from tree SQL or table SQL;
  • whether a Pipe captures tree-model or table-model data;
  • whether a same-name Pipe conflicts with another Pipe under the other dialect;
  • whether mode.double-living=true is valid, except for the remaining forwarding restriction described below.

Examples:

  • A Pipe created from a tree dialect session with capture.table=true is still a tree-only Pipe.
  • A Pipe created from a table dialect session with capture.tree=true is still a table-only Pipe.
  • A tree-only Pipe with capture.tree=true,capture.table=true does not block creating a table-only Pipe with the same name.
  • capture.tree=false or capture.table=false no longer makes a double-living Pipe invalid.

The only remaining double-living parameter restriction is:

  • mode.double-living=true cannot be used with forwarding-pipe-requests=true.

3. Creation rules and same-name Pipes

Pipe name uniqueness is scoped by model visibility.

This means:

  • A tree-only Pipe named p and a table-only Pipe named p can coexist.
  • Creating another tree-only Pipe named p conflicts with the existing tree-only p, unless IF NOT EXISTS is used.
  • Creating another table-only Pipe named p conflicts with the existing table-only p, unless IF NOT EXISTS is used.
  • A Pipe visible in one dialect does not reserve the same name in the other dialect.

For SQL creation:

  • tree SQL creates a tree-only Pipe by setting __system.sql-dialect=tree;
  • table SQL creates a table-only Pipe by setting __system.sql-dialect=table.

For direct ConfigNode RPC creation:

  • callers must set __system.sql-dialect=table explicitly to create a table-only Pipe;
  • if the dialect is absent, the Pipe is treated as tree-only for backward compatibility.

4. mode.double-living=true

mode.double-living=true no longer means one Pipe has both tree and table visibility.

Instead, create is expanded into two physical Pipe metas with the same Pipe name:

  • one tree-only Pipe with __system.sql-dialect=tree;
  • one table-only Pipe with __system.sql-dialect=table.

Each physical Pipe is then handled by the same visibility rules as a normal single-dialect Pipe.

Consequences:

  • SHOW PIPE in tree dialect shows only the tree-side Pipe.
  • SHOW PIPE in table dialect shows only the table-side Pipe.
  • Start, stop, drop, and alter are model-scoped when the request carries the model flag.
  • The two same-name Pipes use distinct creation times, so DataNode and ConfigNode runtime state can still identify the exact Pipe instance.
  • If auto full-sync splitting is enabled, the SQL layer can still split a full-sync Pipe into _realtime and _history; ConfigNode then applies the double-living split to each of those requests.

5. Operation visibility and compatibility

The following operations are scoped by Pipe visibility:

  • SHOW PIPE
  • START PIPE
  • STOP PIPE
  • DROP PIPE
  • ALTER PIPE
  • Pipe existence checks
  • Pipe name conflict checks
  • Pipe status and runtime-meta lookup
  • Pipe meta push/drop on DataNodes

For new/extended RPCs that carry isTableModel:

  • isTableModel=false targets the tree-only Pipe with that name.
  • isTableModel=true targets the table-only Pipe with that name.

For legacy RPCs that do not carry isTableModel:

  • if a tree-only Pipe with that name exists, the operation resolves to the tree Pipe;
  • otherwise, if only a table-only Pipe exists, the operation resolves to the table Pipe.

This keeps old clients compatible while allowing same-name tree/table Pipes for new model-aware callers.

6. Data and metadata capture rules

A Pipe can only capture data and metadata that belong to its own dialect.

Tree-only Pipe:

  • captures tree-model data regions only;
  • captures tree-model schema metadata only;
  • captures tree-model database metadata only;
  • does not capture table-model data, table schema, table-only privileges, or table-model databases.

Table-only Pipe:

  • captures table-model data regions only;
  • captures table-model schema metadata only;
  • captures table-model database metadata only;
  • does not capture tree-model data, timeseries schema, tree-only privileges, or tree-model databases.

Database metadata is explicitly model-isolated:

  • table-model database plans are only listened by table-only Pipes;
  • tree-model database plans are only listened by tree-only Pipes;
  • Pipe send and receive paths keep this model distinction.

Model-neutral metadata is handled as common metadata:

  • common auth base entities, such as user/role create/drop/update, remain eligible for both dialects because model-specific privileges depend on them;
  • model-specific auth content is trimmed by scope visitors so a tree-only Pipe does not transfer table-only privileges and a table-only Pipe does not transfer tree-only privileges.

7. Consensus Pipes

Consensus Pipes are not split by mode.double-living.

When a consensus Pipe is built for a region, its dialect is derived from the region database model:

  • regions under table-model databases use __system.sql-dialect=table;
  • regions under tree-model databases use __system.sql-dialect=tree.

Consensus Pipe requests no longer set capture.tree=true / capture.table=true to force both visibility.

8. Plugin validation and request mutation

For double-living creation, DataNode validates plugins once for the tree-side attributes and once for the table-side attributes.

Each validation uses cloned source, processor, and sink attributes so a temporary plugin validation cannot mutate shared attributes and affect the other side.

9. Compatibility summary

Behavior after this PR:

  • __system.sql-dialect decides Pipe visibility.
  • capture.tree / capture.table are ignored for visibility and capture scope.
  • Existing requests without __system.sql-dialect are treated as tree dialect.
  • Same-name tree/table Pipes can coexist.
  • Same-name same-dialect Pipes still conflict.
  • mode.double-living=true creates two same-name Pipes, not one both-visible Pipe.
  • Legacy no-model lifecycle RPCs prefer the tree Pipe when both same-name Pipes exist.
  • Model-aware RPCs operate on the exact dialect selected by isTableModel.

Implementation Notes

Main changes:

  • Calculate Pipe visibility only from __system.sql-dialect.
  • Remove runtime capture.tree / capture.table validation and visibility semantics.
  • Split non-consensus mode.double-living=true creation into tree/table same-name Pipe requests.
  • Keep alter requests' extractor dialect consistent with the target model.
  • Resolve start/stop/drop/alter by explicit model flag when present, and keep legacy fallback when absent.
  • Build consensus Pipe dialect from the region database model.
  • Clone source/processor/sink attributes during double-living plugin validation.
  • Update and extend IT coverage for dialect isolation, same-name tree/table Pipes, double-living split, and lifecycle isolation.

Verification

Passed:

  • git diff --check
  • grep check: runtime code no longer reads capture.tree / capture.table; only constants remain
  • mvn -o "-Ddevelocity.off=true" -pl iotdb-core/node-commons,iotdb-core/confignode -DskipTests compile
  • mvn -o "-Ddevelocity.off=true" -pl iotdb-core/datanode spotless:apply
  • mvn -o "-Ddevelocity.off=true" -pl iotdb-core/node-commons -Dtest=VisibilityUtilsTest,PipeMetaKeeperTest test
  • mvn -o "-Ddevelocity.off=true" -pl integration-test spotless:apply -P with-integration-tests

Attempted but blocked by current local/base compile issues:

  • mvn -o "-Ddevelocity.off=true" -pl iotdb-core/node-commons,iotdb-core/datanode -DskipTests compile
  • mvn -o "-Ddevelocity.off=true" -pl integration-test -DskipTests test-compile -P with-integration-tests

The DataNode compile reaches javac, then fails on unrelated existing/base errors, including missing ComparatorChain, missing IoTDBConfig#getModeMapSizeThreshold(), DataRegionStateMachine override mismatch, and missing PipeParameters.ValueHider#isHiddenKey. The changed DataNode file is not reported as a compilation error.

The integration-test compile reaches test compilation after refreshing local client snapshots, then fails in unrelated subscription/UDF tests due local/base dependency mismatches such as missing SubscriptionMessage#getResultSets(), SubscriptionMessage#getTsFile(), SubscriptionMessage#getRecordTabletIterator(), and missing org.apache.iotdb.db.queryengine.plan.relational.function classes. The newly added Pipe IT files are not reported as compilation errors before that failure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant