Support strict tree table pipe isolation#17948
Open
Caideyipi wants to merge 7 commits into
Open
Conversation
…ee-table-isolation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR updates Pipe tree/table isolation from
capture.tree/capture.tablebased visibility to SQL dialect based visibility.The short rule is:
Complete Feature Definition
1. The source of truth for Pipe visibility
The decisive attribute is
__system.sql-dialectin the Pipe source/extractor attributes.__system.sql-dialectvaluetreetableA normal user-created Pipe is no longer visible to both tree and table models at the same time.
A Pipe is visible to a dialect if and only if its calculated visibility is compatible with that dialect:
capture.tree/capture.table: ignored for visibility; visibility still follows__system.sql-dialect.2.
capture.treeandcapture.tableno longer define visibilitycapture.treeandcapture.table, including thesource.*andextractor.*aliases, are kept only as accepted compatibility parameters.They do not decide:
mode.double-living=trueis valid, except for the remaining forwarding restriction described below.Examples:
capture.table=trueis still a tree-only Pipe.capture.tree=trueis still a table-only Pipe.capture.tree=true,capture.table=truedoes not block creating a table-only Pipe with the same name.capture.tree=falseorcapture.table=falseno longer makes a double-living Pipe invalid.The only remaining double-living parameter restriction is:
mode.double-living=truecannot be used withforwarding-pipe-requests=true.3. Creation rules and same-name Pipes
Pipe name uniqueness is scoped by model visibility.
This means:
pand a table-only Pipe namedpcan coexist.pconflicts with the existing tree-onlyp, unlessIF NOT EXISTSis used.pconflicts with the existing table-onlyp, unlessIF NOT EXISTSis used.For SQL creation:
__system.sql-dialect=tree;__system.sql-dialect=table.For direct ConfigNode RPC creation:
__system.sql-dialect=tableexplicitly to create a table-only Pipe;4.
mode.double-living=truemode.double-living=trueno longer means one Pipe has both tree and table visibility.Instead, create is expanded into two physical Pipe metas with the same Pipe name:
__system.sql-dialect=tree;__system.sql-dialect=table.Each physical Pipe is then handled by the same visibility rules as a normal single-dialect Pipe.
Consequences:
SHOW PIPEin tree dialect shows only the tree-side Pipe.SHOW PIPEin table dialect shows only the table-side Pipe._realtimeand_history; ConfigNode then applies the double-living split to each of those requests.5. Operation visibility and compatibility
The following operations are scoped by Pipe visibility:
SHOW PIPESTART PIPESTOP PIPEDROP PIPEALTER PIPEFor new/extended RPCs that carry
isTableModel:isTableModel=falsetargets the tree-only Pipe with that name.isTableModel=truetargets the table-only Pipe with that name.For legacy RPCs that do not carry
isTableModel:This keeps old clients compatible while allowing same-name tree/table Pipes for new model-aware callers.
6. Data and metadata capture rules
A Pipe can only capture data and metadata that belong to its own dialect.
Tree-only Pipe:
Table-only Pipe:
Database metadata is explicitly model-isolated:
Model-neutral metadata is handled as common metadata:
7. Consensus Pipes
Consensus Pipes are not split by
mode.double-living.When a consensus Pipe is built for a region, its dialect is derived from the region database model:
__system.sql-dialect=table;__system.sql-dialect=tree.Consensus Pipe requests no longer set
capture.tree=true/capture.table=trueto force both visibility.8. Plugin validation and request mutation
For double-living creation, DataNode validates plugins once for the tree-side attributes and once for the table-side attributes.
Each validation uses cloned source, processor, and sink attributes so a temporary plugin validation cannot mutate shared attributes and affect the other side.
9. Compatibility summary
Behavior after this PR:
__system.sql-dialectdecides Pipe visibility.capture.tree/capture.tableare ignored for visibility and capture scope.__system.sql-dialectare treated as tree dialect.mode.double-living=truecreates two same-name Pipes, not one both-visible Pipe.isTableModel.Implementation Notes
Main changes:
__system.sql-dialect.capture.tree/capture.tablevalidation and visibility semantics.mode.double-living=truecreation into tree/table same-name Pipe requests.Verification
Passed:
git diff --checkcapture.tree/capture.table; only constants remainmvn -o "-Ddevelocity.off=true" -pl iotdb-core/node-commons,iotdb-core/confignode -DskipTests compilemvn -o "-Ddevelocity.off=true" -pl iotdb-core/datanode spotless:applymvn -o "-Ddevelocity.off=true" -pl iotdb-core/node-commons -Dtest=VisibilityUtilsTest,PipeMetaKeeperTest testmvn -o "-Ddevelocity.off=true" -pl integration-test spotless:apply -P with-integration-testsAttempted but blocked by current local/base compile issues:
mvn -o "-Ddevelocity.off=true" -pl iotdb-core/node-commons,iotdb-core/datanode -DskipTests compilemvn -o "-Ddevelocity.off=true" -pl integration-test -DskipTests test-compile -P with-integration-testsThe DataNode compile reaches javac, then fails on unrelated existing/base errors, including missing
ComparatorChain, missingIoTDBConfig#getModeMapSizeThreshold(),DataRegionStateMachineoverride mismatch, and missingPipeParameters.ValueHider#isHiddenKey. The changed DataNode file is not reported as a compilation error.The integration-test compile reaches test compilation after refreshing local client snapshots, then fails in unrelated subscription/UDF tests due local/base dependency mismatches such as missing
SubscriptionMessage#getResultSets(),SubscriptionMessage#getTsFile(),SubscriptionMessage#getRecordTabletIterator(), and missingorg.apache.iotdb.db.queryengine.plan.relational.functionclasses. The newly added Pipe IT files are not reported as compilation errors before that failure.