Skip to content

[#17939] Implement FFT table function for table model#18006

Open
DaZuiZui wants to merge 3 commits into
apache:masterfrom
DaZuiZui:feat/fft-table-v1-17939
Open

[#17939] Implement FFT table function for table model#18006
DaZuiZui wants to merge 3 commits into
apache:masterfrom
DaZuiZui:feat/fft-table-v1-17939

Conversation

@DaZuiZui

Copy link
Copy Markdown
Contributor

Description

FFT table-valued function

This PR implements the v1 built-in FFT / fft table-valued function for the table model.

The function accepts a required DATA table argument with set semantics and ORDER BY time, plus optional SAMPLE_INTERVAL, N, and NORM arguments. It transforms all numeric input columns except time and partition columns, emits the full complex spectrum, and returns frequency_index, frequency, and <column>_real / <column>_imag columns. When partition columns exist, they are kept at the front of the output schema.

The FFT implementation uses JTransforms DoubleFFT_1D. N supports default partition length, truncation, and zero padding. NORM supports backward, forward, and ortho. SAMPLE_INTERVAL can be supplied as a duration literal; otherwise the interval is inferred from the partition time range.

Analyzer, planner, and validation

The function is registered as a built-in table function and gets analyzer/planner handling similar to M4 where needed so the output column order matches the table-function contract.

Validation covers:

  • DATA must have ORDER BY time in ascending order.
  • SAMPLE_INTERVAL, if provided, must be positive.
  • N, if provided, must be positive.
  • NORM must be one of backward, forward, or ortho.
  • At least one numeric transform column is required.
  • Runtime data must be strictly increasing by time and numeric transform values must not be null.
  • If SAMPLE_INTERVAL is omitted, at least two rows are required to infer it.

Tests

This PR adds focused unit coverage for the FFT processor, analyzer/planner coverage for built-in table-function recognition and schema handling, and a table-model IT covering normal execution and failure cases.

Tested locally with:

JAVA_HOME=$(/usr/libexec/java_home -v 21) ./mvnw test -pl iotdb-core/node-commons -am -Dtest=FFTTableFunctionTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
JAVA_HOME=$(/usr/libexec/java_home -v 21) ./mvnw test -pl iotdb-core/datanode -am -Dtest=TableFunctionTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
JAVA_HOME=$(/usr/libexec/java_home -v 21) ./mvnw verify -DskipUTs -Dit.test=IoTDBFFTTableFunctionIT -DfailIfNoTests=false -Dfailsafe.failIfNoSpecifiedTests=false -pl integration-test -am -PTableSimpleIT -P with-integration-tests

This PR has:

  • been self-reviewed.
  • added Javadocs for most classes and all non-trivial methods.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR
  • org.apache.iotdb.commons.udf.builtin.relational.tvf.FFTTableFunction
  • org.apache.iotdb.commons.queryengine.plan.relational.function.TableBuiltinTableFunction
  • org.apache.iotdb.db.queryengine.plan.relational.analyzer.StatementAnalyzer
  • org.apache.iotdb.db.queryengine.plan.relational.planner.RelationPlanner
  • org.apache.iotdb.commons.udf.builtin.relational.tvf.FFTTableFunctionTest
  • org.apache.iotdb.db.queryengine.plan.relational.analyzer.TableFunctionTest
  • org.apache.iotdb.relational.it.db.it.IoTDBFFTTableFunctionIT

Related to #17939.

@DaZuiZui DaZuiZui force-pushed the feat/fft-table-v1-17939 branch from dc535ca to 2faa54b Compare June 23, 2026 06:28
@DaZuiZui

Copy link
Copy Markdown
Contributor Author

Updated this PR for the latest review comments:

  • Capped FFT transform length at 65,536 and added a spectrum buffer-size guard for multi-column input.
  • Validate evenly spaced timestamps when SAMPLE_INTERVAL is omitted, and validate input gaps against SAMPLE_INTERVAL when it is provided.
  • Added UT/IT coverage for oversized N, irregular timestamps, and explicit interval mismatch.

Tests run locally with JDK 21:

  • JAVA_HOME=$( /usr/libexec/java_home -v 21 ) ./mvnw spotless:apply -pl iotdb-core/node-commons,iotdb-core/datanode,integration-test -P with-integration-tests
  • JAVA_HOME=$( /usr/libexec/java_home -v 21 ) ./mvnw test -pl iotdb-core/node-commons -am -Dtest=FFTTableFunctionTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
  • JAVA_HOME=$( /usr/libexec/java_home -v 21 ) ./mvnw test -pl iotdb-core/datanode -am -Dtest=TableFunctionTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
  • JAVA_HOME=$( /usr/libexec/java_home -v 21 ) ./mvnw verify -DskipUTs -Dit.test=IoTDBFFTTableFunctionIT -DfailIfNoTests=false -Dfailsafe.failIfNoSpecifiedTests=false -pl integration-test -am -PTableSimpleIT -P with-integration-tests

@DaZuiZui

Copy link
Copy Markdown
Contributor Author

Follow-up update:

  • Added node-commons unit coverage for actual FFT spectrum output, zero padding, N truncation, and the explicit failure behavior for invalid rows even when they are beyond the truncated N range.
  • Added a release-note level entry documenting SAMPLE_INTERVAL, default/capped N, null rejection, and timestamp spacing behavior.

Pushed as a normal follow-up commit (no force push).

Tests run locally with JDK 21:

  • JAVA_HOME=$( /usr/libexec/java_home -v 21 ) ./mvnw spotless:apply -pl iotdb-core/node-commons
  • JAVA_HOME=$( /usr/libexec/java_home -v 21 ) ./mvnw test -pl iotdb-core/node-commons -am -Dtest=FFTTableFunctionTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant