Skip to content

[WIP][SPARK-56876][SQL] Add TimestampNTZNanosType and TimestampLTZNanosType#55952

Open
MaxGekk wants to merge 1 commit into
apache:masterfrom
MaxGekk:nanos-add-types
Open

[WIP][SPARK-56876][SQL] Add TimestampNTZNanosType and TimestampLTZNanosType#55952
MaxGekk wants to merge 1 commit into
apache:masterfrom
MaxGekk:nanos-add-types

Conversation

@MaxGekk
Copy link
Copy Markdown
Member

@MaxGekk MaxGekk commented May 18, 2026

What changes were proposed in this pull request?

In the PR, I propose to extend the Spark SQL type system, and add new classes to Scala/Java APIs:

  • TimestampNTZNanosType(p)represents the SQL data type TIMESTAMP_NTZ(p)
  • TimestampLTZNanosType(p)represents TIMESTAMP_LTZ(p)

They align with the SQL standard’s direction for optional feature F555, “Enhanced seconds precision”: datetime types can carry fractional seconds with precision p in the SECOND field beyond the traditional six decimal places (microseconds). Here p is restricted to 7, 8, and 9, i.e. the nanosecond-capable band (up to nine fractional digits, nanoseconds in the second field).

The logical layout documented on the classes matches this precision story: epoch microseconds plus nanoseconds within that microsecond, with a default estimated width of 10 bytes for planning (8 + 2).

Parameterless timestamp_ntz / timestamp_ltz are unchanged and remain the existing microsecond-oriented types.

Why are the changes needed?

New timestamp types are useful for Spark SQL users because they allow:

  1. Represent timestamp without time zone and timestamp with local time zone with fractional-second precision 7–9, in line with SQL optional feature F555 (Enhanced seconds precision).
  2. Describe schemas from other systems that already use nanosecond-capable timestamps, without overloading microsecond timestamp_ntz / timestamp_ltz types.
  3. Migrate SQL and metadata that distinguish NTZ and LTZ at sub-microsecond precision toward Spark in small, reviewable steps.
  4. Prepare later work to read and write such columns from datasources and JDBC, and to apply optimizations that depend on precise timestamp types.

Does this PR introduce any user-facing change?

Yes, it extends the type system of Spark SQL.

How was this patch tested?

By extending DataTypeSuite (round-trip and precision bounds for the new types, including invalid precisions).

$ build/sbt "test:testOnly *DataTypeSuite"

Plus SparkThrowableSuite / error-json validation if error-conditions.json is updated.

Was this patch authored or co-authored using generative AI tooling?

No.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant