Overview

Spark SQL, Databricks SQL, and Hive/HiveQL are represented internally as a single value, spark_sql, in the conversion pipeline's SQLDialect enum, alongside tsql (SQL Server), plsql (Oracle), plpgsql (PostgreSQL), mysql, and ansi (generic ANSI SQL). During Discovery, a field validator normalizes dozens of spellings a source file or its LLM classification might use: "Hive", "HiveQL", and "Databricks SQL" all collapse to spark_sql, so variation in self-labeling doesn't affect classification.

What makes this page worth reading on its own: every other supported dialect is a genuinely different execution engine from PySpark, with real semantic gaps to close. Spark SQL / Databricks SQL / Hive is not a different engine: it runs on Spark already. That single fact changes what "conversion" means for this dialect end to end.

Why This Dialect Is Different

For T-SQL, PL/SQL, PL/pgSQL, MySQL, and generic ANSI sources, 3XCode's job is to close real semantic gaps between a foreign engine and Spark: things like DATEDIFF argument order reversing, NULL ordering differences, or cursors that have no direct PySpark equivalent. For Spark SQL / Databricks SQL / Hive, none of that applies: the source SQL already executes correctly on Spark today.

3XCode still runs Spark SQL / Databricks SQL / Hive files through the same pipeline as every other dialect (Discovery, Planning, a deterministic Grouping step, Conversion, Validation, Auto-Fix) and still produces dependency-ordered, structured PySpark modules. What changes is what Validation is actually checking for, and what Conversion is actually deciding.

How Spark SQL / Databricks SQL Is Detected

Detection still runs the same two independent layers as every other dialect during Discovery. You never set a flag telling 3XCode what dialect or platform a file is in.

The platform layer matters more here than for any other dialect: spark_sql is a dialect family, and the specific vendor behind it (genuine Databricks SQL versus plain open-source Hive) determines which syntax extensions are actually valid in the source file. Every generated PySpark file carries a one-line header comment naming the detected platform, confidence, and matched markers, so that distinction is always visible in the output, not just inferred.

Discovery's object inventory can also legitimately come back empty for this dialect. An empty inventory is one of three defined discovery outcomes, and the pipeline explicitly treats "a file already written in Spark SQL" as a valid reason for it: the runner writes a clean no_objects audit rather than a false "success," because there may genuinely be nothing left to convert.

spark.sql() Passthrough vs. DataFrame API

Because there's no correctness gap to close, the Conversion phase's real decision for each construct in a Spark SQL / Databricks SQL / Hive source is an idiom choice: rewrite it as native PySpark DataFrame API, or keep it as a spark.sql("...") passthrough call inside a structured module.

# Ordinary aggregation, rewritten to DataFrame API for readability & testability
# Source: SELECT region, SUM(revenue) AS total FROM sales GROUP BY region

def _step_regional_totals(spark, sales_df):
    return (
        sales_df
        .groupBy("region")
        .agg(F.sum("revenue").alias("total"))
    )


# Databricks-specific QUALIFY dedup, kept as a spark.sql() passthrough
# because the source is already correct, already native Spark SQL

def _step_latest_per_customer(spark):
    return spark.sql("""
        SELECT *
        FROM orders
        QUALIFY ROW_NUMBER() OVER (
            PARTITION BY customer_id ORDER BY order_date DESC
        ) = 1
    """)

Both functions above are equally valid PySpark: the difference is a maintainability judgment, not a correctness fix. That judgment call, applied construct by construct, is what "conversion" actually means for this dialect.

Databricks-Only Syntax Is Preserved, Not Genericized

3XCode does not blindly flatten Databricks/Delta-flavored SQL extensions down to open-source Spark grammar, and it does not force them into a DataFrame-only rewrite that could change what the query actually does. Constructs that are already valid, already correct Spark-SQL/Delta syntax (some of which are also standard SQL more broadly, not exclusive to Databricks) are preserved as spark.sql() passthrough rather than being treated as a translation problem to solve.

Converting a Spark SQL / Databricks File

No flag is needed, dialect and platform are both detected automatically during Discovery, exactly as with every other dialect:

# Convert a single Databricks SQL / Hive script
3xcode pyspark convert warehouse_jobs.sql

# Dry-run: discovery + planning only, no conversion
3xcode pyspark analyze warehouse_jobs.sql

# Convert every .sql file in a directory (bulk mode)
3xcode pyspark convert --dir ./sql/ --session "databricks-cleanup" -w 4

# Find the exact session ID before resuming
3xcode session list

# Resume an interrupted bulk session using the ID from the list above
3xcode pyspark convert --resume <session-id-from-list>

The --session name you pass in is a label, not the resumable ID. Run 3xcode session list to see the actual session ID before using --resume.

Check <stem>_audit.md after conversion the same way you would for any other dialect: for a spark_sql source, expect fewer behavioral-gotcha items and more readability/structure notes, plus a clear record of which constructs were kept as spark.sql() passthrough.

FAQ