Any job that uses Column.withField(...) or dropFields(...) fails during lineage harvest.
The job itself finishes fine (SplineAgent swallows the error), but lineage for that execution is dropped entirely.
Repro:
import org.apache.spark.sql.functions._
val df = spark.range(1)
.select(struct(lit(1).as("a"), lit(2).as("b")).as("s"))
.withColumn("s", col("s").withField("a", lit(99)))
df.write.mode("overwrite").saveAsTable("tmp.repro_table")
Error:
ERROR agent.SplineAgent: Unexpected error occurred during lineage processing for application: ...
java.lang.IllegalStateException: StructFieldsOperation.dataType should not be called.
at org.apache.spark.sql.catalyst.expressions.complexTypeCreator$StructFieldsOperation.dataType(...)
at za.co.absa.spline.harvester.converter.ExpressionConverter.convertDataType(ExpressionConverter.scala)
at za.co.absa.spline.harvester.converter.ExprToRefConverter.convert(ExprToRefConverter.scala:42)
...
WithField/DropField are children of UpdateFields, and Spark deliberately leaves their dataType unimplemented — only the parent folds them to compute the result type. ExpressionConverter calls .dataType on every node uniformly, so it hits this guard.
Same shape as the WindowSpecDefinition/WindowFrame case already handled here:
case _: sparkExprssions.WindowSpecDefinition
| _: sparkExprssions.WindowFrame =>
FunctionalExpression(dataType = None, ...)
Adding StructFieldsOperation (or WithField/DropField directly) to that same match seems like it'd fix it. Can send a PR if that's the right approach.
Env: spline-spark-agent 2.2.3 (spark-3.4-spline-agent-bundle_2.12), Spark 3.4.2, Scala 2.12
Any job that uses Column.withField(...) or dropFields(...) fails during lineage harvest.
The job itself finishes fine (SplineAgent swallows the error), but lineage for that execution is dropped entirely.
Repro:
Error:
WithField/DropField are children of UpdateFields, and Spark deliberately leaves their dataType unimplemented — only the parent folds them to compute the result type. ExpressionConverter calls .dataType on every node uniformly, so it hits this guard.
Same shape as the WindowSpecDefinition/WindowFrame case already handled here:
Adding StructFieldsOperation (or WithField/DropField directly) to that same match seems like it'd fix it. Can send a PR if that's the right approach.
Env: spline-spark-agent 2.2.3 (spark-3.4-spline-agent-bundle_2.12), Spark 3.4.2, Scala 2.12