Search before asking
Linkis Component
Description
The SparkEngineConnExecutor code assumes all executors are SparkSqlExecutor and uses asInstanceOf[SparkSqlExecutor] to retrieve the SparkSession. This causes type conversion errors for other executor types like SparkScalaExecutor, SparkPythonExecutor, and SparkDataCalcExecutor.
Steps to reproduce
- Use SparkScalaExecutor, SparkPythonExecutor, or SparkDataCalcExecutor
- Trigger the spark executor params setting code path
- Type conversion exception occurs due to incorrect casting
Expected behavior
Each executor type should be able to correctly retrieve its own SparkSession without type conversion errors.
Your environment
- Linkis version used: 2.0.0
- Environment name and version:
- hadoop-3.3.4
- hive-2.3.3
- spark-2.4.3 / 3.3.0
- scala-2.11.12 / 2.12.17
- jdk 1.8.0_xxx
Anything else
This PR adds a getSparkEngineSession method to each executor subclass and uses pattern matching to safely retrieve the SparkSession based on the actual executor type.
Are you willing to submit a PR?
Search before asking
Linkis Component
Description
The SparkEngineConnExecutor code assumes all executors are SparkSqlExecutor and uses
asInstanceOf[SparkSqlExecutor]to retrieve the SparkSession. This causes type conversion errors for other executor types like SparkScalaExecutor, SparkPythonExecutor, and SparkDataCalcExecutor.Steps to reproduce
Expected behavior
Each executor type should be able to correctly retrieve its own SparkSession without type conversion errors.
Your environment
Anything else
This PR adds a
getSparkEngineSessionmethod to each executor subclass and uses pattern matching to safely retrieve the SparkSession based on the actual executor type.Are you willing to submit a PR?