Spline producer url unreachable from spark

[learnerr101]

I used the TLDR configuration and packages given in https://absaoss.github.io/spline/. za.co.absa.spline.agent.spark:spark-2.4-spline-agent-bundle_2.12:0.5.2 and 'spark.spline.producer.url' as 'http://localhost:9090/producer'

But I get this error in my spark job:

Configurations:
Spark -> 2.4.2
Scala -> 2.12
Commands used:
spark-submit --packages za.co.absa.spline.agent.spark:spark-2.4-spline-agent-bundle_2.12:0.5.2 --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" --conf "spark.spline.producer.url=http://localhost:9090/producer" AB_2.py

I have also tried spark 3.0.0 with compatible artifacts and Java version but I'm getting the same error.

[cerveada]

Try to follow steps from this guide: #1225

[learnerr101]

So these were the steps I performed:

Troubleshooting Spline Agent:
I was able to see the logs pertaining to the lineage.

Troubleshooting Arango DB:

Troubleshooting Spline Server:

Troubleshooting Spline UI:

Still I'm getting the same error for some reason.

[cerveada]

In your config you use port 9090 but on the picture from server i see port 8080

[learnerr101]

Yeah, I used commands with both the configs.

With 9090:
spark-submit --packages za.co.absa.spline.agent.spark:spark-2.4-spline-agent-bundle_2.12:0.5.2 --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" --conf "spark.spline.producer.url=http://localhost:9090/producer" AB_2.py

With 8080:
spark-submit --packages za.co.absa.spline.agent.spark:spark-2.4-spline-agent-bundle_2.12:0.5.2 --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" --conf "spark.spline.producer.url=http://localhost:8080/producer" AB_2.py

In both the scenarios I get the same error.

[cerveada]

In the log I see output from LoggingLineageDispatcher so that is working. Now switch to http dispatcher.

• You should use the url under Producer API - Base URL that you see in server page.
• I would use the latest release version of the agent: 2.1.0
• make sure the connection from agent to server is possible and nothing is blocking it

[learnerr101]

Yeah I tried the following command:

_spark-submit  
--packages za.co.absa.spline.agent.spark:spark-3.0-spline-agent-bundle_2.12:2.1.0  
--conf spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener 
--conf spark.spline.lineageDispatcher=http  
--conf spark.spline.lineageDispatcher.http.producer.url=http://localhost:8080/producer  
AB_2.py_

So the config as of now is:
Spark 3.0.0
Scala 2.12
Spline : 2.1.0

But error still prevails!

[cerveada]

If everything else works, it must be some networking issue. Are you sure that the server is reachable? Is the spark running locally on the same machine as the severe? If spark is somewhere else, localhost will not work.

Also, if a server or agent is running in the docker container, it may block the connection.

[wajda]

What does the error say, can you show it once again please?

[learnerr101]

Yeah sure!

[wajda]

If as you said the Python example worked then the Spline server should indeed be OK.
The error message on the screenshot you posted says that the agent tried to reach the server but got 404 Not Found response. That in turn suggests that some HTTP server is indeed listening on the Spark driver's localhost on the port 8080, but it might be a different server, thus 404.
All the Spline agent is doing at that particular moment is this:

curl -I http://localhost:8080/producer/status

Try it and see what you've got.

[learnerr101]

Hey,

This is the output of the above curl command:

HTTP/1.1 200
Vary: Origin
Vary: Access-Control-Request-Method
Vary: Access-Control-Request-Headers
ABSA-Spline-API-Version: 1.1
ABSA-Spline-API-Version: 1
ABSA-Spline-API-LTS-Version: 1.1
ABSA-Spline-Accept-Request-Encoding: gzip
Content-Length: 0
Date: Tue, 30 Jul 2024 19:51:16 GMT

[wajda]

So, this means that it works as expected.
Something is wrong with your Spark driver network settings.
Try to use IP instead of localhost