Improve `dev/Dockerfile`

[kevinjqliu]

Feature Request / Improvement

We had issues with downloading spark for the integration test docker image.

• Using archive.apache.org may be unstable sometimes (docker: The archive seems unstable #1425)
• Using dlcdn.apache.org breaks the script and force us to upgrade whenever there's a new release ([infra] download Spark from archive.apache.org #1523)

The issue comes from the need to download spark from an URL.

Since dev/Dockerfile is building a spark docker image with some extra configs.
I was wondering if we can use the dockerhub spark image as the base and configure it.

[kevinjqliu]

possibly reuse for rust as well apache/iceberg-rust#894 (comment)

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[jayceslesar]

I think its as easy as doing a multi stage build where we first build from

FROM apache/spark-py:v3.5.6 AS spark-dist

# stuff in between the existing start of the Dockerfile to COPY spark-defaults.conf /opt/spark/conf

COPY --from=spark-dist /opt/spark /opt/spark
COPY spark-defaults.conf /opt/spark/conf

# stuff after spark-defaults.conf /opt/spark/conf

Would need some ARG/ENV handling so we dont break anything

[kevinjqliu]

a couple progress on this

• added infra: prefer downloads.apache.org, fallback to archive.apache.org #2494 for faster spark download
• added infra: use spark connect to run pytests #2491 to simplify our test setup using spark

We're on our way to use the default Spark docker image without any modifications :)