Add execution engine run scripts: Databricks, Spark Standalone, Dataproc

[Copilot]

Adds shell scripts for submitting and managing Spark workloads across three execution engines, starting from an empty repo.

Structure

databricks/
  run_job.sh          # Trigger existing job by ID or name; poll until done
  submit_notebook.sh  # One-time notebook run on existing or ephemeral cluster
spark-standalone/
  start_cluster.sh    # Start master + N local workers (configurable cores/mem)
  stop_cluster.sh     # Gracefully stop workers then master
  submit_job.sh       # spark-submit wrapper (JAR + Python, --conf, deploy mode)
dataproc/
  create_cluster.sh   # Create cluster (machine types, auto-delete, labels, properties)
  delete_cluster.sh   # Delete cluster with optional confirmation
  submit_job.sh       # Submit spark/pyspark/hadoop/hive/pig/presto jobs

Key design points

• Databricks — uses Jobs API 2.1 (run-now / runs/submit); run_job.sh supports name→ID resolution via URL-encoded list query
• Dataproc — all Spark/job properties collected into a single --properties key=val,key=val argument (multiple --properties flags are silently last-wins in gcloud)
• All scripts: set -euo pipefail, named flags with defaults, --help, dependency checks (require_cmd), clear error messages on missing required inputs

Usage

# Databricks: trigger job by name and wait
DATABRICKS_HOST=https://adb-xxx.azuredatabricks.net DATABRICKS_TOKEN=dapi... \
./databricks/run_job.sh --job-name nightly_etl --params '{"date":"2024-01-01"}' --wait

# Spark Standalone: start cluster, submit, stop
SPARK_HOME=/opt/spark ./spark-standalone/start_cluster.sh --workers 2 --worker-cores 4 --worker-memory 8g
./spark-standalone/submit_job.sh --app /path/to/etl.py
./spark-standalone/stop_cluster.sh

# Dataproc: create, submit, teardown
GCP_PROJECT=my-project GCP_REGION=us-central1 \
./dataproc/create_cluster.sh --cluster-name etl --num-workers 4 --max-idle 30m
./dataproc/submit_job.sh --cluster-name etl --job-type pyspark --app gs://bucket/etl.py -- --date 2024-01-01
./dataproc/delete_cluster.sh --cluster-name etl --yes

Original prompt

I want to add different execution engines run scripts like databricks, Spark-Standalone, Dataproc

[Copilot]

Copilot wasn't able to review any files in this pull request.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.