Using a locally published bom

[ehigham]

Hi all,
I'm a contributor on the hail project. We switched from gradle a year or so ago
and it's been brilliant so far. Thanks so much for all your effort.

Our application is built with pyspark and our dependencies are dicatated by
which managed spark service it's deployed in. I'm trying to create mill
configurations to build our application using the versions of the libraries
each managed service provides.

For example:

• dataproc-2.3.x installs pyspark-3.5.3, uses scala 2.12, and provides lib::P.Q.R
• dataproc-3.0.x installs pyspark-4.0.0, uses scala 2.13 and provides lib::P.Q.S
• hdinsight-5.1 installs pyspark-3.3.1, uses scala 2.12 and provides lib::P.T.U

I'd like to keep the number of dependencies we shade to a minimum to avoid bloat
and reuse as many as possible from the pyspark distributions.

To solve this, I tried to model the libraries shipped in pyspark as a cross scala
BomModule, with scala version on one axis and spark version on the other. You can
see the work-in-progress in hail/mill-build/pyspark-bom/package.mill in this PR.
Unfortunately, spark doesn't produce a bom for the jars it ships in pyspark
distributions so I've begun the mind-numbing task of compiling my own. I'm
sure there's a way to automate this, I'm very new to boms, poms and maven et al.

After publishing all cross versions, I added the bom to the bomMvnDeps here.
The trouble is, when I build my application, I see the following:

$ sh mill 'hail[].compile'
[196/196, 1 failed] ============================== hail[].compile ==============================
1 tasks failed
hail[dataproc-2.3.x].resolvedMvnDeps java.lang.RuntimeException: Cannot find org.json4s:json4s-jackson_2.12: in projectCache

When I use bomModuleDeps, however, the versions of the root module dependencies
are inferred correctly. Can you see what I'm doing wrong?

I'd like to use published boms for a couple of reasons:

• I'd like to maintain these in a separate repository eventually
• Intellij gets very confused with all these cross modules and uses versions of
  libraries it shouldn't.

I hope this makes sense... Thanks so much for your help in advance!

[anntnzrb]

When using bomMvnDeps with a locally published BOM, Mill needs the local repository explicitly configured. Add the Ivy local repo to your module's repositories:

object hail extends CrossSbtModule {
  override def repositories = super.repositories() ++ Seq(
    coursier.LocalRepositories.ivy2Local
  )
  
  override def bomMvnDeps = Seq(
    `pyspark-bom` :: crossSparkVersion
  )
}

Alternatively, publish using Maven format with LocalM2Publisher instead of publishLocal (Ivy format).

[ehigham]

Thanks for the reply. Unfortunately that didn't work but I think I spot the issue.
It seems the ivy2local path format doesn't match the path of the published artefacts:

Given:

  override def repositories: T[Seq[String]] =
    super.repositories() ++ Seq(
      coursier.LocalRepositories.ivy2Local.repr,
    )

or

  override def repositories: T[Seq[String]] =
    super.repositories() ++ Seq(
      "ivy2Local", // from coursier.internal.PlatformRepositoryParser
    )

Then

$ sh mill show 'hail[].repositories'
[1/1] show
[
  "ivy:file:/home/edmund/.ivy2/local/[organisation]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]"
]

But:

$ sh mill show 'ext.pyspark-bom[2.12,3.5.3].publishLocalCached'
[1/1] show
[1-33/33] ext.pyspark-bom[2.12,3.5.3].publishLocalCached
[1-33] Publishing Artifact(org.apache.spark,pyspark-bom_2.12,3.5.3) to ivy repo /home/edmund/.ivy2/local
[
  "ref:v1:5b359a98:/home/edmund/.ivy2/local/org.apache.spark/pyspark-bom_2.12/3.5.3/poms/pyspark-bom_2.12.pom",
  "ref:v1:74414226:/home/edmund/.ivy2/local/org.apache.spark/pyspark-bom_2.12/3.5.3/ivys/ivy.xml",
  "ref:v1:93328a19:/home/edmund/.ivy2/local/org.apache.spark/pyspark-bom_2.12/3.5.3/srcs/pyspark-bom_2.12-sources.jar",
  "ref:v1:93328a19:/home/edmund/.ivy2/local/org.apache.spark/pyspark-bom_2.12/3.5.3/docs/pyspark-bom_2.12-javadoc.jar"
]

I'll try with the local maven publisher as you suggest. Thanks!

[ehigham]

Fantastic news!

$ sh mill __.publishM2Local

and configuring the repositories as:

  override def repositories: T[Seq[String]] =
    Seq("central", "m2Local")

Now my local boms are used! Thank you so much for your help.