Script to automatically update all library versions

[cpovirk]

Similar to Maven commands like:

• mvn versions:use-latest-versions
• mvn versions:use-latest-releases
• mvn versions:update-properties

And just to spam more keywords, here are some Maven commands that display updates but don't make the changes:

• mvn versions:display-property-updates
• mvn versions:display-dependency-updates

I poked around the web and found some a project for keeping Bazel deps up to date, but it has its own system that might not play well with bazel-common:
https://github.com/johnynek/bazel-deps

I notice that it's mentioned in:
https://github.com/bazelbuild/BUILD_file_generator#third-party-maven-dependencies

[ronshapiro]

We could decide to do something with this: https://blog.bazel.build/2019/03/31/rules-jvm-external-maven.html. I'd like to figure out what it does with dependency conflicts first to make sure.

Then we can install that with a private repository name and then wire up processors + exports manually

[jin]

@ronshapiro it has the resolution logic of Coursier. I can imagine instantiating a custom maven_install for bazel-common, like

maven_install(
    name = "bazel_common_maven",
    artifacts = # bazel common artifacts,
    repositories = # repos,
)

And then downstream libraries will depend on @bazel_common_maven//:com_google_guava_guava and @bazel_common_maven//:junit_junit, for example. They can also declare additional dependencies using their own maven_install declarations, which will be namespaced by the repository name.

Let me know if you have other questions or suggestions for rules_jvm_external.

[cpovirk]

Another thing for us to consider is using the Google Cloud Platform Libraries BOM (Maven central, GitHub). It provides a set of libraries that have been statically checked for linkage errors. This could help us avoid errors like this one (fix) (and maybe there was another error that required this?).

If I did my testing right, then the BOM doesn't yet cover everything that we'd like it to. But maybe we can help add some libraries, or at least we can use the BOM as our own starting point.

[edit: The commands below are my attempt to generate a list of libraries that bazel-common includes but the BOM does not specify a version for -- neither explicitly (by listing them) nor implicitly (by listing a library that depends on them).]

$ ( tmp && mvn help:effective-pom -f $HOME/.m2/repository/com/google/cloud/libraries-bom/3.3.0/libraries-bom-3.3.0.pom -Doutput=/tmp/wholeep && ( echo "plugins {"; echo "    id 'java-library'"; echo "}"; echo "repositories {"; echo "    jcenter()"; echo "}"; echo "dependencies {"; xmlstarlet sel -t -m '/_:project/_:dependencyManagement/_:dependencies/_:dependency' -v _:groupId -o : -v _:artifactId -o : -v _:version -n /tmp/wholeep | sort | prepend "    implementation '" | append "'" ; echo '}' ) > build.gradle && $HOME/gradle-6.0.1/bin/gradle :dependencies | grep -v '[-]>' | grep -o '[^: ]*:[^: ]*:[^ ]*' | sort -u | grep -Fv '[' > /tmp/bomversions )

$ ( cd $HOME/clients/bazel-common-gray/bazel-common/ && bazel query --output=xml 'attr(tags, maven_coordinates.*, kind(java_import, deps(//...)))'  | grep -oP 'maven_coordinates=\K[^:]*:[^:]*:[^:"]*' | sort > /tmp/bazelcommonversions )

$ comm -23 <(grep -o '^[^: ]*:[^: ]*' /tmp/bazelcommonversions) <(grep -o '^[^: ]*:[^: ]*' /tmp/bomversions)
com.google.auto:auto-common
com.google.auto.factory:auto-factory
com.google.auto.service:auto-service
com.googlecode.java-diff-utils:diffutils
com.google.common.inject:inject-common
com.google.errorprone:error_prone_annotation
com.google.errorprone:error_prone_check_api
com.google.errorprone:javac-shaded
com.google.googlejavaformat:google-java-format
com.google.guava:guava-beta-checker
com.google.testing.compile:compile-testing
com.google.truth.extensions:truth-java8-extension
com.google.truth:truth
com.squareup:javapoet
io.grpc:grpc-all
javax.annotation:jsr250-api
javax.enterprise:cdi-api
javax.inject:javax.inject-tck
log4j:log4j
net.bytebuddy:byte-buddy
net.bytebuddy:byte-buddy-agent
net.ltgt.gradle.incap:incap
net.ltgt.gradle.incap:incap-processor
org.apache.ant:ant
org.apache.ant:ant-launcher
org.apache.bcel:bcel
org.apache.logging.log4j:log4j-api
org.apache.logging.log4j:log4j-core
org.apache.maven:maven-artifact
org.apache.maven:maven-model
org.apache.maven:maven-plugin-api
org.checkerframework:checker-qual
org.checkerframework:dataflow
org.checkerframework:javacutil
org.codehaus.plexus:plexus-classworlds
org.codehaus.plexus:plexus-component-annotations
org.codehaus.plexus:plexus-utils
org.eclipse.sisu:org.eclipse.sisu.inject
org.eclipse.sisu:org.eclipse.sisu.plexus
org.mockito:mockito-core
org.objenesis:objenesis
org.ow2.asm:asm
org.ow2.asm:asm-commons
org.ow2.asm:asm-tree
org.pantsbuild:jarjar

[cpovirk]

I just saw 842dd97, which fixes a linkage error, fly by.

(One thing worth noting from that is that some of the artifacts in our list are build tools, not "traditional dependencies" in the narrower sense. If we're lucky, we can easily find compatible versions of all those artifacts, build tools and "traditional dependencies" both. But if not, it would be sufficient to find a set of compatible versions for each individual tool (in addition, of course, to the set for all the "traditional dependencies").)

[cpovirk]

Another linkage error fixed in 6db2b87.

[cpovirk]

The even easier thing would be automatic updates, like we get from Dependabot for most of our repos.

As of this writing, Dependabot doesn't support Bazel, but RenovateBot does. Note that RenovateBot does not yet support maven_install, but we are not using that yet.

I see a smallish number of renovate.json files in the third-party section of our internal repository. But I see some evidence that it's used from almost all repos under https://github.com/googleapis, and I see that Google's GitHub admins are willing to grant the needed permissions to the RenovateBot app. So this seems doable.

[cpovirk]

I suspect that this will essentially happen automatically if we switch to rules_jvm_external or whatever the most standard thing is nowadays.

[cpovirk]

(We recently set up Dependabot, but that supports only bumping things like rules_java, not our actual Java dependencies.)