From 554e11c261fe3a66e714cb9224e8cf870064c492 Mon Sep 17 00:00:00 2001 From: Russ White <356303+russwyte@users.noreply.github.com> Date: Tue, 23 Jun 2026 17:08:00 -0500 Subject: [PATCH 1/2] Run compiler in-process for the sbt 2.0 / Scala 3 plugin The sbt and Mill plugins previously embedded a 24 MB marklit-cli fat jar and drove it as a subprocess (or a long-lived JSON-RPC daemon) because a Scala 2.12 / sbt 1.x plugin could not load the Scala 3 compiler as a library. On sbt 2.0 the plugin compiles against Scala 3, so it can depend on marklit-compiler directly and call it in-process. - Extract the CLI's multi-file orchestration into a side-effect-free MarklitRun facade in the compiler module; the CLI becomes a thin shell over it. Add MarklitError.ResolutionError and MarklitRunSpec. - Publish marklit-compiler-api, marklit-core, marklit-compiler (Scala 3). Bundle the two compiler-shim jars as Compile resources of marklit-compiler so CompilerFactory resolves them off the classpath. - Convert sbt-marklit to an sbt 2.0 / Scala 3 AutoPlugin published as sbt-marklit_sbt2_3. It depends on marklit-compiler and holds one warm CompilerFactory for the sbt session (MarklitSession) instead of a daemon. Handle sbt 2.0 specifics: fileConverter for virtual-file classpaths, Def.uncached for side-effecting tasks, and the new target/out/jvm layout in cross-build classpath discovery. Add a scripted test. - Convert the Mill plugin the same way (warm factory in a Task.Worker); built and tested locally, not published. - Delete the subprocess/daemon/JSON-RPC machinery from both plugins. - Update README and plugin/example docs to match. --- README.md | 53 +-- build.sbt | 134 ++++--- .../main/scala/marklit/cli/MarklitCli.scala | 246 +++---------- .../src/main/scala/marklit/MarklitRun.scala | 298 ++++++++++++++++ .../marklit/compiler/CompilerFactory.scala | 25 +- .../marklit/compiler/ScalaCompiler.scala | 30 +- .../test/scala/marklit/MarklitRunSpec.scala | 134 +++++++ .../scala/marklit/model/MarklitError.scala | 2 + examples/mill/README.md | 77 ++-- examples/mill/build.mill | 2 +- examples/sbt/project/build.properties | 2 +- examples/sbt/project/plugins.sbt | 2 +- mill-plugin/README.md | 38 +- mill-plugin/build.mill | 187 ++-------- .../marklit/mill/MarklitDaemonClient.scala | 157 --------- .../src/marklit/mill/MarklitJson.scala | 115 ------ .../src/marklit/mill/MarklitModule.scala | 285 ++++----------- .../src/marklit/mill/MarklitWorker.scala | 61 ++++ project/build.properties | 2 +- project/plugins.sbt | 2 +- .../marklit/sbt/MarklitDaemonClient.scala | 191 ---------- .../marklit/sbt/MarklitDaemonRegistry.scala | 44 --- .../scala/marklit/sbt/MarklitInProcess.scala | 75 ++++ .../main/scala/marklit/sbt/MarklitJson.scala | 138 -------- .../scala/marklit/sbt/MarklitPlugin.scala | 328 +++++++----------- .../scala/marklit/sbt/MarklitRunner.scala | 216 ------------ .../scala/marklit/sbt/MarklitSession.scala | 71 ++++ .../src/sbt-test/marklit/basic/build.sbt | 16 + .../src/sbt-test/marklit/basic/changes/bad.md | 7 + .../sbt-test/marklit/basic/markdown/good.md | 8 + .../marklit/basic/project/build.properties | 1 + .../marklit/basic/project/plugins.sbt | 3 + sbt-plugin/src/sbt-test/marklit/basic/test | 11 + 33 files changed, 1164 insertions(+), 1797 deletions(-) create mode 100644 compiler/src/main/scala/marklit/MarklitRun.scala create mode 100644 compiler/src/test/scala/marklit/MarklitRunSpec.scala delete mode 100644 mill-plugin/src/marklit/mill/MarklitDaemonClient.scala delete mode 100644 mill-plugin/src/marklit/mill/MarklitJson.scala create mode 100644 mill-plugin/src/marklit/mill/MarklitWorker.scala delete mode 100644 sbt-plugin/src/main/scala/marklit/sbt/MarklitDaemonClient.scala delete mode 100644 sbt-plugin/src/main/scala/marklit/sbt/MarklitDaemonRegistry.scala create mode 100644 sbt-plugin/src/main/scala/marklit/sbt/MarklitInProcess.scala delete mode 100644 sbt-plugin/src/main/scala/marklit/sbt/MarklitJson.scala delete mode 100644 sbt-plugin/src/main/scala/marklit/sbt/MarklitRunner.scala create mode 100644 sbt-plugin/src/main/scala/marklit/sbt/MarklitSession.scala create mode 100644 sbt-plugin/src/sbt-test/marklit/basic/build.sbt create mode 100644 sbt-plugin/src/sbt-test/marklit/basic/changes/bad.md create mode 100644 sbt-plugin/src/sbt-test/marklit/basic/markdown/good.md create mode 100644 sbt-plugin/src/sbt-test/marklit/basic/project/build.properties create mode 100644 sbt-plugin/src/sbt-test/marklit/basic/project/plugins.sbt create mode 100644 sbt-plugin/src/sbt-test/marklit/basic/test diff --git a/README.md b/README.md index c8b3e26..ba606ec 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # marklit -[![Maven Repository](https://img.shields.io/maven-central/v/io.github.russwyte/sbt-marklit_2.12_1.0?logo=apachemaven)](https://mvnrepository.com/artifact/io.github.russwyte/sbt-marklit) +[![Maven Repository](https://img.shields.io/maven-central/v/io.github.russwyte/sbt-marklit_sbt2_3?logo=apachemaven)](https://mvnrepository.com/artifact/io.github.russwyte/sbt-marklit) **Typechecked Scala documentation that actually runs your code — across multiple Scala versions, in the same file.** @@ -10,13 +10,13 @@ If your docs claim something works, marklit makes the build break when it doesn' ## Installation -marklit ships as an sbt plugin. Add it to `project/plugins.sbt`: +marklit ships as an sbt plugin (sbt 2.0+, published for Scala 3). Add it to `project/plugins.sbt`: ```scala -addSbtPlugin("io.github.russwyte" % "sbt-marklit" % "0.0.1") +addSbtPlugin("io.github.russwyte" % "sbt-marklit" % "0.1.0") ``` -The plugin is an `AutoPlugin` triggered on every JVM project — no `enablePlugins` needed. See [Build tool integration](#build-tool-integration) for wiring up a docs project. +The plugin is an `AutoPlugin` triggered on every JVM project — no `enablePlugins` needed. It calls marklit's compiler in-process (no subprocess), so the only thing it adds to your build is the plugin dependency. See [Build tool integration](#build-tool-integration) for wiring up a docs project. ## What sets marklit apart @@ -37,7 +37,7 @@ marklit lives in [mdoc](https://scalameta.org/mdoc/)'s neighborhood. Many ideas | **Built-in ZIO runtime** (`zio-app` modifier) | ✗ | **✓** | | Multiple modifiers per block (`silent,id=setup`) | partial | **✓** | | Scoped-by-default — blocks isolated unless you opt in | ✗ | **✓** | -| **Persistent on-disk block cache + warm-classloader daemon** | ✗ | **✓** | +| **Persistent on-disk block cache + warm in-process compilers** | ✗ | **✓** | The multi-version story is the one that pays for the project. You can write a migration guide that shows the *same code* compiled on 2.13 and 3.x, side by side, with both outputs verified by real compilers. @@ -99,7 +99,7 @@ println(scala.util.Properties.versionNumberString) // "2.13.16" actually ``` ```` -The `marklit-cli` jar is built once and bundles only thin shims against the dotc (3.x) and nsc (2.13) compiler APIs. Every per-version classloader gets a fresh copy of the *user-requested* compiler and its matching standard library — so user code is always compiled and run by the version they asked for, never by the bundled shim's version. +marklit ships only thin shims against the dotc (3.x) and nsc (2.13) compiler APIs (bundled as resources inside `marklit-compiler`). Every per-version classloader gets a fresh copy of the *user-requested* compiler and its matching standard library — so user code is always compiled and run by the version they asked for, never by the shim's compile-time version. A worked example with five different versions in one file lives in [examples/base/src/main/markdown/scopes-and-versions.md](examples/base/src/main/markdown/scopes-and-versions.md). @@ -123,14 +123,14 @@ Both blocks reference `example.Greeter`. Both compile. Both produce real output ## Performance -Compiling Scala (especially across multiple versions) is expensive. marklit has two layers of caching that make warm runs roughly **3-4× faster** than cold runs on the included examples: +Compiling Scala (especially across multiple versions) is expensive. marklit has two layers of caching that make warm runs dramatically faster than cold runs on the included examples: -- **Long-lived daemon JVM.** The build plugins talk to a marklit subprocess over JSON-RPC instead of spawning a fresh JVM per task. Per-version compiler classloaders stay warm across `marklitGenerate` invocations within a build session. Cold-start of a new Scala version is ~1-2s; subsequent compiles against the same version reuse that loader. +- **Warm in-process compilers.** The build plugins call marklit's `CompilerFactory` in-process and hold it for the whole build session — sbt keeps one factory for the session; Mill keeps it in a long-lived `Task.Worker`. Per-version compiler classloaders are built once and reused across `marklitGenerate` / `marklitCheck` invocations. Cold-start of a new Scala version is ~1-2s; subsequent compiles against the same version reuse that warm loader. (This replaced the earlier out-of-process daemon — same warmth, no subprocess.) - **Persistent SHA-256 block cache.** Every block's compiled `.class` files are stored on disk keyed by a hash of `(code, prior code, scalaVersion, classpath, scalac options, …)`. A cache hit skips both compile and re-emit — execution loads the cached class files directly. Cache lives at `target/marklit-cache/` (sbt) or `out//marklitCacheDir.dest/marklit-cache/` (Mill); both plugins clean it as part of `/marklitClean`. -Cold-vs-warm on [examples/sbt/](examples/sbt/) (4 markdown files, 46 blocks across 4 Scala versions): **~20s → ~6s**. Mill: **~18s → ~4s**. Same machine, no other changes. +The warm-compiler effect is large within a session: on [examples/sbt/](examples/sbt/), a 23-block multi-version file drops from ~8s on the cold first run to tens of milliseconds on a warm re-check. -Both layers are on by default. To disable the daemon: `marklitDaemon := false` (sbt) / `def marklitDaemonEnabled = false` (Mill). To disable the disk cache: `marklitCacheDirectory := None` (sbt) / `def marklitCacheDir = None` (Mill). +The warm factory is always on (it's just an in-process object). To disable the disk cache: `marklitCacheDirectory := None` (sbt) / `def marklitCacheDir = None` (Mill). ## Modifiers @@ -282,7 +282,7 @@ Default remains off: marklit's per-block isolation is the better default for ref ```scala // project/plugins.sbt -addSbtPlugin("io.github.russwyte" % "sbt-marklit" % "0.0.1") +addSbtPlugin("io.github.russwyte" % "sbt-marklit" % "0.1.0") ``` ```scala @@ -312,11 +312,11 @@ A worked multi-version example lives in [examples/sbt/](examples/sbt/). ### Mill -> **Not published yet.** The Mill plugin is built in this repo but hasn't been released to Maven Central — the snippet below is a preview of the intended usage. A release is coming soon; until then, use the sbt plugin or the standalone CLI. +> **Not published yet.** The Mill plugin is built and tested in this repo but hasn't been released to Maven Central — the snippet below is a preview of the intended usage. Until it's published you can build it locally: publish marklit's libraries and the plugin to your local repo (see [Building from source](#building-from-source)), then depend on the local version. A release is coming soon; until then, use the sbt plugin or the standalone CLI. ```scala //| mvnDeps: -//| - io.github.russwyte::mill-marklit:0.1.0-SNAPSHOT +//| - io.github.russwyte::mill-marklit:0.1.0 import marklit.mill.MarklitModule @@ -339,7 +339,7 @@ A worked example lives in [examples/mill/](examples/mill/). ### CLI -The CLI is bundled inside both build plugins, but you can also run it standalone: +The build plugins call marklit in-process, but the same engine is also available as a standalone CLI (built from this repo — see [Building from source](#building-from-source)): ```sh marklit docs/ --out target/docs/ @@ -371,7 +371,7 @@ You can also declare dependencies inline in a Markdown file using [scala-cli](ht //> using options -feature -deprecation ``` -Precedence (highest to lowest): per-block `scala=` → in-source `//> using scala` → CLI `--scala-version` → bundled shim's compile-time version. +Precedence (highest to lowest): per-block `scala=` → in-source `//> using scala` → CLI `--scala-version` → the shim's compile-time version. ## How it actually works @@ -389,14 +389,14 @@ The per-version classloader pattern is the same one Bloop and Metals use to host | Module | What it is | | --- | --- | -| [core/](core/) | Parser, scope manager, document processor, renderer. Pure types, no compiler dependency. Published. | -| [compiler-api/](compiler-api/) | Java-only interfaces between marklit and the per-version compiler shims. | -| [compiler-shim/](compiler-shim/) | Thin shim against `dotty.tools.dotc.*`. Pinned to the oldest 3.x we support; the API surface is stable across 3.x. | -| [compiler-shim-2/](compiler-shim-2/) | Thin shim against `scala.tools.nsc.*` for 2.13. | -| [compiler/](compiler/) | Orchestration: Coursier-based `CompilerFactory`, classloader management, ZIO layers. Does not directly depend on `scala3-compiler` or `scala-compiler`. | -| [cli/](cli/) | `zio-cli` front-end. Distributed as a fat jar bundled inside the build plugins. | -| [sbt-plugin/](sbt-plugin/) | `sbt-marklit` AutoPlugin. Published. | -| [mill-plugin/](mill-plugin/) | `mill-marklit` module trait. Built with Mill. Published. | +| [core/](core/) | Parser, scope manager, document processor, renderer. Pure types, no compiler dependency. Published (`marklit-core`). | +| [compiler-api/](compiler-api/) | Java-only interfaces between marklit and the per-version compiler shims. Published (`marklit-compiler-api`). | +| [compiler-shim/](compiler-shim/) | Thin shim against `dotty.tools.dotc.*`. Pinned to the oldest 3.x we support; the API surface is stable across 3.x. Bundled as a resource inside `marklit-compiler` (not published separately). | +| [compiler-shim-2/](compiler-shim-2/) | Thin shim against `scala.tools.nsc.*` for 2.13. Bundled inside `marklit-compiler`. | +| [compiler/](compiler/) | Orchestration: the `MarklitRun` facade, Coursier-based `CompilerFactory`, classloader management, ZIO layers. Does not directly depend on `scala3-compiler` or `scala-compiler`. Published (`marklit-compiler`); the build plugins call it in-process. | +| [cli/](cli/) | `zio-cli` front-end — a standalone fat-jar tool built from this repo. Not published; the build plugins no longer embed it. | +| [sbt-plugin/](sbt-plugin/) | `sbt-marklit` AutoPlugin (sbt 2.0, Scala 3). Published as `sbt-marklit_sbt2_3`; depends on `marklit-compiler` and runs it in-process. | +| [mill-plugin/](mill-plugin/) | `mill-marklit` module trait. Built and tested with Mill; calls `marklit-compiler` in-process. Not yet published. | ## Limitations @@ -409,11 +409,14 @@ The per-version classloader pattern is the same one Bloop and Metals use to host ## Building from source ```sh -sbt publishAll # build CLI fat jar + publish sbt plugin locally -(cd mill-plugin && mill plugin.publishLocal) # publish mill plugin locally +sbt publishAll # publish marklit-compiler (+ api, core) and the sbt plugin locally +(cd mill-plugin && mill plugin.publishLocal) # build + publish the mill plugin locally sbt test # full test suite +sbt "plugin/scripted" # sbt-plugin integration tests (real sbt builds) ``` +`publishAll` publishes the libraries the sbt plugin depends on (`marklit-compiler-api`, `marklit-core`, `marklit-compiler`) plus `sbt-marklit` to your local Ivy repo, so a downstream project (or the [examples/](examples/)) can resolve them. The standalone CLI fat jar is built separately with `sbt cli/assembly`. + ## Inspiration marklit owes its shape to [mdoc](https://scalameta.org/mdoc/) and reuses mdoc's modifier vocabulary where it makes sense. The differences — multiple modifiers per block, scope inheritance with append, scoped-by-default semantics, real per-block multi-version compilation, ZIO runtime, cross-built dependency awareness — are the parts worth comparing. diff --git a/build.sbt b/build.sbt index 095163a..c564b2e 100644 --- a/build.sbt +++ b/build.sbt @@ -1,6 +1,8 @@ // Scala version used to compile marklit's own modules (core, compiler, cli, -// the Mill plugin trait — everything except the shim and the sbt plugin). -val marklitScalaVersion = "3.8.3" +// the Mill plugin trait — everything except the shim). Aligned with the Scala +// 3 version the sbt 2.0 plugin compiles against so the plugin consumes +// marklit-compiler at an identical version. +val marklitScalaVersion = "3.8.4" // Scala version the compiler-shim is built against. Pinned to the oldest // supported 3.x to keep the shim's dotc API surface compatible at runtime @@ -58,6 +60,20 @@ usePgpKeyHex( sys.env.getOrElse("PGP_KEY_HEX", "2F64727A87F1BCF42FD307DD8582C4F16659A7D6") ) +// Copy a packaged jar (an sbt 2.0 virtual file ref) into a resource dir. +// fileConverter must be read inside each Def.task — `.value` is a macro that +// only expands in a task/setting scope — but the conversion + copy lives here. +def copyJarResource( + converter: xsbti.FileConverter, + jar: xsbti.HashedVirtualFileRef, + targetDir: File, + name: String +): Seq[File] = { + val targetFile = targetDir / name + IO.copyFile(converter.toPath(jar).toFile, targetFile) + Seq(targetFile) +} + // Only the sbt plugin is published; everything else is internal or bundled. lazy val publishSettings = Seq( publishMavenStyle := true, @@ -92,9 +108,9 @@ lazy val root = project // both the orchestrator's classloader and each per-version compiler classloader. lazy val compilerApi = project .in(file("compiler-api")) + .settings(publishSettings) .settings( name := "marklit-compiler-api", - publish / skip := true, crossPaths := false, autoScalaLibrary := false, Compile / doc / sources := Seq.empty, @@ -173,23 +189,26 @@ lazy val compilerShim2 = project }.taskValue ) -// Alias to rebuild CLI assembly and publish plugin locally +// Publish the plugin and the libraries it depends on to the local Ivy repo. +// The plugin declares marklit-compiler as a libraryDependency, so its deps +// (compiler-api, core, compiler) must be published for that dep to resolve. addCommandAlias( "publishAll", - "; cli/clean; cli/assembly; plugin/clean; plugin/publishLocal" + "; compilerApi/publishLocal; core/publishLocal; compiler/publishLocal; plugin/clean; plugin/publishLocal" ) -// Release the plugin to Sonatype Central (requires git tag + signing key + creds). +// Release the plugin and its published dependencies to Sonatype Central +// (requires git tag + signing key + creds). addCommandAlias( "release", - "; plugin/clean; plugin/publishSigned; sonaRelease" + "; compilerApi/publishSigned; core/publishSigned; compiler/publishSigned; plugin/clean; plugin/publishSigned; sonaRelease" ) lazy val core = project .in(file("core")) + .settings(publishSettings) .settings( name := "marklit-core", - publish / skip := true, libraryDependencies ++= Seq( "dev.zio" %% "zio" % zioVersion, "dev.zio" %% "zio-streams" % zioVersion, @@ -206,9 +225,9 @@ lazy val core = project lazy val compiler = project .in(file("compiler")) .dependsOn(core, compilerApi) + .settings(publishSettings) .settings( name := "marklit-compiler", - publish / skip := true, libraryDependencies ++= Seq( // ZIO "dev.zio" %% "zio" % zioVersion, @@ -220,21 +239,28 @@ lazy val compiler = project "dev.zio" %% "zio-json" % "0.9.2" ), testFrameworks += new TestFramework("zio.test.sbt.ZTestFramework"), - // Both shim jars are wired into the compiler test classpath as resources - // so CompilerFactory tests can load them without going through the CLI. - Test / resourceGenerators += Def.task { - val shimJar = (compilerShim / Compile / packageBin).value - val resourceDir = (Test / resourceManaged).value - val targetFile = resourceDir / "marklit-compiler-shim.jar" - IO.copyFile(shimJar, targetFile) - Seq(targetFile) + // Both shim jars are bundled as Compile resources so they ride inside the + // published marklit-compiler jar. CompilerFactory.copyResource reads them + // via getClass.getResourceAsStream at runtime — so any consumer of + // marklit-compiler (the CLI fat jar, the sbt/Mill plugins in-process) finds + // them on its classpath without a separate published shim artifact. The + // compiler's own tests inherit Compile resources, so CompilerFactory tests + // keep working with no Test-scoped generator. + Compile / resourceGenerators += Def.task { + copyJarResource( + fileConverter.value, + (compilerShim / Compile / packageBin).value, + (Compile / resourceManaged).value, + "marklit-compiler-shim.jar" + ) }.taskValue, - Test / resourceGenerators += Def.task { - val shimJar = (compilerShim2 / Compile / packageBin).value - val resourceDir = (Test / resourceManaged).value - val targetFile = resourceDir / "marklit-compiler-shim-2.jar" - IO.copyFile(shimJar, targetFile) - Seq(targetFile) + Compile / resourceGenerators += Def.task { + copyJarResource( + fileConverter.value, + (compilerShim2 / Compile / packageBin).value, + (Compile / resourceManaged).value, + "marklit-compiler-shim-2.jar" + ) }.taskValue ) @@ -255,25 +281,9 @@ lazy val cli = project "dev.zio" %% "zio-test-sbt" % zioVersion % Test ), testFrameworks += new TestFramework("zio.test.sbt.ZTestFramework"), - // Bundle both compiler-shim jars as resources. CompilerFactory extracts - // the right one at runtime onto each per-version compiler classloader - // depending on the requested major (3.x → dotc shim; 2.13.x → nsc shim). - // Each jar is thin (shim classes only — scala-compiler is Provided), so - // it works against whatever exact version the user requested. - Compile / resourceGenerators += Def.task { - val shimJar = (compilerShim / Compile / packageBin).value - val resourceDir = (Compile / resourceManaged).value - val targetFile = resourceDir / "marklit-compiler-shim.jar" - IO.copyFile(shimJar, targetFile) - Seq(targetFile) - }.taskValue, - Compile / resourceGenerators += Def.task { - val shimJar = (compilerShim2 / Compile / packageBin).value - val resourceDir = (Compile / resourceManaged).value - val targetFile = resourceDir / "marklit-compiler-shim-2.jar" - IO.copyFile(shimJar, targetFile) - Seq(targetFile) - }.taskValue, + // The two compiler-shim jars are bundled as Compile resources of + // marklit-compiler now, so they arrive here transitively via + // dependsOn(compiler) and the assembly fat jar still contains them. // Assembly settings for fat jar assembly / assemblyJarName := "marklit-cli.jar", assembly / mainClass := Some("marklit.cli.MarklitCli"), @@ -304,22 +314,32 @@ lazy val cli = project lazy val plugin = project .in(file("sbt-plugin")) .enablePlugins(SbtPlugin) + .dependsOn(compiler) .settings(publishSettings) .settings( name := "sbt-marklit", - scalaVersion := "2.12.20", - // Copy CLI assembly jar to plugin resources before packaging - Compile / resourceGenerators += Def.task { - val cliJar = (cli / assembly).value - val resourceDir = (Compile / resourceManaged).value - val targetFile = resourceDir / "marklit-cli.jar" - IO.copyFile(cliJar, targetFile) - Seq(targetFile) - }.taskValue, - // The Compile resourceGenerator above pulls in `cli / assembly`, so the CLI - // fat jar is rebuilt before any packaging task (publish, publishLocal, - // publishSigned). These overrides make that dependency explicit. - publish := (publish dependsOn (cli / assembly)).value, - publishLocal := (publishLocal dependsOn (cli / assembly)).value, - scalacOptions := Seq("-deprecation", "-feature") + // sbt 2.0 plugins compile against Scala 3 and publish with the _sbt2_3 + // suffix. The plugin calls marklit-compiler in-process, so it depends on it + // as an ordinary library (dependsOn here for local dev; the published POM + // records the libraryDependency below for downstream resolution). + scalaVersion := marklitScalaVersion, + libraryDependencies += + "io.github.russwyte" %% "marklit-compiler" % version.value, + scalacOptions := Seq("-deprecation", "-feature"), + // Scripted tests: publish this plugin + its libs to the local repo first, + // and pass the version through so each test's project/plugins.sbt can + // resolve it via `sys.props("plugin.version")`. + scriptedDependencies := scriptedDependencies + .dependsOn( + compilerApi / publishLocal, + core / publishLocal, + compiler / publishLocal, + publishLocal + ) + .value, + scriptedLaunchOpts ++= Seq( + "-Xmx1024m", + s"-Dplugin.version=${version.value}" + ), + scriptedBufferLog := false ) diff --git a/cli/src/main/scala/marklit/cli/MarklitCli.scala b/cli/src/main/scala/marklit/cli/MarklitCli.scala index 63e025e..7bad1ec 100644 --- a/cli/src/main/scala/marklit/cli/MarklitCli.scala +++ b/cli/src/main/scala/marklit/cli/MarklitCli.scala @@ -1,14 +1,11 @@ package marklit.cli -import marklit.{Marklit, MarklitResult} -import marklit.compiler.CompilerFactory -import marklit.renderer.{MarkdownRenderer, RenderConfig} -import marklit.resolver.DependencyResolver +import marklit.{MarklitRun, MarklitRunConfig} import zio.* import zio.cli.* import zio.cli.HelpDoc.Span.text -import java.nio.file.{Files, Path} +import java.nio.file.Path /** CLI options */ final case class MarklitOptions( @@ -244,215 +241,58 @@ object MarklitCli extends ZIOCliDefault: effect.as(options) } - def runMarklit(options: MarklitOptions): ZIO[Any, Throwable, Unit] = - for - _ <- ZIO.when(options.verbose)( - Console.printLine(s"Processing ${options.inputFiles.size} file(s)...") - ) - - // Validate input files exist - _ <- ZIO.foreachDiscard(options.inputFiles) { path => - ZIO - .fail(new RuntimeException(s"File not found: $path")) - .unless(Files.exists(path)) - } - - // Effective default Scala version, before per-file using directives - // are applied. Precedence: --scala-version > shim's compile-time version. - shimDefault = CompilerFactory.defaultScalaVersion - cliDefault = options.scalaVersion.getOrElse(shimDefault) - - _ <- ZIO.when(options.verbose)( - Console.printLine(s"Default Scala version: $cliDefault") - ) - - // Read each input file and parse its using directives. Per-file parsing - // (rather than one merged set) lets us emit one notification per file - // when a `//> using scala` directive overrides the CLI default. - perFile <- ZIO.foreach(options.inputFiles) { path => - for - content <- ZIO.attempt(Files.readString(path)) - directives = DependencyResolver.parseUsingDirectives(content) - yield (path, directives) - } - - // Aggregate dependencies across files (kept as a single resolution - // bundle to avoid resolving the same dep N times). - allDeps = options.dependencies.toVector ++ perFile.toVector - .flatMap(_._2.dependencies) - .distinct - allRepos = options.repositories.toVector ++ perFile.toVector - .flatMap(_._2.repositories) - .distinct - allScalacOptions = perFile.toVector.flatMap(_._2.scalacOptions).distinct - - _ <- ZIO.when(options.verbose && allDeps.nonEmpty)( - Console.printLine(s"Resolving dependencies: ${allDeps.mkString(", ")}") - ) - - // Resolve dependencies once, against the CLI default version. Per-file - // overrides only affect compilation, not Coursier resolution of user deps. - resolvedJars <- { - if allDeps.nonEmpty then - DependencyResolver - .resolve(allDeps, cliDefault, allRepos) - .mapError(e => - new RuntimeException( - s"Dependency resolution failed: ${e.getMessage}" - ) - ) - else ZIO.succeed(Vector.empty[String]) - } - - _ <- ZIO.when(options.verbose && resolvedJars.nonEmpty)( - Console.printLine(s"Resolved ${resolvedJars.size} JAR(s)") - ) - - // Filter scala-library/scala3-library out of resolved deps — the - // per-version classloader brings matching copies transitively, and a - // user-resolved copy at a different version causes duplicate-package - // errors. - cliClasspath = options.classpath - .map(_.split("[;:]").toVector) - .getOrElse(Vector.empty) - filteredResolved = resolvedJars.filterNot { jar => - val fileName = java.nio.file.Paths.get(jar).getFileName.toString - fileName.startsWith("scala-library") || fileName.startsWith( - "scala3-library" - ) - } - fullClasspath = cliClasspath ++ filteredResolved + /** Split a `--classpath`-style flag value into entries (colon/semicolon + * separated), or empty when absent. + */ + private def splitClasspath(cp: Option[String]): Vector[String] = + cp.map(_.split("[;:]").toVector).getOrElse(Vector.empty) - // Per-major classpaths from the build plugin's cross-publish: each - // entry's classpath is built against that major and is the right one - // to use for `marklit:scala=.x.y` blocks. The default - // --classpath above remains the path used for blocks compiled at the - // default major. Resolved Coursier deps are version-agnostic enough to - // share across majors here; the per-version classloader's transitive - // scala-library/scala3-library wins by virtue of the filter above. - majorClasspaths = Map( - "2" -> options.classpath2, - "3" -> options.classpath3 - ).flatMap { case (m, cpOpt) => - cpOpt - .map(_.split("[;:]").toVector ++ filteredResolved) - .map(m -> _) - } + def runMarklit(options: MarklitOptions): ZIO[Any, Throwable, Unit] = + val config = MarklitRunConfig( + inputFiles = options.inputFiles.toVector, + outputDir = options.outputDir, + scalaVersion = options.scalaVersion, + classpath = splitClasspath(options.classpath), + classpath2 = splitClasspath(options.classpath2), + classpath3 = splitClasspath(options.classpath3), + deps = options.dependencies.toVector, + repos = options.repositories.toVector, + cacheDir = options.cacheDir, + pageScope = options.pageScope, + check = options.check, + showVersion = options.showVersionInOutput, + showWarnings = options.showWarningsInOutput, + verbose = options.verbose + ) - // Print override notifications: one info line per file whose using - // directive opts into a Scala version different from the CLI default. - _ <- ZIO.foreachDiscard(perFile) { case (path, directives) => - directives.scalaVersion match - case Some(v) if v != cliDefault => - Console.printLine( - s"$path: //> using scala $v overrides default $cliDefault" - ) - case _ => ZIO.unit - } + for + result <- MarklitRun + .run(config) + .mapError(e => new RuntimeException(e.pretty)) - // Process each file with its own effective default version (file - // directive wins over CLI default). Per-block specific versions are - // handled inside the processor via the same factory instance. - results <- ZIO - .foreach(perFile) { case (path, directives) => - val fileDefault = directives.scalaVersion.getOrElse(cliDefault) - // `allScalacOptions` already includes this file's directives via the - // flatMap above; the parsed value is itself a ListSet, so duplicates - // within a file collapsed at parse time. No further dedup needed. - val fileScalacOptions = allScalacOptions - val absPath = path.toAbsolutePath - Marklit - .processFile(absPath) - .provideSome[CompilerFactory]( - Marklit.liveWithFactory( - fileDefault, - fullClasspath, - fileScalacOptions, - majorClasspaths, - options.cacheDir, - if options.pageScope then marklit.model.ScopeMode.Page - else marklit.model.ScopeMode.Isolated - ) - ) - .mapError(e => new RuntimeException(e.pretty)) - } - .provide(CompilerFactory.layer) + // Surface the facade's informational notices (already verbose-gated). + _ <- ZIO.foreachDiscard(result.notices)(Console.printLine(_)) - // Report results - _ <- ZIO.foreachDiscard(results) { result => + // Per-file summary + verbose error/diagnostic detail. + _ <- ZIO.foreachDiscard(result.files) { file => for - _ <- Console.printLine(result.summary) - _ <- ZIO.when(options.verbose && result.errors.nonEmpty)( - ZIO.foreachDiscard(result.errors) { case (block, error) => - Console.printLine( - s" ${block.location.pretty}: ${error.pretty}" - ) + _ <- Console.printLine(file.summary) + _ <- ZIO.when(options.verbose && file.blockErrors.nonEmpty)( + ZIO.foreachDiscard(file.blockErrors) { case (loc, msg) => + Console.printLine(s" $loc: $msg") } ) - failedBlocks = result.processingResult.blockResults.filter(br => - !br.isSuccess && br.error.isEmpty && br.compileResult.exists( - !_.success - ) - ) - _ <- ZIO.when(options.verbose && failedBlocks.nonEmpty)( - ZIO.foreachDiscard(failedBlocks) { br => - val diagnostics = - br.compileResult.map(_.diagnostics).getOrElse(Nil) - val diagStr = diagnostics - .map(d => s"${d.severity}: ${d.message}") - .mkString("; ") - Console.printLine( - s" ${br.block.location.pretty}: Compile failed - $diagStr" - ) + _ <- ZIO.when(options.verbose && file.failedCompiles.nonEmpty)( + ZIO.foreachDiscard(file.failedCompiles) { case (loc, msg) => + Console.printLine(s" $loc: $msg") } ) yield () } - // Check for failures - failures = results.filterNot(_.isSuccess) + // Nonzero exit when any file failed (compile failures are data, not + // exceptions, inside the facade — we translate them to a CLI failure here). _ <- ZIO - .fail(new RuntimeException(s"${failures.size} file(s) failed")) - .unless(failures.isEmpty) - - // Write output if not in check mode - _ <- ZIO.when(!options.check && options.outputDir.isDefined)( - writeOutputs( - results.toVector, - options.outputDir.get, - options.verbose, - options.showVersionInOutput, - options.showWarningsInOutput - ) - ) - yield () - - /** Write final rendered markdown output */ - private def writeOutputs( - results: Vector[MarklitResult], - outputDir: Path, - verbose: Boolean, - showVersion: Boolean, - showWarnings: Boolean - ): ZIO[Any, Throwable, Unit] = - for - _ <- ZIO.attempt(Files.createDirectories(outputDir)) - _ <- ZIO.foreachDiscard(results) { result => - val outputPath = outputDir.resolve(result.sourceFile.getFileName) - val config = RenderConfig( - showScalaVersion = showVersion, - showCompileWarnings = showWarnings - ) - for - rendered <- ZIO.succeed( - MarkdownRenderer.render( - result.document, - result.processingResult, - config - ) - ) - _ <- ZIO.attempt(Files.writeString(outputPath, rendered)) - _ <- ZIO.when(verbose)(Console.printLine(s" Wrote: $outputPath")) - yield () - } + .fail(new RuntimeException(s"${result.failedCount} file(s) failed")) + .unless(result.success) yield () diff --git a/compiler/src/main/scala/marklit/MarklitRun.scala b/compiler/src/main/scala/marklit/MarklitRun.scala new file mode 100644 index 0000000..113f025 --- /dev/null +++ b/compiler/src/main/scala/marklit/MarklitRun.scala @@ -0,0 +1,298 @@ +package marklit + +import marklit.compiler.CompilerFactory +import marklit.model.{MarklitError, ScopeMode} +import marklit.renderer.{MarkdownRenderer, RenderConfig} +import marklit.resolver.DependencyResolver +import zio.* + +import java.nio.file.{Files, Path} + +/** Configuration for an end-to-end marklit run over a set of markdown files. + * + * This is the CLI-independent superset of the options the CLI used to thread + * through command-line flags. Classpaths arrive already split into entries + * (the CLI does the `[;:]` splitting before constructing this); `watch` and + * `daemon` have no place here — they are concerns of the caller's process + * model, not of the per-run orchestration. + */ +final case class MarklitRunConfig( + inputFiles: Vector[Path], + outputDir: Option[Path] = None, + scalaVersion: Option[String] = None, + classpath: Vector[String] = Vector.empty, + classpath2: Vector[String] = Vector.empty, + classpath3: Vector[String] = Vector.empty, + deps: Vector[String] = Vector.empty, + repos: Vector[String] = Vector.empty, + cacheDir: Option[Path] = None, + pageScope: Boolean = false, + check: Boolean = false, + showVersion: Boolean = true, + showWarnings: Boolean = true, + verbose: Boolean = false +) + +/** Per-file outcome with no Console side effects. `rendered` is populated when + * an output directory is configured and the file is not in check mode (so a + * caller can choose to write it itself); `outputPath` is where it would land. + */ +final case class MarklitFileReport( + sourceFile: Path, + success: Boolean, + summary: String, + blockErrors: Vector[(String, String)], + failedCompiles: Vector[(String, String)], + rendered: Option[String], + outputPath: Option[Path] +) + +/** Structured result of a run. `notices` are ordered informational lines the + * caller may surface verbatim (the facade has already applied the verbose + * gate when deciding which lines to include). Compile failures are reported as + * data via [[MarklitFileReport.success]]; only file-read/parse/resolution + * errors fail the effect. + */ +final case class MarklitRunResult( + files: Vector[MarklitFileReport], + notices: Vector[String] +): + def success: Boolean = files.forall(_.success) + def failedCount: Int = files.count(!_.success) + +/** The shared, side-effect-free orchestration that drives marklit over a set of + * files: parse `//> using` directives, aggregate and resolve dependencies once, + * build per-major classpaths, process each file through [[Marklit]], render and + * optionally write outputs. Both the CLI and the build-tool plugins call this + * so the multi-file logic lives in exactly one place. + */ +object MarklitRun: + + /** Run with a freshly-built [[CompilerFactory]] (the CLI path). The factory is + * built once and shared across all files, so per-version compilers are reused + * within the run. + */ + def run( + config: MarklitRunConfig, + writeOutputs: Boolean = true + ): IO[MarklitError, MarklitRunResult] = + ZIO.scoped { + CompilerFactory.layer.build + .mapError(t => + MarklitError.ResolutionError( + s"Failed to initialize compiler factory: ${t.getMessage}" + ) + ) + .flatMap(env => runWith(config, env.get[CompilerFactory], writeOutputs)) + } + + /** Run with a caller-supplied, long-lived [[CompilerFactory]] (the plugin + * path). The factory's per-version cache survives across invocations, keeping + * compilers warm — the in-process replacement for the old daemon. + */ + def runWith( + config: MarklitRunConfig, + factory: CompilerFactory, + writeOutputs: Boolean = true + ): IO[MarklitError, MarklitRunResult] = + for + notices <- Ref.make(Chunk.empty[String]) + addNote = (s: String) => notices.update(_ :+ s) + + _ <- ZIO.when(config.verbose)( + addNote(s"Processing ${config.inputFiles.size} file(s)...") + ) + + // Validate input files exist. + _ <- ZIO.foreachDiscard(config.inputFiles) { path => + ZIO + .fail( + MarklitError.ParseError( + marklit.model.Location(path.toString, 1, 1), + s"File not found: $path" + ) + ) + .unless(Files.exists(path)) + } + + // Effective default Scala version before per-file using directives. + // Precedence: config.scalaVersion > the bundled shim's compile-time version. + shimDefault = CompilerFactory.defaultScalaVersion + cliDefault = config.scalaVersion.getOrElse(shimDefault) + + _ <- ZIO.when(config.verbose)( + addNote(s"Default Scala version: $cliDefault") + ) + + // Read each input file and parse its using directives. Per-file parsing + // lets us emit one notice per file when `//> using scala` overrides the + // default. + perFile <- ZIO.foreach(config.inputFiles) { path => + for + content <- ZIO + .attempt(Files.readString(path)) + .mapError(e => + MarklitError.ParseError( + marklit.model.Location(path.toString, 1, 1), + s"Failed to read file: ${e.getMessage}" + ) + ) + directives = DependencyResolver.parseUsingDirectives(content) + yield (path, directives) + } + + // Aggregate dependencies/repos/scalacOptions across files (resolved once). + allDeps = config.deps ++ perFile + .flatMap(_._2.dependencies) + .distinct + allRepos = config.repos ++ perFile + .flatMap(_._2.repositories) + .distinct + allScalacOptions = perFile.flatMap(_._2.scalacOptions).distinct + + _ <- ZIO.when(config.verbose && allDeps.nonEmpty)( + addNote(s"Resolving dependencies: ${allDeps.mkString(", ")}") + ) + + // Resolve user dependencies once, against the default version. Per-file + // overrides only affect compilation, not Coursier resolution of user deps. + resolvedJars <- + if allDeps.nonEmpty then + DependencyResolver + .resolve(allDeps, cliDefault, allRepos) + .mapError(e => MarklitError.ResolutionError(e.getMessage)) + else ZIO.succeed(Vector.empty[String]) + + _ <- ZIO.when(config.verbose && resolvedJars.nonEmpty)( + addNote(s"Resolved ${resolvedJars.size} JAR(s)") + ) + + // Filter scala-library/scala3-library out of resolved deps — the + // per-version classloader brings matching copies transitively, and a + // user-resolved copy at a different version causes duplicate-package errors. + filteredResolved = resolvedJars.filterNot { jar => + val fileName = java.nio.file.Paths.get(jar).getFileName.toString + fileName.startsWith("scala-library") || + fileName.startsWith("scala3-library") + } + fullClasspath = config.classpath ++ filteredResolved + + // Per-major classpaths from the build plugin's cross-publish. Resolved + // Coursier deps are version-agnostic enough to share across majors; the + // per-version classloader's transitive stdlib wins via the filter above. + majorClasspaths = Map( + "2" -> config.classpath2, + "3" -> config.classpath3 + ).collect { + case (m, cp) if cp.nonEmpty => m -> (cp ++ filteredResolved) + } + + // One notice per file whose using directive opts into a different version. + _ <- ZIO.foreachDiscard(perFile) { case (path, directives) => + directives.scalaVersion match + case Some(v) if v != cliDefault => + addNote(s"$path: //> using scala $v overrides default $cliDefault") + case _ => ZIO.unit + } + + // Process each file with its own effective default version (file directive + // wins over the run default). + reports <- ZIO.foreach(perFile) { case (path, directives) => + val fileDefault = directives.scalaVersion.getOrElse(cliDefault) + val absPath = path.toAbsolutePath + Marklit + .processFile(absPath) + .map(result => buildReport(result, config)) + .provide( + ZLayer.succeed(factory), + Marklit.liveWithFactory( + fileDefault, + fullClasspath, + allScalacOptions, + majorClasspaths, + config.cacheDir, + if config.pageScope then ScopeMode.Page else ScopeMode.Isolated + ) + ) + } + + // Write outputs only when the whole run succeeded — matching the CLI's + // previous all-or-nothing behavior (it failed before writing if any file + // failed). Rendering itself already happened in buildReport. + allSuccess = reports.forall(_.success) + _ <- ZIO.when( + writeOutputs && !config.check && config.outputDir.isDefined && allSuccess + ) { + ZIO.foreachDiscard(reports) { report => + (report.rendered, report.outputPath) match + case (Some(content), Some(out)) => + ZIO + .attempt { + Files.createDirectories(out.getParent) + Files.writeString(out, content) + } + .mapError(e => + MarklitError.ParseError( + marklit.model.Location(out.toString, 1, 1), + s"Failed to write output: ${e.getMessage}" + ) + ) *> ZIO.when(config.verbose)(addNote(s"Wrote: $out")) + case _ => ZIO.unit + } + } + + noticeLines <- notices.get + yield MarklitRunResult(reports, noticeLines.toVector) + + /** Build a side-effect-free per-file report from a [[MarklitResult]], + * rendering output markdown when an output dir is configured and the run is + * not in check mode. + */ + private def buildReport( + result: MarklitResult, + config: MarklitRunConfig + ): MarklitFileReport = + val blockErrors = + result.errors.map((block, error) => + (block.location.pretty, error.pretty) + ) + + val failedCompiles = + result.processingResult.blockResults + .filter(br => + !br.isSuccess && br.error.isEmpty && + br.compileResult.exists(!_.success) + ) + .map { br => + val diags = br.compileResult.map(_.diagnostics).getOrElse(Nil) + val diagStr = + diags.map(d => s"${d.severity}: ${d.message}").mkString("; ") + (br.block.location.pretty, s"Compile failed - $diagStr") + } + + val shouldRender = config.outputDir.isDefined && !config.check + val rendered = + if shouldRender then + Some( + MarkdownRenderer.render( + result.document, + result.processingResult, + RenderConfig( + showScalaVersion = config.showVersion, + showCompileWarnings = config.showWarnings + ) + ) + ) + else None + val outputPath = + config.outputDir.map(_.resolve(result.sourceFile.getFileName)) + + MarklitFileReport( + sourceFile = result.sourceFile, + success = result.isSuccess, + summary = result.summary, + blockErrors = blockErrors, + failedCompiles = failedCompiles, + rendered = rendered, + outputPath = outputPath + ) diff --git a/compiler/src/main/scala/marklit/compiler/CompilerFactory.scala b/compiler/src/main/scala/marklit/compiler/CompilerFactory.scala index d39717f..327e56a 100644 --- a/compiler/src/main/scala/marklit/compiler/CompilerFactory.scala +++ b/compiler/src/main/scala/marklit/compiler/CompilerFactory.scala @@ -189,18 +189,35 @@ object CompilerFactory: private final class ApiOnlyParent(host: ClassLoader) extends ClassLoader(null): override def loadClass(name: String, resolve: Boolean): Class[?] = + // Delegate the marklit<->shim bridge API plus everything the JDK platform + // classloader owns. The platform check is authoritative and covers java.*, + // javax.*, sun.*, jdk.* AND platform-module packages such as org.w3c.* / + // org.xml.* (java.xml) that JDBC drivers reach for at runtime — so we never + // again dead-end a JDK class that wasn't in a hand-maintained prefix list. + // `scala.*` / `dotty.*` are deliberately NOT shared: the child must find + // those in its own per-version URL list. val shared = name.startsWith("marklit.compiler.api.") || - name.startsWith("java.") || - name.startsWith("javax.") || - name.startsWith("sun.") || - name.startsWith("jdk.") + ApiOnlyParent.isPlatformClass(name) if shared then val c = host.loadClass(name) if resolve then resolveClass(c) c else throw new ClassNotFoundException(name) + private object ApiOnlyParent: + private val platformLoader: ClassLoader = + ClassLoader.getPlatformClassLoader + + /** True when the JDK platform classloader can load `name` — the + * authoritative, version-current test for "is this a JDK platform class?". + */ + def isPlatformClass(name: String): Boolean = + try + platformLoader.loadClass(name) + true + catch case _: ClassNotFoundException => false + private final class Live( shim3Jar: Path, shim2Jar: Path, diff --git a/compiler/src/main/scala/marklit/compiler/ScalaCompiler.scala b/compiler/src/main/scala/marklit/compiler/ScalaCompiler.scala index 41e6dbf..95c9f54 100644 --- a/compiler/src/main/scala/marklit/compiler/ScalaCompiler.scala +++ b/compiler/src/main/scala/marklit/compiler/ScalaCompiler.scala @@ -396,12 +396,16 @@ private class ChildFirstClassLoader( ) extends java.net.URLClassLoader(urls, parent): override def loadClass(name: String, resolve: Boolean): Class[?] = - if name.startsWith("java.") || - name.startsWith("javax.") || - name.startsWith("sun.") || - name.startsWith("jdk.") || - name.startsWith("scala.") || - name.startsWith("dotty.") + // `scala.*` / `dotty.*` must come from the parent (the per-version compiler + // loader), never child-first — otherwise we get duplicate `scala` package + // definitions. Everything the JDK platform classloader owns is delegated + // too: that authoritative check covers java.*, javax.*, sun.*, jdk.* AND + // platform-module packages like org.w3c.* / org.xml.* (java.xml) that JDBC + // drivers reach for. Consulting the platform loader instead of a + // hand-maintained prefix list means new platform packages never regress. + if name.startsWith("scala.") || + name.startsWith("dotty.") || + ChildFirstClassLoader.isPlatformClass(name) then super.loadClass(name, resolve) else try @@ -414,3 +418,17 @@ private class ChildFirstClassLoader( catch case _: ClassNotFoundException => super.loadClass(name, resolve) + +private object ChildFirstClassLoader: + /** The platform classloader can load every JDK platform-module class + * (java.base, java.xml, java.sql, java.naming, the CORBA/RMI packages, …) + * regardless of package. It is the authoritative, version-current source for + * "is this a JDK class?" — far safer than enumerating package prefixes. + */ + private val platformLoader: ClassLoader = ClassLoader.getPlatformClassLoader + + def isPlatformClass(name: String): Boolean = + try + platformLoader.loadClass(name) + true + catch case _: ClassNotFoundException => false diff --git a/compiler/src/test/scala/marklit/MarklitRunSpec.scala b/compiler/src/test/scala/marklit/MarklitRunSpec.scala new file mode 100644 index 0000000..eaa1cb5 --- /dev/null +++ b/compiler/src/test/scala/marklit/MarklitRunSpec.scala @@ -0,0 +1,134 @@ +package marklit + +import zio.* +import zio.test.* + +import java.nio.file.{Files, Path} + +object MarklitRunSpec extends ZIOSpecDefault: + + /** Write `content` to a fresh temp `.md` file and yield its path. */ + private def tempMd(content: String): ZIO[Scope, Throwable, Path] = + ZIO.acquireRelease( + ZIO.attempt { + val p = Files.createTempFile("marklit-run-spec-", ".md") + Files.writeString(p, content) + p + } + )(p => ZIO.attempt(Files.deleteIfExists(p)).ignore) + + private def tempDir: ZIO[Scope, Throwable, Path] = + ZIO.acquireRelease( + ZIO.attempt(Files.createTempDirectory("marklit-run-out-")) + )(p => + ZIO + .attempt { + if Files.exists(p) then + Files + .walk(p) + .sorted(java.util.Comparator.reverseOrder()) + .forEach(f => Files.deleteIfExists(f): Unit) + } + .ignore + ) + + def spec = suite("MarklitRun")( + test("generate writes rendered output and reports success") { + val md = + """# Title + | + |```scala + |val x = 1 + 1 + |println(x) + |``` + |""".stripMargin + ZIO.scoped { + for + file <- tempMd(md) + outDir <- tempDir + result <- MarklitRun.run( + MarklitRunConfig( + inputFiles = Vector(file), + outputDir = Some(outDir) + ) + ) + report = result.files.head + outPath = report.outputPath.get + wroteFile <- ZIO.attempt(Files.exists(outPath)) + written <- ZIO.attempt(Files.readString(outPath)) + yield assertTrue( + result.success, + result.files.size == 1, + report.success, + report.rendered.exists(_.nonEmpty), + wroteFile, + written.contains("2") + ) + } + }, + test("check mode reports success without writing output") { + val md = + """```scala + |val ok = "fine" + |println(ok) + |``` + |""".stripMargin + ZIO.scoped { + for + file <- tempMd(md) + outDir <- tempDir + result <- MarklitRun.run( + MarklitRunConfig( + inputFiles = Vector(file), + outputDir = Some(outDir), + check = true + ) + ) + report = result.files.head + // Check mode never renders and never writes to the output dir. + contents <- ZIO.attempt( + Files.list(outDir).count() + ) + yield assertTrue( + result.success, + report.success, + report.rendered.isEmpty, + contents == 0L + ) + } + }, + test("a failing block is reported as data, not a failed effect") { + val md = + """```scala + |val n: Int = "not an int" + |``` + |""".stripMargin + ZIO.scoped { + for + file <- tempMd(md) + outDir <- tempDir + result <- MarklitRun.run( + MarklitRunConfig( + inputFiles = Vector(file), + outputDir = Some(outDir) + ) + ) + report = result.files.head + // Compile failure => report.success false, but the effect succeeded. + wroteAnything <- ZIO.attempt(Files.list(outDir).count()) + yield assertTrue( + !result.success, + result.failedCount == 1, + !report.success, + report.failedCompiles.nonEmpty, + // Run did not succeed overall, so no output is written. + wroteAnything == 0L + ) + } + }, + test("missing input file fails the effect") { + val missing = Path.of("/tmp/marklit-this-does-not-exist-12345.md") + for exit <- MarklitRun.run(MarklitRunConfig(inputFiles = Vector(missing))).exit + yield assertTrue(exit.isFailure) + } + ) @@ TestAspect.sequential @@ TestAspect.timeout(120.seconds) diff --git a/core/src/main/scala/marklit/model/MarklitError.scala b/core/src/main/scala/marklit/model/MarklitError.scala index ad2f319..b1bea67 100644 --- a/core/src/main/scala/marklit/model/MarklitError.scala +++ b/core/src/main/scala/marklit/model/MarklitError.scala @@ -21,6 +21,7 @@ enum MarklitError: case CompileError(diagnostics: List[ScalaDiagnostic]) case RuntimeError(exception: Throwable, output: String) case ValidationError(location: Location, message: String) + case ResolutionError(message: String) def pretty: String = this match case ParseError(loc, msg) => s"Parse error at ${loc.pretty}: $msg" @@ -28,3 +29,4 @@ enum MarklitError: case RuntimeError(ex, output) => s"Runtime error: ${ex.getMessage}\nOutput: $output" case ValidationError(loc, msg) => s"Validation error at ${loc.pretty}: $msg" + case ResolutionError(msg) => s"Dependency resolution failed: $msg" diff --git a/examples/mill/README.md b/examples/mill/README.md index 6c95be9..4be4791 100644 --- a/examples/mill/README.md +++ b/examples/mill/README.md @@ -4,26 +4,37 @@ This example shows how to use marklit with the [Mill](https://mill-build.org/) b ## Prerequisites +The Mill plugin calls marklit's compiler **in-process** — there's no CLI jar to +build. You just need the plugin (and the libraries it depends on) published to +your local repo. Until `mill-marklit` is on Maven Central, publish them locally: + 1. Install Mill: https://mill-build.org/mill/Installation_IDE_Support.html -2. Build the marklit CLI jar from the root project: +2. Publish marklit's libraries and the Mill plugin locally from the repo root: ```bash cd ../.. - sbt 'project cli' assembly + sbt '; set every version := "0.1.0-LOCAL" ; compilerApi/publishLocal ; core/publishLocal ; compiler/publishLocal' + (cd mill-plugin && mill plugin.publishLocal) ``` + This example's `build.mill` depends on `mill-marklit:0.1.0-LOCAL`. +3. marklit's published artifacts are built for JDK 25, so run Mill on a JDK 25 + (e.g. `export JAVA_HOME=$(/usr/libexec/java_home -v 25)` or via your JDK + manager) for these examples. ## Usage Generate documentation: ```bash -mill root.marklitGenerate +mill docs.marklitGenerate ``` Check that code compiles without generating output: ```bash -mill root.marklitCheck +mill docs.marklitCheck ``` -Output is written to `out/root/marklitGenerate.dest/`. +This build also defines a page-scoped module: `mill pageDocs.marklitGenerate`. + +Output is written to `out/docs/marklitGenerate.dest/`. ### Watch mode @@ -47,42 +58,28 @@ default `mill -w docs.marklitGenerate` is preferred. ## Configuration -The `build.mill` file defines: +`build.mill` mixes `MarklitModule` into the `docs` and `pageDocs` modules. The +trait provides the `marklitGenerate` / `marklitCheck` tasks plus configurable +settings (`marklitSourceDir`, `marklitShowVersion`, `marklitShowWarnings`, +`marklitVerbose`, `marklitPageScope`, `marklitClasspath`, +`marklitCrossModuleDeps`, `marklitMajorClasspaths`, `marklitCacheDir`). See +[mill-plugin/README.md](../../mill-plugin/README.md) for the full table. -- `marklitSourceDir` - Task.Source for the markdown source directory -- `marklitCliJar` - Task.Source for the marklit CLI jar -- `marklitGenerate` - Task that generates markdown with executed output -- `marklitCheck` - Task that verifies code compiles without generating output +This example also demonstrates **cross-built dependencies**: `core` is built for +both 2.13 and 3.x, and `docs` exposes both builds via `marklitCrossModuleDeps = +core.crossModules`, so a `marklit:scala=2.13.16` block can reach the 2.13 build +of `core`. The plugin buckets each cross-module's classpath by major +automatically. ## Customizing -To use marklit in your own Mill project, copy the marklit tasks from `build.mill` and adjust: - -1. Set `marklitSourceDir` to your markdown source directory (relative to `moduleDir`) -2. Either embed the CLI jar or reference it from a published location -3. Add any project classpath entries via `--classpath` if your examples use project code - -## Example with Project Classpath - -If your markdown examples use code from your project: - -```scala -def marklitGenerate = Task { - val cp = runClasspath().map(_.path.toString).mkString(":") - val sourceDir = marklitSourceDir().path - val targetDir = Task.dest - val cliJar = marklitCliJar().path - - val sources = os.walk(sourceDir).filter(_.ext == "md") - - val args = Seq( - "java", "-jar", cliJar.toString, - "--verbose", - "--classpath", cp, - "--out", targetDir.toString - ) ++ sources.map(_.toString) - - os.proc(args).call(check = false) - // ... -} -``` +To use marklit in your own Mill project: + +1. Add `mill-marklit` to your `build.mill` `//| mvnDeps:` header and `import + marklit.mill.MarklitModule`. +2. Mix `MarklitModule` into your `ScalaModule`. +3. Point `marklitSourceDir` at your markdown directory. Your module's compile + classpath is forwarded automatically, so blocks can use your project code + — no manual `--classpath` wiring needed. +4. For cross-version blocks, set `marklitCrossModuleDeps` to your cross-built + sibling modules' `.crossModules`. diff --git a/examples/mill/build.mill b/examples/mill/build.mill index 66a3984..90c676f 100644 --- a/examples/mill/build.mill +++ b/examples/mill/build.mill @@ -1,5 +1,5 @@ //| mvnDeps: -//| - io.github.russwyte::mill-marklit:0.1.0-SNAPSHOT +//| - io.github.russwyte::mill-marklit:0.1.0-LOCAL package build diff --git a/examples/sbt/project/build.properties b/examples/sbt/project/build.properties index 08a6fc0..666624a 100644 --- a/examples/sbt/project/build.properties +++ b/examples/sbt/project/build.properties @@ -1 +1 @@ -sbt.version=1.12.8 +sbt.version=2.0.0 diff --git a/examples/sbt/project/plugins.sbt b/examples/sbt/project/plugins.sbt index eec8393..77fe05f 100644 --- a/examples/sbt/project/plugins.sbt +++ b/examples/sbt/project/plugins.sbt @@ -1 +1 @@ -addSbtPlugin("io.github.russwyte" % "sbt-marklit" % "0.1.0-SNAPSHOT") +addSbtPlugin("io.github.russwyte" % "sbt-marklit" % "0.1.0-LOCAL") diff --git a/mill-plugin/README.md b/mill-plugin/README.md index 5832932..82ea396 100644 --- a/mill-plugin/README.md +++ b/mill-plugin/README.md @@ -2,12 +2,20 @@ A Mill plugin for [marklit](https://github.com/russwyte/marklit) - typechecked Scala documentation. +> **Not yet published.** Build and publish it locally first (see +> [Building from Source](#building-from-source)), then depend on the local +> version. marklit's compiler runs **in-process** — the plugin depends on +> `marklit-compiler` and calls it directly; there is no CLI subprocess or +> daemon. + ## Installation -Add the plugin to your `build.mill`: +Declare the plugin in your `build.mill` header: ```scala -import $ivy.`io.github.russwyte::mill-marklit:0.1.0` +//| mvnDeps: +//| - io.github.russwyte::mill-marklit:0.1.0 + import marklit.mill.MarklitModule ``` @@ -16,14 +24,18 @@ import marklit.mill.MarklitModule Mix `MarklitModule` into your module: ```scala -import $ivy.`io.github.russwyte::mill-marklit:0.1.0` +//| mvnDeps: +//| - io.github.russwyte::mill-marklit:0.1.0 + +import mill._ +import mill.scalalib._ import marklit.mill.MarklitModule object docs extends ScalaModule with MarklitModule { def scalaVersion = "3.8.2" - - // Optional: customize source directory (defaults to millSourcePath / "markdown") - // override def marklitSourceDir = Task.Source(millSourcePath / "docs") + + // Optional: customize source directory (defaults to moduleDir / "markdown") + // override def marklitSourceDir = Task.Source(moduleDir / "docs") } ``` @@ -41,10 +53,15 @@ mill docs.marklitCheck | Setting | Type | Default | Description | |---------|------|---------|-------------| -| `marklitSourceDir` | `T[PathRef]` | `millSourcePath / "markdown"` | Directory containing markdown source files | +| `marklitSourceDir` | `T[PathRef]` | `moduleDir / "markdown"` | Directory containing markdown source files | | `marklitShowVersion` | `T[Boolean]` | `true` | Show Scala version in output blocks | +| `marklitShowWarnings` | `T[Boolean]` | `true` | Render compile warnings in output blocks | | `marklitVerbose` | `T[Boolean]` | `false` | Enable verbose output | -| `marklitClasspath` | `T[Seq[PathRef]]` | `runClasspath()` | Classpath for code execution | +| `marklitPageScope` | `T[Boolean]` | `false` | Share scope across all anonymous blocks per file (mdoc-style) | +| `marklitClasspath` | `T[Seq[PathRef]]` | `compileClasspath()` (stdlib filtered) | Classpath made available inside code blocks | +| `marklitCrossModuleDeps` | `Seq[CrossModuleBase]` | `Seq.empty` | Cross-built sibling modules to expose to cross-version blocks | +| `marklitMajorClasspaths` | `T[Map[String, Seq[PathRef]]]` | derived from `marklitCrossModuleDeps` | Per-major (`"2"`/`"3"`) classpaths for cross-version blocks | +| `marklitCacheDir` | `T[Option[PathRef]]` | a dest under `out/` | Persistent on-disk block compile cache (`None` disables) | ## Example with Project Code @@ -66,11 +83,12 @@ object docs extends ScalaModule with MarklitModule { ## Building from Source -1. Build the CLI jar: +1. Publish marklit's libraries locally (the plugin depends on `marklit-compiler`): ```bash cd /path/to/marklit - sbt 'project cli' assembly + sbt '; set every version := "0.1.0-LOCAL" ; compilerApi/publishLocal ; core/publishLocal ; compiler/publishLocal' ``` + (Match the `marklitVersion` in `mill-plugin/build.mill`.) 2. Build and publish the Mill plugin locally: ```bash diff --git a/mill-plugin/build.mill b/mill-plugin/build.mill index 41cd9c0..5db974c 100644 --- a/mill-plugin/build.mill +++ b/mill-plugin/build.mill @@ -4,192 +4,49 @@ import mill._ import mill.scalalib._ import mill.scalalib.publish._ -// Scala version used to compile marklit's own modules (cli aggregate of -// core/compiler/cli, plus the Mill plugin trait). User code is never compiled -// against this — it's just the version of marklit's orchestration code. -def marklitScalaVersion = "3.8.3" - -// Scala version the compiler-shim is built against. Pinned to the oldest -// supported 3.x to keep the shim's dotc API surface compatible at runtime -// against any user-requested 3.x compiler. The shim's compile-time version -// also serves as the default fallback when no document, CLI flag, or build -// plugin picks a version. Bump only when we drop support for a 3.x line. -def shimScalaVersion = "3.3.7" - -// Scala version the 2.13 compiler shim is built against. The 2.13 nsc API -// is stable, so we pin to the latest patch. User-requested 2.13.x version -// is still resolved fresh per-block via Coursier. -def shim2ScalaVersion = "2.13.16" - -// Project root - all top-level modules live one level above mill-plugin/. -private def projectRoot(moduleDir: os.Path) = moduleDir / os.up / os.up - -// compiler-api: Java-only, version-neutral interfaces shared between the -// orchestrator and the per-version compiler shim. Must NOT depend on Scala -// or any Scala library — that's what makes it loadable from a custom parent -// classloader visible to both the orchestrator and each per-version loader. -object compilerApi extends JavaModule { - override def sources = Task.Sources( - projectRoot(moduleDir) / "compiler-api" / "src" / "main" / "java" - ) -} - -// compiler-shim: the ONLY module that imports dotty.tools.dotc.*. Compiled -// against a known scala3-compiler version, but at runtime its dotc references -// are resolved against whatever scala3-compiler the orchestrator loaded into -// the per-version URLClassLoader. The dotc API surface used here (Driver, -// Reporter, Diagnostic) is stable across 3.x. -// -// Packaged as a thin jar — scala3-compiler is provided/runtime-resolved, and -// compiler-api is loaded via its own classloader at runtime. -object compilerShim extends ScalaModule { - def scalaVersion = shimScalaVersion - - override def moduleDeps = Seq(compilerApi) - - // scala3-compiler at compile time only; at runtime the shim is loaded onto - // a classloader where the per-version compiler jars live. - override def compileMvnDeps = Seq( - mvn"org.scala-lang::scala3-compiler:$shimScalaVersion" - ) - - override def sources = Task.Sources( - projectRoot(moduleDir) / "compiler-shim" / "src" / "main" / "scala" - ) - - // Embed the shim's compile-time scala3 version as a plain text resource so - // CompilerFactory.defaultScalaVersion can read it WITHOUT loading any shim - // classes (which would transitively need scala3-compiler on the classpath - // — defeating the whole point of the per-version classloader). - override def resources = Task { - val versionFile = Task.dest / "marklit-shim-version.txt" - os.write.over(versionFile, shimScalaVersion) - super.resources() ++ Seq(PathRef(Task.dest)) - } -} - -// compiler-shim-2: the 2.13 sibling of compilerShim. Imports scala.tools.nsc.* -// (the classic Scala compiler), compiled against shim2ScalaVersion. The -// orchestrator picks this shim when a block requests `scala=2.x.y` or -// `scala=2`, mirroring the dotc shim path on the 3.x side. -object compilerShim2 extends ScalaModule { - def scalaVersion = shim2ScalaVersion - - override def moduleDeps = Seq(compilerApi) - - override def compileMvnDeps = Seq( - mvn"org.scala-lang:scala-compiler:$shim2ScalaVersion" - ) - - override def sources = Task.Sources( - projectRoot(moduleDir) / "compiler-shim-2" / "src" / "main" / "scala" - ) - - // Embed the 2.13 shim's compile-time scala version so the orchestrator - // can read it without touching shim classes (which would require - // scala-compiler on the probe classpath). - override def resources = Task { - val versionFile = Task.dest / "marklit-shim-2-version.txt" - os.write.over(versionFile, shim2ScalaVersion) - super.resources() ++ Seq(PathRef(Task.dest)) - } -} - -// CLI module - builds the fat jar that does the actual work. Bundles the -// shim's thin jar as a resource (NOT on the orchestrator classpath); the -// CompilerFactory extracts and loads it on per-version classloaders at -// runtime. -object cli extends ScalaModule { - def scalaVersion = marklitScalaVersion - - override def moduleDeps = Seq(compilerApi) - - def mvnDeps = Seq( - // ZIO - mvn"dev.zio::zio:2.1.24", - mvn"dev.zio::zio-streams:2.1.24", - mvn"dev.zio::zio-json:0.9.0", - mvn"dev.zio::zio-cli:0.8.0", - // Markdown parsing - mvn"com.lihaoyi::fastparse:3.1.1", - // Coursier for dependency resolution - mvn"io.get-coursier:interface:1.0.9" - ) - - // Include core, compiler, and cli sources from the main project. Note: we - // do NOT include compiler-shim sources here — that would pull dotty.* onto - // the orchestrator's classpath and break per-version isolation. - override def sources = Task.Sources( - projectRoot(moduleDir) / "core" / "src" / "main" / "scala", - projectRoot(moduleDir) / "compiler" / "src" / "main" / "scala", - projectRoot(moduleDir) / "cli" / "src" / "main" / "scala" - ) - - // Bundle both shim thin jars as resources. CompilerFactory extracts the - // right one at runtime onto each per-version compiler classloader, picking - // by the requested major (3.x → dotc shim; 2.13.x → nsc shim). Each jar is - // thin (shim classes only — scala-compiler is provided), so it works - // against whatever exact version the user requested. - override def resources = Task { - val shim3 = compilerShim.jar() - val shim2 = compilerShim2.jar() - os.copy(shim3.path, Task.dest / "marklit-compiler-shim.jar", replaceExisting = true) - os.copy(shim2.path, Task.dest / "marklit-compiler-shim-2.jar", replaceExisting = true) - super.resources() ++ Seq(PathRef(Task.dest)) - } - - // Assembly configuration - exclude signature files - override def assemblyRules = Seq( - Assembly.Rule.ExcludePattern(".*\\.SF"), - Assembly.Rule.ExcludePattern(".*\\.DSA"), - Assembly.Rule.ExcludePattern(".*\\.RSA"), - Assembly.Rule.ExcludePattern(".*module-info\\.class"), - Assembly.Rule.Append("reference.conf") - ) - - override def mainClass = Some("marklit.cli.MarklitCli") -} - -// Plugin module - provides the Mill integration, shells out to CLI jar +// Scala version used to compile the Mill plugin trait. Aligned with the Scala +// 3 version marklit's own modules are built against so the plugin consumes +// marklit-compiler at an identical version. +def marklitScalaVersion = "3.8.4" + +// The marklit-compiler version to depend on. For local build + test, publish +// the libraries to ~/.ivy2/local first: +// sbt '; set every version := "0.1.0-LOCAL" ; compilerApi/publishLocal ; core/publishLocal ; compiler/publishLocal' +def marklitVersion = "0.1.0-LOCAL" + +// Plugin module — provides the Mill integration and calls marklit-compiler +// IN-PROCESS (no CLI fat jar, no daemon). marklit-compiler carries the two +// per-version compiler-shim jars as resources, so CompilerFactory resolves +// them off the plugin classpath at runtime. object plugin extends ScalaModule with PublishModule { def scalaVersion = marklitScalaVersion - // Project root - plugin's moduleDir is mill-plugin/plugin - private def projectRoot = moduleDir / os.up / os.up - override def artifactName = "mill-marklit" - // Only need Mill libs for the plugin trait def mvnDeps = Seq( - mvn"com.lihaoyi::mill-libs:1.1.5" + mvn"com.lihaoyi::mill-libs:1.1.5", + mvn"io.github.russwyte::marklit-compiler:$marklitVersion" ) - // Plugin source only - the actual work is done by the CLI jar + // Plugin source only — the orchestration lives in marklit-compiler. override def sources = Task.Sources( moduleDir / os.up / "src" ) - // Include the CLI assembly jar as a resource - // Use the Mill-built CLI assembly - override def resources = Task { - val cliJar = cli.assembly() - val resourceDir = Task.dest / "marklit-cli.jar" - os.copy(cliJar.path, resourceDir, replaceExisting = true) - super.resources() ++ Seq(PathRef(Task.dest)) - } - // Skip scaladoc generation to avoid compiler conflicts override def docJar = Task { val jarFile = Task.dest / "javadoc.jar" os.write(jarFile, Array.emptyByteArray) // Create minimal valid jar - val jos = new java.util.jar.JarOutputStream(new java.io.FileOutputStream(jarFile.toIO)) + val jos = new java.util.jar.JarOutputStream( + new java.io.FileOutputStream(jarFile.toIO) + ) jos.close() PathRef(jarFile) } - // Publishing configuration - def publishVersion = "0.1.0-SNAPSHOT" + // Publishing configuration (not published this round — kept for parity). + def publishVersion = marklitVersion def pomSettings = PomSettings( description = "Mill plugin for marklit - typechecked Scala documentation", diff --git a/mill-plugin/src/marklit/mill/MarklitDaemonClient.scala b/mill-plugin/src/marklit/mill/MarklitDaemonClient.scala deleted file mode 100644 index 216cc43..0000000 --- a/mill-plugin/src/marklit/mill/MarklitDaemonClient.scala +++ /dev/null @@ -1,157 +0,0 @@ -package marklit.mill - -import java.io.{ - BufferedReader, - BufferedWriter, - InputStreamReader, - OutputStreamWriter, - PrintWriter -} -import java.nio.charset.StandardCharsets -import java.util.concurrent.atomic.AtomicReference - -/** Long-lived marklit daemon process and a thin RPC client around its - * stdin/stdout. One client per Mill `Task.Worker` slot — Mill caches the - * worker for the life of the daemon (or until inputs invalidate it) and calls - * [[close]] when displacing or shutting down. - * - * Mirrors `marklit.sbt.MarklitDaemonClient`. The two implementations stay - * intentionally similar — every protocol shape, error path, and respawn rule - * lives in both. Keep them in sync by hand until the surface grows large - * enough to justify a shared Java module. - */ -private[mill] final class MarklitDaemonClient( - marklitJar: os.Path, - idleTimeoutSeconds: Long, - log: mill.api.daemon.Logger -) extends AutoCloseable: - - private val active = new AtomicReference[ActiveDaemon](null) - private val rpcLock = new AnyRef - - /** Send a `compile-document` RPC. Returns `None` on success, `Some(msg)` on - * protocol-level error. Throws on transport failure (broken pipe, malformed - * response, daemon refused to spawn) — the caller catches and falls back to - * one-shot mode. - */ - def compileDocument( - inputFiles: Seq[String], - outputDir: Option[String], - verbose: Boolean, - check: Boolean, - showVersionInOutput: Boolean, - showWarningsInOutput: Boolean, - classpath: Option[String], - classpath2: Option[String], - classpath3: Option[String], - scalaVersion: Option[String], - cacheDir: Option[String], - pageScope: Boolean - ): Option[String] = rpcLock.synchronized { - val daemon = ensureAlive() - val request = MarklitJson.compileDocumentRequest( - inputFiles = inputFiles, - outputDir = outputDir, - verbose = verbose, - check = check, - showVersionInOutput = showVersionInOutput, - showWarningsInOutput = showWarningsInOutput, - classpath = classpath, - classpath2 = classpath2, - classpath3 = classpath3, - scalaVersion = scalaVersion, - cacheDir = cacheDir, - pageScope = pageScope - ) - sendRequest(daemon, request) - parseAck(readResponse(daemon)) - } - - /** `clear-cache` RPC — wipe the on-disk cache at `cacheDir`. */ - def clearCache(cacheDir: String): Option[String] = rpcLock.synchronized { - val daemon = ensureAlive() - sendRequest(daemon, MarklitJson.clearCacheRequest(cacheDir)) - parseAck(readResponse(daemon)) - } - - /** Mill calls this when the worker is displaced or the build server is - * shutting down. Best-effort: any IOException is logged and ignored — the - * daemon's stdout-EOF will end the process anyway. - */ - override def close(): Unit = rpcLock.synchronized { - val daemon = active.getAndSet(null) - if daemon != null then - try - sendRequest(daemon, MarklitJson.shutdownRequest) - readResponse(daemon) - catch case _: Throwable => () - try daemon.process.destroy() - catch case _: Throwable => () - } - - private def ensureAlive(): ActiveDaemon = - val current = active.get() - if current != null && current.process.isAlive then current - else - if current != null && !current.process.isAlive then - log.info("[marklit] daemon exited; respawning") - val fresh = spawn() - active.set(fresh) - fresh - - private def spawn(): ActiveDaemon = - val cmd = new java.util.ArrayList[String]() - cmd.add("java") - cmd.add("-jar") - cmd.add(marklitJar.toString) - cmd.add("--daemon") - cmd.add("--idle-timeout") - cmd.add(idleTimeoutSeconds.toString) - - val pb = new ProcessBuilder(cmd) - pb.redirectError(ProcessBuilder.Redirect.INHERIT) - pb.redirectInput(ProcessBuilder.Redirect.PIPE) - pb.redirectOutput(ProcessBuilder.Redirect.PIPE) - - val process = pb.start() - val writer = new PrintWriter( - new BufferedWriter( - new OutputStreamWriter(process.getOutputStream, StandardCharsets.UTF_8) - ), - /* autoFlush = */ true - ) - val reader = new BufferedReader( - new InputStreamReader(process.getInputStream, StandardCharsets.UTF_8) - ) - - log.debug( - s"[marklit] spawned daemon (pid=${process.pid()}, idle-timeout=${idleTimeoutSeconds}s)" - ) - ActiveDaemon(process, reader, writer) - - private def sendRequest(d: ActiveDaemon, line: String): Unit = - d.writer.println(line) - if d.writer.checkError() then - active.compareAndSet(d, null) - sys.error("marklit daemon: broken pipe on send") - - private def readResponse(d: ActiveDaemon): String = - val line = d.reader.readLine() - if line == null then - active.compareAndSet(d, null) - sys.error("marklit daemon: stdout closed before response arrived") - line - - private def parseAck(line: String): Option[String] = - MarklitJson.extractStatus(line) match - case Some("ok") => None - case Some("error") => - Some(MarklitJson.extractMessage(line).getOrElse("unknown error")) - case Some(other) => Some(s"unexpected status '$other' in response: $line") - case None => Some(s"unparseable response: $line") - - private case class ActiveDaemon( - process: Process, - reader: BufferedReader, - writer: PrintWriter - ) diff --git a/mill-plugin/src/marklit/mill/MarklitJson.scala b/mill-plugin/src/marklit/mill/MarklitJson.scala deleted file mode 100644 index c7e7b8b..0000000 --- a/mill-plugin/src/marklit/mill/MarklitJson.scala +++ /dev/null @@ -1,115 +0,0 @@ -package marklit.mill - -/** Minimal JSON encoder for daemon-RPC requests, plus a tiny extractor for the - * response fields we care about. Hand-rolled to avoid pulling a JSON library - * onto the Mill plugin classpath. Mirrors the sbt-plugin twin in - * `marklit.sbt.MarklitJson`; if either needs to grow much past trivial shapes, - * swap in a real JSON dep instead of duplicating again. - */ -private[mill] object MarklitJson: - - def compileDocumentRequest( - inputFiles: Seq[String], - outputDir: Option[String], - verbose: Boolean, - check: Boolean, - showVersionInOutput: Boolean, - showWarningsInOutput: Boolean, - classpath: Option[String], - classpath2: Option[String], - classpath3: Option[String], - scalaVersion: Option[String], - cacheDir: Option[String], - pageScope: Boolean - ): String = - val params = new StringBuilder - params.append("{") - appendField(params, "inputFiles", arr(inputFiles.map(str))) - params.append(",") - appendField(params, "outputDir", outputDir.map(str).getOrElse("null")) - params.append(",") - appendField(params, "verbose", bool(verbose)) - params.append(",") - appendField(params, "check", bool(check)) - params.append(",") - appendField(params, "showVersionInOutput", bool(showVersionInOutput)) - params.append(",") - appendField(params, "showWarningsInOutput", bool(showWarningsInOutput)) - classpath.foreach { v => - params.append(",") - appendField(params, "classpath", str(v)) - } - classpath2.foreach { v => - params.append(",") - appendField(params, "classpath2", str(v)) - } - classpath3.foreach { v => - params.append(",") - appendField(params, "classpath3", str(v)) - } - scalaVersion.foreach { v => - params.append(",") - appendField(params, "scalaVersion", str(v)) - } - cacheDir.foreach { v => - params.append(",") - appendField(params, "cacheDir", str(v)) - } - params.append(",") - appendField(params, "pageScope", bool(pageScope)) - params.append("}") - - s"""{"method":"compile-document","params":${params.toString}}""" - - def clearCacheRequest(cacheDir: String): String = - val params = new StringBuilder - params.append("{") - appendField(params, "cacheDir", str(cacheDir)) - params.append("}") - s"""{"method":"clear-cache","params":${params.toString}}""" - - def shutdownRequest: String = """{"method":"shutdown"}""" - - def extractStatus(line: String): Option[String] = - extractStringField(line, "status") - - def extractMessage(line: String): Option[String] = - extractStringField(line, "message") - - private def appendField( - sb: StringBuilder, - key: String, - rawValue: String - ): Unit = - sb.append('"').append(key).append('"').append(':').append(rawValue) - - private def str(s: String): String = - val sb = new StringBuilder(s.length + 2) - sb.append('"') - var i = 0 - while i < s.length do - val c = s.charAt(i) - c match - case '"' => sb.append("\\\"") - case '\\' => sb.append("\\\\") - case '\n' => sb.append("\\n") - case '\r' => sb.append("\\r") - case '\t' => sb.append("\\t") - case ch if ch < 0x20 => sb.append(f"\\u${ch.toInt}%04x") - case ch => sb.append(ch) - i += 1 - sb.append('"') - sb.toString - - private def bool(b: Boolean): String = if b then "true" else "false" - - private def arr(parts: Seq[String]): String = parts.mkString("[", ",", "]") - - private def extractStringField(line: String, key: String): Option[String] = - val needle = "\"" + key + "\":\"" - val idx = line.indexOf(needle) - if idx < 0 then None - else - val start = idx + needle.length - val end = line.indexOf('"', start) - if end < 0 then None else Some(line.substring(start, end)) diff --git a/mill-plugin/src/marklit/mill/MarklitModule.scala b/mill-plugin/src/marklit/mill/MarklitModule.scala index 1f5a1fd..59f34dd 100644 --- a/mill-plugin/src/marklit/mill/MarklitModule.scala +++ b/mill-plugin/src/marklit/mill/MarklitModule.scala @@ -3,17 +3,19 @@ package marklit.mill import mill._ import mill.api.PathRef import mill.scalalib._ +import marklit.MarklitRunConfig /** A Mill module trait that provides marklit documentation generation * capabilities. * * Mix this into a ScalaModule to add markdown documentation with executable - * Scala code blocks. + * Scala code blocks. marklit's compiler runs IN-PROCESS (the trait depends on + * marklit-compiler) — no CLI fat jar, no daemon subprocess. * * Example usage: * {{{ * //| mvnDeps: - * //| - io.github.russwyte::mill-marklit:0.1.0-SNAPSHOT + * //| - io.github.russwyte::mill-marklit:0.1.0-LOCAL * * import marklit.mill.MarklitModule * @@ -55,15 +57,13 @@ trait MarklitModule extends ScalaModule { /** Additional classpath entries to pass to marklit. By default, uses this * module's compile classpath, filtering out Scala standard library jars to - * avoid version conflicts (the CLI resolves its own Scala library). + * avoid version conflicts (marklit resolves its own Scala library). */ def marklitClasspath: T[Seq[PathRef]] = Task { - // Filter out: - // - marklit-cli jar (contains bundled compiler) - // - scala-library and scala3-library (CLI resolves these to avoid version conflicts) + // Filter out scala-library and scala3-library (marklit resolves these per + // requested version to avoid version conflicts). compileClasspath().filterNot { pr => val name = pr.path.last - name.contains("marklit-cli") || name.startsWith("scala-library") || name.startsWith("scala3-library") } @@ -84,9 +84,9 @@ trait MarklitModule extends ScalaModule { * }}} * * Each entry's compile classpath is bucketed by its `crossScalaVersion` - * major and forwarded as `--classpath-2` / `--classpath-3` to the CLI. The - * bucket matching the docs module's own scalaVersion is skipped (those - * classes are already in `marklitClasspath`). + * major and forwarded to cross-version blocks. The bucket matching the docs + * module's own scalaVersion is skipped (those classes are already in + * `marklitClasspath`). */ def marklitCrossModuleDeps: Seq[CrossModuleBase] = Seq.empty @@ -102,8 +102,7 @@ trait MarklitModule extends ScalaModule { // Use runClasspath so we get this dep's own compiled classes plus its // transitive deps. compileClasspath excludes the module's own output // (since "compile this module" doesn't need its own classes), which is - // exactly the wrong answer here — we want to *use* the dep's classes - // from a separate process. + // exactly the wrong answer here — we want to *use* the dep's classes. val perDepCps: Seq[Seq[PathRef]] = Task.traverse(marklitCrossModuleDeps)(_.runClasspath)() val pairs: Seq[(String, Seq[PathRef])] = @@ -111,7 +110,6 @@ trait MarklitModule extends ScalaModule { val major = dep.crossScalaVersion.takeWhile(_ != '.') val filtered = cp.toSeq.filterNot { pr => val name = pr.path.last - name.contains("marklit-cli") || name.startsWith("scala-library") || name.startsWith("scala3-library") } @@ -125,77 +123,24 @@ trait MarklitModule extends ScalaModule { .toMap } - /** Whether to talk to a long-lived marklit daemon JVM instead of spawning a - * fresh subprocess per task. The daemon survives across `marklitGenerate` / - * `marklitCheck` invocations within one Mill server lifetime, keeping the - * per-version compiler classloaders warm. Default: true. - */ - def marklitDaemonEnabled: T[Boolean] = true - - /** Idle timeout (seconds) before an inactive daemon shuts itself down. - * Default: 900 (15 minutes). - */ - def marklitDaemonIdleTimeoutSeconds: T[Long] = 900L - - /** Persistent on-disk compile cache directory. Defaults to - * `moduleDir / "out" / "marklit-cache"`-style under Mill's `Task.dest` for a - * dedicated worker, so it survives across `marklitGenerate` / `marklitCheck` - * invocations within and across Mill server lifetimes. Set to `None` to - * disable caching entirely. + /** Persistent on-disk compile cache directory. Defaults to a dedicated + * worker dest under Mill's `out/`, so it survives across `marklitGenerate` / + * `marklitCheck` invocations within and across Mill server lifetimes. Set to + * `None` to disable caching entirely. */ def marklitCacheDir: T[Option[PathRef]] = Task { Some(PathRef(Task.dest / "marklit-cache")) } - /** Long-lived RPC client to a marklit daemon. Cached by Mill for the life of - * the build server (or until inputs invalidate it); Mill calls - * `MarklitDaemonClient.close()` on displacement, which sends a `shutdown` - * RPC and tears down the helper JVM. + /** Long-lived in-process marklit worker. Holds one warm CompilerFactory + ZIO + * runtime for the life of the Mill build server (Mill calls `close()` on + * displacement). Replaces the old out-of-process daemon worker. * - * Inputs are intentionally narrow — `marklitCliJar` and the idle-timeout - * setting. Changing the docs module's classpath or sources should NOT spin - * up a new daemon; the per-request `compile-document` payload carries those. + * Inputs are intentionally empty — the per-request config carries sources + * and classpaths, so changing them must NOT spin up a new worker. */ - def marklitDaemon: Worker[MarklitDaemonClient] = Task.Worker { - new MarklitDaemonClient( - marklitCliJar().path, - marklitDaemonIdleTimeoutSeconds(), - Task.log - ) - } - - /** Path to the marklit CLI jar. By default, extracts the bundled jar from - * plugin resources. - */ - private def marklitCliJar: T[PathRef] = Task { - val dest = Task.dest / "marklit-cli.jar" - // Try multiple classloader strategies to find the resource - val resourceStream = - Option(classOf[MarklitModule].getResourceAsStream("/marklit-cli.jar")) - .orElse( - Option( - classOf[MarklitModule].getClassLoader - .getResourceAsStream("marklit-cli.jar") - ) - ) - .orElse( - Option( - Thread.currentThread.getContextClassLoader - .getResourceAsStream("marklit-cli.jar") - ) - ) - .getOrElse { - throw new Exception( - "marklit-cli.jar not found in plugin resources. " + - "Plugin may not be packaged correctly." - ) - } - try { - os.write(dest, resourceStream) - } finally { - resourceStream.close() - } - PathRef(dest) + def marklitWorker: Worker[MarklitWorker] = Task.Worker { + new MarklitWorker } /** Generate markdown documentation with executed code output. Output is @@ -204,7 +149,6 @@ trait MarklitModule extends ScalaModule { def marklitGenerate: T[Seq[PathRef]] = Task { val sourceDir = marklitSourceDir().path val targetDir = Task.dest - val cliJar = marklitCliJar().path val cpEntries = marklitClasspath().map(_.path.toString) val majorCps = marklitMajorClasspaths().view .mapValues(_.map(_.path.toString)) @@ -213,10 +157,9 @@ trait MarklitModule extends ScalaModule { val showWarnings = marklitShowWarnings() val verbose = marklitVerbose() val scalaVer = scalaVersion() - val cacheDirOpt = marklitCacheDir().map(_.path.toString) + val cacheDirOpt = marklitCacheDir().map(_.path) val pageScope = marklitPageScope() - val daemonOpt = - if (marklitDaemonEnabled()) Some(marklitDaemon()) else None + val worker = marklitWorker() if (!os.exists(sourceDir)) { Task.log.info(s"[marklit] No source directory: $sourceDir") @@ -230,7 +173,7 @@ trait MarklitModule extends ScalaModule { Task.log.info(s"[marklit] Generating ${sources.size} file(s)...") runMarklit( - cliJar = cliJar, + worker = worker, sources = sources, outputDir = Some(targetDir), classpath = cpEntries, @@ -240,7 +183,6 @@ trait MarklitModule extends ScalaModule { showWarnings = showWarnings, verbose = verbose, check = false, - daemon = daemonOpt, taskLabel = "generation", log = Task.log, cacheDir = cacheDirOpt, @@ -256,17 +198,15 @@ trait MarklitModule extends ScalaModule { */ def marklitCheck: T[Unit] = Task { val sourceDir = marklitSourceDir().path - val cliJar = marklitCliJar().path val cpEntries = marklitClasspath().map(_.path.toString) val majorCps = marklitMajorClasspaths().view .mapValues(_.map(_.path.toString)) .toMap val verbose = marklitVerbose() val scalaVer = scalaVersion() - val cacheDirOpt = marklitCacheDir().map(_.path.toString) + val cacheDirOpt = marklitCacheDir().map(_.path) val pageScope = marklitPageScope() - val daemonOpt = - if (marklitDaemonEnabled()) Some(marklitDaemon()) else None + val worker = marklitWorker() if (!os.exists(sourceDir)) { Task.log.info(s"[marklit] No source directory: $sourceDir") @@ -278,7 +218,7 @@ trait MarklitModule extends ScalaModule { Task.log.info(s"[marklit] Checking ${sources.size} file(s)...") runMarklit( - cliJar = cliJar, + worker = worker, sources = sources, outputDir = None, classpath = cpEntries, @@ -288,7 +228,6 @@ trait MarklitModule extends ScalaModule { showWarnings = true, verbose = verbose, check = true, - daemon = daemonOpt, taskLabel = "check", log = Task.log, cacheDir = cacheDirOpt, @@ -298,13 +237,11 @@ trait MarklitModule extends ScalaModule { } } - /** Send a compile or check request through the daemon when one is provided; - * fall back to a one-shot subprocess on transport failure. Mirrors the - * sbt-plugin's MarklitRunner.runViaDaemon shape so behavior is consistent - * across build tools. + /** Build a run config and execute it in-process on the warm worker, logging + * the facade's notices/summaries and failing the task on any file failure. */ private def runMarklit( - cliJar: os.Path, + worker: MarklitWorker, sources: Seq[os.Path], outputDir: Option[os.Path], classpath: Seq[String], @@ -314,148 +251,44 @@ trait MarklitModule extends ScalaModule { showWarnings: Boolean, verbose: Boolean, check: Boolean, - daemon: Option[MarklitDaemonClient], taskLabel: String, log: mill.api.daemon.Logger, - cacheDir: Option[String], + cacheDir: Option[os.Path], pageScope: Boolean ): Unit = { - val sep = java.io.File.pathSeparator - val cpStr = if (classpath.isEmpty) None else Some(classpath.mkString(sep)) - def cpFor(major: String) = - majorClasspaths.get(major).map(_.mkString(sep)).filter(_.nonEmpty) + val config = MarklitRunConfig( + inputFiles = sources.map(_.toNIO).toVector, + outputDir = outputDir.map(_.toNIO), + scalaVersion = Some(scalaVer), + classpath = classpath.toVector, + classpath2 = majorClasspaths.getOrElse("2", Seq.empty).toVector, + classpath3 = majorClasspaths.getOrElse("3", Seq.empty).toVector, + cacheDir = cacheDir.map(_.toNIO), + pageScope = pageScope, + check = check, + showVersion = showVersion, + showWarnings = showWarnings, + verbose = verbose + ) - daemon match { - case Some(client) => - try { - val ack = client.compileDocument( - inputFiles = sources.map(_.toString), - outputDir = outputDir.map(_.toString), - verbose = verbose, - check = check, - showVersionInOutput = showVersion, - showWarningsInOutput = showWarnings, - classpath = cpStr, - classpath2 = cpFor("2"), - classpath3 = cpFor("3"), - scalaVersion = Some(scalaVer), - cacheDir = cacheDir, - pageScope = pageScope - ) - ack match { - case None => () - case Some(message) => - throw new Exception(s"marklit $taskLabel failed: $message") - } - } catch { - case e: Exception - if e.getMessage != null && e.getMessage.startsWith( - "marklit " + taskLabel - ) => - throw e - case t: Throwable => - log.warn( - s"[marklit] daemon RPC failed (${t.getMessage}); falling back to one-shot" - ) - runOneShot( - cliJar, - sources, - outputDir, - classpath, - majorClasspaths, - scalaVer, - showVersion, - showWarnings, - verbose, - check, - taskLabel, - log, - cacheDir, - pageScope - ) - } - case None => - runOneShot( - cliJar, - sources, - outputDir, - classpath, - majorClasspaths, - scalaVer, - showVersion, - showWarnings, - verbose, - check, - taskLabel, - log, - cacheDir, - pageScope - ) - } - } + val result = worker.run(config) - private def runOneShot( - cliJar: os.Path, - sources: Seq[os.Path], - outputDir: Option[os.Path], - classpath: Seq[String], - majorClasspaths: Map[String, Seq[String]], - scalaVer: String, - showVersion: Boolean, - showWarnings: Boolean, - verbose: Boolean, - check: Boolean, - taskLabel: String, - log: mill.api.daemon.Logger, - cacheDir: Option[String], - pageScope: Boolean - ): Unit = { - val sep = java.io.File.pathSeparator - val args = Seq.newBuilder[String] - args += "java" - args += "-jar" - args += cliJar.toString - outputDir match { - case Some(dir) => - args += "--out" - args += dir.toString - case None => - args += "--check" - } - args += "--scala-version" - args += scalaVer - if (classpath.nonEmpty) { - args += "--classpath" - args += classpath.mkString(sep) - } - majorClasspaths.foreach { case (major, cps) => - if (cps.nonEmpty) { - args += s"--classpath-$major" - args += cps.mkString(sep) + result.notices.foreach(n => log.info(s"[marklit] $n")) + result.files.foreach { file => + log.info(s"[marklit] ${file.summary}") + if (verbose) { + file.blockErrors.foreach { case (loc, msg) => + log.error(s"[marklit] $loc: $msg") + } + file.failedCompiles.foreach { case (loc, msg) => + log.error(s"[marklit] $loc: $msg") + } } } - cacheDir.foreach { d => - args += "--cache-dir" - args += d - } - if (outputDir.isDefined && !showVersion) args += "--no-show-version" - if (outputDir.isDefined) { - args += "--show-warnings" - args += showWarnings.toString - } - if (pageScope) args += "--page-scope" - if (verbose) args += "--verbose" - args ++= sources.map(_.toString) - - val result = - os.proc(args.result()) - .call(check = false, stderr = os.Pipe, stdout = os.Pipe) - if (result.exitCode != 0) { - log.error(s"marklit stdout: ${result.out.text()}") - log.error(s"marklit stderr: ${result.err.text()}") + if (!result.success) { throw new Exception( - s"marklit $taskLabel failed with exit code ${result.exitCode}" + s"marklit $taskLabel failed: ${result.failedCount} file(s) failed" ) } } diff --git a/mill-plugin/src/marklit/mill/MarklitWorker.scala b/mill-plugin/src/marklit/mill/MarklitWorker.scala new file mode 100644 index 0000000..ffa6bff --- /dev/null +++ b/mill-plugin/src/marklit/mill/MarklitWorker.scala @@ -0,0 +1,61 @@ +package marklit.mill + +import marklit.compiler.CompilerFactory +import marklit.{MarklitRun, MarklitRunConfig, MarklitRunResult} +import zio.{Exit, Runtime, Scope, Unsafe} + +/** Holds a single warm [[CompilerFactory]] + ZIO runtime for the life of the + * Mill build server. This is the in-process replacement for the old marklit + * daemon: `CompilerFactory.layer` extracts the shim jars and caches per-version + * classloaders, so keeping one instance alive across `marklitGenerate` / + * `marklitCheck` invocations keeps compilers warm. + * + * Mill caches this via a `Task.Worker`; when the worker is displaced Mill calls + * [[close]], which releases the factory's scope (temp shim jars, classloaders). + * + * Execution is serialized on a single monitor because + * [[marklit.compiler.ScalaCompiler]] redirects `System.out`/`System.err` at the + * JVM level while a block runs — unsafe under concurrent Mill tasks. + */ +class MarklitWorker extends AutoCloseable { + + private val runtime: Runtime[Any] = Runtime.default + + private val scope: Scope.Closeable = + Unsafe.unsafe { implicit u => + runtime.unsafe.run(Scope.make).getOrThrowFiberFailure() + } + + private val factory: CompilerFactory = + Unsafe.unsafe { implicit u => + runtime.unsafe + .run(scope.extend[Any](CompilerFactory.layer.build)) + .getOrThrowFiberFailure() + .get[CompilerFactory] + } + + private val lock = new AnyRef + + /** Run marklit over `config` on the warm factory, returning the structured + * result. Throws on file-read / parse / resolution errors (compile failures + * are reported as data on the result). + */ + def run(config: MarklitRunConfig): MarklitRunResult = lock.synchronized { + Unsafe.unsafe { implicit u => + runtime.unsafe + .run( + MarklitRun + .runWith(config, factory) + .mapError(e => new RuntimeException(e.pretty)) + ) + .getOrThrowFiberFailure() + } + } + + override def close(): Unit = + try + Unsafe.unsafe { implicit u => + runtime.unsafe.run(scope.close(Exit.unit)).getOrThrowFiberFailure() + } + catch { case _: Throwable => () } +} diff --git a/project/build.properties b/project/build.properties index dabdb15..666624a 100644 --- a/project/build.properties +++ b/project/build.properties @@ -1 +1 @@ -sbt.version=1.12.11 +sbt.version=2.0.0 diff --git a/project/plugins.sbt b/project/plugins.sbt index 3cbf5b8..5361893 100644 --- a/project/plugins.sbt +++ b/project/plugins.sbt @@ -1,4 +1,4 @@ addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "2.3.1") -addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.5.5") +addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.6.1") addSbtPlugin("com.github.sbt" % "sbt-dynver" % "5.1.1") addSbtPlugin("com.github.sbt" % "sbt-pgp" % "2.3.1") diff --git a/sbt-plugin/src/main/scala/marklit/sbt/MarklitDaemonClient.scala b/sbt-plugin/src/main/scala/marklit/sbt/MarklitDaemonClient.scala deleted file mode 100644 index f04eddb..0000000 --- a/sbt-plugin/src/main/scala/marklit/sbt/MarklitDaemonClient.scala +++ /dev/null @@ -1,191 +0,0 @@ -package marklit.sbt - -import sbt._ - -import java.io.{ - BufferedReader, - BufferedWriter, - InputStreamReader, - OutputStreamWriter, - PrintWriter -} -import java.nio.charset.StandardCharsets -import java.util.concurrent.atomic.AtomicReference - -/** Long-lived marklit daemon process and a thin RPC client around its - * stdin/stdout. One client per sbt session — the same JVM is reused across - * `marklitGenerate` / `marklitCompile` invocations so per-version compiler - * classloaders stay warm. - * - * Lifecycle: - * - First RPC lazy-spawns the daemon and stores the [[Process]] handle. - * - Subsequent RPCs reuse the same process. - * - If a prior process died (parent exit, idle timeout, crash), the next RPC - * respawns transparently. - * - [[shutdown]] sends a graceful `shutdown` RPC and closes the streams. - * - * Thread-safe: the active [[Process]] is held in an [[AtomicReference]] and - * one mutex serializes the request/response window so two concurrent tasks - * can't interleave RPCs on the same pipe. - */ -private[sbt] final class MarklitDaemonClient( - marklitJar: File, - idleTimeoutSeconds: Long, - log: Logger -) { - - private val active = new AtomicReference[ActiveDaemon](null) - private val rpcLock = new AnyRef - - /** Send a `compile-document` RPC. Returns `None` on success, `Some(msg)` on - * protocol-level error. Throws on transport failure (broken pipe, malformed - * response, daemon refused to spawn) — the caller catches and falls back to - * one-shot mode. - */ - def compileDocument( - inputFiles: Seq[String], - outputDir: Option[String], - verbose: Boolean, - check: Boolean, - showVersionInOutput: Boolean, - showWarningsInOutput: Boolean, - classpath: Option[String], - classpath2: Option[String], - classpath3: Option[String], - scalaVersion: Option[String], - cacheDir: Option[String], - pageScope: Boolean - ): Option[String] = rpcLock.synchronized { - val daemon = ensureAlive() - val request = MarklitJson.compileDocumentRequest( - inputFiles = inputFiles, - outputDir = outputDir, - verbose = verbose, - check = check, - showVersionInOutput = showVersionInOutput, - showWarningsInOutput = showWarningsInOutput, - classpath = classpath, - classpath2 = classpath2, - classpath3 = classpath3, - scalaVersion = scalaVersion, - cacheDir = cacheDir, - pageScope = pageScope - ) - sendRequest(daemon, request) - parseAck(readResponse(daemon)) - } - - /** Send a `clear-cache` RPC for the given on-disk cache directory. Returns - * `None` on success, `Some(msg)` on protocol-level error. Throws on - * transport failure (caller falls back to filesystem-level cleanup). - */ - def clearCache(cacheDir: String): Option[String] = rpcLock.synchronized { - val daemon = ensureAlive() - sendRequest(daemon, MarklitJson.clearCacheRequest(cacheDir)) - parseAck(readResponse(daemon)) - } - - /** Graceful shutdown. Best-effort: any IOException is logged and ignored (the - * daemon's stdout-EOF will end the process anyway). - */ - def shutdown(): Unit = rpcLock.synchronized { - val daemon = active.getAndSet(null) - if (daemon != null) { - try { - sendRequest(daemon, MarklitJson.shutdownRequest) - // Drain the ack so the daemon's `Console.printLine` doesn't EPIPE - // before its own loop exits. - readResponse(daemon) - } catch { - case _: Throwable => // best-effort — process is going away regardless - } - try daemon.process.destroy() - catch { case _: Throwable => () } - } - } - - private def ensureAlive(): ActiveDaemon = { - val current = active.get() - if (current != null && current.process.isAlive) current - else { - // Either uninitialized or the prior daemon died (idle timeout, crash, - // user kill). Spawn a fresh one. - if (current != null && !current.process.isAlive) { - log.info("[marklit] daemon exited; respawning") - } - val fresh = spawn() - active.set(fresh) - fresh - } - } - - private def spawn(): ActiveDaemon = { - val cmd = new java.util.ArrayList[String]() - cmd.add("java") - cmd.add("-jar") - cmd.add(marklitJar.getAbsolutePath) - cmd.add("--daemon") - cmd.add("--idle-timeout") - cmd.add(idleTimeoutSeconds.toString) - - val pb = new ProcessBuilder(cmd) - // Inherit stderr so the daemon's diagnostic output (per-file SUCCESS/ - // FAIL lines, override notifications) flows to the user's console. - // stdin/stdout are pipes for the JSON-RPC channel. - pb.redirectError(ProcessBuilder.Redirect.INHERIT) - pb.redirectInput(ProcessBuilder.Redirect.PIPE) - pb.redirectOutput(ProcessBuilder.Redirect.PIPE) - - val process = pb.start() - val writer = new PrintWriter( - new BufferedWriter( - new OutputStreamWriter(process.getOutputStream, StandardCharsets.UTF_8) - ), - /* autoFlush = */ true - ) - val reader = new BufferedReader( - new InputStreamReader(process.getInputStream, StandardCharsets.UTF_8) - ) - - log.debug( - s"[marklit] spawned daemon (pid=${process.pid()}, idle-timeout=${idleTimeoutSeconds}s)" - ) - ActiveDaemon(process, reader, writer) - } - - private def sendRequest(d: ActiveDaemon, line: String): Unit = { - d.writer.println(line) - if (d.writer.checkError()) { - // PrintWriter swallows IOExceptions and only surfaces them via - // checkError. Treat as broken pipe — clear the slot so the next call - // respawns. - active.compareAndSet(d, null) - sys.error("marklit daemon: broken pipe on send") - } - } - - private def readResponse(d: ActiveDaemon): String = { - val line = d.reader.readLine() - if (line == null) { - active.compareAndSet(d, null) - sys.error("marklit daemon: stdout closed before response arrived") - } - line - } - - private def parseAck(line: String): Option[String] = { - MarklitJson.extractStatus(line) match { - case Some("ok") => None - case Some("error") => - Some(MarklitJson.extractMessage(line).getOrElse("unknown error")) - case Some(other) => Some(s"unexpected status '$other' in response: $line") - case None => Some(s"unparseable response: $line") - } - } - - private case class ActiveDaemon( - process: Process, - reader: BufferedReader, - writer: PrintWriter - ) -} diff --git a/sbt-plugin/src/main/scala/marklit/sbt/MarklitDaemonRegistry.scala b/sbt-plugin/src/main/scala/marklit/sbt/MarklitDaemonRegistry.scala deleted file mode 100644 index 35840a8..0000000 --- a/sbt-plugin/src/main/scala/marklit/sbt/MarklitDaemonRegistry.scala +++ /dev/null @@ -1,44 +0,0 @@ -package marklit.sbt - -import sbt._ - -import java.util.concurrent.ConcurrentHashMap - -/** Process-wide registry of marklit daemons, one per `(jarPath, - * idleTimeoutSeconds)` pair. The sbt JVM may host multiple loaded builds - * across reloads; keying by jar path means a build reload reuses the same - * daemon, but two builds pointing at different CLI jars stay isolated. - * - * `Global / onUnload` shuts down all entries — fires when the build session - * ends or sbt itself exits. - */ -private[sbt] object MarklitDaemonRegistry { - - private val clients = - new ConcurrentHashMap[Key, MarklitDaemonClient]() - - private case class Key(jarPath: String, idleTimeoutSeconds: Long) - - def get( - marklitJar: File, - idleTimeoutSeconds: Long, - log: Logger - ): MarklitDaemonClient = { - val key = Key(marklitJar.getAbsolutePath, idleTimeoutSeconds) - clients.computeIfAbsent( - key, - _ => new MarklitDaemonClient(marklitJar, idleTimeoutSeconds, log) - ) - } - - /** Shut down every registered daemon. Idempotent. */ - def shutdownAll(): Unit = { - val it = clients.values().iterator() - while (it.hasNext) { - val client = it.next() - try client.shutdown() - catch { case _: Throwable => () } - } - clients.clear() - } -} diff --git a/sbt-plugin/src/main/scala/marklit/sbt/MarklitInProcess.scala b/sbt-plugin/src/main/scala/marklit/sbt/MarklitInProcess.scala new file mode 100644 index 0000000..a88395c --- /dev/null +++ b/sbt-plugin/src/main/scala/marklit/sbt/MarklitInProcess.scala @@ -0,0 +1,75 @@ +package marklit.sbt + +import marklit.{MarklitRun, MarklitRunConfig, MarklitRunResult} +import sbt.{Logger, MessageOnlyException} +import zio.Unsafe + +/** Bridges the sbt plugin's task values to the in-process marklit orchestrator + * ([[marklit.MarklitRun]]) and translates the structured result back into sbt + * logging + failure semantics. + * + * Execution is serialized on a single monitor: [[marklit.compiler.ScalaCompiler]] + * redirects `System.out`/`System.err` at the JVM level while a block executes, + * which is unsafe if two marklit tasks run concurrently inside the same sbt + * JVM. The old daemon serialized via its RPC lock; this preserves that. + */ +object MarklitInProcess { + + // Process-wide lock: only one marklit run touches System.out at a time. + private val executionLock = new AnyRef + + /** Run marklit over `config` on the warm session factory. Logs the facade's + * notices and per-file summaries, and throws [[MessageOnlyException]] (the + * sbt idiom for a clean task failure) when any file failed. + * + * @param taskLabel + * "compilation" / "generation" — used in the failure message. + */ + def run( + config: MarklitRunConfig, + taskLabel: String, + log: Logger + ): MarklitRunResult = executionLock.synchronized { + val runtime = MarklitSession.runtime() + val factory = MarklitSession.factory() + + // MarklitRun fails with MarklitError (file-read / parse / resolution + // failures); compile failures are data on the result, handled below. Map + // the typed error to a MessageOnlyException so the task fails cleanly. + val result = + Unsafe.unsafe { implicit u => + runtime.unsafe + .run( + MarklitRun + .runWith(config, factory) + .mapError(e => new MessageOnlyException(e.pretty)) + ) + .getOrThrowFiberFailure() + } + + // Surface informational notices (the facade already applied the verbose + // gate when choosing which lines to include). + result.notices.foreach(n => log.info(s"[marklit] $n")) + + // Per-file summary + (verbose) diagnostic detail. + result.files.foreach { file => + log.info(s"[marklit] ${file.summary}") + if (config.verbose) { + file.blockErrors.foreach { case (loc, msg) => + log.error(s"[marklit] $loc: $msg") + } + file.failedCompiles.foreach { case (loc, msg) => + log.error(s"[marklit] $loc: $msg") + } + } + } + + if (!result.success) { + throw new MessageOnlyException( + s"marklit $taskLabel failed: ${result.failedCount} file(s) failed" + ) + } + + result + } +} diff --git a/sbt-plugin/src/main/scala/marklit/sbt/MarklitJson.scala b/sbt-plugin/src/main/scala/marklit/sbt/MarklitJson.scala deleted file mode 100644 index d102936..0000000 --- a/sbt-plugin/src/main/scala/marklit/sbt/MarklitJson.scala +++ /dev/null @@ -1,138 +0,0 @@ -package marklit.sbt - -/** Minimal JSON encoder for daemon-RPC requests, plus a tiny extractor for the - * response fields we care about. Hand-rolled to avoid pulling a JSON library - * onto the sbt-plugin classpath (sbt 1.x is on Scala 2.12). - * - * Only handles the shapes used by [[DaemonProtocol]] in the CLI. Don't - * generalize this — if the protocol grows past trivial types, add a real JSON - * dep. - */ -private[sbt] object MarklitJson { - - /** Build a `compile-document` request line. */ - def compileDocumentRequest( - inputFiles: Seq[String], - outputDir: Option[String], - verbose: Boolean, - check: Boolean, - showVersionInOutput: Boolean, - showWarningsInOutput: Boolean, - classpath: Option[String], - classpath2: Option[String], - classpath3: Option[String], - scalaVersion: Option[String], - cacheDir: Option[String], - pageScope: Boolean - ): String = { - val params = new StringBuilder - params.append("{") - appendField(params, "inputFiles", arr(inputFiles.map(str))) - params.append(",") - appendField(params, "outputDir", outputDir.map(str).getOrElse("null")) - params.append(",") - appendField(params, "verbose", bool(verbose)) - params.append(",") - appendField(params, "check", bool(check)) - params.append(",") - appendField(params, "showVersionInOutput", bool(showVersionInOutput)) - params.append(",") - appendField(params, "showWarningsInOutput", bool(showWarningsInOutput)) - if (classpath.isDefined) { - params.append(",") - appendField(params, "classpath", str(classpath.get)) - } - if (classpath2.isDefined) { - params.append(",") - appendField(params, "classpath2", str(classpath2.get)) - } - if (classpath3.isDefined) { - params.append(",") - appendField(params, "classpath3", str(classpath3.get)) - } - if (scalaVersion.isDefined) { - params.append(",") - appendField(params, "scalaVersion", str(scalaVersion.get)) - } - if (cacheDir.isDefined) { - params.append(",") - appendField(params, "cacheDir", str(cacheDir.get)) - } - params.append(",") - appendField(params, "pageScope", bool(pageScope)) - params.append("}") - - s"""{"method":"compile-document","params":${params.toString}}""" - } - - /** `clear-cache` RPC body. Caller supplies the directory to wipe. */ - def clearCacheRequest(cacheDir: String): String = { - val params = new StringBuilder - params.append("{") - appendField(params, "cacheDir", str(cacheDir)) - params.append("}") - s"""{"method":"clear-cache","params":${params.toString}}""" - } - - def shutdownRequest: String = """{"method":"shutdown"}""" - - /** Extract `status` from a response JSON line. Returns `None` if the line - * isn't well-formed enough to find a `"status"` field. - */ - def extractStatus(line: String): Option[String] = - extractStringField(line, "status") - - /** Extract `message` from an error response. */ - def extractMessage(line: String): Option[String] = - extractStringField(line, "message") - - private def appendField( - sb: StringBuilder, - key: String, - rawValue: String - ): Unit = { - sb.append('"').append(key).append('"').append(':').append(rawValue) - () - } - - private def str(s: String): String = { - val sb = new StringBuilder(s.length + 2) - sb.append('"') - var i = 0 - while (i < s.length) { - val c = s.charAt(i) - c match { - case '"' => sb.append("\\\"") - case '\\' => sb.append("\\\\") - case '\n' => sb.append("\\n") - case '\r' => sb.append("\\r") - case '\t' => sb.append("\\t") - case ch if ch < 0x20 => - sb.append("\\u%04x".format(ch.toInt)) - case ch => sb.append(ch) - } - i += 1 - } - sb.append('"') - sb.toString - } - - private def bool(b: Boolean): String = if (b) "true" else "false" - - private def arr(parts: Seq[String]): String = parts.mkString("[", ",", "]") - - /** Naive `"key":"value"` extractor. Doesn't handle escapes inside the value - * (good enough for `status: "ok" | "error"` and short messages the daemon - * sends back). - */ - private def extractStringField(line: String, key: String): Option[String] = { - val needle = "\"" + key + "\":\"" - val idx = line.indexOf(needle) - if (idx < 0) None - else { - val start = idx + needle.length - val end = line.indexOf('"', start) - if (end < 0) None else Some(line.substring(start, end)) - } - } -} diff --git a/sbt-plugin/src/main/scala/marklit/sbt/MarklitPlugin.scala b/sbt-plugin/src/main/scala/marklit/sbt/MarklitPlugin.scala index 04cf02e..5f27e1c 100644 --- a/sbt-plugin/src/main/scala/marklit/sbt/MarklitPlugin.scala +++ b/sbt-plugin/src/main/scala/marklit/sbt/MarklitPlugin.scala @@ -1,7 +1,8 @@ package marklit.sbt -import sbt._ -import sbt.Keys._ +import marklit.MarklitRunConfig +import sbt.* +import sbt.Keys.* object MarklitPlugin extends AutoPlugin { @@ -20,18 +21,6 @@ object MarklitPlugin extends AutoPlugin { "Share scope across all anonymous blocks in each file (default: false)" ) - // Daemon settings — when enabled, marklit tasks talk to a long-lived - // marklit JVM that survives across task invocations within the sbt - // session. This keeps the per-Scala-version compiler classloaders warm - // (cold-start ≈ 1-2s per fresh major), so the second `marklitGenerate` - // run is substantially faster than the first. - val marklitDaemon = settingKey[Boolean]( - "Enable the long-lived marklit daemon for warm-classloader reuse across tasks" - ) - val marklitDaemonIdleTimeout = settingKey[Long]( - "Idle timeout in seconds before an inactive daemon shuts itself down (default: 900 = 15 minutes)" - ) - // Persistent block-compile cache. SHA-256-keyed entries live under this // directory and survive across sbt sessions. Default sits inside the // project's `target/` so `sbt clean` removes it the same as other build @@ -44,9 +33,9 @@ object MarklitPlugin extends AutoPlugin { // Map from Scala major ("2", "3") to a per-major classpath. When a // markdown block opts into a cross-version compile (e.g. // `marklit:scala=2.13.16` from a project whose own scalaVersion is - // 3.x), the matching entry is forwarded as `--classpath-`. This - // is the place to wire `(otherModule / Compile / fullClasspath)` from - // a sibling module that's cross-built for the other major. + // 3.x), the matching entry is forwarded to the cross-version compiler. + // This is the place to wire `(otherModule / Compile / fullClasspath)` + // from a sibling module that's cross-built for the other major. val marklitMajorClasspaths = taskKey[Map[String, Seq[File]]]( "Per-major classpath overrides for cross-version blocks (key = Scala major like \"2\" or \"3\")" ) @@ -58,63 +47,68 @@ object MarklitPlugin extends AutoPlugin { val marklitClean = taskKey[Unit]("Clean marklit output directory") } - import autoImport._ + import autoImport.* override def requires: Plugins = plugins.JvmPlugin override def trigger: PluginTrigger = allRequirements - // Extract the embedded CLI jar on first use - private lazy val extractedJar: File = { - val tempDir = IO.createTemporaryDirectory - val jarFile = tempDir / "marklit-cli.jar" - - val resourceStream = getClass.getResourceAsStream("/marklit-cli.jar") - if (resourceStream == null) { - sys.error( - "marklit-cli.jar not found in plugin resources. Plugin may not be packaged correctly." - ) - } - - try { - IO.transfer(resourceStream, jarFile) - } finally { - resourceStream.close() - } - - // Mark for cleanup on JVM exit - tempDir.deleteOnExit() - jarFile.deleteOnExit() - - jarFile - } - - /** Filter scala-library / scala3-library jars (and the bundled marklit-cli) - * out of a forwarded classpath. The CLI resolves the per-version stdlib via - * Coursier; leaving the host project's stdlib on the classpath leaks - * 3.8.x-bin TASTy into 3.7.x compile contexts. + /** Filter scala-library / scala3-library jars out of a forwarded classpath. + * marklit resolves the per-version stdlib via Coursier; leaving the host + * project's stdlib on the classpath leaks its bin-TASTy into a different + * compile context. */ private def filterForwardedClasspath(cp: Seq[File]): Seq[File] = cp.filterNot { f => val name = f.getName - name.contains("marklit-cli") || name.startsWith("scala-library") || name.startsWith("scala3-library") } - /** Resolve a dep project's per-cross-version classes directory. sbt's - * cross-build target naming is inconsistent: 2.13 → "scala-2.13" (binary - * version), 3.x → "scala-3.x.y" (full version). Returns the candidate dirs - * in order — caller picks an existing one or, when checking readiness, - * accepts the binary form as the canonical location. + /** Resolve the real on-disk paths of a Classpath. sbt 2.0 classpaths are + * `Seq[Attributed[xsbti.HashedVirtualFileRef]]`; the converter turns each + * virtual ref into a `java.nio.file.Path` (then a `File`). + */ + private def classpathFiles( + cp: Def.Classpath, + converter: xsbti.FileConverter + ): Seq[File] = + cp.map(af => converter.toPath(af.data).toFile) + + /** Resolve a dep project's per-cross-version classes directory. + * + * In sbt 2.0 the build output layout is + * `/out/jvm/scala-//classes`, and the `target` + * key for the dep is already scoped to the *current* session's Scala version + * (e.g. `.../out/jvm/scala-3.8.2/marklit-example-core`). To reach a different + * cross version we substitute the `scala-` path segment. + * + * For robustness we also emit the sbt 1.x candidates (`target/scala-` + * and `target/scala-`); the caller filters by existence and + * picks the first that's present. */ private def crossClassesDir(target: File, version: String): Seq[File] = { val major = version.takeWhile(_ != '.') val binV = if (major == "2") version.split('.').take(2).mkString(".") else version - val binDir = target / s"scala-$binV" / "classes" - val fullDir = target / s"scala-$version" / "classes" - Seq(binDir, fullDir).distinct + + // sbt 2.0 layout: target == .../out/jvm/scala-/. + // Replace the scala- grandparent segment with scala-. + val parent = Option(target.getParentFile) + val sbt2Candidate: Seq[File] = + parent match { + case Some(p) if p.getName.startsWith("scala-") => + Seq(p.getParentFile / s"scala-$version" / target.getName / "classes") + case _ => Seq.empty + } + + // sbt 1.x layout: target/scala-/classes. + val sbt1Candidates = Seq( + target / s"scala-$binV" / "classes", + target / s"scala-$version" / "classes" + ) + + (sbt2Candidate ++ sbt1Candidates).distinct } /** Auto-discover per-major classpaths from this project's dependsOn graph. @@ -128,9 +122,6 @@ object MarklitPlugin extends AutoPlugin { * invocation). When invoking the task directly via project scoping * (`docs/marklitGenerate`), the user is responsible for having compiled the * cross-builds first. - * - * The discovered map is then merged with any user override; user keys win so - * explicit `marklitMajorClasspaths += ...` still works. */ private def autoMajorClasspaths : Def.Initialize[Task[Map[String, Seq[File]]]] = @@ -140,7 +131,7 @@ object MarklitPlugin extends AutoPlugin { .classpath(thisProjectRef.value) .map(_.project) - val perDepFilter = ScopeFilter(inProjects(deps: _*)) + val perDepFilter = ScopeFilter(inProjects(deps*)) Def.task { val depCrosses = (Keys.crossScalaVersions ?? Seq.empty) .all(perDepFilter) @@ -177,15 +168,6 @@ object MarklitPlugin extends AutoPlugin { * task runs). * 2. `/` per marklit-enabled project. * - * Why `+` (cross-prefix) and not `++ !` (set-version)? The `++` form - * mutates the session globally and there's no public API to reliably - * un-mutate without going through `Cross.SwitchCommand`. The `+` form uses - * sbt's internal session-stash machinery to compile *all* of the project's - * `crossScalaVersions` and restore the original state. That's stronger than - * we need (it cross-builds even the major already in the docs scope), but - * the resulting compile is incremental, so the cost of re-touching the same - * major is near-zero on a warm build. - * * Each dep project is cross-compiled at most once (de-duplicated across docs * projects). */ @@ -238,17 +220,7 @@ object MarklitPlugin extends AutoPlugin { * Why a command (and not a task)? sbt's `scalaVersion` is a setting, not a * parameter to `compile`. The `+` cross-prefix is a *command-level* loop * that re-applies the build with a different `scalaVersion` per cross - * version, then runs the suffix command. There is no public API to do this - * from a `Def.task` body — it must be done by prepending commands to - * `state.remainingCommands`. This is the same machinery `sbt.Cross` uses - * internally for the `+` command. - * - * Users who run the project-scoped task directly (`docs/marklitGenerate`) - * still need to ensure cross-builds are present (e.g., via prior - * `+depModule/compile`). The build-level commands `marklitGenerate` / - * `marklitCompile` (defined here, no project selector) handle that - * automatically by detecting which projects have markdown sources, which - * deps they have, and what cross versions those deps declare. + * version, then runs the suffix command. */ private val marklitGenerateCommand: Command = Command.command("marklitGenerate") { state => @@ -264,21 +236,33 @@ object MarklitPlugin extends AutoPlugin { else cmds ::: state } - override lazy val buildSettings: Seq[Setting[_]] = Seq( - // Tear daemons down when the build session ends. `onUnload` runs on - // sbt exit and on `reload`. We chain after any existing onUnload so we - // don't clobber other plugins' cleanup. + /** Build a per-major classpath map of real paths for the run config. */ + private def majorClasspathStrings( + majorCps: Map[String, Seq[File]] + ): (Vector[String], Vector[String]) = { + def entries(major: String): Vector[String] = + filterForwardedClasspath(majorCps.getOrElse(major, Seq.empty)) + .map(_.getAbsolutePath) + .toVector + (entries("2"), entries("3")) + } + + override lazy val buildSettings: Seq[Setting[?]] = Seq( + // Release the in-process CompilerFactory (temp shim jars, cached + // classloaders) when the build session ends. `onUnload` runs on sbt exit + // and on `reload`. We chain after any existing onUnload so we don't clobber + // other plugins' cleanup. Global / onUnload := { (s: State) => val previous = (Global / onUnload).value val next = previous(s) - try MarklitDaemonRegistry.shutdownAll() + try MarklitSession.shutdown() catch { case _: Throwable => () } next }, Global / commands ++= Seq(marklitGenerateCommand, marklitCompileCommand) ) - override lazy val projectSettings: Seq[Setting[_]] = Seq( + override lazy val projectSettings: Seq[Setting[?]] = Seq( // Default settings marklitSourceDirectory := (Compile / sourceDirectory).value / "markdown", marklitTargetDirectory := target.value / "marklit", @@ -286,104 +270,86 @@ object MarklitPlugin extends AutoPlugin { marklitShowWarnings := true, marklitVerbose := false, marklitPageScope := false, - marklitDaemon := true, - marklitDaemonIdleTimeout := 900L, marklitCacheDirectory := Some(target.value / "marklit-cache"), - marklitMajorClasspaths := autoMajorClasspaths.value, + // Uncached: yields Map[String, Seq[File]], not a cacheable output type. + marklitMajorClasspaths := Def.uncached(autoMajorClasspaths.value), // Make `sbt ~marklitGenerate` (and friends) re-trigger when a markdown - // source under marklitSourceDirectory is edited. Without this, sbt only - // watches Scala/Java sources and a `.md` save would not retrigger the - // task. - Compile / watchSources += new WatchSource( - marklitSourceDirectory.value, - "*.md" || "*.markdown", - HiddenFileFilter - ), - - // Compile task - check markdown files compile successfully - marklitCompile := { + // source under marklitSourceDirectory is edited. Uncached: WatchSource has + // no JsonFormat for sbt 2.0's task-result cache. + Compile / watchSources := Def.uncached { + (Compile / watchSources).value :+ new WatchSource( + marklitSourceDirectory.value, + "*.md" || "*.markdown", + HiddenFileFilter + ) + }, + + // Compile task - check markdown files compile successfully. + // Def.uncached: this task drives the compiler and has File-typed inputs that + // sbt 2.0 can't hash for its default task-result cache; it must always run. + marklitCompile := Def.uncached { val log = streams.value.log val sourceDir = marklitSourceDirectory.value val verbose = marklitVerbose.value - // Get the project's full classpath (includes dependencies) - val cp = filterForwardedClasspath((Compile / fullClasspath).value.files) + val converter = fileConverter.value + val cp = filterForwardedClasspath( + classpathFiles((Compile / fullClasspath).value, converter) + ).map(_.getAbsolutePath).toVector val scalaVer = scalaVersion.value - val majorCps = marklitMajorClasspaths.value + val (cp2, cp3) = majorClasspathStrings(marklitMajorClasspaths.value) val cacheDir = marklitCacheDirectory.value val pageScope = marklitPageScope.value - val daemon = - if (marklitDaemon.value) - Some( - MarklitDaemonRegistry.get( - extractedJar, - marklitDaemonIdleTimeout.value, - log - ) - ) - else None if (!sourceDir.exists()) { log.info(s"[marklit] No source directory: $sourceDir") } else { - val sources = (sourceDir ** "*.md").get + val sources = (sourceDir ** "*.md").get() if (sources.isEmpty) { log.info(s"[marklit] No markdown files in $sourceDir") } else { log.info(s"[marklit] Checking ${sources.size} file(s)...") - val exitCode = - MarklitRunner.check( - extractedJar, - sources, - cp, - scalaVer, - verbose, - log, - majorCps, - daemon, - cacheDir, - pageScope - ) - if (exitCode != 0) { - throw new MessageOnlyException( - s"marklit compilation failed with exit code $exitCode" - ) - } + val config = MarklitRunConfig( + inputFiles = sources.map(_.toPath).toVector, + scalaVersion = Some(scalaVer), + classpath = cp, + classpath2 = cp2, + classpath3 = cp3, + cacheDir = cacheDir.map(_.toPath), + pageScope = pageScope, + check = true, + verbose = verbose + ) + val _ = MarklitInProcess.run(config, "compilation", log) } } }, - // Generate task - render output markdown files - marklitGenerate := { + // Generate task - render output markdown files. + // Def.uncached: side-effecting (runs the compiler, writes files) and returns + // Seq[File], which is not a cacheable output type in sbt 2.0. + marklitGenerate := Def.uncached { val log = streams.value.log val sourceDir = marklitSourceDirectory.value val targetDir = marklitTargetDirectory.value val showVersion = marklitShowVersion.value val showWarnings = marklitShowWarnings.value val verbose = marklitVerbose.value - // Get the project's full classpath (includes dependencies) - val cp = filterForwardedClasspath((Compile / fullClasspath).value.files) + val converter = fileConverter.value + val cp = filterForwardedClasspath( + classpathFiles((Compile / fullClasspath).value, converter) + ).map(_.getAbsolutePath).toVector val scalaVer = scalaVersion.value - val majorCps = marklitMajorClasspaths.value + val (cp2, cp3) = majorClasspathStrings(marklitMajorClasspaths.value) val cacheDir = marklitCacheDirectory.value val pageScope = marklitPageScope.value - val daemon = - if (marklitDaemon.value) - Some( - MarklitDaemonRegistry.get( - extractedJar, - marklitDaemonIdleTimeout.value, - log - ) - ) - else None if (!sourceDir.exists()) { log.info(s"[marklit] No source directory: $sourceDir") Seq.empty } else { - val sources = (sourceDir ** "*.md").get + val sources = (sourceDir ** "*.md").get() if (sources.isEmpty) { log.info(s"[marklit] No markdown files in $sourceDir") Seq.empty @@ -392,53 +358,35 @@ object MarklitPlugin extends AutoPlugin { log.info(s"[marklit] Generating ${sources.size} file(s)...") log.info(s"[marklit] Scala version: $scalaVer") - val exitCode = MarklitRunner.generate( - extractedJar, - sources, - targetDir, - cp, - scalaVer, - showVersion, - showWarnings, - verbose, - log, - majorCps, - daemon, - cacheDir, - pageScope + val config = MarklitRunConfig( + inputFiles = sources.map(_.toPath).toVector, + outputDir = Some(targetDir.toPath), + scalaVersion = Some(scalaVer), + classpath = cp, + classpath2 = cp2, + classpath3 = cp3, + cacheDir = cacheDir.map(_.toPath), + pageScope = pageScope, + check = false, + showVersion = showVersion, + showWarnings = showWarnings, + verbose = verbose ) - if (exitCode != 0) { - throw new MessageOnlyException( - s"marklit generation failed with exit code $exitCode" - ) - } + val _ = MarklitInProcess.run(config, "generation", log) // Return generated files - sources.map { source => - targetDir / source.getName - } + sources.map(source => targetDir / source.getName) } } }, // Clean task — removes both rendered output and the persistent compile - // cache. We try the daemon's `clear-cache` RPC first when one is running - // (so the daemon's in-memory file handles are released cleanly on - // Windows); on any failure we fall back to a filesystem delete. - marklitClean := { + // cache via a plain filesystem delete (the in-process factory holds no OS + // locks on the cache dir; the disk cache opens/closes per entry). + marklitClean := Def.uncached { val log = streams.value.log val targetDir = marklitTargetDirectory.value val cacheDir = marklitCacheDirectory.value - val daemon = - if (marklitDaemon.value) - Some( - MarklitDaemonRegistry.get( - extractedJar, - marklitDaemonIdleTimeout.value, - log - ) - ) - else None if (targetDir.exists()) { IO.delete(targetDir) @@ -447,17 +395,7 @@ object MarklitPlugin extends AutoPlugin { cacheDir.foreach { dir => if (dir.exists()) { - val cleared = daemon match { - case Some(client) => - try { - client.clearCache(dir.getAbsolutePath) match { - case None => true - case Some(_) => false - } - } catch { case _: Throwable => false } - case None => false - } - if (!cleared) IO.delete(dir) + IO.delete(dir) log.info(s"[marklit] Cleared cache: $dir") } } diff --git a/sbt-plugin/src/main/scala/marklit/sbt/MarklitRunner.scala b/sbt-plugin/src/main/scala/marklit/sbt/MarklitRunner.scala deleted file mode 100644 index ada0e6b..0000000 --- a/sbt-plugin/src/main/scala/marklit/sbt/MarklitRunner.scala +++ /dev/null @@ -1,216 +0,0 @@ -package marklit.sbt - -import sbt._ -import scala.sys.process._ - -/** Runner for marklit operations - thin wrapper around marklit CLI. - */ -object MarklitRunner { - - /** Run marklit CLI with given arguments. Returns exit code. - */ - def run( - marklitJar: File, - args: Seq[String], - log: Logger - ): Int = { - val cmd = Seq("java", "-jar", marklitJar.getAbsolutePath) ++ args - - log.debug(s"[marklit] Running: ${cmd.mkString(" ")}") - - val exitCode = Process(cmd) ! ProcessLogger( - out => log.info(s"[marklit] $out"), - err => log.error(s"[marklit] $err") - ) - - exitCode - } - - private def scalaVersionArg(scalaVersion: String): Seq[String] = - Seq("--scala-version", scalaVersion) - - private def classpathArg(flag: String, classpath: Seq[File]): Seq[String] = - if (classpath.nonEmpty) - Seq( - flag, - classpath.map(_.getAbsolutePath).mkString(java.io.File.pathSeparator) - ) - else Seq.empty - - private def majorClasspathArgs( - majorClasspaths: Map[String, Seq[File]] - ): Seq[String] = - majorClasspaths.toSeq.flatMap { case (major, cp) => - classpathArg(s"--classpath-$major", cp) - } - - private def cacheDirArg(cacheDir: Option[File]): Seq[String] = - cacheDir.toSeq.flatMap(d => Seq("--cache-dir", d.getAbsolutePath)) - - /** Run marklit in check mode. - */ - def check( - marklitJar: File, - sources: Seq[File], - classpath: Seq[File], - scalaVersion: String, - verbose: Boolean, - log: Logger, - majorClasspaths: Map[String, Seq[File]] = Map.empty, - daemon: Option[MarklitDaemonClient] = None, - cacheDir: Option[File] = None, - pageScope: Boolean = false - ): Int = daemon match { - case Some(client) => - runViaDaemon( - client, - sources = sources, - outputDir = None, - classpath = classpath, - scalaVersion = scalaVersion, - showVersion = true, - showWarnings = true, - check = true, - verbose = verbose, - log = log, - majorClasspaths = majorClasspaths, - marklitJar = marklitJar, - cacheDir = cacheDir, - pageScope = pageScope - ) - case None => - val args = Seq("--check") ++ - scalaVersionArg(scalaVersion) ++ - classpathArg("--classpath", classpath) ++ - majorClasspathArgs(majorClasspaths) ++ - cacheDirArg(cacheDir) ++ - (if (pageScope) Seq("--page-scope") else Seq.empty) ++ - (if (verbose) Seq("--verbose") else Seq.empty) ++ - sources.map(_.getAbsolutePath) - run(marklitJar, args, log) - } - - /** Run marklit to generate output. - */ - def generate( - marklitJar: File, - sources: Seq[File], - outputDir: File, - classpath: Seq[File], - scalaVersion: String, - showVersion: Boolean, - showWarnings: Boolean, - verbose: Boolean, - log: Logger, - majorClasspaths: Map[String, Seq[File]] = Map.empty, - daemon: Option[MarklitDaemonClient] = None, - cacheDir: Option[File] = None, - pageScope: Boolean = false - ): Int = daemon match { - case Some(client) => - runViaDaemon( - client, - sources = sources, - outputDir = Some(outputDir), - classpath = classpath, - scalaVersion = scalaVersion, - showVersion = showVersion, - showWarnings = showWarnings, - check = false, - verbose = verbose, - log = log, - majorClasspaths = majorClasspaths, - marklitJar = marklitJar, - cacheDir = cacheDir, - pageScope = pageScope - ) - case None => - val args = Seq("--out", outputDir.getAbsolutePath) ++ - scalaVersionArg(scalaVersion) ++ - classpathArg("--classpath", classpath) ++ - majorClasspathArgs(majorClasspaths) ++ - cacheDirArg(cacheDir) ++ - (if (showVersion) Seq.empty else Seq("--no-show-version")) ++ - Seq("--show-warnings", showWarnings.toString) ++ - (if (pageScope) Seq("--page-scope") else Seq.empty) ++ - (if (verbose) Seq("--verbose") else Seq.empty) ++ - sources.map(_.getAbsolutePath) - run(marklitJar, args, log) - } - - /** Send the request as a JSON-RPC `compile-document` to a long-lived daemon. - * On any transport-level failure (broken pipe, malformed response, daemon - * won't spawn), log the cause and fall back to a one-shot subprocess so the - * build still produces output. - */ - private def runViaDaemon( - client: MarklitDaemonClient, - sources: Seq[File], - outputDir: Option[File], - classpath: Seq[File], - scalaVersion: String, - showVersion: Boolean, - showWarnings: Boolean, - check: Boolean, - verbose: Boolean, - log: Logger, - majorClasspaths: Map[String, Seq[File]], - marklitJar: File, - cacheDir: Option[File], - pageScope: Boolean - ): Int = { - val cpStr = pathString(classpath) - val cp2 = majorClasspaths.get("2").flatMap(pathString) - val cp3 = majorClasspaths.get("3").flatMap(pathString) - try { - client.compileDocument( - inputFiles = sources.map(_.getAbsolutePath), - outputDir = outputDir.map(_.getAbsolutePath), - verbose = verbose, - check = check, - showVersionInOutput = showVersion, - showWarningsInOutput = showWarnings, - classpath = cpStr, - classpath2 = cp2, - classpath3 = cp3, - scalaVersion = Some(scalaVersion), - cacheDir = cacheDir.map(_.getAbsolutePath), - pageScope = pageScope - ) match { - case None => - 0 - case Some(message) => - log.error(s"[marklit] daemon error: $message") - 1 - } - } catch { - case t: Throwable => - log.warn( - s"[marklit] daemon RPC failed (${t.getMessage}); falling back to one-shot" - ) - // One-shot fallback uses the same args we'd have built normally. - val baseArgs = scalaVersionArg(scalaVersion) ++ - classpathArg("--classpath", classpath) ++ - majorClasspathArgs(majorClasspaths) ++ - cacheDirArg(cacheDir) ++ - Seq("--show-warnings", showWarnings.toString) ++ - (if (pageScope) Seq("--page-scope") else Seq.empty) ++ - (if (verbose) Seq("--verbose") else Seq.empty) ++ - sources.map(_.getAbsolutePath) - val args = outputDir match { - case Some(dir) => - Seq("--out", dir.getAbsolutePath) ++ - (if (showVersion) Seq.empty else Seq("--no-show-version")) ++ - baseArgs - case None => - Seq("--check") ++ baseArgs - } - run(marklitJar, args, log) - } - } - - private def pathString(cp: Seq[File]): Option[String] = - if (cp.isEmpty) None - else - Some(cp.map(_.getAbsolutePath).mkString(java.io.File.pathSeparator)) -} diff --git a/sbt-plugin/src/main/scala/marklit/sbt/MarklitSession.scala b/sbt-plugin/src/main/scala/marklit/sbt/MarklitSession.scala new file mode 100644 index 0000000..c75bbb3 --- /dev/null +++ b/sbt-plugin/src/main/scala/marklit/sbt/MarklitSession.scala @@ -0,0 +1,71 @@ +package marklit.sbt + +import marklit.compiler.CompilerFactory +import zio.{Exit, Runtime, Scope, Unsafe, ZIO} + +/** Session-scoped holder for a single [[CompilerFactory]] and a ZIO runtime. + * + * This is the in-process replacement for the old out-of-process marklit + * daemon. `CompilerFactory.layer` is a *scoped* layer — it extracts the two + * shim jars to temp files and maintains a per-Scala-version cache of + * classloaders + reflective dotc invokers. By building it once and keeping its + * Scope open for the whole sbt session, every marklit task reuses the same + * warm per-version compilers; the second `marklitGenerate` skips the + * cold-start classloader/Coursier work entirely. + * + * `shutdown()` is wired to the build's `Global / onUnload` so the scope (and + * its temp files) is released on sbt exit / reload. + */ +object MarklitSession { + + private final class Live( + val factory: CompilerFactory, + val runtime: Runtime[Any], + val scope: Scope.Closeable + ) + + // Guarded by `this`. Null until first use. + private var live: Live = null + + private def ensure(): Live = synchronized { + if (live == null) { + val runtime = Runtime.default + val scope = Unsafe.unsafe { implicit u => + runtime.unsafe.run(Scope.make).getOrThrowFiberFailure() + } + // Build the factory inside the session scope so its acquireRelease + // resources (shim temp jars) stay alive until shutdown() closes the scope. + val factory = Unsafe.unsafe { implicit u => + runtime.unsafe + .run(scope.extend[Any](CompilerFactory.layer.build)) + .getOrThrowFiberFailure() + .get[CompilerFactory] + } + live = new Live(factory, runtime, scope) + } + live + } + + /** The shared, warm CompilerFactory for this sbt session. */ + def factory(): CompilerFactory = ensure().factory + + /** The runtime marklit effects are executed on. */ + def runtime(): Runtime[Any] = ensure().runtime + + /** Release the session's CompilerFactory scope (temp shim jars, cached + * classloaders). Safe to call when nothing was ever initialized. + */ + def shutdown(): Unit = synchronized { + if (live != null) { + val l = live + live = null + try + Unsafe.unsafe { implicit u => + l.runtime.unsafe + .run(l.scope.close(Exit.unit)) + .getOrThrowFiberFailure() + } + catch { case _: Throwable => () } + } + } +} diff --git a/sbt-plugin/src/sbt-test/marklit/basic/build.sbt b/sbt-plugin/src/sbt-test/marklit/basic/build.sbt new file mode 100644 index 0000000..9a549f7 --- /dev/null +++ b/sbt-plugin/src/sbt-test/marklit/basic/build.sbt @@ -0,0 +1,16 @@ +lazy val docs = project + .in(file(".")) + .settings( + scalaVersion := "3.8.2", + marklitSourceDirectory := baseDirectory.value / "markdown", + marklitTargetDirectory := baseDirectory.value / "target" / "docs", + marklitVerbose := true + ) + +// Assert the generated output actually contains the executed block's stdout. +TaskKey[Unit]("checkOutput") := { + val out = baseDirectory.value / "target" / "docs" / "good.md" + val content = IO.read(out) + if (!content.contains("answer = 2")) + sys.error(s"expected executed output 'answer = 2' in $out, got:\n$content") +} diff --git a/sbt-plugin/src/sbt-test/marklit/basic/changes/bad.md b/sbt-plugin/src/sbt-test/marklit/basic/changes/bad.md new file mode 100644 index 0000000..2013472 --- /dev/null +++ b/sbt-plugin/src/sbt-test/marklit/basic/changes/bad.md @@ -0,0 +1,7 @@ +# A deliberately broken block + +This block does not compile (type mismatch): + +```scala +val n: Int = "not an int" +``` diff --git a/sbt-plugin/src/sbt-test/marklit/basic/markdown/good.md b/sbt-plugin/src/sbt-test/marklit/basic/markdown/good.md new file mode 100644 index 0000000..4aff2fc --- /dev/null +++ b/sbt-plugin/src/sbt-test/marklit/basic/markdown/good.md @@ -0,0 +1,8 @@ +# Basic marklit scripted test + +A simple block that compiles and executes: + +```scala +val x = 1 + 1 +println(s"answer = $x") +``` diff --git a/sbt-plugin/src/sbt-test/marklit/basic/project/build.properties b/sbt-plugin/src/sbt-test/marklit/basic/project/build.properties new file mode 100644 index 0000000..666624a --- /dev/null +++ b/sbt-plugin/src/sbt-test/marklit/basic/project/build.properties @@ -0,0 +1 @@ +sbt.version=2.0.0 diff --git a/sbt-plugin/src/sbt-test/marklit/basic/project/plugins.sbt b/sbt-plugin/src/sbt-test/marklit/basic/project/plugins.sbt new file mode 100644 index 0000000..c0fd9e5 --- /dev/null +++ b/sbt-plugin/src/sbt-test/marklit/basic/project/plugins.sbt @@ -0,0 +1,3 @@ +addSbtPlugin( + "io.github.russwyte" % "sbt-marklit" % sys.props("plugin.version") +) diff --git a/sbt-plugin/src/sbt-test/marklit/basic/test b/sbt-plugin/src/sbt-test/marklit/basic/test new file mode 100644 index 0000000..fb04242 --- /dev/null +++ b/sbt-plugin/src/sbt-test/marklit/basic/test @@ -0,0 +1,11 @@ +# Check mode passes for the good block. +> docs/marklitCompile + +# Generate renders output and writes it to the target dir. +> docs/marklitGenerate +$ exists target/docs/good.md +> checkOutput + +# Adding a block that does not compile makes marklitCompile fail. +$ copy-file changes/bad.md markdown/bad.md +-> docs/marklitCompile From 1ae296faa6ec6c8b3fc2f76df6753152f16e5e5e Mon Sep 17 00:00:00 2001 From: Russ White <356303+russwyte@users.noreply.github.com> Date: Tue, 23 Jun 2026 17:15:22 -0500 Subject: [PATCH 2/2] fmt --- .../src/main/scala/marklit/MarklitRun.scala | 34 +++++++++---------- .../test/scala/marklit/MarklitRunSpec.scala | 28 +++++++-------- .../scala/marklit/sbt/MarklitInProcess.scala | 9 ++--- .../scala/marklit/sbt/MarklitPlugin.scala | 4 +-- 4 files changed, 37 insertions(+), 38 deletions(-) diff --git a/compiler/src/main/scala/marklit/MarklitRun.scala b/compiler/src/main/scala/marklit/MarklitRun.scala index 113f025..b875158 100644 --- a/compiler/src/main/scala/marklit/MarklitRun.scala +++ b/compiler/src/main/scala/marklit/MarklitRun.scala @@ -48,10 +48,10 @@ final case class MarklitFileReport( ) /** Structured result of a run. `notices` are ordered informational lines the - * caller may surface verbatim (the facade has already applied the verbose - * gate when deciding which lines to include). Compile failures are reported as - * data via [[MarklitFileReport.success]]; only file-read/parse/resolution - * errors fail the effect. + * caller may surface verbatim (the facade has already applied the verbose gate + * when deciding which lines to include). Compile failures are reported as data + * via [[MarklitFileReport.success]]; only file-read/parse/resolution errors + * fail the effect. */ final case class MarklitRunResult( files: Vector[MarklitFileReport], @@ -61,16 +61,16 @@ final case class MarklitRunResult( def failedCount: Int = files.count(!_.success) /** The shared, side-effect-free orchestration that drives marklit over a set of - * files: parse `//> using` directives, aggregate and resolve dependencies once, - * build per-major classpaths, process each file through [[Marklit]], render and - * optionally write outputs. Both the CLI and the build-tool plugins call this - * so the multi-file logic lives in exactly one place. + * files: parse `//> using` directives, aggregate and resolve dependencies + * once, build per-major classpaths, process each file through [[Marklit]], + * render and optionally write outputs. Both the CLI and the build-tool plugins + * call this so the multi-file logic lives in exactly one place. */ object MarklitRun: - /** Run with a freshly-built [[CompilerFactory]] (the CLI path). The factory is - * built once and shared across all files, so per-version compilers are reused - * within the run. + /** Run with a freshly-built [[CompilerFactory]] (the CLI path). The factory + * is built once and shared across all files, so per-version compilers are + * reused within the run. */ def run( config: MarklitRunConfig, @@ -87,8 +87,8 @@ object MarklitRun: } /** Run with a caller-supplied, long-lived [[CompilerFactory]] (the plugin - * path). The factory's per-version cache survives across invocations, keeping - * compilers warm — the in-process replacement for the old daemon. + * path). The factory's per-version cache survives across invocations, + * keeping compilers warm — the in-process replacement for the old daemon. */ def runWith( config: MarklitRunConfig, @@ -118,7 +118,7 @@ object MarklitRun: // Effective default Scala version before per-file using directives. // Precedence: config.scalaVersion > the bundled shim's compile-time version. shimDefault = CompilerFactory.defaultScalaVersion - cliDefault = config.scalaVersion.getOrElse(shimDefault) + cliDefault = config.scalaVersion.getOrElse(shimDefault) _ <- ZIO.when(config.verbose)( addNote(s"Default Scala version: $cliDefault") @@ -199,7 +199,7 @@ object MarklitRun: // wins over the run default). reports <- ZIO.foreach(perFile) { case (path, directives) => val fileDefault = directives.scalaVersion.getOrElse(cliDefault) - val absPath = path.toAbsolutePath + val absPath = path.toAbsolutePath Marklit .processFile(absPath) .map(result => buildReport(result, config)) @@ -253,9 +253,7 @@ object MarklitRun: config: MarklitRunConfig ): MarklitFileReport = val blockErrors = - result.errors.map((block, error) => - (block.location.pretty, error.pretty) - ) + result.errors.map((block, error) => (block.location.pretty, error.pretty)) val failedCompiles = result.processingResult.blockResults diff --git a/compiler/src/test/scala/marklit/MarklitRunSpec.scala b/compiler/src/test/scala/marklit/MarklitRunSpec.scala index eaa1cb5..ec314bc 100644 --- a/compiler/src/test/scala/marklit/MarklitRunSpec.scala +++ b/compiler/src/test/scala/marklit/MarklitRunSpec.scala @@ -21,15 +21,13 @@ object MarklitRunSpec extends ZIOSpecDefault: ZIO.acquireRelease( ZIO.attempt(Files.createTempDirectory("marklit-run-out-")) )(p => - ZIO - .attempt { - if Files.exists(p) then - Files - .walk(p) - .sorted(java.util.Comparator.reverseOrder()) - .forEach(f => Files.deleteIfExists(f): Unit) - } - .ignore + ZIO.attempt { + if Files.exists(p) then + Files + .walk(p) + .sorted(java.util.Comparator.reverseOrder()) + .forEach(f => Files.deleteIfExists(f): Unit) + }.ignore ) def spec = suite("MarklitRun")( @@ -44,7 +42,7 @@ object MarklitRunSpec extends ZIOSpecDefault: |""".stripMargin ZIO.scoped { for - file <- tempMd(md) + file <- tempMd(md) outDir <- tempDir result <- MarklitRun.run( MarklitRunConfig( @@ -55,7 +53,7 @@ object MarklitRunSpec extends ZIOSpecDefault: report = result.files.head outPath = report.outputPath.get wroteFile <- ZIO.attempt(Files.exists(outPath)) - written <- ZIO.attempt(Files.readString(outPath)) + written <- ZIO.attempt(Files.readString(outPath)) yield assertTrue( result.success, result.files.size == 1, @@ -75,7 +73,7 @@ object MarklitRunSpec extends ZIOSpecDefault: |""".stripMargin ZIO.scoped { for - file <- tempMd(md) + file <- tempMd(md) outDir <- tempDir result <- MarklitRun.run( MarklitRunConfig( @@ -105,7 +103,7 @@ object MarklitRunSpec extends ZIOSpecDefault: |""".stripMargin ZIO.scoped { for - file <- tempMd(md) + file <- tempMd(md) outDir <- tempDir result <- MarklitRun.run( MarklitRunConfig( @@ -128,7 +126,9 @@ object MarklitRunSpec extends ZIOSpecDefault: }, test("missing input file fails the effect") { val missing = Path.of("/tmp/marklit-this-does-not-exist-12345.md") - for exit <- MarklitRun.run(MarklitRunConfig(inputFiles = Vector(missing))).exit + for exit <- MarklitRun + .run(MarklitRunConfig(inputFiles = Vector(missing))) + .exit yield assertTrue(exit.isFailure) } ) @@ TestAspect.sequential @@ TestAspect.timeout(120.seconds) diff --git a/sbt-plugin/src/main/scala/marklit/sbt/MarklitInProcess.scala b/sbt-plugin/src/main/scala/marklit/sbt/MarklitInProcess.scala index a88395c..7023473 100644 --- a/sbt-plugin/src/main/scala/marklit/sbt/MarklitInProcess.scala +++ b/sbt-plugin/src/main/scala/marklit/sbt/MarklitInProcess.scala @@ -8,10 +8,11 @@ import zio.Unsafe * ([[marklit.MarklitRun]]) and translates the structured result back into sbt * logging + failure semantics. * - * Execution is serialized on a single monitor: [[marklit.compiler.ScalaCompiler]] - * redirects `System.out`/`System.err` at the JVM level while a block executes, - * which is unsafe if two marklit tasks run concurrently inside the same sbt - * JVM. The old daemon serialized via its RPC lock; this preserves that. + * Execution is serialized on a single monitor: + * [[marklit.compiler.ScalaCompiler]] redirects `System.out`/`System.err` at + * the JVM level while a block executes, which is unsafe if two marklit tasks + * run concurrently inside the same sbt JVM. The old daemon serialized via its + * RPC lock; this preserves that. */ object MarklitInProcess { diff --git a/sbt-plugin/src/main/scala/marklit/sbt/MarklitPlugin.scala b/sbt-plugin/src/main/scala/marklit/sbt/MarklitPlugin.scala index 5f27e1c..d241f45 100644 --- a/sbt-plugin/src/main/scala/marklit/sbt/MarklitPlugin.scala +++ b/sbt-plugin/src/main/scala/marklit/sbt/MarklitPlugin.scala @@ -79,8 +79,8 @@ object MarklitPlugin extends AutoPlugin { * In sbt 2.0 the build output layout is * `/out/jvm/scala-//classes`, and the `target` * key for the dep is already scoped to the *current* session's Scala version - * (e.g. `.../out/jvm/scala-3.8.2/marklit-example-core`). To reach a different - * cross version we substitute the `scala-` path segment. + * (e.g. `.../out/jvm/scala-3.8.2/marklit-example-core`). To reach a + * different cross version we substitute the `scala-` path segment. * * For robustness we also emit the sbt 1.x candidates (`target/scala-` * and `target/scala-`); the caller filters by existence and