The future of this project

[huwaireb]

Hello again 👋,

I'm curious in regards to your plans for typst.js @jcbhmr. From my understanding, currently it's only a wrapper around the typst binary distributed in npm. Are there any plans to go any further for e.g JS bindings? Like what is done by the typst.ts project. Or do your goals end here?

[jcbhmr]

Yes, I have been thinking along those lines.

There are a few ideas for how this typst package could be used:

1. To auto-install the typst binary when a user runs npm install with dependencies having typst: "^1.0.0" in it.
2. To provide functions like compile() that do exec("typst", args) similar to how other CLI JS-API wrapper works (Docker, Pulumi, tar, apt, etc.)
3. To provide a WASM/browser build of Typst so that you can run typst.compile("my typst text") or something idk in your browser

All of these can be part of the same package.

The first one is relatively easy. Hence, it's the first one that I did.
The second is just tedious JS -> arg mapping.
and third is much tougher.

I've attempted and succeeded in compiling the typst-cli package to WASIX and getting it to run on my local PC with wasmer run typst.wasm https://github.com/jcbhmr/typst/actions but I'm still having trouble getting it to work with the Wasmer WASIX JavaScript browser API https://wasmerio.github.io/wasmer-js/

so that lead me to search "typst javascript" and you're absolutely right i did find typst.ts.

I've been trudging through the absolute monster of a codebase (there's like 10 rust crates and 10 npm packages in it lol) https://github.com/Myriad-Dreamin/typst.ts project to try and figure out how it was done! i dont need all that complexity; just enough to mirror the exposed cli API. particularly typst compile.

typst compile: needs all the compiler infra. very tough.
typst watch: idk if relevant in programmatic context? maybe. basically typst compile on loop tho.
typst query: very tough. needs compiler infra plus metadata serialization. ugh.
typst fonts: ezpz just use existing web fonts API in browser! https://developer.mozilla.org/en-US/docs/Web/API/CSS_Font_Loading_API https://developer.mozilla.org/en-US/docs/Web/API/Local_Font_Access_API
typst update: not relevant.
typst help: ezpz just print help text.

(this might be more for my benefit than yours)
so far my understanding is this:

• there's the main @myriaddreamin/typst.ts which depends on @myriaddreamin/typst-ts-web-compiler. That's the main juicy part
• @myriaddreamin/typst-ts-web-compiler does the actual typst -> SVG magic
• in packages/compiler/src/lib.rs there's a compile() function that takes in a path (ok weird; how are files emulated) and a format "vector" to provide some binary data (probably the SVG source text as utf8? idk)
• then it uses some internal compiler magic (some complex indirection and dep injection 😨)
• then that delegates to typst_ts_svg_exporter::SvgModuleExporter::default()
• that's actually from the typst-ts-svg-exporter rust crate under core/exporter/svg/src/frontend/flat.rs
• idk what magic lies in there
• then that gets somehow export_module()-ed to bytes from the SvgDocument type
• and tada! not complex at all /s

so to answer your question, yes, I have thought about it and yes I would like to add such functionality but I'm still a complete novice in WASM Rust world lol. the Node.js exec("typst", ...) stuff; that I can do. the wasm browser stuff; that's more of an experiment and I'll see how it goes.

any input or ideas or help or code or education or resources or wasm magic is much appreciated lol. ❤️

[huwaireb]

Sounds great,

I'm curious to if you thought about merging step 1 and step 2 with the use of (napi-rs/deno-ops) to bind to the typst crates for cli/lib usage. To allow for a more cohesive, direct approach without the headache of managing Installations.

This would warrant a new cli(js) to act as a frontend to what will be the lib(ops/napi/wasm) that would for one wrap the typst crates.

1. It would be running natively in the perspective runtime
2. It would require the cli interface to be reimplemented in js, which is by no means bad as long as we can offer more functionality along the way.

This would entail separating the packages to be of that:

• typst (cli & rexport of lib + other functionality)
• typst-(core/lib)

For shipping typst to the web, I have a general idea in regards to how it could be implemented (WASM, of course), but we'll leave that for later. Of course a custom parser is out-of-scope due to its unnecessary added complexity. As if SSR is the primary use of the end user this won't pose major problems in terms of the drawbacks of WASM such as bundle size. Since we can always rotate between the native addon (if implemented), and using WASM. Primary issue is of course ensuring, in the case of the browser & WASM, minimal bundle sizes are shipped.

However, I have to ask if you've evaluated the need for typst.js in the first place with the existence of typst.ts. To avoid unnecessary fragmentation of the community so early on. Existence can be always justified if you e.g have a different approach to the implementation consider LaTeX JS eco with the various different implementations.

[jcbhmr]

I'm curious to if you thought about merging step 1 and step 2 with the use of (napi-rs/deno-ops) to bind to the typst crates for cli/lib usage. To allow for a more cohesive, direct approach without the headache of managing Installations.

Since napi-rs effectively builds platform-specific mylib.node files (one for Windows x64, one for Linux x64, etc.) you'd run into the same "distributing a big fat binary" problem again. Their solutions is to, guess what, do #3 🤣

so effectively: instead of one typst binary for each platform, you'd end up with:

• recreating bindings for the cli interface through https://napi.rs to make sure arguments, env, etc. get forwarded to the main() rust function
• figuring out how to publish all those fat typst.node things (they're actually just dll/so renamed .node)
• now maintaining a weird nodejs-only binding in addition to the wasm browser binding

So I think that's a lot of work for not-so-much gain. Using exec("typst", ["compile",...]) is already what https://www.npmjs.com/package/docker-cli-js, https://github.com/minhhh/node-uncompress, etc. do: they wrap some fancy exec() calls in nice JS functions.

on another note lol: pulumi has a similar "distribute binaries as npm package" issue that might be related: pulumi/pulumi#11744

However, I have to ask if you've evaluated the need for typst.js in the first place with the existence of typst.ts. To avoid unnecessary fragmentation of the community so early on. Existence can be always justified if you e.g have a different approach to the implementation consider LaTeX JS eco with the various different implementations.

"hey; what if i used wasix instead of trying to wrangle with wrappers?" <-- who knows how far that will get.

as of now the browser API is indeterminate. i have no idea. it could not exist! or it could be great! or it could just re-export typst.ts. or it could combine with typst.ts. who knows?!?

right now im just having a silly time trying to wrangle wasm and learn more about wasm/rust magic. typst seemed like a great project to do that since it compiles relatively cleanly to wasix using cargo wasix build with minimal mods.

so far I've:

• gotten it to build reliably
• published it to https://wasmer.io/typst-community/typst
• redownloaded the raw .wasm file and done wasmer run ./typst.wasm --help and had it work
• had my computer run out of memory when i do wasmer run ./typst.wasm compile test.typ
• run the ./typst.wasm in the browser only to have the browser wasix runtime run out of memory and exit code 1. at least i think that's what it did. it was rather opaque and indeterminate.

point being dont worry about fragmentation. right now typst.js https://npm.im/typst is just a redist installable version of the plain old typst binary. that's ok for now.

bottom line: my next step is to wrap those cli commands in pretty js/ts typed functions so you get autocomplete for them lol
dont worry about browser wasm; im having fun experimenting with that. for now, use typst.ts as its the clear "already works" in-browser typst magic that works.

[jcbhmr]

note to self: i was pointed to napi-rs/napi-rs#796
wasm support seems to be on the horizon but not here until napi-rs v3? napi-rs/napi-rs#1493

[huwaireb]

Since napi-rs effectively builds platform-specific mylib.node files (one for Windows x64, one for Linux x64, etc.) you'd run into the same "distributing a big fat binary" problem again. Their solutions is to, guess what, do #3 🤣

There's a huge difference between interfacing with a binary and library. Yes, you are distributing platform-specific software in all cases, however you miss the point of going the route of a library. When you wrap a cli you're essentially restricted with the interface, this is fine if of course the purpose is simple, you can just go with this unpleasant way of simply wrapping the cli. However, if you need granularity over the interface and reach over the underlying interface/functions which are not exposed, that is not the case. You'll have to either manually replicate these features in javascript, which is not going to end well, so you'll end up just wrapping the library anyways with napi.

Ah and did I mention that if you want to ship WASM you'll need to do it anyways 🙂? But yeah, at least you can distribute .wasm blobs instead, however as i will mention below, napi-rs supports wasm-bindgen as a target.

• recreating bindings for the cli interface through napi.rs to make sure arguments, env, etc. get forwarded to the main() rust function

This, sure is tedious. However since you'll have to implement the library wrapper anyways, is not the biggest problem, and you probably don't want to distribute both binary and library. Plus you get the advantage of having again as I mentioned control over the cli itself, features specific to typst in js could be introduced. A cli built on top of the runtime people are using to run typst offers its own merits.

• figuring out how to publish all those fat typst.node things (they're actually just dll/so renamed .node)

Handled by napi-rs, great cli offered.

• now maintaining a weird nodejs-only binding in addition to the wasm browser binding

napi-rs supports targeting wasm-bindgen, so you can avoid rewriting the bindings. And It's partially supported by the bun runtime, and there is work on going to get it to work in deno.

So I think that's a lot of work for not-so-much gain. Using exec("typst", ["compile",...]) is already what npmjs.com/package/docker-cli-js, minhhh/node-uncompress, etc. do: they wrap some fancy exec() calls in nice JS functions.

Think again

[jcbhmr]

youre right a direct nodejs <=> rust binding would be more powerful. in the future, that might be what happens! and the idea to avoid binding a typst.node and a typst.exe by just using a main.js => typst.node for the binary instead of a separate typst.exe is a good one.

but im not there yet. ill get there. still working on it. right now it works great with the cli wrapper! s othat's where it's at now. now that I've achieved getting it to have mirror with all the typst --help commands available as js typst.*() functions lol i feel good moving to next challenge! 👍

[Myriad-Dreamin]

@jcbhmr @huwaireb web compiler (bundle size 17mb) and renderer (bundle size 290kb) separately runs in browser, which specially considers SSR frameworks, so it introduces extra overhead to understand. I created a typst.node yesterday, which does simpler binding to typst.

[Myriad-Dreamin]

For benefit of a node binding rather than a binary shim to typst-cli, there is at least an advantage: you can run compiler with font loading once and comemo cache. Please see typst/typst#2738.

• Load font once. For each exec('typst'), there would introduce a 80ms (windows default fonts) ~ 100ms (user installing fonts) overhead to load font. With node binding you can load it once and run multiple compiler actions.
  • in my PC I install some heavy fonts that brings time to font loading to 400ms.
• Share comemo cache across compiler actions. typst is fast, but become 10x slower if we disable its comemo cache. If we run actions on similar files, we can gain benefit from comemo cache.

[Myriad-Dreamin]

in packages/compiler/src/lib.rs there's a compile() function that takes in a path ...

It is done in by a hierarchical access model, indexing data by a standard unix path.

https://github.com/Myriad-Dreamin/typst.ts/blob/main/compiler/src/vfs/mod.rs#L125

type VfsAccessModel<M> = CachedAccessModel<OverlayAccessModel<NotifyAccessModel<M>>, Source>;

/// Create a new `Vfs` harnessing over the given `access_model` specific for
/// [`crate::world::CompilerWorld`]. With vfs, we can minimize the
/// implementation overhead for [`AccessModel`] trait.
pub struct Vfs<M: AccessModel + Sized> {
    ...

    /// The wrapped access model.
    access_model: VfsAccessModel<M>,

  ...
}

Interpreting CachedAccessModel<OverlayAccessModel<NotifyAccessModel<M>>, Source>.

• NotifyAccessModel is for system notifying a vfs that some physical files are changed.
  • In browser, "system" is some remote http service.
  • In operating system, "system" is a physical file system.
• OverlayAccessModel is for you shadowing memory content above NotifyAccessModel correctly. There are some synchronization disaster detected long time ago, and I have solved it for typst-preview and get well-tested by preview users.
• CachedAccessModel is internally for compiler storing cache within a single compilation.