Native Node.js Addons do not work

[Ralf12358]

Trying to compile a "Native Node.js Addon" does not work. I get the error:

Error: Dynamic loading not supported

I assume it should work, because in the example there is this line:

# eval "$("$(npm bin)/musl-exports")" # sets CC CXX and LD in the shell

which is only needed when compiling native AddOns.

Am I doing something wrong?

I tried compiling node without --fully-static to solve this problem, but then I get an error when compiling node:

> make -C out BUILDTYPE=Release V=1 touch 34387d1d5929e5a61662c04af32f67604a0a34b9.intermediate LD_LIBRARY_PATH=/home/XXX/Node-Musl/src/out/Release/lib.host:/home/XXX/Node-Musl/src/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../deps/v8/gypfiles; mkdir -p /home/XXX/Node-Musl/src/out/Release/obj/gen/torque-generated; "/home/XXX/Node-Musl/src/out/Release/torque" ../src/builtins/base.tq ../src/builtins/array.tq ../src/builtins/array-copywithin.tq ../src/builtins/array-foreach.tq ../src/builtins/array-reverse.tq ../src/builtins/typed-array.tq ../src/builtins/data-view.tq ../third_party/v8/builtins/array-sort.tq -o "/home/XXX/Node-Musl/src/out/Release/obj/gen/torque-generated" /bin/sh: /home/XXX/Node-Musl/src/out/Release/torque: No such file or directory make[1]: *** [deps/v8/gypfiles/v8_torque.host.mk:17: 34387d1d5929e5a61662c04af32f67604a0a34b9.intermediate] Error 127 rm 34387d1d5929e5a61662c04af32f67604a0a34b9.intermediate

[bruce-one]

Yep, the Error: Dynamic loading not supported error message will occur due to the --fully-static flag as you inferred. Fully statically linking node means that dlopen is not supported, hence dynamically loading native addons won't be supported.

To build without the --fully-static flag, I've had success replacing it with the --partly-static flag instead (rather than removing it). This fixes the error you've seen IME, but still allows dlopen to work, and hence native modules to be loaded.

Note, though, that native modules will need to be compiled with Musl to be loadable. Which module are you trying to use?

Here's an example using sqlite3:

npm i node-musl

mkdir node; cd node
curl -L https://github.com/nodejs/node/archive/v10.14.2.tar.gz | tar xvz --strip-components=1
eval "$("$(npm bin)/musl-exports")"
./configure --partly-static
make -j$(node -p 'os.cpus().length + 1')

# --scripts-prepend-node-path is to ensure we use the just-built node executable; not necessary if you've `make install`ed
# -static-libgcc in particular is required for sqlite3 to build correctly; I suspect this to be very commonly required
LDFLAGS='-static-libgcc -static-libstdc++' ./node ./deps/npm/bin/npx-cli.js npm install sqlite3 --build-from-source --scripts-prepend-node-path
./node -e 'sqlite3 = require("sqlite3"); db = new sqlite3.Database(":memory:"); db.get("select 1", (err, r) => console.log(r));'
> { '1': 1 }

[Ralf12358]

Hmmm. Thats strange. I copied line by line from your example.
But, when building node I still get the same error:

.../node/out/Release/torque: No such file or directory make[1]: *** [deps/v8/gypfiles/v8_torque.host.mk:17: ad93465f913f1cc78fbbf3a831235ebe1a33a7d2.intermediate] Error 127 make[1]: *** Waiting for unfinished jobs.... rm 84c39e3bad9a7e7383c8bdb3e00b6ba92c2f3239.intermediate ad93465f913f1cc78fbbf3a831235ebe1a33a7d2.intermediate make: *** [Makefile:99: node] Error

When looking at out/Release/torque the file exists and the executable flag is set, but when trying to execute it I get: -bash: ./torque: No such file or directory

readelf -d torque looks OK, too:

Dynamic section at offset 0x352878 contains 21 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x000000000000000c (INIT)               0x108dd8
 0x000000000000000d (FINI)               0x2bfe87
 0x0000000000000019 (INIT_ARRAY)         0x53eeb8
 0x000000000000001b (INIT_ARRAYSZ)       1344 (bytes)
 0x0000000000000004 (HASH)               0x258
 0x0000000000000005 (STRTAB)             0x4b5d0
 0x0000000000000006 (SYMTAB)             0x11c40
 0x000000000000000a (STRSZ)              560365 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x553000
 0x0000000000000002 (PLTRELSZ)           3504 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x108028
 0x0000000000000007 (RELA)               0xd42c0
 0x0000000000000008 (RELASZ)             212328 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffb (FLAGS_1)            Flags: PIE
 0x000000006ffffff9 (RELACOUNT)          8836
 0x0000000000000000 (NULL)               0x0

For some unknow reason I can only execute binaries generated by a musl-toolchain if I build them completely static... :-(

If your example works for you the reason might be somewhere else. I'm using Manjaro Linux. CFLAGS, CXXFLAGS, LDFLAGS were not declared when trying your example.

I'm interested, because I would like to build my own node addons. Everythings works fine with a glibc
node.js. I'm very interested in using a musl toolchain, because this way my application will run on most Linux "Out of the box" without any containers.

[bruce-one]

Hmm, odd; just as a sanity check did you make clean between runs?

I ran those steps verbatim in a clean (node) container; and also on my normal dev machine :-s (Which is Arch, so shouldn't be that different?)

Personally, I actually use the static linking with native modules -> rebuilding them all with musl, and then doing some code replacement to use process._linkedBinding('node_sqlite3') (for example, rather than require(binding_path)); and then modifying node.gyp to add those libraries to the node executable link step (just adding them as libraries).

When looking at out/Release/torque the file exists and the executable flag is set, but when trying to execute it I get: -bash: ./torque: No such file or directory

readelf -d torque looks OK, too:

It's dynamically linked against musl, but the musl libc isn't in the /lib/... location (hence it fails); it shouldn't be dynamically linked :-s (this makes me think a make clean might be in order?)

[Ralf12358]

I think we found the reason of the problem:

It's dynamically linked against musl, but the musl libc isn't in the /lib/... location (hence it fails); it shouldn't be dynamically linked :-s (this makes me think a make clean might be in order?)

I used --partly-static for building node, so I expected dynamically linked binaries. The problem is - as you mentioned: the search path /lib. I was aware of this problem so I used LD_LIBRARY_PATH, but it seems linux is searching in the /lib directory first and there is a libc.so file from the glibc library!

It worked in your example, because you used a clean node container, probably completely musl based.

So I will investigate if there is a way to force a library search path. I'm aware of -Wl,-rpath=$ORIGIN/lib as a linker flag, but I'm not sure if it tries at this position first.

Basically I like your idea of static link my modules to node, but I have to check how much fiddling it is to mix cmake and gyp when building node.

Yes, I did make clean between my tests and I set up a completely fresh directory when copying your example verbatim.

Thanks for your help!

[Ralf12358]

Well, this is not the problem. But I'm getting closer:

$ file torque 
torque: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, with debug_info, not stripped

This part concerns me: /lib/ld-musl-x86_64.so.1

[bruce-one]

Yeah, cool; are you happy to bundle the linker with your app? You can move the linker path using patchelf, eg patchelf --set-interpreter ./my-ld node - which would mean you could bundle everything you need together and then ship it?

And you're right; I did have a stray /lib64/ld-linux-x86-64.so.2 that made this magically work for me.

I'm going to have a look into that...

(Just for reference, the linker that is built into the package is available at a path akin to .../node_modules/node-musl/x64-linux-musl/x86_64-linux-musl/lib/libc.so if you wanted to use it, hence you could cp .../node_modules/node-musl/x64-linux-musl/x86_64-linux-musl/lib/libc.so . && patchelf --set-interpreter ./libc.so node ---- actually, my reading makes me think the path can't be relative, but you could potentially run the linker directly instead for ease, eg ./libc.so myprogram)

[Ralf12358]

Thanks a lot!

Relative pathes like: patchelf --set-interpreter ./libc.so torque seems to work fine!

Now there is just the remaining problem: When node is build it creates these binaries like torque and others. In the build process these binaries are used, which fails even if I - before - copy the libc.so into the .../out/Release driectory, because the binaires still reference the wrong linker path. I then have to manually set the linker with patchelf on the newly created binaries and try to continue the build process.

I probably need a whole musl based container to make this work without manual intervention and in this case node-musl does not help me that much. If I find a simple solution using node-musl I let you know.

Edit: Well, I only have to build node if I switch to a new node release, not for every build of my AddOn! So I'll try now if I can complete the build process of node with patchelf

Edit2: No I still can not finish the build process. It re-creates torque, so my patchelf-version of torque is lost and the build stops at the same position.

[bruce-one]

I did something akin to the following, if it helps any?

    npm i node-musl

    mkdir node; cd node
    cp "$(find ../node_modules -name libc.so -type f -executable -print -quit)" libc.so
    curl -L https://nodejs.org/dist/node-latest.tar.gz | tar xvz --strip-components=1
    while IFS='' read -r line || [[ -n "$line" ]]; do
      if ! [ -z "$line" ]; then
        eval ${line%\'} -Wl,-dynamic-linker=$(pwd)/libc.so\'
      fi
    done < <("$(npm bin)/musl-exports")
    ./configure --partly-static # fix linking with libgcc
    make -j$(node -p 'os.cpus().length + 1')

Which is using -dynamic-linker to say to use the specified libc.so/linker. Potentially then you'd want to do the patchelf to set the relative path.

(I haven't tried all those steps in sequence, so there might be gremlins, but it was along those lines.)

[Ralf12358]

Yes! Finally, there is a partly dnyamically linked node build with the musl toolchain.

Your script contained no gremlins and worked on the first paste.

Thanks, again!