Felix (latest) build error on Linux

[razetime]

Felix (as of the latest commit in 26th October) throws a build error when installed.

OS: Ubuntu 20.04.5 LTS x86_64

Python version: 3.8.10

OCaml version: 4.14.0

g++ version: 9.4.0

Full command output: gist

Previous discussion has been done in the Felix mail group, and in #174.

[skaller]

So: the bootstrap build has worked successfully. However I cannot see the exact commands, which are in the file

build/fbuild.log

I think. I also cannot see the commands for the second build, these can be exposed by

export FLX_SHELL_ECHO=1

Both build processes just provide a summary of each step, rather than the exact shell command.

So in the second build we get two different errors when trying to link:

/usr/local/bin/ld: error: build/release/host/lib/rtl/libflx_static.a(flx_world_config_static.o): requires unsupported dynamic reloc 11; recompile with -fPIC
/usr/local/bin/ld: error: build/release/host/lib/rtl/libflx_static.a(flx_svc_static.o): requires unsupported dynamic reloc 11; recompile with -fPIC
/usr/local/bin/ld: error: build/release/host/lib/rtl/libflx_static.a(flx_ioutil_static.o): requires dynamic R_X86_64_32 reloc which may overflow at runtime; recompile with -fPIC

As far as I can tell this is a bug in the linker. The linker thinks it should be linking a shared library OR an executable that depends on a shared library. But it's SUPPOSED to be doing a full static link and all the objects file SHOULD have been compiled for that. Every one of them ends in the suffix

_static.o

which means precisely compiled WITHOUT -fPIC for static linkage.

Now the thing is, the fbuild (bootstrap) build successfully linked bootflx (which is renamed flx) which is almost exactly the same code as the full flx except it doesn't support plugins.

Now, plugin support just loads shared libraries. This uses dlopen. However dlopen and dlsym should NOT require the executable to be -fPIC. That would be required if we did LOAD TIME linkage to a shared library but plugins are loaded at RUN TIME. It's possible the Linux library object file containing dlopen is incorrectly compiled.

So at the moment it looks like gcc, ld, or something in LInux is broken. It can be easily fixed by simply building even the static objects with -fPIC but the whole point of having static objects at all is to avoid precisely that.

It's POSSIBLE that the error is in the Felix build code but very unlikely because it has worked before, and, the compiles are done with a wildcarded Felix script that builds everything the same way. The difference in error messages is almost certainly because of this: some modules have no dependencies, other does. So the reloc 11 error is when there are no dependencies, and the reloc X86 error is when there are.

The fact that EVERY object file gives an error suggests it its the linker command that is wrong: either the wrong switches given by the Felix build script, OR, the linker itself is broken.

Linkage by default should be static.

However .. there is one issue here: the C library on most platforms is a shared library END OF STORY. There is no static link shared library any more. Linux is one of the archaic holdouts on this one. It's possible modern Linux has removed static link C libraries (because they prevent upgrading the system!)

MacOS doesn't support static link system libraries: not just the C library either, almost everything in a Framework on a Mac is dynamic link, and dynamic link ONLY. Similarly Windows. Everything's a DLL.

So the bottom line is this:

Step 1: set

export FLX_SHELL_ECHO=1

Step 2: delete the build artefacts

rm -rf build

Step 3: rebuild it (sorry!)
make

You need to somehow save the output. Both stderr and stdout.

Step 4: Find the linker commands used to build bootflx and flx.
The first one is in the fbuild.log file.
The second in console output from the second stage build.
This has all the errors in it.

The two commands may be different. The first one works, the second doesn't.
However the fbuild system uses Python for compiling and linking, whereas
the second stage does compiling and linking all over again using Felix script
instead of Python.

I need to see both linker commands to see if I got the switches right.

Based on the advice of the error message it is possible to patch it so that the compiles are done with -fPIC even when they should not need to be. The actual compiler toolchains are in this file:

https://github.com/felix-lang/felix/blob/master/src/packages/toolchain.fdoc

around line 935 you will see the Felix code that is used to launch g++ for compilation and a bit later for linkage.

[skaller]

So I'm now building Felix on Ubuntu using GitHub Workflows! Hopefully this will help identify the problem.

[skaller]

Well you can look in the Actions tag in GitHub for the repository. The bootstrap, for some arcane reason is using clang++. I can fix that. But the Felix build is using g++ (so it says anyhow) and it's building without any problem.

I need to add some stuff to be sure it's actually using g++ and what version.

[skaller]

ok everything builds with clang++ I've fixed the script finally to use g++ instead

[skaller]

https://github.com/felix-lang/felix/actions/runs/3337443837/jobs/5523792114

This is using g++ version 9.4 and it builds just fine. I wonder why that works and your build does not?

[razetime]

I am not sure why, but I do have the result of a fresh build from scratch.

• log of the full build, stdout and stderr
• even though FLX_SHELL_ECHO was set to 1, there was not build/fbuild.log
  • I did find `build/release/fbuild.log'. this is the file: link

[skaller]

maybe build/release/fbuild.log

This is always produced. It is unrelated to FLX_SHELL_ECHO.

[skaller]

It seems to have worked!

[razetime]

Oh, that's great. It must have been the artefacts that hurt the build. Thanks for the support.

[skaller]

Just to explain the build process:

1a. A program written in Python, called fbuild, is used to build a bootstrap version of Felix.
The log for that part of the build processes is here:

~/felix>ls build/release/fbuild.log
build/release/fbuild.log

This contains more detailed information than shown on the console.

Looking at the file you posted, fbuild decided to use clang++, not g++, as the compiler for this phase.
This shouldn't matter .. but it might if any object files are left lying around because whilst clang and gcc use compatible C libraries, they use quite different, incompatible C++ libraries.

The main result of this process is to build flxg the Felix compiler (written in Ocaml).

1b. Now, the flxg compiler is used to compile bootflx, which is written in Felix.

1c. Now bootflx is renamed to flx, and is used to build flx_build_boot, flx_build_prep, flx_build_flxg
and flx_build_rtl, the Felix programs used to build Felix.

2. We now have a cut down Felix system, with enough things built to rebuild Felix from scratch.
   So we use these tools to rebuild the compiler flxg and the run time library, and finally the
   driver flx. Then we scrap the boostrap Felix altogether and use flx to rebuild the build
   tools, and then use these tools to rebuild the rest of the system.

3. Finally we run the tests.

Now if the repository is changed you can try

make rebuild

and it rebuild the system starting at phase 2. If the ONLY changes are to test code, or most
of the library, or the grammar, you don't need to rebuild ANYTHING. It will be rebuilt automatically.
The rebuild is needed if the compiler is changed, or the code for flx itself is changed.
And sometimes other things, so if you're not sure, just do a phase 2 rebuild.

Occasionally a clean rebuild from phase 1 is required.

There are other targets in the GNUmakefile:

make test

is one that can be run every now and then. There will be a

make rosetta

at some stage to rebuild and check all the Rosetta tests.

Finally if you say

flx --help

you will get a list of switches and environment variables. This one:

FLX_SHELL_ECHO=1

causes ANY Felix program that calls the shell to display the text of the shell call. Of course flx does this a lot:

~/felix>FLX_SHELL_ECHO=1 flx hello
[system] "/Users/skaller/felix/build/release/host/bin/flxg" "-q" "--inline=25" "--output_dir=/Users/skaller/.felix/cache/text" "--cache_dir=/Users/skaller/.felix/cache/binary" "-I/Users/skaller/felix/build/release/share/lib" "-I/Users/skaller/felix/build/release/host/lib" "--syntax=@/Users/skaller/felix/build/release/share/lib/grammar/grammar.files" "--automaton=/Users/skaller/.felix/cache/binary/Users/skaller/felix/build/release/share/lib/grammar/grammar.files/syntax.automaton" "--import=plat/flx.flxh" "--import=concordance/concordance.flxh" "std" "/Users/skaller/felix/hello.flx"
[get_stdout] "clang++" "-MM" "-std=c++14" "-O1" "-I/Users/skaller/felix/build/release/share/lib/rtl" "-I/Users/skaller/felix/build/release/host/lib/rtl" "/Users/skaller/.felix/cache/text/Users/skaller/felix/hello.cpp"
[system] "clang++" "-fPIC" "-fvisibility=hidden" "-g" "-c" "-O1" "-fno-common" "-fno-strict-aliasing" "-std=c++14" "-w" "-Wfatal-errors" "-Wno-return-type-c-linkage" "-Wno-invalid-offsetof" "-O1" "-I/Users/skaller/felix/build/release/share/lib/rtl" "-I/Users/skaller/felix/build/release/host/lib/rtl" "/Users/skaller/.felix/cache/text/Users/skaller/felix/hello.cpp" -o "/Users/skaller/.felix/cache/text/Users/skaller/felix/hello_dynamic.o"
[system] "clang++" "-dynamiclib" "/Users/skaller/.felix/cache/text/Users/skaller/felix/hello_dynamic.o" -o "/Users/skaller/.felix/cache/binary/Users/skaller/felix/hello.dylib" "-L/Users/skaller/felix/build/release/host/lib/rtl" "-lflx_dynamic" "-lflx_pthread_dynamic" "-lflx_dynlink_dynamic" "-lflx_strutil_dynamic" "-lflx_gc_dynamic" "-ljudy_dynamic" "-lflx_exceptions_dynamic"
[system] env DYLD_LIBRARY_PATH=/Users/skaller/felix/build/release/host/lib/rtl:$DYLD_LIBRARY_PATH "/Users/skaller/felix/build/release/host/bin/flx_run" "/Users/skaller/.felix/cache/binary/Users/skaller/felix/hello.dylib"
[load_library] /Users/skaller/.felix/cache/binary/Users/skaller/felix/hello.dylib
Hello World!

You can see flx calling the compiler flxg and then the c++ compiler clang++ to compile the file, and then clang++ again to link the file .. SURPRISE! It is making a shared library! It then runs the shared library hello.dylib from the executable flx_run.

By default, Felix generates shared libraries. If you want an executable:

~/felix>FLX_SHELL_ECHO=1 flx --static hello
[get_stdout] "clang++" "-MM" "-std=c++14" "-O1" "-I/Users/skaller/felix/build/release/share/lib/rtl" "-I/Users/skaller/felix/build/release/host/lib/rtl" "/Users/skaller/.felix/cache/text/Users/skaller/felix/hello_static_link_thunk.cpp"
[system] "clang++" "-g" "-c" "-O1" "-fno-common" "-fno-strict-aliasing" "-std=c++14" "-w" "-Wfatal-errors" "-Wno-return-type-c-linkage" "-Wno-invalid-offsetof" "-O1" "-I/Users/skaller/felix/build/release/share/lib/rtl" "-I/Users/skaller/felix/build/release/host/lib/rtl" "/Users/skaller/.felix/cache/text/Users/skaller/felix/hello_static_link_thunk.cpp" -o "/Users/skaller/.felix/cache/text/Users/skaller/felix/hello_static_link_thunk_static.o"
[get_stdout] "clang++" "-MM" "-std=c++14" "-O1" "-I/Users/skaller/felix/build/release/share/lib/rtl" "-I/Users/skaller/felix/build/release/host/lib/rtl" "/Users/skaller/.felix/cache/text/Users/skaller/felix/hello.cpp"
[system] "clang++" "-g" "-c" "-O1" "-fno-common" "-fno-strict-aliasing" "-std=c++14" "-w" "-Wfatal-errors" "-Wno-return-type-c-linkage" "-Wno-invalid-offsetof" "-O1" "-I/Users/skaller/felix/build/release/share/lib/rtl" "-I/Users/skaller/felix/build/release/host/lib/rtl" "/Users/skaller/.felix/cache/text/Users/skaller/felix/hello.cpp" -o "/Users/skaller/.felix/cache/text/Users/skaller/felix/hello_static.o"
[system] "clang++"  -o "/Users/skaller/.felix/cache/binary/Users/skaller/felix/hello" "/Users/skaller/felix/build/release/host/lib/rtl/flx_run_lib_static_static.o" "/Users/skaller/felix/build/release/host/lib/rtl/flx_run_main_static.o" "/Users/skaller/.felix/cache/text/Users/skaller/felix/hello_static_link_thunk_static.o" "/Users/skaller/.felix/cache/text/Users/skaller/felix/hello_static.o" "-L/Users/skaller/felix/build/release/host/lib/rtl" "-lflx_static" "-lflx_pthread_static" "-lflx_dynlink_static" "-lflx_strutil_static" "-lflx_gc_static" "-ljudy_static" "-lflx_exceptions_static"
[system] "/Users/skaller/.felix/cache/binary/Users/skaller/felix/hello"
Hello World!

You may notice above flxg isn't called this time! That's because the output is cached and the input hasn't changed. You can also see clang++ --MM there. That's calculating the C++ dependencies. Watch this:

~/felix>FLX_SHELL_ECHO=1 flx --static hello
[get_stdout] "clang++" "-MM" "-std=c++14" "-O1" "-I/Users/skaller/felix/build/release/share/lib/rtl" "-I/Users/skaller/felix/build/release/host/lib/rtl" "/Users/skaller/.felix/cache/text/Users/skaller/felix/hello_static_link_thunk.cpp"
[get_stdout] "clang++" "-MM" "-std=c++14" "-O1" "-I/Users/skaller/felix/build/release/share/lib/rtl" "-I/Users/skaller/felix/build/release/host/lib/rtl" "/Users/skaller/.felix/cache/text/Users/skaller/felix/hello.cpp"
[system] "/Users/skaller/.felix/cache/binary/Users/skaller/felix/hello"
Hello World!

See? The -MM steps calculate dependencies but there is no Felix compile, there is no C++ compile, and there is no link! Felix just runs the executable. Everything is cached. In fact notice WHERE the executable is .. it's in the cache too.

[razetime]

When i attempt to run programs with flx, however I get this error:

$ flx factorial.flx
/home/razetime/Software/felix/build/release/host/bin/flx_run: symbol lookup error: /home/razetime/Software/felix/build/release/host/lib/rtl/libflx_gc_dynamic.so: undefined symbol: pthread_create
Error 127 in flx: [strerror_r] Failed to find text for error number 127

[skaller]

AH. I think I know what that is. Actually all of the test cases you had failed. I didn't notice. Hmm.

I think the problem is that the linker needs -lpthread.

[razetime]

flx --static factorial.flx runs it perfectly.