How to correctly add dependencies for embedded generated files

[dpirch]

I want compile shaders from their glsl source to spir-v and then embed them in C sources (using C23 #embed).

This seems to work if I create custom_targets like

spirv_shaders = []
foreach shader: shader_sources
    spirv_shaders += custom_target(
        shader + '.spv',
        input: files(shader),
        output: shader + '.spv',
        command: [glslang, '-V', '@INPUT@', '-o', '@OUTPUT@'] + extra_args,
    )
endforeach

and then either add spirv_shaders directly to the sources of an executable, or create a dependency with shaders_dep = declare_dependency(sources: spirv_shaders) and add that to the executable as a dependency.

But in both cases, whenever I make changes to a shader, apparently all C files are recompiled, not just the ones which actually embed the updated spv. How do I make meson/ninja use the compiler's -MD output correctly to only recompile the C sources that need to be recompiled?

[dcbaker]

glslang is a pain.

what does ninja $builddir -d explain say after you touch one of the GLSL sources?

As an aside to your question, here's the code from mesa (mostly written by me) to check if glslang supports depfile generation of its own, which you may find helpful.

# GLSL has interesting version output and Meson doesn't parse it correctly as of                                                                                                                                                                                                           
# Meson 1.4.0                                                                                                                                                                                                                                                                              
prog_glslang = find_program('glslangValidator', native : true)                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                           
if prog_glslang.found()                                                                                                                                                                                                                                                                    
  _glslang_version = run_command(prog_glslang, ['--version'], check : false).stdout().split(':')[2]                                                                                                                                                                                        
  # Check if glslang has depfile support. Support was added in 11.3.0, but                                                                                                                                                                                                                 
  # Windows path support was broken until 11.9.0.                                                                                                                                                                                                                                          
  #                                                                                                                                                                                                                                                                                        
  # It is intentional to check the build machine, since we need to ensure that                                                                                                                                                                                                             
  # glslang will output valid paths on the build platform                                                                                                                                                                                                                                  
  _glslang_check = build_machine.system() == 'windows' ? '>= 11.9.0' : '>= 11.3.0'                                                                                                                                                                                                         
  if _glslang_version.version_compare(_glslang_check)                                                                                                                                                                                                                                      
    glslang_depfile = ['--depfile', '@DEPFILE@']                                                                                                                                                                                                                                           
  else                                                                                                                                                                                                                                                                                     
    glslang_depfile = []                                                                                                                                                                                                                                                                   
  endif                                                                                                                                                                                                                                                                   
  if run_command(prog_glslang, [ '--quiet', '--version' ], check : false).returncode() == 0                                                                                                                                                                                                
    glslang_quiet = ['--quiet']                                                                                                                                                                                                                                                            
  else                                                                                                                                                                                                                                                                                     
    glslang_quiet = []                                                                                                                                                                                                                                                                     
  endif                                                                                                                                                                                                                                                                                    
endif

[dpirch]

Actually I was testing with an older version of Meson, with Meson 1.9.1 the problem is no longer that everything is rebuilt, but instead I have to run ninja two times after changing the source file, because the first time only generates the embedded file again but doesn't recompile the program.

I don't think it is related to glslang, the shaders don't include anything for now, so their only dependency is the input file.

Here is another minimal example without glslang:

meson.build:

project('demo', 'c',
  version: '1.0.0',
  default_options: ['c_std=c23']
)

generated_file = custom_target('generate_output',
  input: 'demo.txt.in',
  output: 'demo.txt',
  command: ['sed', 's/Meson/World/g', '@INPUT@'],
  capture: true,
)

exe = executable('demo',
  'main.c',
  dependencies: [declare_dependency(sources: generated_file)],
  c_args: ['--embed-dir=' + meson.current_build_dir()],
)

main.c:

#include <stdio.h>

static const unsigned char text[] = {
    #embed "demo.txt" suffix(,)
    0
};

int main(void) {
    printf("%s", text);
}

demo.txt.in:

Hello, Meson!

The build after echo Line2 >> demo.txt.in regenerates only demo.txt but doesn't compile the C program, this happens only when I call ninja again without any other changes:

$ CC=gcc-15 meson setup build
The Meson build system
Version: 1.9.1
Source dir: /home/dp/buffer/mesontest
Build dir: /home/dp/buffer/mesontest/build
Build type: native build
Project name: demo
Project version: 1.0.0
C compiler for the host machine: gcc-15 (gcc 15.2.0 "gcc-15 (Debian 15.2.0-7) 15.2.0")
C linker for the host machine: gcc-15 ld.bfd 2.44
Host machine cpu family: x86_64
Host machine cpu: x86_64
Build targets in project: 2

Found ninja-1.12.1 at /usr/bin/ninja
$ ninja -C build -v -d explain
ninja: Entering directory `build'
ninja explain: deps for 'demo.p/main.c.o' are missing
ninja explain: output demo.txt doesn't exist
ninja explain: demo.p/main.c.o is dirty
ninja explain: demo is dirty
[1/3] /usr/bin/meson --internal exe --capture demo.txt -- /usr/bin/sed s/Meson/World/g ../demo.txt.in
[2/3] gcc-15 -Idemo.p -I. -I.. -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c23 -O0 -g --embed-dir=/home/dp/buffer/mesontest/build -MD -MQ demo.p/main.c.o -MF demo.p/main.c.o.d -o demo.p/main.c.o -c ../main.c
[3/3] gcc-15  -o demo demo.p/main.c.o -Wl,--as-needed -Wl,--no-undefined
$ echo Line2 >> demo.txt.in 
$ ninja -C build -v -d explain
ninja: Entering directory `build'
ninja explain: recorded mtime of demo.txt older than most recent input ../demo.txt.in (1764612299739331626 vs 1764612306975281923)
[1/1] /usr/bin/meson --internal exe --capture demo.txt -- /usr/bin/sed s/Meson/World/g ../demo.txt.in
$ ./build/demo
Hello, World!
$ ninja -C build -v -d explain
ninja: Entering directory `build'
ninja explain: output demo.p/main.c.o older than most recent input /home/dp/buffer/mesontest/build/demo.txt (1764612299755331515 vs 1764612315115225929)
ninja explain: demo.p/main.c.o is dirty
ninja explain: demo is dirty
[1/2] gcc-15 -Idemo.p -I. -I.. -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c23 -O0 -g --embed-dir=/home/dp/buffer/mesontest/build -MD -MQ demo.p/main.c.o -MF demo.p/main.c.o.d -o demo.p/main.c.o -c ../main.c
[2/2] gcc-15  -o demo demo.p/main.c.o -Wl,--as-needed -Wl,--no-undefined
$ ./build/demo
Hello, World!
Line2

[bonzini]

Reproduced with both Meson 1.8.5 and 1.9.1, but it seems to be a ninja bug. I cannot reproduce it with samurai and I could even reproduce without Meson at all, replacing its output-capture wrapper with a script like this (out.py):

#! /usr/bin/env python3
import sys
with open(sys.argv[1], 'w') as f:
    print('Hello, World!', file=f)
    with open(sys.argv[2], 'r') as g:
        for line in g:
            print(line, file=f)

and a minimal build.ninja:

ninja_required_version = 1.8.2

rule c_COMPILER
 command = cc $ARGS -MD -MQ $out -MF $DEPFILE -o $out -c $in
 deps = gcc
 depfile = $DEPFILE_UNQUOTED
rule c_LINKER
 command = cc $ARGS -o $out $in
rule CUSTOM_COMMAND
 command = $COMMAND
 restat = 1

build demo.txt: CUSTOM_COMMAND ../demo.txt.in | ../out.py
 COMMAND = /usr/bin/python3 ../out.py demo.txt ../demo.txt.in
build main.c.o: c_COMPILER ../main.c || demo.txt
 DEPFILE = main.c.o.d
 DEPFILE_UNQUOTED = main.c.o.d
 ARGS = -I. -I.. -O0 -g --embed-dir=/home/pbonzini/work/upstream/meson/ff/build
build demo: c_LINKER main.c.o
default demo

Note that wihle build.ninja has an order-only dependency on demo.txt, the expectation that the actual demo.txt dependency is recorded in the dependencies, which it is (as found with both ninja -t deps and ninja -d keepdepfile:

main.c.o: ../main.c [snip] /home/pbonzini/work/upstream/meson/ff/build/demo.txt

Removing restat =1 does not fix the bug, either. I'm on Fedora 43 with ninja versions 1.13.0.git.kitware.jobserver-pipe-1 (from PyPI) and 1.13.1.

[dcbaker]

Thanks for doing the leg work on this.

If I'm reading this right GCC is producing a valid depfile, but ninja isn't reading it back in correctly?

[dpirch]

I see the same issue with samurai, actually with all combinations of meson/muon and ninja/samurai (with muon 0.6.0-942374c5)

But it does seem to work correctly with clang instead of gcc. Apparently gcc writes the absolute path of demo.txt into the depfile, and clang writes the relative path (i.e. just demo.txt).

[dcbaker]

@thesamesam. I see the same behavior, but that doesn't make sense to me because both clang and GCC write all of the headers out as absolute paths. So there's a bug here, I'm leaning toward a ninja bug, but maybe you have a different opinion?

[dpirch]

Clang seems to search the working directory for embed files before --embed-dir, unlike gcc which doesn't search the working directory at all. This is why clang simply wrote a relative "demo.txt" into the depfile; it didn't need the --embed-dir in this case.

I think normally the paths the compiler writes into the depfile are just like the search path passed as -I or --embed-dir where the file is found.

For generated header files, meson automatically adds -I flags with relative paths, so the relative depfile paths match the relative paths in the Ninja file.

But for embed files, I need to set '--embed-dir=' + meson.current_build_dir() manually, that is an absolute path so the depfile has absolute paths too, which don't match the relative paths in the Ninja file.

I think this is where the problem comes from; Ninja treats a relative and absolute path to the same file as separate files, it says so in the manual and there is also this issue with >100 comments about it: ninja-build/ninja#1251

[dcbaker]

So, one option would be for us to add an embedded_directory option, that passes --embed-dir with relative paths. That wouldn't be too hard, I wonder how many lanuages apart from the C family would having something equivalent to that.

[dpirch]

As discussed above, my problem was adding an absolute path with '--embed-dir=' + meson.current_build_dir(), so a solution for now is not to use meson.current_build_dir(), and instead either simply hardcode the correct relative path from the project build directory (e.g. '--embed-dir=shaders'), or to use fs.relative_to.

This seems to work well, even with includes between glsl sources:

shader_sources = ['test.vert', 'test.frag']
spirv_shaders = []
foreach shader: shader_sources
    spirv_shaders += custom_target(
        shader + '.spv',
        input: shader,
        output: shader + '.spv',
        depfile: shader + '.d',
        command: [glslang, '-V', '@INPUT@', '-o', '@OUTPUT@', '--depfile', '@DEPFILE@'],
    )
endforeach

shaders_dep = declare_dependency(
    sources: spirv_shaders,
    compile_args: '--embed-dir=' + import('fs').relative_to(meson.current_build_dir(), meson.project_build_root()),
)

# possibly in another subdir:
executable(
    # .....
    dependencies: [shaders_dep],
)