Nested dune call fix (executable resolution part)

[Leonidas-from-XIV]

#10019 mentions that Dune should run itself when encountering package rules, which is quite sensible to do. This PR fixes part of the problem - the run command resolution. When encountering a command dune attempts to look up the programs path. But if the command is dune, we can immediately resolve this to the current executable (was we are Dune, even if the command is not called dune but dune.exe or main.exe or something else). This prevents capture of an accidental dune executable (as exhibited by the repro fix).

This PR thus solves the first case of #10020, where dune is called directly. It does not fix the second part (dune not part of the path), which requires a different fix that I intend to submit as a separate PR (as the fix is somewhat independent from this one).

[rgrinberg]

It does not fix the second part (dune not part of the path), which requires a different fix that I intend to submit as a separate PR (as the fix is somewhat independent from this one).

Why is it independent? I think if you were to insert dune in the PATH correctly, it would be enough to respect this PATH when doing binary look ups. I think it would make this work redundant.

[Alizter]

Why is it independent? I think if you were to insert dune in the PATH correctly, it would be enough to respect this PATH when doing binary look ups. I think it would make this work redundant.

Inserting dune in PATH is obviously the right thing to do. The issues arise when you actually try to do it. We have two cases:

1. Dune is in PATH, in which case there is nothing special to do and this should already work.
2. Dune is not in PATH, in which case the dune executable that is being run is living somewhere in the file system. The naive thing to do would be to add the parent folder to PATH. This has the side-effect of adding whatever else is living in the same directory as dune to PATH which is probably not a good idea.

To further address the issues in 2 we need to probably create a temporary home for dune executable, which this PR allows us to find. We then link it to live there and can add that temporary directory to PATH during the running of dune. That way we can cleanly add the current dune to PATH.

The technical details of the temporary directory need to be correctly sorted out. i.e. where does it live? Should be on the same file system, should be hardlinked etc.

[Leonidas-from-XIV]

Discussing this with @Alizter we noticed that we need a way to call a specific dune (the dune under test) but in a way where argv[0] is not set to the absolute path (otherwise Sys.executable_name will return just that), so I've expanded the test to emulate calling dune build call by setting argv[0] to dune instead of the absolute path.

It turns out our initial solution exec -a is not available in sh on Ubuntu and while there's plenty of other options, we can't really depend on them. So the test now builds a small utility which emulates what exec -a does.

[Alizter]

Sys.executable_name uses argv[0] and then checks proc (at least on linux) for a better candidate. I'm not yet certain what happens on macos or Windows.

We still need to check that this does something sane on the platforms we are interested in, but if it turns out not to be the case, I think introducing our own way of determining this is better.

Generally relying on argv[0] is not a great idea because various programs, can mutate its value with no real consequence.

[Leonidas-from-XIV]

I've rebased this on main as #12935 was merged. It is the remaining piece to get #10019 fixed.

The reason why this PR is independent from the other is because Dune does the resolution of the program executable in one place (expand_exe_value) and the computation of the PATH for subcommands in another place (Run_with_path.action). I could change the code to inject a temporary folder into in the path passed to Which.which a bit like it is done in #12935, but given we already know what binary we want "dune" to resolve to, might as well resolve it directly and avoid writing into the file system.

As the exec-a tests show this works correctly no matter what argv[0] is set to.

[Sudha247]

This looks like a sensible fix!