[RFC] loading: always append bundled depots to DEPOT_PATH unless JULIA_DEPOT_PATH_BUNDLED is false

[IanButterworth]

The requirement to have a trailing : or ; (windows) on JULIA_DEPOT_PATH is a really unfriendly footgun.

With this proposal when JULIA_DEPOT_PATH is set, the bundled depots are now always appended to Base.DEPOT_PATH, removing the need for the trailing : or ; (but allowing it if it is there).

A new environment variable JULIA_DEPOT_PATH_BUNDLED can be set to false to disable this behavior for users who explicitly want to exclude bundled resources (who should be very rare).

cc. @vchuravy @maleadt @Keno

[maleadt]

LGTM; I was always in favor of this behavior anyway.

x-ref #51448.

[vchuravy]

I am not a fan, but I understand where you are coming from.

The semantics of environment variables has some long traditions.

As an example

PATH="/a/b/c"
PATH="$PATH:/a/b/c"
PATH="/a/b/c:$PATH"

Are three very different operations. Now with JULIA_DEPOT_PATH we have the issue that we would like to populate with some defaults and that we can't use:

JULIA_DEPOT_PATH="/a/b/c:$JULIA_DEPOT_PATH"

so instead we use

JULIA_DEPOT_PATH="/a/b/c:"

(but note that $JULIA_DEPOT_PATH defaults to empty, so it isn't as much of a pun).

So currently we have two semantically distinct operations:

1. Append/Prepend by using the IMO pretty standard : operator
2. Setting the path

So this PR makes the second one harder to access and is contra to every other UNIX program I ever used.

[giordano]

and is contra to every other UNIX program I ever used.

I believe one difference here is that PATH doesn't have (as far as I know!) an implicit default set of directories where executables are searched in, while Julia's depot path has the implicit default depot (the one bundled with Julia itself), but only when the : is appended to JULIA_DEPOT_PATH, which I don't think is very intuitive for most people and constitutes a footgun (forgetting to include the : causes recompilation of stdlibs, and in turn of many other packages).

[IanButterworth]

What use case is there for not including the bundled depot? Understanding that would help here.

[vtjnash]

So this PR makes the second one harder to access and is contra to every other UNIX program I ever used.

Eh, I don't think unix is the one to follow here, as the most popular unix systems out there implement all 5 of the 3 options you outlined. (linux has assumed current working directory to be either at the end (old behavior) or the start (even older behavior) or drops it entirely (newer behavior), while the empty variable means default path but empty string means current working directory, while BSD uses the empty string to indicate where to put the current working directory in the path, even for empty string, with the default path having : at the beginning, until : was moved to the end--but the empty string also always meant current working directory, not the whole default argument. So BSD libc's have a consistent meaning for path:$PATH, but glibc has always had an inconsistent meaning for it)

[StefanKarpinski]

I don't really see what the issue here is. Is the issue that leaving off the "bundled" depot is too disruptive and rarely what anyone wants? I suppose we could have no empty entry still include the bundled depot since that's the real issue here. I think an explicit @default makes this worse. I also dislike the change in #51448 and think we're rapidly careening towards the way this variable works being incoherent.

[StefanKarpinski]

It seems to me that #51448 was probably a mistake—it was a different attempt to solve this problem, namely that you almost never want to not include the bundled depot. When you write JULIA_DEPOT_PATH=/some/path you want the actual depot path to be ["/some/path", bundled_depot]. That change was motivated by wanting to do JULIA_DEPOT_PATH=/some/path: and have it expand to roughly that, which means not including the user depot. But what it actually wanted was this change. So basically, I'm in favor of this change but only if we revert the other one.