fix: sandbox tag are unique (and monotone).

[luc-hiverge]

Some software stacks using docker will look at the version and assume that if the version is the same it must point to the same image. This is a fallacy because tags aren't immutable. Unfortunately, it's the default for gcloud/k8s. Therefore, the current choice leads to outdated code running in the sandbox.

Since the sandbox version tag has no real meaning and users are unlikely to push at the exact same time, we can use a simple timestamp.

[luc-hiverge]

/kind bug

[ky-hiverge]

Generally LGTM with this approach but still unclear why this will lead to outdated code. Is that a bug with hash[:7] which leads to tag conflicts?

[ky-hiverge]

I would suggest to add a git hash as well, you never know when we need the hash, it will look like "1772722212345678-a1b2c3d", use the first 7 characters of the hash. WDYT?

[luc-hiverge]

The dashboard already contains all the source code. I think it includes the .git directory. Either way, if we want the git commit, then we should make it part of the metadata visible from the dashboard.

If we add it how does one know it's the git commit and not the random tag appended to (most) runs? Do we need to add -dirty? Should we hash the non-tracked files ourselves? What about git ignored files?

Personally, I like that there's only one variation. If it's buggy it affects everyone equally and therefore gets spotted more reliably.

[luc-hiverge]

I think I don't see the benefit; but I also don't see a real negative of adding it. So if it's a big plus for you, we can add it.

[ky-hiverge]

I forgot the fact that the code is not related to the git anymore, since you won't commit the change, forget the suggestion then.

[luc-hiverge]

Generally LGTM with this approach but still unclear why this will lead to outdated code. Is that a bug with hash[:7] which leads to tag conflicts?

I think the steps are:

1. Checkout commit ab23 and run hive create exp ... which creates a docker tag gcr.io/.../dummy:ab23.
2. K8s creates a sandbox that loads this docker image using this tag gcr.io/.../dummy:ab23. It doesn't have it so it downloads it.
3. I make some modifications to the repo.
4. I don't commit my changes and run hive create exp again. This is a slow operation and genuinely builds the correct docker image with the correct contents, it then uploads them all correctly. Everything feels right at this point. Note that the tag is still: gcr.io/.../dummy:ab23. However, that tag points to different "blobs"/"digests". It's a different image with the same tag.
5. K8s creates a sandbox (which might be on the same node). K8s sees that it needs gcr.io/.../dummy:ab23 it already has a copy of an image with that tag. By default it will not check if this tag has been updated and just use what it already has.
6. The sandbox then runs on the old docker image, with outdated source code.

[ky-hiverge]

/lgtm
/approve