ci: disable gha caching for docker builder image #653

Job	Run time
ci-data	7s
Compile native gem for x64-mingw-ucrt	2m 13s
Compile native gem for aarch64-linux	2m 9s
Compile native gem for x86_64-darwin	1m 52s
Compile native gem for arm64-darwin	1m 54s
Compile native gem for aarch64-linux-musl	1m 54s
Compile native gem for x86_64-linux	5m 24s
Compile native gem for x86_64-linux-musl	2m 14s
	17m 47s

Provide feedback