diff --git a/87_lazy_pulling/README.md b/87_lazy_pulling/README.md index bb4759f..bca47f9 100644 --- a/87_lazy_pulling/README.md +++ b/87_lazy_pulling/README.md @@ -8,6 +8,8 @@ TODO: * nerdctl * does slim ai use it? * Test time to pull and run from registry. +* NOTE: This does not work on linux - even using containerd. +* Check how to get these examples working on ubuntu with containerd ## Reason @@ -52,7 +54,7 @@ export BASEIMAGE=scratch #export BASEIMAGE=gcr.io/distroless/nodejs16-debian11 export IMAGE_NAME_ZSTD=ttl.sh/$(uuidgen):1h export IMAGE_NAME_ZSTD=${IMAGE_NAME_ZSTD:l} -export COMPRESSION_LEVEL=15 +export COMPRESSION_LEVEL=5 docker build --build-arg=baseimage=$BASEIMAGE --build-arg=NIX_FILE=ffmpeg-full.nix --build-arg=PROGRAM_FILE=ffmpeg --progress=plain -f Dockerfile.ffmpeg --target PRODUCTION --output type=image,name=$IMAGE_NAME_ZSTD,oci-mediatypes=true,compression=zstd,compression-level=$COMPRESSION_LEVEL,force-compression=true,push=true . diff --git a/90_containerd/.gitignore b/90_containerd/.gitignore new file mode 100644 index 0000000..10d00b5 --- /dev/null +++ b/90_containerd/.gitignore @@ -0,0 +1 @@ +*.gz diff --git a/90_containerd/README.md b/90_containerd/README.md new file mode 100644 index 0000000..4736d77 --- /dev/null +++ b/90_containerd/README.md @@ -0,0 +1,226 @@ +# CONTAINERD + +Demonstrate how to use `containerd` + +TODO: + +* firecracker can be containerd plugin +* dive does not work with containerd. Why is this? +* Get Estargz compression working on linux. +* https://www.usenix.org/system/files/conference/fast16/fast16-papers-harter.pdf +* https://link.medium.com/Jl8qGEQPGyb +*https://github.com/containerd/stargz-snapshotter/issues/258 +* https://github.com/containerd/stargz-snapshotter/blob/main/docs/ctr-remote.md +* https://github.com/containerd/containerd/blob/main/docs/rootless.md +* https://github.com/containerd/stargz-snapshotter/blob/main/docs/INSTALL.md +* Checked vendor/modules.txt to see what version was included https://github.com/moby/moby/tree/219f21bf07502b447095649b5a2764661737f164 +* https://github.com/containerd/stargz-snapshotter/blob/main/docs/overview.md +* https://manpages.ubuntu.com/manpages/jammy/man8/mount.fuse.8.html + +## Install Docker Desktop + +On MacOS Docker Desktop it is possible to set docker to use containerd as the default backend. + +## Install Ubuntu + +```sh +# install docker prereqs +sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin + +# check containerd is running +sudo systemctl status containerd + +# sort outaa group for using non-root containerd socket +sudo groupadd containerd +sudo usermod -a -G containerd $(whoami) +sudo chown :containerd /var/run/containerd/containerd.sock +sudo chmod 660 /var/run/containerd/containerd.sock +sudo ls -l /var/run/containerd/containerd.sock +id +ctr version + +# ctr will be installed +sudo ctr version + +ctr +ctr plugins ls + + +# run nginx +docker run -d -p 8080:80 nginx + +# docker +docker ps +# runc +runc list +# containerd +sudo ctr containers ls +sudo ctr namespaces ls +sudo ctr --namespace moby containers ls +sudo ctr --namespace moby containers info 0a5ef6d3a499e420c00692ac04cfe84fbc5f1d96ebde8bf4765c0f5f3c9088dd + + +sudo /home/linuxbrew/.linuxbrew/bin/nerdctl ps + +dockerd --help + +# it looks like it is running containerd sock +ps -aux | grep dockerd +``` + +## Upgrade to Kernel 5.19 + +```sh +sudo apt-get install --install-recommends linux-generic-hwe-22.04 +``` + +## Configuration + +```sh +cat /etc/docker/daemon.json +cat /etc/containerd/config.toml +cat /etc/containerd-stargz-grpc/config.toml +``` + +## Plugins + +Instructions for installing estargz snapshotter [here](https://github.com/containerd/stargz-snapshotter/blob/main/docs/INSTALL.md) +https://github.com/containerd/stargz-snapshotter/blob/main/script/config/etc/containerd/config.toml + +```sh +sudo nano /etc/docker/daemon.json +sudo nano /etc/containerd/config.toml + +# list plugins +sudo ctr plugins ls +``` + +Goto stargz-snapshotter [releases](https://github.com/containerd/stargz-snapshotter/releases) + +```sh +sudo -s + +version=v0.14.3 +arch=amd64 +# get plugin +curl -Lo stargz-snapshotter-${version}-linux-${arch}.tar.gz https://github.com/containerd/stargz-snapshotter/releases/download/${version}/stargz-snapshotter-${version}-linux-${arch}.tar.gz + +tar -C /usr/local/bin -xvf stargz-snapshotter-${version}-linux-${arch}.tar.gz containerd-stargz-grpc ctr-remote +curl -Lo ./stargz-snapshotter.service https://raw.githubusercontent.com/containerd/stargz-snapshotter/$version/script/config/etc/systemd/system/stargz-snapshotter.service +cp ./stargz-snapshotter.service /etc/systemd/system/stargz-snapshotter.service + +systemctl enable --now stargz-snapshotter +systemctl restart containerd +systemctl restart docker +``` + +Example stargz-snapshotter.service + +```ini +[Unit] +Description=stargz snapshotter +After=network.target +Before=containerd.service + +[Service] +Type=notify +Environment=HOME=/root +ExecStart=/usr/local/bin/containerd-stargz-grpc --log-level=debug --config=/etc/containerd-stargz-grpc/config.toml +Restart=always +RestartSec=1 + +[Install] +WantedBy=multi-user.target +``` + +```toml +# explicitly use v2 config format +version = 2 + +# - Set default runtime handler to v2, which has a per-pod shim +# - Enable to use stargz snapshotter +[plugins."io.containerd.grpc.v1.cri".containerd] + default_runtime_name = "runc" + snapshotter = "stargz" + disable_snapshot_annotations = false +[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] + runtime_type = "io.containerd.runc.v2" + +# Setup a runtime with the magic name ("test-handler") used for Kubernetes +# runtime class tests ... +[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler] + runtime_type = "io.containerd.runc.v2" + +# Use stargz snapshotter +[proxy_plugins] + [proxy_plugins.stargz] + type = "snapshot" + address = "/run/containerd-stargz-grpc/containerd-stargz-grpc.sock" +``` + +journalctl -xeu stargz-snapshotter.service + +sudo /usr/local/bin/containerd-stargz-grpc +sudo dmesg + +{"error":"failed to mount overlay: invalid argument","level":"fatal","msg":"snapshotter is not supported","time":"2023-04-08T19:53:06.916442927+01:00"} + +overlayfs: upper fs does not support RENAME_WHITEOUT. +overlayfs: upper fs is missing required features. + + + + +https://github.com/canonical/microk8s/issues/1378 + +https://discuss.linuxcontainers.org/t/run-docker-on-lxd-container/11575/7 + +https://discuss.linuxcontainers.org/t/docker-overlay2-on-btrfs-supported/16771 + + + +# it seems to create this folder. +sudo ls -la /var/lib/containerd-stargz-grpc/snapshotter + +sudo strace /usr/local/bin/containerd-stargz-grpc + +ZFS is sadly quite painful with Docker in Docker and similar scenarios. It might be best to avoid the problem by creating a volume in your ZFS pool, formatting that volume to ext4, and having docker use "overlay2" on top of that, instead of "zfs". + +zfs create -s -V 20GB rpool/docker +mkfs.ext4 /dev/zvol/rpool/docker + +mkdir -p /mnt/docker + +sudo mount /dev/zvol/zroot/docker /mnt/docker + + +# add the mount to /etc/fstab +mount /dev/zvol/zroot/docker /var/lib/docker +The zfs create -s is for sparse volumes. Analogous to thin provisioning on LVM. + +I just finished setting this up and it nicely solves my problems with k3s and also kind. I use these for testing and development, and there the volume should be just fine. + +https://github.com/k3s-io/k3s/issues/66 + + + +## nerdctl + +```sh +# on linux or mac +brew install nerdctl +``` + +## Resources + +* ctr - Man Page [here](https://www.mankier.com/8/ctr) +* nerdctl: Docker-compatible CLI for containerd [here](https://github.com/containerd/nerdctl) +* Getting started with containerd [here](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) +* runc is a CLI tool for spawning and running containers on Linux according to the OCI specification. [here](https://github.com/opencontainers/runc) +* Why and How to Use containerd From Command Line [here](https://iximiuz.com/en/posts/containerd-command-line-clients/) +* Extending Docker’s Integration with containerd [here](https://www.docker.com/blog/extending-docker-integration-with-containerd/) +* https://docs.docker.com/engine/reference/commandline/dockerd/ +* https://medium.com/nttlabs/startup-containers-in-lightning-speed-with-lazy-image-distribution-on-containerd-243d94522361 +* https://www.tutorialworks.com/difference-docker-containerd-runc-crio-oci/?utm_content=cmp-true +* https://docs.docker.com/engine/install/linux-postinstall/ +* https://github.com/containerd/nerdctl/blob/main/docs/stargz.md diff --git a/90_containerd/flake.lock b/90_containerd/flake.lock new file mode 100644 index 0000000..f7b02fa --- /dev/null +++ b/90_containerd/flake.lock @@ -0,0 +1,27 @@ +{ + "nodes": { + "nixpkgs": { + "locked": { + "lastModified": 1742069588, + "narHash": "sha256-C7jVfohcGzdZRF6DO+ybyG/sqpo1h6bZi9T56sxLy+k=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "c80f6a7e10b39afcc1894e02ef785b1ad0b0d7e5", + "type": "github" + }, + "original": { + "owner": "NixOS", + "ref": "nixos-unstable", + "repo": "nixpkgs", + "type": "github" + } + }, + "root": { + "inputs": { + "nixpkgs": "nixpkgs" + } + } + }, + "root": "root", + "version": 7 +} diff --git a/90_containerd/flake.nix b/90_containerd/flake.nix new file mode 100644 index 0000000..fda0f72 --- /dev/null +++ b/90_containerd/flake.nix @@ -0,0 +1,35 @@ +# flake.nix +{ + description = "A flake for working with containerd"; + inputs = { + nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; + }; + + outputs = {nixpkgs, ...}: let + system = "x86_64-linux"; + # ↑ Swap it for your system if needed + # "aarch64-linux" / "x86_64-darwin" / "aarch64-darwin" + pkgs = nixpkgs.legacyPackages.${system}; + in { + devShells.${system}.default = pkgs.mkShell { + + packages = [ + pkgs.containerd + pkgs.runc + ]; + + env = { + #NIXPKGS_ALLOW_UNFREE=1; + }; + + shellHook = '' + echo "***************************************************" + echo "*** Welcome to containerd" + echo "***************************************************" + echo "" + ctr --version + runc --version + ''; + }; + }; +} diff --git a/90_containerd/justfile b/90_containerd/justfile new file mode 100644 index 0000000..490e1aa --- /dev/null +++ b/90_containerd/justfile @@ -0,0 +1,11 @@ +set dotenv-load := true + +# default lists actions +default: + @just -f example.justfile --list + +nix: + #!/usr/bin/env bash + set -eufo pipefail + nix develop --command zsh + diff --git a/TODO.md b/TODO.md index 3e50109..60f4dd1 100644 --- a/TODO.md +++ b/TODO.md @@ -1,19 +1,20 @@ # 📝 TODO -1. pipewire?? -1. cgroups and namespaces -1. Start a cgroup manually using an unpacked container. -1. Can I use a cgroup command inside a container? -1. quay.io and special labels -1. botb - break out the box image https://github.com/brompwnie/botb -1. Show how multistage builds work for testing -1. Demo using cache-from..... Use a timed layer. -1. Use different CAPS_SYS parameters. -1. Build a apt-mirror https://www.tecmint.com/setup-local-repositories-in-ubuntu/ -1. Speed of different file system layers. i.e. If we have a 1000 layers how fast is it to find a file? -1. Diffing using different tools https://docs.docker.com/engine/reference/commandline/diff/ -1. Kaniko. Caching, speed? -2. Docker container escape with docker copy. +1. Use packer to create an image. +2. cgroups and namespaces +3. Start a cgroup manually using an unpacked container. +4. Can I use a cgroup command inside a container? +5. quay.io and special labels +6. botb - break out the box image https://github.com/brompwnie/botb +7. Show how multistage builds work for testing +8. Demo using cache-from..... Use a timed layer. +9. Use different CAPS_SYS parameters. +10. Build a apt-mirror https://www.tecmint.com/setup-local-repositories-in-ubuntu/ +11. Speed of different file system layers. i.e. If we have a 1000 layers how fast is it to find a file? +12. Diffing using different tools https://docs.docker.com/engine/reference/commandline/diff/ +13. Kaniko. Caching, speed? +14. Calculating accurate image sizes - looks like container-diff can output this. +15. Docker container escape with docker copy. https://www.andreafortuna.org/2019/11/26/cve-2019-14271-a-docker-cp-container-escape-vulnerability/ 1. Add procdump to the sidecar 2. mounting volumes and permissions. @@ -35,4 +36,4 @@ https://www.andreafortuna.org/2019/11/26/cve-2019-14271-a-docker-cp-container-es 18. https://github.com/tonistiigi/xx 19. https://pythonspeed.com/articles/podman-buildkit/ 20. https://github.com/reproducible-containers/repro-sources-list.sh -21. https://github.com/containers/toolbox \ No newline at end of file +21. https://github.com/containers/toolbox