JGFOperator

reference

Overview

1. Create CRD and controller with kubebuilder link
2. Create clientset, informer, etc with code-generator link
3. Solve conflicts of two generators
   • Modify Makefile to prevent kubebuilder scan the folder generated by code-generator. kubebuilder won't work with the .go files generated by code-generator.
   • Add folder created by code-generator to Dockerfile generated by kubebuilder. Otherwise, *.go files won't compile in the docker build process.
   • PodInfo is actually a poor choice of CRD name. kubebuilder think its plural form is podinfoes while code-generator thinks it should be podinfos. This will cause a permission error.

CRD with ClientSet

Initialize Project

MODULE=fluxframework.io/jgfoperator
go mod init $MODULE
kubebuilder init --domain fluxframework.io
kubebuilder edit --multigroup=true

Generate Resources and Manifests (PodInfo Controller)

kubebuilder create api --group flux --version v1 --kind PodInfo

Create Resource[y/n]y

Create Controller[y/n]y

• Do some editing to the type definition.

make manifests

Docker Build

make docker-build

Local Deploy

make docker-push-local

make deploy-local

Pod Controller

kubebuilder create api --group core --version v1 --kind Pod

Create Resource[y/n]n

Create Controller[y/n]y

make manifests

Modify pod_controller.go

• It seems that make docker-build will automatically run the following cmd

go get k8s.io/api/core/v1@v0.22.1

Code-Generator

• source

  MODULE and go.mod bring into correspondence with API_PKG=apis, consistent with the API directory OUTPUT_PKG=generated/flux, the group specified when generating the Resource is the same GROUP_VERSION=flux:v1 Corresponds to the group version specified when the Resource is generated

Prepare Scripts

hack/tools.go

// +build tools
package tools

import _ "k8s.io/code-generator"

update-codegen.sh

#!/usr/bin/env bash

set -o errexit
set -o nounset
set -o pipefail

# corresponding to go mod init <module>
MODULE=fluxframework.io/jgfoperator
# api package
APIS_PKG=apis
# generated output package
OUTPUT_PKG=generated/flux
# group-version such as foo:v1alpha1
GROUP_VERSION=flux:v1

SCRIPT_ROOT=$(dirname "${BASH_SOURCE[0]}")/..
CODEGEN_PKG=${CODEGEN_PKG:-$(cd "${SCRIPT_ROOT}"; ls -d -1 ./vendor/k8s.io/code-generator 2>/dev/null || echo ../code-generator)}

# generate the code with:
# --output-base    because this script should also be able to run inside the vendor dir of
#                  k8s.io/kubernetes. The output-base is needed for the generators to output into the vendor dir
#                  instead of the $GOPATH directly. For normal projects this can be dropped.
bash "${CODEGEN_PKG}"/generate-groups.sh "client,lister,informer" \
  ${MODULE}/${OUTPUT_PKG} ${MODULE}/${APIS_PKG} \
  ${GROUP_VERSION} \
  --go-header-file "${SCRIPT_ROOT}"/hack/boilerplate.go.txt \
  --output-base "${SCRIPT_ROOT}"
#  --output-base "${SCRIPT_ROOT}/../../.." \

verify-codegen.sh

#!/usr/bin/env bash

set -o errexit
set -o nounset
set -o pipefail

OUTPUT_PKG=generated/flux
MODULE=fluxframework.io/jgfoperator

SCRIPT_ROOT=$(dirname "${BASH_SOURCE[0]}")/..

DIFFROOT="${SCRIPT_ROOT}/${OUTPUT_PKG}"
TMP_DIFFROOT="${SCRIPT_ROOT}/_tmp/${OUTPUT_PKG}"
_tmp="${SCRIPT_ROOT}/_tmp"

cleanup() {
  rm -rf "${_tmp}"
}
trap "cleanup" EXIT SIGINT

cleanup

mkdir -p "${TMP_DIFFROOT}"
cp -a "${DIFFROOT}"/* "${TMP_DIFFROOT}"

"${SCRIPT_ROOT}/hack/update-codegen.sh"
echo "copying generated ${SCRIPT_ROOT}/${MODULE}/${OUTPUT_PKG} to ${DIFFROOT}"
cp -r "${SCRIPT_ROOT}/${MODULE}/${OUTPUT_PKG}"/* "${DIFFROOT}"

echo "diffing ${DIFFROOT} against freshly generated codegen"
ret=0
diff -Naupr "${DIFFROOT}" "${TMP_DIFFROOT}" || ret=$?
cp -a "${TMP_DIFFROOT}"/* "${DIFFROOT}"
if [[ $ret -eq 0 ]]
then
  echo "${DIFFROOT} up to date."
else
  echo "${DIFFROOT} is out of date. Please run hack/update-codegen.sh"
  exit 1
fi

Download Code-Generator

K8S_VERSION=v0.22.1
go get k8s.io/code-generator@$K8S_VERSION
go mod vendor

chmod +x vendor/k8s.io/code-generator/generate-groups.sh

Generate Code

• add // +clientgen to types.go
• add doc.go and register.go according to reference

./hack/update-codegen.sh

• move generated folder to project root
  • don't forget to add cmd to copy generated folder in Dockerfile

Add clientset to Pod Controller

• Notes on debug

  • The Makefile generated kubebuilder will scan all the paths as defined in paths=./.... But controller-gen won't work with *.go files generated by code-generator. So in order to ignore folder that generated by code-generator, argument paths=./... is changed to paths=./apis/... and paths=./controllers/...

  • PodInfo is a terrible choice of Kind name. Since kubebuilder thinks it's plural form is podinfoes, while code-generator thinks it's plural form is podinfos. As a result, clientset generated by code-generator complains it doesn't have permission of resource podinfos, while the actual resource declared by kubebuilder is podinfoes. I decided to manually find and replace "podinfos" to "podinfoes" in the code-generator-generated code since I think this part will have less changes after the type is defined.

  • After the above step, don't forget to add permission to pod controller. Oneway to check if a new permission is added is that, after make manifests, open role.yaml to see if there are new changes.

Test

• Initialize Kind local cluster with local docker registry
• make undeploy
• ./deploy-local.sh
• deploy pi-test
• kubectl get podinfoes
• command to check the log

kubectl logs -n jgfoperator-system $(kubectl get pods -n jgfoperator-system -o jsonpath="{.items[0].metadata.name}") manager