From 42fd7039e6ea64d2e8e924221b8029a7efaf3faa Mon Sep 17 00:00:00 2001 From: Bonusree Date: Thu, 15 Jan 2026 13:07:58 +0600 Subject: [PATCH 1/7] kafka gitops Signed-off-by: Bonusree --- docs/guides/kafka/gitops/_index.md | 10 + docs/guides/kafka/gitops/gitops.md | 908 +++++++++++++++++++++++++++ docs/guides/kafka/gitops/overview.md | 50 ++ 3 files changed, 968 insertions(+) create mode 100644 docs/guides/kafka/gitops/_index.md create mode 100644 docs/guides/kafka/gitops/gitops.md create mode 100644 docs/guides/kafka/gitops/overview.md diff --git a/docs/guides/kafka/gitops/_index.md b/docs/guides/kafka/gitops/_index.md new file mode 100644 index 000000000..165e948d7 --- /dev/null +++ b/docs/guides/kafka/gitops/_index.md @@ -0,0 +1,10 @@ +--- +title: Gitops Kafka +menu: + docs_{{ .version }}: + identifier: kf-gitops-kafka + name: Gitops + parent: kf-kafka-guides + weight: 25 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/kafka/gitops/gitops.md b/docs/guides/kafka/gitops/gitops.md new file mode 100644 index 000000000..d9cd087f1 --- /dev/null +++ b/docs/guides/kafka/gitops/gitops.md @@ -0,0 +1,908 @@ +--- +title: GitOps KafkaQL +menu: + docs_{{ .version }}: + identifier: kf-using-gitops + name: GitOps KafkaQL + parent: kf-gitops-Kafka + weight: 15 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# GitOps Kafka using KubeDB GitOps Operator + +This guide will show you how to use `KubeDB` GitOps operator to create Kafka database and manage updates using GitOps workflow. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Install `KubeDB` operator in your cluster following the steps [here](/docs/setup/README.md). Pass `--set kubedb-crd-manager.installGitOpsCRDs=true` in the kubedb installation process to enable `GitOps` operator. + +- You need to install GitOps tools like `ArgoCD` or `FluxCD` and configure with your Git Repository to monitor the Git repository and synchronize the state of the Kubernetes cluster with the desired state defined in Git. + + ```bash + $ kubectl create ns monitoring + namespace/monitoring created + + $ kubectl create ns demo + namespace/demo created + ``` +> Note: YAML files used in this tutorial are stored in [docs/examples/kafka](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/kafka) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +We are going to use `ArgoCD` in this tutorial. You can install `ArgoCD` in your cluster by following the steps [here](https://argo-cd.readthedocs.io/en/stable/getting_started/). Also, you need to install `argocd` CLI in your local machine. You can install `argocd` CLI by following the steps [here](https://argo-cd.readthedocs.io/en/stable/cli_installation/). + +## Creating Apps via CLI + +### For Public Repository +```bash +argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace +``` + +### For Private Repository +#### Using HTTPS +```bash +argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace --username --password +``` + +#### Using SSH +```bash +argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace --ssh-private-key-path ~/.ssh/id_rsa +``` + +## Create Kafka Database using GitOps + +### Create a Kafka GitOps CR +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Kafka +metadata: + name: kafka-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: Standard + controller: + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Create a directory like below, +```bash +$ tree . +├── kubedb + └── Kafka.yaml +1 directories, 1 files +``` + +Now commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is created in your cluster. + +Our `gitops` operator will create an actual `Kafka` database CR in the cluster. List the resources created by `gitops` operator in the `demo` namespace. + + +```bash +$ kubectl get kafka.gitops.kubedb.com,kafka.kubedb.com -n demo +NAME AGE +kafka.gitops.kubedb.com/kafka-prod 2m56s + +NAME TYPE VERSION STATUS AGE +kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 2m56s +``` + +List the resources created by `kubedb` operator created for `kubedb.com/v1` Kafka. + +```bash +$ kubectl get petset,pod,secret,service,appbinding -n demo -l 'app.kubernetes.io/instance=kafka-prod' +NAME AGE +petset.apps.k8s.appscode.com/kafka-prod-broker 4m49s +petset.apps.k8s.appscode.com/kafka-prod-controller 4m47s + +NAME READY STATUS RESTARTS AGE +pod/kafka-prod-broker-0 1/1 Running 0 4m48s +pod/kafka-prod-broker-1 1/1 Running 0 4m42s +pod/kafka-prod-controller-0 1/1 Running 0 4m47s +pod/kafka-prod-controller-1 1/1 Running 0 4m40s + +NAME TYPE DATA AGE +secret/kafka-prod-auth kubernetes.io/basic-auth 2 4m51s + +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +service/kafka-prod-pods ClusterIP None 9092/TCP,9093/TCP,29092/TCP 4m51s + +NAME TYPE VERSION AGE +appbinding.appcatalog.appscode.com/kafka-prod kubedb.com/kafka 3.9.0 4m47s +``` + +## Update Kafka Database using GitOps + +### Scale Kafka Database Resources + +Update the `Kafka.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Kafka +metadata: + name: kafka-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + controller: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + storageType: Durable + deletionPolicy: WipeOut + ``` + +Resource Requests and Limits are updated to `700m` CPU and `2Gi` Memory. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. + +Now, `gitops` operator will detect the resource changes and create a `KafkaOpsRequest` to update the `Kafka` database. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ $ kubectl get kf,kafka,kfops -n demo +NAME TYPE VERSION STATUS AGE +kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 22h + +NAME AGE +kafka.gitops.kubedb.com/kafka-prod 22h + +NAME TYPE STATUS AGE +Kafkaopsrequest.ops.kubedb.com/kafka-prod-verticalscaling-i0kr1l VerticalScaling Successful 2s +``` + +After Ops Request becomes `Successful`, We can validate the changes by checking the one of the pod, +```bash +$ kubectl get pod -n demo Kafka-prod-broker-0 -o json | jq '.spec.containers[0].resources' +{ + "limits": { + "memory": "1536Mi" + }, + "requests": { + "cpu": "500m", + "memory": "1536Mi" + } +} +``` + +### Scale Kafka Replicas +Update the `Kafka.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Kafka +metadata: + name: kafka-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 3 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + controller: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 3 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + storageType: Durable + deletionPolicy: WipeOut +``` + +Update the `replicas` to `3`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. +Now, `gitops` operator will detect the replica changes and create a `HorizontalScaling` KafkaOpsRequest to update the `Kafka` database replicas. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,kafka,kfops -n demo +NAME TYPE VERSION STATUS AGE +kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 22h + +NAME AGE +kafka.gitops.kubedb.com/kafka-prod 22h + +NAME TYPE STATUS AGE +kafkaopsrequest.ops.kubedb.com/kafka-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m +kafkaopsrequest.ops.kubedb.com/kafka-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s +``` + +After Ops Request becomes `Successful`, We can validate the changes by checking the number of pods, +```bash +$ kubectl get pod -n demo -l 'app.kubernetes.io/instance=kafka-prod' +NAME READY STATUS RESTARTS AGE +kafka-prod-broker-0 1/1 Running 0 34m +kafka-prod-broker-1 1/1 Running 0 33m +kafka-prod-broker-2 1/1 Running 0 33m +kafka-prod-controller-0 1/1 Running 0 32m +kafka-prod-controller-1 1/1 Running 0 31m +kafka-prod-controller-2 1/1 Running 0 31m +``` + +We can also scale down the replicas by updating the `replicas` fields. + +### Expand Kafka Volume + +Update the `Kafka.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Kafka +metadata: + name: kafka-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Update the `storage.resources.requests.storage` to `2Gi`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. + +Now, `gitops` operator will detect the volume changes and create a `VolumeExpansion` KafkaOpsRequest to update the `Kafka` database volume. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,kafka,kfops -n demo +NAME TYPE VERSION STATUS AGE +kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 23m + +NAME AGE +kafka.gitops.kubedb.com/kafka-prod 23m + +NAME TYPE STATUS AGE +kafkaopsrequest.ops.kubedb.com/kafka-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m +kafkaopsrequest.ops.kubedb.com/kafka-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s +kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 19m +``` + +After Ops Request becomes `Successful`, We can validate the changes by checking the pvc size, +```bash +$ kubectl get pvc -n demo -l 'app.kubernetes.io/instance=kafka-prod' +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE +kafka-prod-data-kafka-prod-broker-0 Bound pvc-2afd4835-5686-492b-be93-c6e040e0a6c6 2Gi RWO Standard 3h39m +kafka-prod-data-kafka-prod-broker-1 Bound pvc-aaf994cc-6b04-4c37-80d5-5e966dad8487 2Gi RWO Standard 3h39m +kafka-prod-data-kafka-prod-controller-0 Bound pvc-82d2b233-203d-4df2-a0fd-ecedbc0825b7 2Gi RWO Standard 3h39m +kafka-prod-data-kafka-prod-controller-1 Bound pvc-91852c29-ab1a-48ad-9255-a0b15d5a7515 2Gi RWO Standard 3h39m + +``` + +## Reconfigure Kafka + +At first, we will create a secret containing `user.conf` file with required configuration settings. +To know more about this configuration file, check [here](/docs/guides/kafka/configuration/kafka-combined.md) +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: new-kf-combined-custom-config + namespace: demo +stringData: + server.properties: |- + log.retention.hours=125 +``` + +Now, we will add this file to `kubedb /kf-configuration.yaml`. + +```bash +$ tree . +├── kubedb +│ ├── kf-configuration.yaml +│ └── Kafka.yaml +1 directories, 2 files +``` + +Update the `Kafka.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Kafka +metadata: + name: kafka-prod + namespace: demo +spec: + configSecret: + name: new-kf-combined-custom-config + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. + +Now, `gitops` operator will detect the configuration changes and create a `Reconfigure` KafkaOpsRequest to update the `Kafka` database configuration. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,kafka,kfops -n demo +NAME TYPE VERSION STATUS AGE +kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 74m + +NAME AGE +kafka.gitops.kubedb.com/kafka-prod 74m + +NAME TYPE STATUS AGE +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfigure-ukj41o Reconfigure Successful 24m +kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 70m + +``` + + + +> We can also reconfigure the parameters creating another secret and reference the secret in the `configSecret` field. Also you can remove the `configSecret` field to use the default parameters. + +### Rotate Kafka Auth + +To do that, create a `kubernetes.io/basic-auth` type k8s secret with the new username and password. + +We will do that using gitops, create the file `kubedb /kf-auth.yaml` with the following content, + +```bash +kubectl create secret generic kf-rotate-auth -n demo \ +--type=kubernetes.io/basic-auth \ +--from-literal=username=kafka \ +--from-literal=password=kafka-secret +secret/kf-rotate-auth created + +``` + + + +Update the `Kafka.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Kafka +metadata: + name: kafka-prod + namespace: demo +spec: + authSecret: + kind: Secret + name: kf-rotate-auth + configSecret: + name: new-kf-combined-custom-config + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Change the `authSecret` field to `kf-rotate-auth`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. + +Now, `gitops` operator will detect the auth changes and create a `RotateAuth` KafkaOpsRequest to update the `Kafka` database auth. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,kafka,kfops -n demo +NAME TYPE VERSION STATUS AGE +kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 7m11s + +NAME AGE +kafka.gitops.kubedb.com/kafka-prod 7m11s + +NAME TYPE STATUS AGE +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfigure-ukj41o Reconfigure Successful 17h +kafkaopsrequest.ops.kubedb.com/kafka-prod-rotate-auth-43ris8 RotateAuth Successful 28m +kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 17h + +``` + + +### TLS configuration + +We can add, rotate or remove TLS configuration using `gitops`. + +To add tls, we are going to create an example `Issuer` that will be used to enable SSL/TLS in Kafka. Alternatively, you can follow this [cert-manager tutorial](https://cert-manager.io/docs/configuration/ca/) to create your own `Issuer`. + +- Start off by generating a ca certificates using openssl. + +```bash +$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ca.key -out ./ca.crt -subj "/CN=ca/O=kubedb" +Generating a RSA private key +................+++++ +........................+++++ +writing new private key to './ca.key' +----- +``` + +- Now we are going to create a ca-secret using the certificate files that we have just generated. + +```bash +$ kubectl create secret tls kafka-ca \ + --cert=ca.crt \ + --key=ca.key \ + --namespace=demo +secret/Kafka-ca created +``` + +Now, Let's create an `Issuer` using the `Kafka-ca` secret that we have just created. The `YAML` file looks like this: + +```yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: kf-issuer + namespace: demo +spec: + ca: + secretName: kafka-ca +``` + +Let's add that to our `kubedb /kf-issuer.yaml` file. File structure will look like this, +```bash +$ tree . +├── kubedb +│ ├── kf-configuration.yaml +│ ├── kf-issuer.yaml +│ └── Kafka.yaml +1 directories, 4 files +``` + +Update the `Kafka.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Kafka +metadata: + name: kafka-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Add `sslMode` and `tls` fields in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. + +Now, `gitops` operator will detect the tls changes and create a `ReconfigureTLS` KafkaOpsRequest to update the `Kafka` database tls. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,kafka,kfops,pods -n demo +NAME TYPE VERSION STATUS AGE +kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 41m + +NAME AGE +kafka.gitops.kubedb.com/kafka-prod 75m + +NAME TYPE STATUS AGE +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfigure-ukj41o Reconfigure Successful 5d18h +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 9m18s +kafkaopsrequest.ops.kubedb.com/kafka-prod-rotate-auth-43ris8 RotateAuth Successful 5d1h +kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d19h + +``` + + +> We can also rotate the certificates updating `.spec.tls.certificates` field. Also you can remove the `.spec.tls` field to remove tls for Kafka. + +### Update Version + +List Kafka versions using `kubectl get Kafkaversion` and choose desired version that is compatible for upgrade from current version. Check the version constraints and ops request [here](/docs/guides/kafka/update-version/versionupgrading/index.md). + +Let's choose `4.0.0` in this example. + +Update the `Kafka.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Kafka +metadata: + name: kafka-prod + namespace: demo +spec: + version: 4.0.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Update the `version` field to `17.4`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. + +Now, `gitops` operator will detect the version changes and create a `VersionUpdate` KafkaOpsRequest to update the `Kafka` database version. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,kafka,kfops -n demo +NAME TYPE VERSION STATUS AGE +kafka.kubedb.com/kafka-prod kubedb.com/v1 4.0.0 Ready 3h47m + +NAME AGE +kafka.gitops.kubedb.com/kafka-prod 3h47m + +NAME TYPE STATUS AGE +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfigure-ukj41o Reconfigure Successful 5d22h +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 4h16m +kafkaopsrequest.ops.kubedb.com/kafka-prod-rotate-auth-43ris8 RotateAuth Successful 5d6h +kafkaopsrequest.ops.kubedb.com/kafka-prod-versionupdate-wyn2dp UpdateVersion Successful 3h51m +kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d23h +``` + + +Now, we are going to verify whether the `Kafka`, `PetSet` and it's `Pod` have updated with new image. Let's check, + +```bash +$ kubectl get Kafka -n demo kafka-prod -o=jsonpath='{.spec.version}{"\n"}' +4.0.0 + +$ kubectl get petset -n demo kafka-prod-broker -o=jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/kafka:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 + +$ kubectl get pod -n demo kafka-prod-broker-0 -o=jsonpath='{.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/kafka:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 +``` + +### Enable Monitoring + +If you already don't have a Prometheus server running, deploy one following tutorial from [here](https://github.com/appscode/third-party-tools/blob/master/monitoring/prometheus/operator/README.md#deploy-prometheus-server). + +Update the `Kafka.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Kafka +metadata: + name: kafka-prod + namespace: demo +spec: + version: 4.0.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: kafka + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + monitor: + agent: prometheus.io/operator + prometheus: + exporter: + port: 9091 + serviceMonitor: + labels: + release: prometheus + interval: 10s + deletionPolicy: WipeOut +``` + +Add `monitor` field in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. + +Now, `gitops` operator will detect the monitoring changes and create a `Restart` KafkaOpsRequest to add the `Kafka` database monitoring. List the resources created by `gitops` operator in the `demo` namespace. +```bash +$ kubectl get Kafkaes.gitops.kubedb.com,Kafkaes.kubedb.com,Kafkaopsrequest -n demo +NAME TYPE VERSION STATUS AGE +kafka.kubedb.com/kafka-prod kubedb.com/v1 4.0.0 Ready 5h12m + +NAME AGE +kafka.gitops.kubedb.com/kafka-prod 5h12m + +NAME TYPE STATUS AGE +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfigure-ukj41o Reconfigure Successful 6d +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 5h42m +kafkaopsrequest.ops.kubedb.com/kafka-prod-restart-ljpqih Restart Successful 3m51s +kafkaopsrequest.ops.kubedb.com/kafka-prod-rotate-auth-43ris8 RotateAuth Successful 5d7h +kafkaopsrequest.ops.kubedb.com/kafka-prod-versionupdate-wyn2dp UpdateVersion Successful 5h16m +kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 6d + +``` + +Verify the monitoring is enabled by checking the prometheus targets. + +There are some other fields that will trigger `Restart` ops request. +- `.spec.monitor` +- `.spec.spec.archiver` +- `.spec.remoteReplica` +- `.spec.leaderElection` +- `spec.replication` +- `.spec.standbyMode` +- `.spec.streamingMode` +- `.spec.enforceGroup` +- `.spec.sslMode` etc. + + +## Next Steps + +[//]: # (- Learn Kafka [GitOps](/docs/guides/kafka/concepts/Kafka-gitops.md)) +- Learn Kafka Scaling + - [Horizontal Scaling](/docs/guides/kafka/scaling/horizontal-scaling/combined.md) + - [Vertical Scaling](/docs/guides/kafka/scaling/vertical-scaling/combined.md) +- Learn Version Update Ops Request and Constraints [here](/docs/guides/kafka/update-version/overview.md) +- Monitor your KafkaQL database with KubeDB using [built-in Prometheus](/docs/guides/kafka/monitoring/using-builtin-prometheus.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/kafka/gitops/overview.md b/docs/guides/kafka/gitops/overview.md new file mode 100644 index 000000000..c04150659 --- /dev/null +++ b/docs/guides/kafka/gitops/overview.md @@ -0,0 +1,50 @@ +--- +title: Kafka GitOps Overview +description: Kafka GitOps Overview +menu: + docs_{{ .version }}: + identifier: kf-gitops-overview + name: Overview + parent: kf-gitops-kafka + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# GitOps Overview for Kafka + +This guide will give you an overview of how KubeDB `gitops` operator works with Kafka databases using the `gitops.kubedb.com/v1alpha1` API. It will help you understand the GitOps workflow for +managing Kafka databases in Kubernetes. + +## Before You Begin + +- You should be familiar with the following `KubeDB` concepts: + - [Kafka](/docs/guides/kafka/concepts/kafka.md) + - [KafkaOpsRequest](/docs/guides/kafka/concepts/kafkaopsrequest.md) + + + +## Workflow GitOps with Kafka + +The following diagram shows how the `KubeDB` GitOps Operator used to sync with your database. Open the image in a new tab to see the enlarged version. + + +
+ + GitOps Flow + +
Fig: GitOps process of Kafka
+ +
+ +1. **Define GitOps Kafka**: Create Custom Resource (CR) of kind `Kafka` using the `gitops.kubedb.com/v1alpha1` API. +2. **Store in Git**: Push the CR to a Git repository. +3. **Automated Deployment**: Use a GitOps tool (like `ArgoCD` or `FluxCD`) to monitor the Git repository and synchronize the state of the Kubernetes cluster with the desired state defined in Git. +4. **Create Database**: The GitOps operator creates a corresponding KubeDB Kafka CR in the Kubernetes cluster to deploy the database. +5. **Handle Updates**: When you update the KafkaGitOps CR, the operator generates an Ops Request to safely apply the update(e.g. `VerticalScaling`, `HorizontalScaling`, `VolumeExapnsion`, `Reconfigure`, `RotateAuth`, `ReconfigureTLS`, `VersionUpdate`, ans `Restart`. + +This flow makes managing Kafka databases efficient, reliable, and fully integrated with GitOps practices. + +In the next doc, we are going to show a step by step guide on running Kafka using GitOps. From b056af0d7138958f7ccf91c3f64492eee5c98c5e Mon Sep 17 00:00:00 2001 From: Bonusree Date: Thu, 15 Jan 2026 13:20:14 +0600 Subject: [PATCH 2/7] url fixed Signed-off-by: Bonusree --- docs/guides/kafka/gitops/gitops.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/kafka/gitops/gitops.md b/docs/guides/kafka/gitops/gitops.md index d9cd087f1..509243a7d 100644 --- a/docs/guides/kafka/gitops/gitops.md +++ b/docs/guides/kafka/gitops/gitops.md @@ -707,7 +707,7 @@ kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansi ### Update Version -List Kafka versions using `kubectl get Kafkaversion` and choose desired version that is compatible for upgrade from current version. Check the version constraints and ops request [here](/docs/guides/kafka/update-version/versionupgrading/index.md). +List Kafka versions using `kubectl get Kafkaversion` and choose desired version that is compatible for upgrade from current version. Check the version constraints and ops request [here](/docs/guides/kafka/update-version/update-version.md). Let's choose `4.0.0` in this example. From 376a00224d98d07fb4cbe620b3605c69f3c006fb Mon Sep 17 00:00:00 2001 From: Bonusree Date: Thu, 15 Jan 2026 17:16:44 +0600 Subject: [PATCH 3/7] title Signed-off-by: Bonusree --- docs/guides/kafka/gitops/gitops.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/guides/kafka/gitops/gitops.md b/docs/guides/kafka/gitops/gitops.md index 509243a7d..bfcb9ef60 100644 --- a/docs/guides/kafka/gitops/gitops.md +++ b/docs/guides/kafka/gitops/gitops.md @@ -1,15 +1,17 @@ --- -title: GitOps KafkaQL +title: Kafka GitOps Guides +description: Kafka GitOps menu: docs_{{ .version }}: - identifier: kf-using-gitops - name: GitOps KafkaQL - parent: kf-gitops-Kafka - weight: 15 + identifier: kf-gitops-guides + name: Gitops Kafka + parent: kf-gitops-kafka + weight: 20 menu_name: docs_{{ .version }} section_menu_id: guides --- + > New to KubeDB? Please start [here](/docs/README.md). # GitOps Kafka using KubeDB GitOps Operator @@ -889,7 +891,6 @@ There are some other fields that will trigger `Restart` ops request. - `.spec.monitor` - `.spec.spec.archiver` - `.spec.remoteReplica` -- `.spec.leaderElection` - `spec.replication` - `.spec.standbyMode` - `.spec.streamingMode` From 3efed82e8237da0fbd8a350f86f98c90c095863a Mon Sep 17 00:00:00 2001 From: Bonusree Date: Thu, 15 Jan 2026 17:53:50 +0600 Subject: [PATCH 4/7] changed weight Signed-off-by: Bonusree --- docs/guides/kafka/autoscaler/_index.md | 2 +- docs/guides/kafka/cli/_index.md | 2 +- docs/guides/kafka/clustering/_index.md | 2 +- docs/guides/kafka/configuration/_index.md | 2 +- docs/guides/kafka/connectcluster/_index.md | 2 +- docs/guides/kafka/gitops/_index.md | 4 ++-- docs/guides/kafka/migration/_index.md | 2 +- docs/guides/kafka/monitoring/_index.md | 2 +- docs/guides/kafka/reconfigure-tls/_index.md | 2 +- docs/guides/kafka/reconfigure/_index.md | 2 +- docs/guides/kafka/restart/_index.md | 2 +- docs/guides/kafka/restproxy/_index.md | 2 +- docs/guides/kafka/rotate-auth/_index.md | 2 +- docs/guides/kafka/scaling/_index.md | 2 +- docs/guides/kafka/schemaregistry/_index.md | 2 +- docs/guides/kafka/tls/_index.md | 2 +- docs/guides/kafka/update-version/_index.md | 2 +- docs/guides/kafka/volume-expansion/_index.md | 2 +- 18 files changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/guides/kafka/autoscaler/_index.md b/docs/guides/kafka/autoscaler/_index.md index 22a1e3830..c6d7e89ab 100644 --- a/docs/guides/kafka/autoscaler/_index.md +++ b/docs/guides/kafka/autoscaler/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-auto-scaling name: Autoscaling parent: kf-kafka-guides - weight: 46 + weight: 90 menu_name: docs_{{ .version }} --- \ No newline at end of file diff --git a/docs/guides/kafka/cli/_index.md b/docs/guides/kafka/cli/_index.md index 155f42086..e879f4f43 100755 --- a/docs/guides/kafka/cli/_index.md +++ b/docs/guides/kafka/cli/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-cli-kafka name: CLI parent: kf-kafka-guides - weight: 100 + weight: 95 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/kafka/clustering/_index.md b/docs/guides/kafka/clustering/_index.md index 21b81a743..6cab222d4 100755 --- a/docs/guides/kafka/clustering/_index.md +++ b/docs/guides/kafka/clustering/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-clustering name: Clustering parent: kf-kafka-guides - weight: 20 + weight: 25 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/kafka/configuration/_index.md b/docs/guides/kafka/configuration/_index.md index 81167c2af..1ef769c0d 100644 --- a/docs/guides/kafka/configuration/_index.md +++ b/docs/guides/kafka/configuration/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-configuration name: Custom Configuration parent: kf-kafka-guides - weight: 30 + weight: 35 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/kafka/connectcluster/_index.md b/docs/guides/kafka/connectcluster/_index.md index 7d833caaa..44991d14d 100644 --- a/docs/guides/kafka/connectcluster/_index.md +++ b/docs/guides/kafka/connectcluster/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-connectcluster-guides name: ConnectCluster parent: kf-kafka-guides - weight: 25 + weight: 40 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/kafka/gitops/_index.md b/docs/guides/kafka/gitops/_index.md index 165e948d7..098093afa 100644 --- a/docs/guides/kafka/gitops/_index.md +++ b/docs/guides/kafka/gitops/_index.md @@ -2,9 +2,9 @@ title: Gitops Kafka menu: docs_{{ .version }}: - identifier: kf-gitops-kafka + identifier: kf-gitops name: Gitops parent: kf-kafka-guides - weight: 25 + weight: 30 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/kafka/migration/_index.md b/docs/guides/kafka/migration/_index.md index 0f8b4d2e1..84e319492 100644 --- a/docs/guides/kafka/migration/_index.md +++ b/docs/guides/kafka/migration/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-migration-kafka name: Migration parent: kf-kafka-guides - weight: 27 + weight: 60 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/kafka/monitoring/_index.md b/docs/guides/kafka/monitoring/_index.md index 05429b728..16cc76ef2 100755 --- a/docs/guides/kafka/monitoring/_index.md +++ b/docs/guides/kafka/monitoring/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-monitoring-kafka name: Monitoring parent: kf-kafka-guides - weight: 50 + weight: 60 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/kafka/reconfigure-tls/_index.md b/docs/guides/kafka/reconfigure-tls/_index.md index 5b2552a6d..e4f77b489 100644 --- a/docs/guides/kafka/reconfigure-tls/_index.md +++ b/docs/guides/kafka/reconfigure-tls/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-reconfigure-tls name: Reconfigure TLS/SSL parent: kf-kafka-guides - weight: 46 + weight: 55 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/kafka/reconfigure/_index.md b/docs/guides/kafka/reconfigure/_index.md index 86d8888b6..7a9181a21 100644 --- a/docs/guides/kafka/reconfigure/_index.md +++ b/docs/guides/kafka/reconfigure/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-reconfigure name: Reconfigure parent: kf-kafka-guides - weight: 46 + weight: 45 menu_name: docs_{{ .version }} --- \ No newline at end of file diff --git a/docs/guides/kafka/restart/_index.md b/docs/guides/kafka/restart/_index.md index d0d4240b4..1199fe4ae 100644 --- a/docs/guides/kafka/restart/_index.md +++ b/docs/guides/kafka/restart/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-restart name: Restart parent: kf-kafka-guides - weight: 46 + weight: 75 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/kafka/restproxy/_index.md b/docs/guides/kafka/restproxy/_index.md index b02df6b52..1e1e16820 100644 --- a/docs/guides/kafka/restproxy/_index.md +++ b/docs/guides/kafka/restproxy/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-rest-proxy-guides name: RestProxy parent: kf-kafka-guides - weight: 25 + weight: 60 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/kafka/rotate-auth/_index.md b/docs/guides/kafka/rotate-auth/_index.md index 840907389..ed2cf8a38 100644 --- a/docs/guides/kafka/rotate-auth/_index.md +++ b/docs/guides/kafka/rotate-auth/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-rotate-auth name: Rotate Authentication parent: kf-kafka-guides - weight: 45 + weight: 85 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/kafka/scaling/_index.md b/docs/guides/kafka/scaling/_index.md index 98b83c710..48781a47d 100644 --- a/docs/guides/kafka/scaling/_index.md +++ b/docs/guides/kafka/scaling/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-scaling name: Scaling parent: kf-kafka-guides - weight: 43 + weight: 70 menu_name: docs_{{ .version }} --- \ No newline at end of file diff --git a/docs/guides/kafka/schemaregistry/_index.md b/docs/guides/kafka/schemaregistry/_index.md index 31f884753..64a121265 100644 --- a/docs/guides/kafka/schemaregistry/_index.md +++ b/docs/guides/kafka/schemaregistry/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-schema-registry-guides name: SchemaRegistry parent: kf-kafka-guides - weight: 25 + weight: 60 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/kafka/tls/_index.md b/docs/guides/kafka/tls/_index.md index 1b5fe9d94..d92320672 100755 --- a/docs/guides/kafka/tls/_index.md +++ b/docs/guides/kafka/tls/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-tls name: TLS/SSL Encryption parent: kf-kafka-guides - weight: 45 + weight: 50 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/kafka/update-version/_index.md b/docs/guides/kafka/update-version/_index.md index 08f8af5d4..2581cfd25 100644 --- a/docs/guides/kafka/update-version/_index.md +++ b/docs/guides/kafka/update-version/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-update-version name: UpdateVersion parent: kf-kafka-guides - weight: 42 + weight: 65 menu_name: docs_{{ .version }} --- \ No newline at end of file diff --git a/docs/guides/kafka/volume-expansion/_index.md b/docs/guides/kafka/volume-expansion/_index.md index 27c8e1f8b..dddfb1563 100644 --- a/docs/guides/kafka/volume-expansion/_index.md +++ b/docs/guides/kafka/volume-expansion/_index.md @@ -5,6 +5,6 @@ menu: identifier: kf-volume-expansion name: Volume Expansion parent: kf-kafka-guides - weight: 44 + weight: 80 menu_name: docs_{{ .version }} --- \ No newline at end of file From 748ea45119d8245336ccdad7a481187af21c976f Mon Sep 17 00:00:00 2001 From: Bonusree Date: Fri, 16 Jan 2026 15:33:51 +0600 Subject: [PATCH 5/7] organize front Signed-off-by: Bonusree --- docs/guides/druid/gitops/_index.md | 11 + docs/guides/druid/gitops/gitops.md | 906 +++++++++++++++++++++++++++ docs/guides/druid/gitops/overview.md | 50 ++ docs/guides/kafka/gitops/gitops.md | 1 - docs/guides/kafka/gitops/overview.md | 6 +- 5 files changed, 970 insertions(+), 4 deletions(-) create mode 100644 docs/guides/druid/gitops/_index.md create mode 100644 docs/guides/druid/gitops/gitops.md create mode 100644 docs/guides/druid/gitops/overview.md diff --git a/docs/guides/druid/gitops/_index.md b/docs/guides/druid/gitops/_index.md new file mode 100644 index 000000000..c7207d6eb --- /dev/null +++ b/docs/guides/druid/gitops/_index.md @@ -0,0 +1,11 @@ +--- +title: Druid Gitops +menu: + docs_{{ .version }}: + identifier: guides-druid-gitops + name: Gitops + parent: guides-druid + weight: 45 +menu_name: docs_{{ .version }} +--- + diff --git a/docs/guides/druid/gitops/gitops.md b/docs/guides/druid/gitops/gitops.md new file mode 100644 index 000000000..e974f667e --- /dev/null +++ b/docs/guides/druid/gitops/gitops.md @@ -0,0 +1,906 @@ +--- +title: Druid Gitops Guide +menu: + docs_{{ .version }}: + identifier: druid-gitops-guide + name: Druid Gitops + parent: guides-druid-gitops + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- +> New to KubeDB? Please start [here](/docs/README.md). + +# GitOps Druid using KubeDB GitOps Operator + +This guide will show you how to use `KubeDB` GitOps operator to create Druid database and manage updates using GitOps workflow. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Install `KubeDB` operator in your cluster following the steps [here](/docs/setup/README.md). Pass `--set kubedb-crd-manager.installGitOpsCRDs=true` in the kubedb installation process to enable `GitOps` operator. + +- You need to install GitOps tools like `ArgoCD` or `FluxCD` and configure with your Git Repository to monitor the Git repository and synchronize the state of the Kubernetes cluster with the desired state defined in Git. + + ```bash + $ kubectl create ns monitoring + namespace/monitoring created + + $ kubectl create ns demo + namespace/demo created + ``` +> Note: YAML files used in this tutorial are stored in [docs/examples/druid](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/druid) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +We are going to use `ArgoCD` in this tutorial. You can install `ArgoCD` in your cluster by following the steps [here](https://argo-cd.readthedocs.io/en/stable/getting_started/). Also, you need to install `argocd` CLI in your local machine. You can install `argocd` CLI by following the steps [here](https://argo-cd.readthedocs.io/en/stable/cli_installation/). + +## Creating Apps via CLI + +### For Public Repository +```bash +argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace +``` + +### For Private Repository +#### Using HTTPS +```bash +argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace --username --password +``` + +#### Using SSH +```bash +argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace --ssh-private-key-path ~/.ssh/id_rsa +``` + +## Create Druid Database using GitOps + +### Create a Druid GitOps CR +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Druid +metadata: + name: Druid-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: Standard + controller: + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Create a directory like below, +```bash +$ tree . +├── kubedb + └── Druid.yaml +1 directories, 1 files +``` + +Now commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is created in your cluster. + +Our `gitops` operator will create an actual `Druid` database CR in the cluster. List the resources created by `gitops` operator in the `demo` namespace. + + +```bash +$ kubectl get Druid.gitops.kubedb.com,Druid.kubedb.com -n demo +NAME AGE +Druid.gitops.kubedb.com/Druid-prod 2m56s + +NAME TYPE VERSION STATUS AGE +Druid.kubedb.com/Druid-prod kubedb.com/v1 3.9.0 Ready 2m56s +``` + +List the resources created by `kubedb` operator created for `kubedb.com/v1` Druid. + +```bash +$ kubectl get petset,pod,secret,service,appbinding -n demo -l 'app.kubernetes.io/instance=Druid-prod' +NAME AGE +petset.apps.k8s.appscode.com/Druid-prod-broker 4m49s +petset.apps.k8s.appscode.com/Druid-prod-controller 4m47s + +NAME READY STATUS RESTARTS AGE +pod/Druid-prod-broker-0 1/1 Running 0 4m48s +pod/Druid-prod-broker-1 1/1 Running 0 4m42s +pod/Druid-prod-controller-0 1/1 Running 0 4m47s +pod/Druid-prod-controller-1 1/1 Running 0 4m40s + +NAME TYPE DATA AGE +secret/Druid-prod-auth kubernetes.io/basic-auth 2 4m51s + +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +service/Druid-prod-pods ClusterIP None 9092/TCP,9093/TCP,29092/TCP 4m51s + +NAME TYPE VERSION AGE +appbinding.appcatalog.appscode.com/Druid-prod kubedb.com/Druid 3.9.0 4m47s +``` + +## Update Druid Database using GitOps + +### Scale Druid Database Resources + +Update the `Druid.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Druid +metadata: + name: Druid-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + controller: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + storageType: Durable + deletionPolicy: WipeOut + ``` + +Resource Requests and Limits are updated to `700m` CPU and `2Gi` Memory. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. + +Now, `gitops` operator will detect the resource changes and create a `DruidOpsRequest` to update the `Druid` database. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ $ kubectl get kf,Druid,kfops -n demo +NAME TYPE VERSION STATUS AGE +Druid.kubedb.com/Druid-prod kubedb.com/v1 3.9.0 Ready 22h + +NAME AGE +Druid.gitops.kubedb.com/Druid-prod 22h + +NAME TYPE STATUS AGE +Druidopsrequest.ops.kubedb.com/Druid-prod-verticalscaling-i0kr1l VerticalScaling Successful 2s +``` + +After Ops Request becomes `Successful`, We can validate the changes by checking the one of the pod, +```bash +$ kubectl get pod -n demo Druid-prod-broker-0 -o json | jq '.spec.containers[0].resources' +{ + "limits": { + "memory": "1536Mi" + }, + "requests": { + "cpu": "500m", + "memory": "1536Mi" + } +} +``` + +### Scale Druid Replicas +Update the `Druid.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Druid +metadata: + name: Druid-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 3 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + controller: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 3 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + storageType: Durable + deletionPolicy: WipeOut +``` + +Update the `replicas` to `3`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. +Now, `gitops` operator will detect the replica changes and create a `HorizontalScaling` DruidOpsRequest to update the `Druid` database replicas. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Druid,kfops -n demo +NAME TYPE VERSION STATUS AGE +Druid.kubedb.com/Druid-prod kubedb.com/v1 3.9.0 Ready 22h + +NAME AGE +Druid.gitops.kubedb.com/Druid-prod 22h + +NAME TYPE STATUS AGE +Druidopsrequest.ops.kubedb.com/Druid-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m +Druidopsrequest.ops.kubedb.com/Druid-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s +``` + +After Ops Request becomes `Successful`, We can validate the changes by checking the number of pods, +```bash +$ kubectl get pod -n demo -l 'app.kubernetes.io/instance=Druid-prod' +NAME READY STATUS RESTARTS AGE +Druid-prod-broker-0 1/1 Running 0 34m +Druid-prod-broker-1 1/1 Running 0 33m +Druid-prod-broker-2 1/1 Running 0 33m +Druid-prod-controller-0 1/1 Running 0 32m +Druid-prod-controller-1 1/1 Running 0 31m +Druid-prod-controller-2 1/1 Running 0 31m +``` + +We can also scale down the replicas by updating the `replicas` fields. + +### Expand Druid Volume + +Update the `Druid.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Druid +metadata: + name: Druid-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Update the `storage.resources.requests.storage` to `2Gi`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. + +Now, `gitops` operator will detect the volume changes and create a `VolumeExpansion` DruidOpsRequest to update the `Druid` database volume. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Druid,kfops -n demo +NAME TYPE VERSION STATUS AGE +Druid.kubedb.com/Druid-prod kubedb.com/v1 3.9.0 Ready 23m + +NAME AGE +Druid.gitops.kubedb.com/Druid-prod 23m + +NAME TYPE STATUS AGE +Druidopsrequest.ops.kubedb.com/Druid-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m +Druidopsrequest.ops.kubedb.com/Druid-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s +Druidopsrequest.ops.kubedb.com/Druid-prod-volumeexpansion-41xthr VolumeExpansion Successful 19m +``` + +After Ops Request becomes `Successful`, We can validate the changes by checking the pvc size, +```bash +$ kubectl get pvc -n demo -l 'app.kubernetes.io/instance=Druid-prod' +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE +Druid-prod-data-Druid-prod-broker-0 Bound pvc-2afd4835-5686-492b-be93-c6e040e0a6c6 2Gi RWO Standard 3h39m +Druid-prod-data-Druid-prod-broker-1 Bound pvc-aaf994cc-6b04-4c37-80d5-5e966dad8487 2Gi RWO Standard 3h39m +Druid-prod-data-Druid-prod-controller-0 Bound pvc-82d2b233-203d-4df2-a0fd-ecedbc0825b7 2Gi RWO Standard 3h39m +Druid-prod-data-Druid-prod-controller-1 Bound pvc-91852c29-ab1a-48ad-9255-a0b15d5a7515 2Gi RWO Standard 3h39m + +``` + +## Reconfigure Druid + +At first, we will create a secret containing `user.conf` file with required configuration settings. +To know more about this configuration file, check [here](/docs/guides/Druid/configuration/Druid-combined.md) +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: new-kf-combined-custom-config + namespace: demo +stringData: + server.properties: |- + log.retention.hours=125 +``` + +Now, we will add this file to `kubedb /kf-configuration.yaml`. + +```bash +$ tree . +├── kubedb +│ ├── kf-configuration.yaml +│ └── Druid.yaml +1 directories, 2 files +``` + +Update the `Druid.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Druid +metadata: + name: Druid-prod + namespace: demo +spec: + configSecret: + name: new-kf-combined-custom-config + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. + +Now, `gitops` operator will detect the configuration changes and create a `Reconfigure` DruidOpsRequest to update the `Druid` database configuration. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Druid,kfops -n demo +NAME TYPE VERSION STATUS AGE +Druid.kubedb.com/Druid-prod kubedb.com/v1 3.9.0 Ready 74m + +NAME AGE +Druid.gitops.kubedb.com/Druid-prod 74m + +NAME TYPE STATUS AGE +Druidopsrequest.ops.kubedb.com/Druid-prod-reconfigure-ukj41o Reconfigure Successful 24m +Druidopsrequest.ops.kubedb.com/Druid-prod-volumeexpansion-41xthr VolumeExpansion Successful 70m + +``` + + + +> We can also reconfigure the parameters creating another secret and reference the secret in the `configSecret` field. Also you can remove the `configSecret` field to use the default parameters. + +### Rotate Druid Auth + +To do that, create a `kubernetes.io/basic-auth` type k8s secret with the new username and password. + +We will do that using gitops, create the file `kubedb /kf-auth.yaml` with the following content, + +```bash +kubectl create secret generic kf-rotate-auth -n demo \ +--type=kubernetes.io/basic-auth \ +--from-literal=username=Druid \ +--from-literal=password=Druid-secret +secret/kf-rotate-auth created + +``` + + + +Update the `Druid.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Druid +metadata: + name: Druid-prod + namespace: demo +spec: + authSecret: + kind: Secret + name: kf-rotate-auth + configSecret: + name: new-kf-combined-custom-config + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Change the `authSecret` field to `kf-rotate-auth`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. + +Now, `gitops` operator will detect the auth changes and create a `RotateAuth` DruidOpsRequest to update the `Druid` database auth. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Druid,kfops -n demo +NAME TYPE VERSION STATUS AGE +Druid.kubedb.com/Druid-prod kubedb.com/v1 3.9.0 Ready 7m11s + +NAME AGE +Druid.gitops.kubedb.com/Druid-prod 7m11s + +NAME TYPE STATUS AGE +Druidopsrequest.ops.kubedb.com/Druid-prod-reconfigure-ukj41o Reconfigure Successful 17h +Druidopsrequest.ops.kubedb.com/Druid-prod-rotate-auth-43ris8 RotateAuth Successful 28m +Druidopsrequest.ops.kubedb.com/Druid-prod-volumeexpansion-41xthr VolumeExpansion Successful 17h + +``` + + +### TLS configuration + +We can add, rotate or remove TLS configuration using `gitops`. + +To add tls, we are going to create an example `Issuer` that will be used to enable SSL/TLS in Druid. Alternatively, you can follow this [cert-manager tutorial](https://cert-manager.io/docs/configuration/ca/) to create your own `Issuer`. + +- Start off by generating a ca certificates using openssl. + +```bash +$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ca.key -out ./ca.crt -subj "/CN=ca/O=kubedb" +Generating a RSA private key +................+++++ +........................+++++ +writing new private key to './ca.key' +----- +``` + +- Now we are going to create a ca-secret using the certificate files that we have just generated. + +```bash +$ kubectl create secret tls Druid-ca \ + --cert=ca.crt \ + --key=ca.key \ + --namespace=demo +secret/Druid-ca created +``` + +Now, Let's create an `Issuer` using the `Druid-ca` secret that we have just created. The `YAML` file looks like this: + +```yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: kf-issuer + namespace: demo +spec: + ca: + secretName: Druid-ca +``` + +Let's add that to our `kubedb /kf-issuer.yaml` file. File structure will look like this, +```bash +$ tree . +├── kubedb +│ ├── kf-configuration.yaml +│ ├── kf-issuer.yaml +│ └── Druid.yaml +1 directories, 4 files +``` + +Update the `Druid.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Druid +metadata: + name: Druid-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Add `sslMode` and `tls` fields in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. + +Now, `gitops` operator will detect the tls changes and create a `ReconfigureTLS` DruidOpsRequest to update the `Druid` database tls. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Druid,kfops,pods -n demo +NAME TYPE VERSION STATUS AGE +Druid.kubedb.com/Druid-prod kubedb.com/v1 3.9.0 Ready 41m + +NAME AGE +Druid.gitops.kubedb.com/Druid-prod 75m + +NAME TYPE STATUS AGE +Druidopsrequest.ops.kubedb.com/Druid-prod-reconfigure-ukj41o Reconfigure Successful 5d18h +Druidopsrequest.ops.kubedb.com/Druid-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 9m18s +Druidopsrequest.ops.kubedb.com/Druid-prod-rotate-auth-43ris8 RotateAuth Successful 5d1h +Druidopsrequest.ops.kubedb.com/Druid-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d19h + +``` + + +> We can also rotate the certificates updating `.spec.tls.certificates` field. Also you can remove the `.spec.tls` field to remove tls for Druid. + +### Update Version + +List Druid versions using `kubectl get Druidversion` and choose desired version that is compatible for upgrade from current version. Check the version constraints and ops request [here](/docs/guides/Druid/update-version/update-version.md). + +Let's choose `4.0.0` in this example. + +Update the `Druid.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Druid +metadata: + name: Druid-prod + namespace: demo +spec: + version: 4.0.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Update the `version` field to `17.4`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. + +Now, `gitops` operator will detect the version changes and create a `VersionUpdate` DruidOpsRequest to update the `Druid` database version. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Druid,kfops -n demo +NAME TYPE VERSION STATUS AGE +Druid.kubedb.com/Druid-prod kubedb.com/v1 4.0.0 Ready 3h47m + +NAME AGE +Druid.gitops.kubedb.com/Druid-prod 3h47m + +NAME TYPE STATUS AGE +Druidopsrequest.ops.kubedb.com/Druid-prod-reconfigure-ukj41o Reconfigure Successful 5d22h +Druidopsrequest.ops.kubedb.com/Druid-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 4h16m +Druidopsrequest.ops.kubedb.com/Druid-prod-rotate-auth-43ris8 RotateAuth Successful 5d6h +Druidopsrequest.ops.kubedb.com/Druid-prod-versionupdate-wyn2dp UpdateVersion Successful 3h51m +Druidopsrequest.ops.kubedb.com/Druid-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d23h +``` + + +Now, we are going to verify whether the `Druid`, `PetSet` and it's `Pod` have updated with new image. Let's check, + +```bash +$ kubectl get Druid -n demo Druid-prod -o=jsonpath='{.spec.version}{"\n"}' +4.0.0 + +$ kubectl get petset -n demo Druid-prod-broker -o=jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/Druid:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 + +$ kubectl get pod -n demo Druid-prod-broker-0 -o=jsonpath='{.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/Druid:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 +``` + +### Enable Monitoring + +If you already don't have a Prometheus server running, deploy one following tutorial from [here](https://github.com/appscode/third-party-tools/blob/master/monitoring/prometheus/operator/README.md#deploy-prometheus-server). + +Update the `Druid.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Druid +metadata: + name: Druid-prod + namespace: demo +spec: + version: 4.0.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Druid + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + monitor: + agent: prometheus.io/operator + prometheus: + exporter: + port: 9091 + serviceMonitor: + labels: + release: prometheus + interval: 10s + deletionPolicy: WipeOut +``` + +Add `monitor` field in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. + +Now, `gitops` operator will detect the monitoring changes and create a `Restart` DruidOpsRequest to add the `Druid` database monitoring. List the resources created by `gitops` operator in the `demo` namespace. +```bash +$ kubectl get Druides.gitops.kubedb.com,Druides.kubedb.com,Druidopsrequest -n demo +NAME TYPE VERSION STATUS AGE +Druid.kubedb.com/Druid-prod kubedb.com/v1 4.0.0 Ready 5h12m + +NAME AGE +Druid.gitops.kubedb.com/Druid-prod 5h12m + +NAME TYPE STATUS AGE +Druidopsrequest.ops.kubedb.com/Druid-prod-reconfigure-ukj41o Reconfigure Successful 6d +Druidopsrequest.ops.kubedb.com/Druid-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 5h42m +Druidopsrequest.ops.kubedb.com/Druid-prod-restart-ljpqih Restart Successful 3m51s +Druidopsrequest.ops.kubedb.com/Druid-prod-rotate-auth-43ris8 RotateAuth Successful 5d7h +Druidopsrequest.ops.kubedb.com/Druid-prod-versionupdate-wyn2dp UpdateVersion Successful 5h16m +Druidopsrequest.ops.kubedb.com/Druid-prod-volumeexpansion-41xthr VolumeExpansion Successful 6d + +``` + +Verify the monitoring is enabled by checking the prometheus targets. + +There are some other fields that will trigger `Restart` ops request. +- `.spec.monitor` +- `.spec.spec.archiver` +- `.spec.remoteReplica` +- `spec.replication` +- `.spec.standbyMode` +- `.spec.streamingMode` +- `.spec.enforceGroup` +- `.spec.sslMode` etc. + + +## Next Steps + +[//]: # (- Learn Druid [GitOps](/docs/guides/Druid/concepts/Druid-gitops.md)) +- Learn Druid Scaling + - [Horizontal Scaling](/docs/guides/Druid/scaling/horizontal-scaling/combined.md) + - [Vertical Scaling](/docs/guides/Druid/scaling/vertical-scaling/combined.md) +- Learn Version Update Ops Request and Constraints [here](/docs/guides/Druid/update-version/overview.md) +- Monitor your DruidQL database with KubeDB using [built-in Prometheus](/docs/guides/Druid/monitoring/using-builtin-prometheus.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/gitops/overview.md b/docs/guides/druid/gitops/overview.md new file mode 100644 index 000000000..05e8c5433 --- /dev/null +++ b/docs/guides/druid/gitops/overview.md @@ -0,0 +1,50 @@ +--- +title: Druid GitOps Overview +description: Druid GitOps Overview +menu: + docs_{{ .version }}: + identifier: kf-gitops-overview + name: Overview + parent: kf-gitops-Druid + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# GitOps Overview for Druid + +This guide will give you an overview of how KubeDB `gitops` operator works with Druid databases using the `gitops.kubedb.com/v1alpha1` API. It will help you understand the GitOps workflow for +managing Druid databases in Kubernetes. + +## Before You Begin + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/druid/concepts/druid.md) + - [DruidOpsRequest](/docs/guides/druid/concepts/druidopsrequest.md) + + + +## Workflow GitOps with Druid + +The following diagram shows how the `KubeDB` GitOps Operator used to sync with your database. Open the image in a new tab to see the enlarged version. + + +
+ + GitOps Flow + +
Fig: GitOps process of Druid
+ +
+ +1. **Define GitOps Druid**: Create Custom Resource (CR) of kind `Druid` using the `gitops.kubedb.com/v1alpha1` API. +2. **Store in Git**: Push the CR to a Git repository. +3. **Automated Deployment**: Use a GitOps tool (like `ArgoCD` or `FluxCD`) to monitor the Git repository and synchronize the state of the Kubernetes cluster with the desired state defined in Git. +4. **Create Database**: The GitOps operator creates a corresponding KubeDB Druid CR in the Kubernetes cluster to deploy the database. +5. **Handle Updates**: When you update the DruidGitOps CR, the operator generates an Ops Request to safely apply the update(e.g. `VerticalScaling`, `HorizontalScaling`, `VolumeExapnsion`, `Reconfigure`, `RotateAuth`, `ReconfigureTLS`, `VersionUpdate`, ans `Restart`. + +This flow makes managing Druid databases efficient, reliable, and fully integrated with GitOps practices. + +In the next doc, we are going to show a step by step guide on running Druid using GitOps. diff --git a/docs/guides/kafka/gitops/gitops.md b/docs/guides/kafka/gitops/gitops.md index bfcb9ef60..e5b5b2d96 100644 --- a/docs/guides/kafka/gitops/gitops.md +++ b/docs/guides/kafka/gitops/gitops.md @@ -1,6 +1,5 @@ --- title: Kafka GitOps Guides -description: Kafka GitOps menu: docs_{{ .version }}: identifier: kf-gitops-guides diff --git a/docs/guides/kafka/gitops/overview.md b/docs/guides/kafka/gitops/overview.md index c04150659..cf959a479 100644 --- a/docs/guides/kafka/gitops/overview.md +++ b/docs/guides/kafka/gitops/overview.md @@ -1,11 +1,11 @@ --- -title: Kafka GitOps Overview -description: Kafka GitOps Overview +title: Kafka Gitops Overview +description: Kafka Gitops Overview menu: docs_{{ .version }}: identifier: kf-gitops-overview name: Overview - parent: kf-gitops-kafka + parent: kf-gitops weight: 10 menu_name: docs_{{ .version }} section_menu_id: guides From eda323b67493ee6935e5116a244408d465c0d7ac Mon Sep 17 00:00:00 2001 From: Bonusree Date: Wed, 21 Jan 2026 15:05:03 +0600 Subject: [PATCH 6/7] Signed-off-by: Bonusree --- docs/guides/druid/gitops/_index.md | 11 - docs/guides/elasticsearch/gitops/_index.md | 10 + .../{druid => elasticsearch}/gitops/gitops.md | 360 +++++++++--------- .../gitops/overview.md | 30 +- 4 files changed, 206 insertions(+), 205 deletions(-) delete mode 100644 docs/guides/druid/gitops/_index.md create mode 100644 docs/guides/elasticsearch/gitops/_index.md rename docs/guides/{druid => elasticsearch}/gitops/gitops.md (71%) rename docs/guides/{druid => elasticsearch}/gitops/overview.md (63%) diff --git a/docs/guides/druid/gitops/_index.md b/docs/guides/druid/gitops/_index.md deleted file mode 100644 index c7207d6eb..000000000 --- a/docs/guides/druid/gitops/_index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -title: Druid Gitops -menu: - docs_{{ .version }}: - identifier: guides-druid-gitops - name: Gitops - parent: guides-druid - weight: 45 -menu_name: docs_{{ .version }} ---- - diff --git a/docs/guides/elasticsearch/gitops/_index.md b/docs/guides/elasticsearch/gitops/_index.md new file mode 100644 index 000000000..a13bc5830 --- /dev/null +++ b/docs/guides/elasticsearch/gitops/_index.md @@ -0,0 +1,10 @@ +--- +title: Elasticsearch Gitops +menu: + docs_{{ .version }}: + identifier: es-gitops + name: Elasticsearch Gitops + parent: es-elasticsearch-guides + weight: 65 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/druid/gitops/gitops.md b/docs/guides/elasticsearch/gitops/gitops.md similarity index 71% rename from docs/guides/druid/gitops/gitops.md rename to docs/guides/elasticsearch/gitops/gitops.md index e974f667e..e5b5b2d96 100644 --- a/docs/guides/druid/gitops/gitops.md +++ b/docs/guides/elasticsearch/gitops/gitops.md @@ -1,19 +1,21 @@ --- -title: Druid Gitops Guide +title: Kafka GitOps Guides menu: - docs_{{ .version }}: - identifier: druid-gitops-guide - name: Druid Gitops - parent: guides-druid-gitops - weight: 10 + docs_{{ .version }}: + identifier: kf-gitops-guides + name: Gitops Kafka + parent: kf-gitops-kafka + weight: 20 menu_name: docs_{{ .version }} section_menu_id: guides --- + + > New to KubeDB? Please start [here](/docs/README.md). -# GitOps Druid using KubeDB GitOps Operator +# GitOps Kafka using KubeDB GitOps Operator -This guide will show you how to use `KubeDB` GitOps operator to create Druid database and manage updates using GitOps workflow. +This guide will show you how to use `KubeDB` GitOps operator to create Kafka database and manage updates using GitOps workflow. ## Before You Begin @@ -30,7 +32,7 @@ This guide will show you how to use `KubeDB` GitOps operator to create Druid dat $ kubectl create ns demo namespace/demo created ``` -> Note: YAML files used in this tutorial are stored in [docs/examples/druid](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/druid) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). +> Note: YAML files used in this tutorial are stored in [docs/examples/kafka](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/kafka) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). We are going to use `ArgoCD` in this tutorial. You can install `ArgoCD` in your cluster by following the steps [here](https://argo-cd.readthedocs.io/en/stable/getting_started/). Also, you need to install `argocd` CLI in your local machine. You can install `argocd` CLI by following the steps [here](https://argo-cd.readthedocs.io/en/stable/cli_installation/). @@ -52,14 +54,14 @@ argocd app create kubedb --repo --path kubedb --dest-server https://k argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace --ssh-private-key-path ~/.ssh/id_rsa ``` -## Create Druid Database using GitOps +## Create Kafka Database using GitOps -### Create a Druid GitOps CR +### Create a Kafka GitOps CR ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Druid +kind: Kafka metadata: - name: Druid-prod + name: kafka-prod namespace: demo spec: version: 3.9.0 @@ -90,58 +92,58 @@ Create a directory like below, ```bash $ tree . ├── kubedb - └── Druid.yaml + └── Kafka.yaml 1 directories, 1 files ``` -Now commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is created in your cluster. +Now commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is created in your cluster. -Our `gitops` operator will create an actual `Druid` database CR in the cluster. List the resources created by `gitops` operator in the `demo` namespace. +Our `gitops` operator will create an actual `Kafka` database CR in the cluster. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ kubectl get Druid.gitops.kubedb.com,Druid.kubedb.com -n demo +$ kubectl get kafka.gitops.kubedb.com,kafka.kubedb.com -n demo NAME AGE -Druid.gitops.kubedb.com/Druid-prod 2m56s +kafka.gitops.kubedb.com/kafka-prod 2m56s NAME TYPE VERSION STATUS AGE -Druid.kubedb.com/Druid-prod kubedb.com/v1 3.9.0 Ready 2m56s +kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 2m56s ``` -List the resources created by `kubedb` operator created for `kubedb.com/v1` Druid. +List the resources created by `kubedb` operator created for `kubedb.com/v1` Kafka. ```bash -$ kubectl get petset,pod,secret,service,appbinding -n demo -l 'app.kubernetes.io/instance=Druid-prod' +$ kubectl get petset,pod,secret,service,appbinding -n demo -l 'app.kubernetes.io/instance=kafka-prod' NAME AGE -petset.apps.k8s.appscode.com/Druid-prod-broker 4m49s -petset.apps.k8s.appscode.com/Druid-prod-controller 4m47s +petset.apps.k8s.appscode.com/kafka-prod-broker 4m49s +petset.apps.k8s.appscode.com/kafka-prod-controller 4m47s NAME READY STATUS RESTARTS AGE -pod/Druid-prod-broker-0 1/1 Running 0 4m48s -pod/Druid-prod-broker-1 1/1 Running 0 4m42s -pod/Druid-prod-controller-0 1/1 Running 0 4m47s -pod/Druid-prod-controller-1 1/1 Running 0 4m40s +pod/kafka-prod-broker-0 1/1 Running 0 4m48s +pod/kafka-prod-broker-1 1/1 Running 0 4m42s +pod/kafka-prod-controller-0 1/1 Running 0 4m47s +pod/kafka-prod-controller-1 1/1 Running 0 4m40s NAME TYPE DATA AGE -secret/Druid-prod-auth kubernetes.io/basic-auth 2 4m51s +secret/kafka-prod-auth kubernetes.io/basic-auth 2 4m51s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE -service/Druid-prod-pods ClusterIP None 9092/TCP,9093/TCP,29092/TCP 4m51s +service/kafka-prod-pods ClusterIP None 9092/TCP,9093/TCP,29092/TCP 4m51s NAME TYPE VERSION AGE -appbinding.appcatalog.appscode.com/Druid-prod kubedb.com/Druid 3.9.0 4m47s +appbinding.appcatalog.appscode.com/kafka-prod kubedb.com/kafka 3.9.0 4m47s ``` -## Update Druid Database using GitOps +## Update Kafka Database using GitOps -### Scale Druid Database Resources +### Scale Kafka Database Resources -Update the `Druid.yaml` with the following, +Update the `Kafka.yaml` with the following, ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Druid +kind: Kafka metadata: - name: Druid-prod + name: kafka-prod namespace: demo spec: version: 3.9.0 @@ -150,7 +152,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1540Mi @@ -169,7 +171,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1540Mi @@ -188,25 +190,25 @@ spec: deletionPolicy: WipeOut ``` -Resource Requests and Limits are updated to `700m` CPU and `2Gi` Memory. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. +Resource Requests and Limits are updated to `700m` CPU and `2Gi` Memory. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. -Now, `gitops` operator will detect the resource changes and create a `DruidOpsRequest` to update the `Druid` database. List the resources created by `gitops` operator in the `demo` namespace. +Now, `gitops` operator will detect the resource changes and create a `KafkaOpsRequest` to update the `Kafka` database. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ $ kubectl get kf,Druid,kfops -n demo +$ $ kubectl get kf,kafka,kfops -n demo NAME TYPE VERSION STATUS AGE -Druid.kubedb.com/Druid-prod kubedb.com/v1 3.9.0 Ready 22h +kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 22h NAME AGE -Druid.gitops.kubedb.com/Druid-prod 22h +kafka.gitops.kubedb.com/kafka-prod 22h NAME TYPE STATUS AGE -Druidopsrequest.ops.kubedb.com/Druid-prod-verticalscaling-i0kr1l VerticalScaling Successful 2s +Kafkaopsrequest.ops.kubedb.com/kafka-prod-verticalscaling-i0kr1l VerticalScaling Successful 2s ``` After Ops Request becomes `Successful`, We can validate the changes by checking the one of the pod, ```bash -$ kubectl get pod -n demo Druid-prod-broker-0 -o json | jq '.spec.containers[0].resources' +$ kubectl get pod -n demo Kafka-prod-broker-0 -o json | jq '.spec.containers[0].resources' { "limits": { "memory": "1536Mi" @@ -218,13 +220,13 @@ $ kubectl get pod -n demo Druid-prod-broker-0 -o json | jq '.spec.containers[0]. } ``` -### Scale Druid Replicas -Update the `Druid.yaml` with the following, +### Scale Kafka Replicas +Update the `Kafka.yaml` with the following, ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Druid +kind: Kafka metadata: - name: Druid-prod + name: kafka-prod namespace: demo spec: version: 3.9.0 @@ -233,7 +235,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1540Mi @@ -252,7 +254,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1540Mi @@ -271,44 +273,44 @@ spec: deletionPolicy: WipeOut ``` -Update the `replicas` to `3`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. -Now, `gitops` operator will detect the replica changes and create a `HorizontalScaling` DruidOpsRequest to update the `Druid` database replicas. List the resources created by `gitops` operator in the `demo` namespace. +Update the `replicas` to `3`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. +Now, `gitops` operator will detect the replica changes and create a `HorizontalScaling` KafkaOpsRequest to update the `Kafka` database replicas. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ kubectl get kf,Druid,kfops -n demo +$ kubectl get kf,kafka,kfops -n demo NAME TYPE VERSION STATUS AGE -Druid.kubedb.com/Druid-prod kubedb.com/v1 3.9.0 Ready 22h +kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 22h NAME AGE -Druid.gitops.kubedb.com/Druid-prod 22h +kafka.gitops.kubedb.com/kafka-prod 22h NAME TYPE STATUS AGE -Druidopsrequest.ops.kubedb.com/Druid-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m -Druidopsrequest.ops.kubedb.com/Druid-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s +kafkaopsrequest.ops.kubedb.com/kafka-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m +kafkaopsrequest.ops.kubedb.com/kafka-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s ``` After Ops Request becomes `Successful`, We can validate the changes by checking the number of pods, ```bash -$ kubectl get pod -n demo -l 'app.kubernetes.io/instance=Druid-prod' +$ kubectl get pod -n demo -l 'app.kubernetes.io/instance=kafka-prod' NAME READY STATUS RESTARTS AGE -Druid-prod-broker-0 1/1 Running 0 34m -Druid-prod-broker-1 1/1 Running 0 33m -Druid-prod-broker-2 1/1 Running 0 33m -Druid-prod-controller-0 1/1 Running 0 32m -Druid-prod-controller-1 1/1 Running 0 31m -Druid-prod-controller-2 1/1 Running 0 31m +kafka-prod-broker-0 1/1 Running 0 34m +kafka-prod-broker-1 1/1 Running 0 33m +kafka-prod-broker-2 1/1 Running 0 33m +kafka-prod-controller-0 1/1 Running 0 32m +kafka-prod-controller-1 1/1 Running 0 31m +kafka-prod-controller-2 1/1 Running 0 31m ``` We can also scale down the replicas by updating the `replicas` fields. -### Expand Druid Volume +### Expand Kafka Volume -Update the `Druid.yaml` with the following, +Update the `Kafka.yaml` with the following, ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Druid +kind: Kafka metadata: - name: Druid-prod + name: kafka-prod namespace: demo spec: version: 3.9.0 @@ -317,7 +319,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1536Mi @@ -336,7 +338,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1536Mi @@ -355,39 +357,39 @@ spec: deletionPolicy: WipeOut ``` -Update the `storage.resources.requests.storage` to `2Gi`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. +Update the `storage.resources.requests.storage` to `2Gi`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. -Now, `gitops` operator will detect the volume changes and create a `VolumeExpansion` DruidOpsRequest to update the `Druid` database volume. List the resources created by `gitops` operator in the `demo` namespace. +Now, `gitops` operator will detect the volume changes and create a `VolumeExpansion` KafkaOpsRequest to update the `Kafka` database volume. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ kubectl get kf,Druid,kfops -n demo +$ kubectl get kf,kafka,kfops -n demo NAME TYPE VERSION STATUS AGE -Druid.kubedb.com/Druid-prod kubedb.com/v1 3.9.0 Ready 23m +kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 23m NAME AGE -Druid.gitops.kubedb.com/Druid-prod 23m +kafka.gitops.kubedb.com/kafka-prod 23m NAME TYPE STATUS AGE -Druidopsrequest.ops.kubedb.com/Druid-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m -Druidopsrequest.ops.kubedb.com/Druid-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s -Druidopsrequest.ops.kubedb.com/Druid-prod-volumeexpansion-41xthr VolumeExpansion Successful 19m +kafkaopsrequest.ops.kubedb.com/kafka-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m +kafkaopsrequest.ops.kubedb.com/kafka-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s +kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 19m ``` After Ops Request becomes `Successful`, We can validate the changes by checking the pvc size, ```bash -$ kubectl get pvc -n demo -l 'app.kubernetes.io/instance=Druid-prod' +$ kubectl get pvc -n demo -l 'app.kubernetes.io/instance=kafka-prod' NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE -Druid-prod-data-Druid-prod-broker-0 Bound pvc-2afd4835-5686-492b-be93-c6e040e0a6c6 2Gi RWO Standard 3h39m -Druid-prod-data-Druid-prod-broker-1 Bound pvc-aaf994cc-6b04-4c37-80d5-5e966dad8487 2Gi RWO Standard 3h39m -Druid-prod-data-Druid-prod-controller-0 Bound pvc-82d2b233-203d-4df2-a0fd-ecedbc0825b7 2Gi RWO Standard 3h39m -Druid-prod-data-Druid-prod-controller-1 Bound pvc-91852c29-ab1a-48ad-9255-a0b15d5a7515 2Gi RWO Standard 3h39m +kafka-prod-data-kafka-prod-broker-0 Bound pvc-2afd4835-5686-492b-be93-c6e040e0a6c6 2Gi RWO Standard 3h39m +kafka-prod-data-kafka-prod-broker-1 Bound pvc-aaf994cc-6b04-4c37-80d5-5e966dad8487 2Gi RWO Standard 3h39m +kafka-prod-data-kafka-prod-controller-0 Bound pvc-82d2b233-203d-4df2-a0fd-ecedbc0825b7 2Gi RWO Standard 3h39m +kafka-prod-data-kafka-prod-controller-1 Bound pvc-91852c29-ab1a-48ad-9255-a0b15d5a7515 2Gi RWO Standard 3h39m ``` -## Reconfigure Druid +## Reconfigure Kafka At first, we will create a secret containing `user.conf` file with required configuration settings. -To know more about this configuration file, check [here](/docs/guides/Druid/configuration/Druid-combined.md) +To know more about this configuration file, check [here](/docs/guides/kafka/configuration/kafka-combined.md) ```yaml apiVersion: v1 kind: Secret @@ -405,16 +407,16 @@ Now, we will add this file to `kubedb /kf-configuration.yaml`. $ tree . ├── kubedb │ ├── kf-configuration.yaml -│ └── Druid.yaml +│ └── Kafka.yaml 1 directories, 2 files ``` -Update the `Druid.yaml` with the following, +Update the `Kafka.yaml` with the following, ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Druid +kind: Kafka metadata: - name: Druid-prod + name: kafka-prod namespace: demo spec: configSecret: @@ -425,7 +427,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1536Mi @@ -444,7 +446,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1536Mi @@ -463,21 +465,21 @@ spec: deletionPolicy: WipeOut ``` -Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. +Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. -Now, `gitops` operator will detect the configuration changes and create a `Reconfigure` DruidOpsRequest to update the `Druid` database configuration. List the resources created by `gitops` operator in the `demo` namespace. +Now, `gitops` operator will detect the configuration changes and create a `Reconfigure` KafkaOpsRequest to update the `Kafka` database configuration. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ kubectl get kf,Druid,kfops -n demo +$ kubectl get kf,kafka,kfops -n demo NAME TYPE VERSION STATUS AGE -Druid.kubedb.com/Druid-prod kubedb.com/v1 3.9.0 Ready 74m +kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 74m NAME AGE -Druid.gitops.kubedb.com/Druid-prod 74m +kafka.gitops.kubedb.com/kafka-prod 74m NAME TYPE STATUS AGE -Druidopsrequest.ops.kubedb.com/Druid-prod-reconfigure-ukj41o Reconfigure Successful 24m -Druidopsrequest.ops.kubedb.com/Druid-prod-volumeexpansion-41xthr VolumeExpansion Successful 70m +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfigure-ukj41o Reconfigure Successful 24m +kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 70m ``` @@ -485,7 +487,7 @@ Druidopsrequest.ops.kubedb.com/Druid-prod-volumeexpansion-41xthr VolumeExpansi > We can also reconfigure the parameters creating another secret and reference the secret in the `configSecret` field. Also you can remove the `configSecret` field to use the default parameters. -### Rotate Druid Auth +### Rotate Kafka Auth To do that, create a `kubernetes.io/basic-auth` type k8s secret with the new username and password. @@ -494,20 +496,20 @@ We will do that using gitops, create the file `kubedb /kf-auth.yaml` with the fo ```bash kubectl create secret generic kf-rotate-auth -n demo \ --type=kubernetes.io/basic-auth \ ---from-literal=username=Druid \ ---from-literal=password=Druid-secret +--from-literal=username=kafka \ +--from-literal=password=kafka-secret secret/kf-rotate-auth created ``` -Update the `Druid.yaml` with the following, +Update the `Kafka.yaml` with the following, ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Druid +kind: Kafka metadata: - name: Druid-prod + name: kafka-prod namespace: demo spec: authSecret: @@ -521,7 +523,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1536Mi @@ -540,7 +542,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1536Mi @@ -559,22 +561,22 @@ spec: deletionPolicy: WipeOut ``` -Change the `authSecret` field to `kf-rotate-auth`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. +Change the `authSecret` field to `kf-rotate-auth`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. -Now, `gitops` operator will detect the auth changes and create a `RotateAuth` DruidOpsRequest to update the `Druid` database auth. List the resources created by `gitops` operator in the `demo` namespace. +Now, `gitops` operator will detect the auth changes and create a `RotateAuth` KafkaOpsRequest to update the `Kafka` database auth. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ kubectl get kf,Druid,kfops -n demo +$ kubectl get kf,kafka,kfops -n demo NAME TYPE VERSION STATUS AGE -Druid.kubedb.com/Druid-prod kubedb.com/v1 3.9.0 Ready 7m11s +kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 7m11s NAME AGE -Druid.gitops.kubedb.com/Druid-prod 7m11s +kafka.gitops.kubedb.com/kafka-prod 7m11s NAME TYPE STATUS AGE -Druidopsrequest.ops.kubedb.com/Druid-prod-reconfigure-ukj41o Reconfigure Successful 17h -Druidopsrequest.ops.kubedb.com/Druid-prod-rotate-auth-43ris8 RotateAuth Successful 28m -Druidopsrequest.ops.kubedb.com/Druid-prod-volumeexpansion-41xthr VolumeExpansion Successful 17h +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfigure-ukj41o Reconfigure Successful 17h +kafkaopsrequest.ops.kubedb.com/kafka-prod-rotate-auth-43ris8 RotateAuth Successful 28m +kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 17h ``` @@ -583,7 +585,7 @@ Druidopsrequest.ops.kubedb.com/Druid-prod-volumeexpansion-41xthr VolumeExpansi We can add, rotate or remove TLS configuration using `gitops`. -To add tls, we are going to create an example `Issuer` that will be used to enable SSL/TLS in Druid. Alternatively, you can follow this [cert-manager tutorial](https://cert-manager.io/docs/configuration/ca/) to create your own `Issuer`. +To add tls, we are going to create an example `Issuer` that will be used to enable SSL/TLS in Kafka. Alternatively, you can follow this [cert-manager tutorial](https://cert-manager.io/docs/configuration/ca/) to create your own `Issuer`. - Start off by generating a ca certificates using openssl. @@ -599,14 +601,14 @@ writing new private key to './ca.key' - Now we are going to create a ca-secret using the certificate files that we have just generated. ```bash -$ kubectl create secret tls Druid-ca \ +$ kubectl create secret tls kafka-ca \ --cert=ca.crt \ --key=ca.key \ --namespace=demo -secret/Druid-ca created +secret/Kafka-ca created ``` -Now, Let's create an `Issuer` using the `Druid-ca` secret that we have just created. The `YAML` file looks like this: +Now, Let's create an `Issuer` using the `Kafka-ca` secret that we have just created. The `YAML` file looks like this: ```yaml apiVersion: cert-manager.io/v1 @@ -616,7 +618,7 @@ metadata: namespace: demo spec: ca: - secretName: Druid-ca + secretName: kafka-ca ``` Let's add that to our `kubedb /kf-issuer.yaml` file. File structure will look like this, @@ -625,16 +627,16 @@ $ tree . ├── kubedb │ ├── kf-configuration.yaml │ ├── kf-issuer.yaml -│ └── Druid.yaml +│ └── Kafka.yaml 1 directories, 4 files ``` -Update the `Druid.yaml` with the following, +Update the `Kafka.yaml` with the following, ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Druid +kind: Kafka metadata: - name: Druid-prod + name: kafka-prod namespace: demo spec: version: 3.9.0 @@ -643,7 +645,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1536Mi @@ -662,7 +664,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1536Mi @@ -681,41 +683,41 @@ spec: deletionPolicy: WipeOut ``` -Add `sslMode` and `tls` fields in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. +Add `sslMode` and `tls` fields in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. -Now, `gitops` operator will detect the tls changes and create a `ReconfigureTLS` DruidOpsRequest to update the `Druid` database tls. List the resources created by `gitops` operator in the `demo` namespace. +Now, `gitops` operator will detect the tls changes and create a `ReconfigureTLS` KafkaOpsRequest to update the `Kafka` database tls. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ kubectl get kf,Druid,kfops,pods -n demo +$ kubectl get kf,kafka,kfops,pods -n demo NAME TYPE VERSION STATUS AGE -Druid.kubedb.com/Druid-prod kubedb.com/v1 3.9.0 Ready 41m +kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 41m NAME AGE -Druid.gitops.kubedb.com/Druid-prod 75m +kafka.gitops.kubedb.com/kafka-prod 75m NAME TYPE STATUS AGE -Druidopsrequest.ops.kubedb.com/Druid-prod-reconfigure-ukj41o Reconfigure Successful 5d18h -Druidopsrequest.ops.kubedb.com/Druid-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 9m18s -Druidopsrequest.ops.kubedb.com/Druid-prod-rotate-auth-43ris8 RotateAuth Successful 5d1h -Druidopsrequest.ops.kubedb.com/Druid-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d19h +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfigure-ukj41o Reconfigure Successful 5d18h +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 9m18s +kafkaopsrequest.ops.kubedb.com/kafka-prod-rotate-auth-43ris8 RotateAuth Successful 5d1h +kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d19h ``` -> We can also rotate the certificates updating `.spec.tls.certificates` field. Also you can remove the `.spec.tls` field to remove tls for Druid. +> We can also rotate the certificates updating `.spec.tls.certificates` field. Also you can remove the `.spec.tls` field to remove tls for Kafka. ### Update Version -List Druid versions using `kubectl get Druidversion` and choose desired version that is compatible for upgrade from current version. Check the version constraints and ops request [here](/docs/guides/Druid/update-version/update-version.md). +List Kafka versions using `kubectl get Kafkaversion` and choose desired version that is compatible for upgrade from current version. Check the version constraints and ops request [here](/docs/guides/kafka/update-version/update-version.md). Let's choose `4.0.0` in this example. -Update the `Druid.yaml` with the following, +Update the `Kafka.yaml` with the following, ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Druid +kind: Kafka metadata: - name: Druid-prod + name: kafka-prod namespace: demo spec: version: 4.0.0 @@ -724,7 +726,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1536Mi @@ -743,7 +745,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1536Mi @@ -762,50 +764,50 @@ spec: deletionPolicy: WipeOut ``` -Update the `version` field to `17.4`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. +Update the `version` field to `17.4`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. -Now, `gitops` operator will detect the version changes and create a `VersionUpdate` DruidOpsRequest to update the `Druid` database version. List the resources created by `gitops` operator in the `demo` namespace. +Now, `gitops` operator will detect the version changes and create a `VersionUpdate` KafkaOpsRequest to update the `Kafka` database version. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ kubectl get kf,Druid,kfops -n demo +$ kubectl get kf,kafka,kfops -n demo NAME TYPE VERSION STATUS AGE -Druid.kubedb.com/Druid-prod kubedb.com/v1 4.0.0 Ready 3h47m +kafka.kubedb.com/kafka-prod kubedb.com/v1 4.0.0 Ready 3h47m NAME AGE -Druid.gitops.kubedb.com/Druid-prod 3h47m +kafka.gitops.kubedb.com/kafka-prod 3h47m NAME TYPE STATUS AGE -Druidopsrequest.ops.kubedb.com/Druid-prod-reconfigure-ukj41o Reconfigure Successful 5d22h -Druidopsrequest.ops.kubedb.com/Druid-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 4h16m -Druidopsrequest.ops.kubedb.com/Druid-prod-rotate-auth-43ris8 RotateAuth Successful 5d6h -Druidopsrequest.ops.kubedb.com/Druid-prod-versionupdate-wyn2dp UpdateVersion Successful 3h51m -Druidopsrequest.ops.kubedb.com/Druid-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d23h +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfigure-ukj41o Reconfigure Successful 5d22h +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 4h16m +kafkaopsrequest.ops.kubedb.com/kafka-prod-rotate-auth-43ris8 RotateAuth Successful 5d6h +kafkaopsrequest.ops.kubedb.com/kafka-prod-versionupdate-wyn2dp UpdateVersion Successful 3h51m +kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d23h ``` -Now, we are going to verify whether the `Druid`, `PetSet` and it's `Pod` have updated with new image. Let's check, +Now, we are going to verify whether the `Kafka`, `PetSet` and it's `Pod` have updated with new image. Let's check, ```bash -$ kubectl get Druid -n demo Druid-prod -o=jsonpath='{.spec.version}{"\n"}' +$ kubectl get Kafka -n demo kafka-prod -o=jsonpath='{.spec.version}{"\n"}' 4.0.0 -$ kubectl get petset -n demo Druid-prod-broker -o=jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' -ghcr.io/appscode-images/Druid:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 +$ kubectl get petset -n demo kafka-prod-broker -o=jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/kafka:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 -$ kubectl get pod -n demo Druid-prod-broker-0 -o=jsonpath='{.spec.containers[0].image}{"\n"}' -ghcr.io/appscode-images/Druid:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 +$ kubectl get pod -n demo kafka-prod-broker-0 -o=jsonpath='{.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/kafka:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 ``` ### Enable Monitoring If you already don't have a Prometheus server running, deploy one following tutorial from [here](https://github.com/appscode/third-party-tools/blob/master/monitoring/prometheus/operator/README.md#deploy-prometheus-server). -Update the `Druid.yaml` with the following, +Update the `Kafka.yaml` with the following, ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Druid +kind: Kafka metadata: - name: Druid-prod + name: kafka-prod namespace: demo spec: version: 4.0.0 @@ -814,7 +816,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1536Mi @@ -833,7 +835,7 @@ spec: podTemplate: spec: containers: - - name: Druid + - name: kafka resources: limits: memory: 1536Mi @@ -861,24 +863,24 @@ spec: deletionPolicy: WipeOut ``` -Add `monitor` field in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Druid` CR is updated in your cluster. +Add `monitor` field in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. -Now, `gitops` operator will detect the monitoring changes and create a `Restart` DruidOpsRequest to add the `Druid` database monitoring. List the resources created by `gitops` operator in the `demo` namespace. +Now, `gitops` operator will detect the monitoring changes and create a `Restart` KafkaOpsRequest to add the `Kafka` database monitoring. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ kubectl get Druides.gitops.kubedb.com,Druides.kubedb.com,Druidopsrequest -n demo +$ kubectl get Kafkaes.gitops.kubedb.com,Kafkaes.kubedb.com,Kafkaopsrequest -n demo NAME TYPE VERSION STATUS AGE -Druid.kubedb.com/Druid-prod kubedb.com/v1 4.0.0 Ready 5h12m +kafka.kubedb.com/kafka-prod kubedb.com/v1 4.0.0 Ready 5h12m NAME AGE -Druid.gitops.kubedb.com/Druid-prod 5h12m +kafka.gitops.kubedb.com/kafka-prod 5h12m NAME TYPE STATUS AGE -Druidopsrequest.ops.kubedb.com/Druid-prod-reconfigure-ukj41o Reconfigure Successful 6d -Druidopsrequest.ops.kubedb.com/Druid-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 5h42m -Druidopsrequest.ops.kubedb.com/Druid-prod-restart-ljpqih Restart Successful 3m51s -Druidopsrequest.ops.kubedb.com/Druid-prod-rotate-auth-43ris8 RotateAuth Successful 5d7h -Druidopsrequest.ops.kubedb.com/Druid-prod-versionupdate-wyn2dp UpdateVersion Successful 5h16m -Druidopsrequest.ops.kubedb.com/Druid-prod-volumeexpansion-41xthr VolumeExpansion Successful 6d +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfigure-ukj41o Reconfigure Successful 6d +kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 5h42m +kafkaopsrequest.ops.kubedb.com/kafka-prod-restart-ljpqih Restart Successful 3m51s +kafkaopsrequest.ops.kubedb.com/kafka-prod-rotate-auth-43ris8 RotateAuth Successful 5d7h +kafkaopsrequest.ops.kubedb.com/kafka-prod-versionupdate-wyn2dp UpdateVersion Successful 5h16m +kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 6d ``` @@ -897,10 +899,10 @@ There are some other fields that will trigger `Restart` ops request. ## Next Steps -[//]: # (- Learn Druid [GitOps](/docs/guides/Druid/concepts/Druid-gitops.md)) -- Learn Druid Scaling - - [Horizontal Scaling](/docs/guides/Druid/scaling/horizontal-scaling/combined.md) - - [Vertical Scaling](/docs/guides/Druid/scaling/vertical-scaling/combined.md) -- Learn Version Update Ops Request and Constraints [here](/docs/guides/Druid/update-version/overview.md) -- Monitor your DruidQL database with KubeDB using [built-in Prometheus](/docs/guides/Druid/monitoring/using-builtin-prometheus.md). +[//]: # (- Learn Kafka [GitOps](/docs/guides/kafka/concepts/Kafka-gitops.md)) +- Learn Kafka Scaling + - [Horizontal Scaling](/docs/guides/kafka/scaling/horizontal-scaling/combined.md) + - [Vertical Scaling](/docs/guides/kafka/scaling/vertical-scaling/combined.md) +- Learn Version Update Ops Request and Constraints [here](/docs/guides/kafka/update-version/overview.md) +- Monitor your KafkaQL database with KubeDB using [built-in Prometheus](/docs/guides/kafka/monitoring/using-builtin-prometheus.md). - Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/gitops/overview.md b/docs/guides/elasticsearch/gitops/overview.md similarity index 63% rename from docs/guides/druid/gitops/overview.md rename to docs/guides/elasticsearch/gitops/overview.md index 05e8c5433..cf959a479 100644 --- a/docs/guides/druid/gitops/overview.md +++ b/docs/guides/elasticsearch/gitops/overview.md @@ -1,11 +1,11 @@ --- -title: Druid GitOps Overview -description: Druid GitOps Overview +title: Kafka Gitops Overview +description: Kafka Gitops Overview menu: docs_{{ .version }}: identifier: kf-gitops-overview name: Overview - parent: kf-gitops-Druid + parent: kf-gitops weight: 10 menu_name: docs_{{ .version }} section_menu_id: guides @@ -13,20 +13,20 @@ section_menu_id: guides > New to KubeDB? Please start [here](/docs/README.md). -# GitOps Overview for Druid +# GitOps Overview for Kafka -This guide will give you an overview of how KubeDB `gitops` operator works with Druid databases using the `gitops.kubedb.com/v1alpha1` API. It will help you understand the GitOps workflow for -managing Druid databases in Kubernetes. +This guide will give you an overview of how KubeDB `gitops` operator works with Kafka databases using the `gitops.kubedb.com/v1alpha1` API. It will help you understand the GitOps workflow for +managing Kafka databases in Kubernetes. ## Before You Begin - You should be familiar with the following `KubeDB` concepts: - - [Druid](/docs/guides/druid/concepts/druid.md) - - [DruidOpsRequest](/docs/guides/druid/concepts/druidopsrequest.md) + - [Kafka](/docs/guides/kafka/concepts/kafka.md) + - [KafkaOpsRequest](/docs/guides/kafka/concepts/kafkaopsrequest.md) -## Workflow GitOps with Druid +## Workflow GitOps with Kafka The following diagram shows how the `KubeDB` GitOps Operator used to sync with your database. Open the image in a new tab to see the enlarged version. @@ -35,16 +35,16 @@ The following diagram shows how the `KubeDB` GitOps Operator used to sync with y GitOps Flow -
Fig: GitOps process of Druid
+
Fig: GitOps process of Kafka
-1. **Define GitOps Druid**: Create Custom Resource (CR) of kind `Druid` using the `gitops.kubedb.com/v1alpha1` API. +1. **Define GitOps Kafka**: Create Custom Resource (CR) of kind `Kafka` using the `gitops.kubedb.com/v1alpha1` API. 2. **Store in Git**: Push the CR to a Git repository. 3. **Automated Deployment**: Use a GitOps tool (like `ArgoCD` or `FluxCD`) to monitor the Git repository and synchronize the state of the Kubernetes cluster with the desired state defined in Git. -4. **Create Database**: The GitOps operator creates a corresponding KubeDB Druid CR in the Kubernetes cluster to deploy the database. -5. **Handle Updates**: When you update the DruidGitOps CR, the operator generates an Ops Request to safely apply the update(e.g. `VerticalScaling`, `HorizontalScaling`, `VolumeExapnsion`, `Reconfigure`, `RotateAuth`, `ReconfigureTLS`, `VersionUpdate`, ans `Restart`. +4. **Create Database**: The GitOps operator creates a corresponding KubeDB Kafka CR in the Kubernetes cluster to deploy the database. +5. **Handle Updates**: When you update the KafkaGitOps CR, the operator generates an Ops Request to safely apply the update(e.g. `VerticalScaling`, `HorizontalScaling`, `VolumeExapnsion`, `Reconfigure`, `RotateAuth`, `ReconfigureTLS`, `VersionUpdate`, ans `Restart`. -This flow makes managing Druid databases efficient, reliable, and fully integrated with GitOps practices. +This flow makes managing Kafka databases efficient, reliable, and fully integrated with GitOps practices. -In the next doc, we are going to show a step by step guide on running Druid using GitOps. +In the next doc, we are going to show a step by step guide on running Kafka using GitOps. From cd9376a28561f3f03bd0ab593c80b9b01860b0d0 Mon Sep 17 00:00:00 2001 From: Bonusree Date: Tue, 27 Jan 2026 17:16:47 +0600 Subject: [PATCH 7/7] elasticsearch without tls Signed-off-by: Bonusree --- docs/guides/elasticsearch/gitops/_index.md | 2 +- docs/guides/elasticsearch/gitops/gitops.md | 957 ++++++++----------- docs/guides/elasticsearch/gitops/overview.md | 32 +- docs/guides/kafka/gitops/gitops.md | 6 +- docs/guides/mariadb/gitops/_index.md | 10 + docs/guides/mariadb/gitops/gitops.md | 910 ++++++++++++++++++ docs/guides/mariadb/gitops/overview.md | 50 + docs/guides/mongodb/gitops/_index.md | 10 + docs/guides/mongodb/gitops/gitops.md | 900 +++++++++++++++++ docs/guides/mongodb/gitops/overview.md | 50 + docs/guides/mssqlserver/gitops/_index.md | 10 + docs/guides/mssqlserver/gitops/gitops.md | 900 +++++++++++++++++ docs/guides/mssqlserver/gitops/overview.md | 50 + 13 files changed, 3317 insertions(+), 570 deletions(-) create mode 100644 docs/guides/mariadb/gitops/_index.md create mode 100644 docs/guides/mariadb/gitops/gitops.md create mode 100644 docs/guides/mariadb/gitops/overview.md create mode 100644 docs/guides/mongodb/gitops/_index.md create mode 100644 docs/guides/mongodb/gitops/gitops.md create mode 100644 docs/guides/mongodb/gitops/overview.md create mode 100644 docs/guides/mssqlserver/gitops/_index.md create mode 100644 docs/guides/mssqlserver/gitops/gitops.md create mode 100644 docs/guides/mssqlserver/gitops/overview.md diff --git a/docs/guides/elasticsearch/gitops/_index.md b/docs/guides/elasticsearch/gitops/_index.md index a13bc5830..cc82b5f6b 100644 --- a/docs/guides/elasticsearch/gitops/_index.md +++ b/docs/guides/elasticsearch/gitops/_index.md @@ -5,6 +5,6 @@ menu: identifier: es-gitops name: Elasticsearch Gitops parent: es-elasticsearch-guides - weight: 65 + weight: 70 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/elasticsearch/gitops/gitops.md b/docs/guides/elasticsearch/gitops/gitops.md index e5b5b2d96..e3a2e57d2 100644 --- a/docs/guides/elasticsearch/gitops/gitops.md +++ b/docs/guides/elasticsearch/gitops/gitops.md @@ -1,29 +1,34 @@ --- -title: Kafka GitOps Guides +title: Elasticsearch Gitops Overview +description: Elasticsearch Gitops Overview menu: docs_{{ .version }}: - identifier: kf-gitops-guides - name: Gitops Kafka - parent: kf-gitops-kafka - weight: 20 + identifier: es-gitops-overview + name: Overview + parent: es-gitops + weight: 10 menu_name: docs_{{ .version }} section_menu_id: guides --- + > New to KubeDB? Please start [here](/docs/README.md). -# GitOps Kafka using KubeDB GitOps Operator +# GitOps Elasticsearch using KubeDB GitOps Operator -This guide will show you how to use `KubeDB` GitOps operator to create Kafka database and manage updates using GitOps workflow. +This guide will show you how to use `KubeDB` GitOps operator to create Elasticsearch database and manage updates using GitOps woreslow. ## Before You Begin -- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, +you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). -- Install `KubeDB` operator in your cluster following the steps [here](/docs/setup/README.md). Pass `--set kubedb-crd-manager.installGitOpsCRDs=true` in the kubedb installation process to enable `GitOps` operator. +- Install `KubeDB` operator in your cluster following the steps [here](/docs/setup/README.md). Pass `--set kubedb-crd-manager.installGitOpsCRDs=true` in the kubedb installation +process to enable `GitOps` operator. -- You need to install GitOps tools like `ArgoCD` or `FluxCD` and configure with your Git Repository to monitor the Git repository and synchronize the state of the Kubernetes cluster with the desired state defined in Git. +- You need to install GitOps tools like `ArgoCD` or `FluxCD` and configure with your Git Repository to monitor the Git repository and synchronize the state of the Kubernetes +cluster with the desired state defined in Git. ```bash $ kubectl create ns monitoring @@ -32,7 +37,7 @@ This guide will show you how to use `KubeDB` GitOps operator to create Kafka dat $ kubectl create ns demo namespace/demo created ``` -> Note: YAML files used in this tutorial are stored in [docs/examples/kafka](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/kafka) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). +> Note: YAML files used in this tutorial are stored in [docs/examples/elasticsearch](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/elasticsearch) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). We are going to use `ArgoCD` in this tutorial. You can install `ArgoCD` in your cluster by following the steps [here](https://argo-cd.readthedocs.io/en/stable/getting_started/). Also, you need to install `argocd` CLI in your local machine. You can install `argocd` CLI by following the steps [here](https://argo-cd.readthedocs.io/en/stable/cli_installation/). @@ -54,37 +59,27 @@ argocd app create kubedb --repo --path kubedb --dest-server https://k argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace --ssh-private-key-path ~/.ssh/id_rsa ``` -## Create Kafka Database using GitOps +## Create Elasticsearch Database using GitOps -### Create a Kafka GitOps CR +### Create a Elasticsearch GitOps CR ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Kafka +kind: Elasticsearch metadata: - name: kafka-prod + name: es-gitops namespace: demo spec: - version: 3.9.0 - topology: - broker: - replicas: 2 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 1Gi - storageClassName: Standard - controller: - replicas: 2 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 1Gi - storageClassName: Standard + version: xpack-8.2.3 + enableSSL: true + replicas: 2 storageType: Durable + storage: + storageClassName: longhorn + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi deletionPolicy: WipeOut ``` @@ -92,492 +87,383 @@ Create a directory like below, ```bash $ tree . ├── kubedb - └── Kafka.yaml + └── Elasticsearch.yaml 1 directories, 1 files ``` -Now commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is created in your cluster. +Now commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is created in your cluster. -Our `gitops` operator will create an actual `Kafka` database CR in the cluster. List the resources created by `gitops` operator in the `demo` namespace. +Our `gitops` operator will create an actual `Elasticsearch` database CR in the cluster. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ kubectl get kafka.gitops.kubedb.com,kafka.kubedb.com -n demo -NAME AGE -kafka.gitops.kubedb.com/kafka-prod 2m56s +$ kubectl get Elasticsearch.gitops.kubedb.com,Elasticsearch.kubedb.com -n demo +NAME AGE +elasticsearch.gitops.kubedb.com/es-gitops 20m -NAME TYPE VERSION STATUS AGE -kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 2m56s +NAME VERSION STATUS AGE +elasticsearch.kubedb.com/es-gitops xpack-8.2.3 Ready 20m ``` -List the resources created by `kubedb` operator created for `kubedb.com/v1` Kafka. +List the resources created by `kubedb` operator created for `kubedb.com/v1` Elasticsearch. ```bash -$ kubectl get petset,pod,secret,service,appbinding -n demo -l 'app.kubernetes.io/instance=kafka-prod' -NAME AGE -petset.apps.k8s.appscode.com/kafka-prod-broker 4m49s -petset.apps.k8s.appscode.com/kafka-prod-controller 4m47s +$ kubectl get petset,pod,secret,service,appbinding -n demo -l 'app.kubernetes.io/instance=es-gitops' +NAME AGE +petset.apps.k8s.appscode.com/es-gitops 20m + +NAME READY STATUS RESTARTS AGE +pod/es-gitops-0 1/1 Running 0 20m +pod/es-gitops-1 1/1 Running 0 19m + +NAME TYPE DATA AGE +secret/es-gitops-a1d7d4 Opaque 1 13m +secret/es-gitops-apm-system-cred kubernetes.io/basic-auth 2 20m +secret/es-gitops-auth kubernetes.io/basic-auth 2 20m +secret/es-gitops-beats-system-cred kubernetes.io/basic-auth 2 20m +secret/es-gitops-ca-cert kubernetes.io/tls 2 20m +secret/es-gitops-client-cert kubernetes.io/tls 3 20m +secret/es-gitops-config Opaque 1 20m +secret/es-gitops-http-cert kubernetes.io/tls 3 20m +secret/es-gitops-kibana-system-cred kubernetes.io/basic-auth 2 20m +secret/es-gitops-logstash-system-cred kubernetes.io/basic-auth 2 20m +secret/es-gitops-remote-monitoring-user-cred kubernetes.io/basic-auth 2 20m +secret/es-gitops-transport-cert kubernetes.io/tls 3 20m + +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +service/es-gitops ClusterIP 10.43.235.79 9200/TCP 20m +service/es-gitops-master ClusterIP None 9300/TCP 20m +service/es-gitops-pods ClusterIP None 9200/TCP 20m + +NAME TYPE VERSION AGE +appbinding.appcatalog.appscode.com/es-gitops kubedb.com/elasticsearch 8.2.3 20m +``` + +## Update Elasticsearch Database using GitOps +### Scale Elasticsearch Replicas +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: es-gitops + namespace: demo +spec: + version: xpack-8.2.3 + enableSSL: true + replicas: 3 + storageType: Durable + storage: + storageClassName: longhorn + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut +``` -NAME READY STATUS RESTARTS AGE -pod/kafka-prod-broker-0 1/1 Running 0 4m48s -pod/kafka-prod-broker-1 1/1 Running 0 4m42s -pod/kafka-prod-controller-0 1/1 Running 0 4m47s -pod/kafka-prod-controller-1 1/1 Running 0 4m40s +Update the `replicas` to `3`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. +Now, `gitops` operator will detect the replica changes and create a `HorizontalScaling` ElasticsearchOpsRequest to update the `Elasticsearch` database replicas. List the +resources created by `gitops` operator in the `demo` namespace. -NAME TYPE DATA AGE -secret/kafka-prod-auth kubernetes.io/basic-auth 2 4m51s +```bash +$ kubectl get es,esops -n demo +NAME VERSION STATUS AGE +elasticsearch.kubedb.com/es-gitops xpack-8.2.3 Ready 64m -NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE -service/kafka-prod-pods ClusterIP None 9092/TCP,9093/TCP,29092/TCP 4m51s +NAME TYPE STATUS AGE +elasticsearchopsrequest.ops.kubedb.com/es-gitops-horizontalscaling-32p116 HorizontalScaling Successful 39m +``` -NAME TYPE VERSION AGE -appbinding.appcatalog.appscode.com/kafka-prod kubedb.com/kafka 3.9.0 4m47s +After Ops Request becomes `Successful`, We can validate the changes by checking the number of pods, +```bash +$ kubectl get pod -n demo -l 'app.kubernetes.io/instance=es-gitops' +NAME READY STATUS RESTARTS AGE +es-gitops-0 1/1 Running 0 36m +es-gitops-1 1/1 Running 0 16m +es-gitops-2 1/1 Running 0 15m ``` -## Update Kafka Database using GitOps -### Scale Kafka Database Resources +We can also scale down the replicas by updating the `replicas` fields. +### Scale Elasticsearch Database Resources -Update the `Kafka.yaml` with the following, +Update the `Elasticsearch.yaml` with the following, ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Kafka +kind: Elasticsearch metadata: - name: kafka-prod + name: es-gitops namespace: demo spec: - version: 3.9.0 - topology: - broker: - podTemplate: - spec: - containers: - - name: kafka - resources: - limits: - memory: 1540Mi - requests: - cpu: 500m - memory: 1536Mi - replicas: 2 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 1Gi - storageClassName: local-path - controller: - podTemplate: - spec: - containers: - - name: kafka - resources: - limits: - memory: 1540Mi - requests: - cpu: 500m - memory: 1536Mi - replicas: 2 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 1Gi - storageClassName: local-path + version: xpack-8.2.3 + enableSSL: true + replicas: 3 + podTemplate: + spec: + containers: + - name: elasticsearch + resources: + limits: + cpu: 500m + memory: 1Gi + requests: + cpu: 100m + memory: 1Gi storageType: Durable + storage: + storageClassName: longhorn + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi deletionPolicy: WipeOut ``` -Resource Requests and Limits are updated to `700m` CPU and `2Gi` Memory. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. +Resource Requests and Limits are updated to `500m` CPU and `1Gi` Memory. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. -Now, `gitops` operator will detect the resource changes and create a `KafkaOpsRequest` to update the `Kafka` database. List the resources created by `gitops` operator in the `demo` namespace. +Now, `gitops` operator will detect the resource changes and create a `ElasticsearchOpsRequest` to update the `Elasticsearch` database. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ $ kubectl get kf,kafka,kfops -n demo -NAME TYPE VERSION STATUS AGE -kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 22h +$ kubectl get es,esops -n demo +NAME VERSION STATUS AGE +elasticsearch.kubedb.com/es-gitops xpack-8.2.3 Ready 64m -NAME AGE -kafka.gitops.kubedb.com/kafka-prod 22h +NAME TYPE STATUS AGE +elasticsearchopsrequest.ops.kubedb.com/es-gitops-horizontalscaling-injx1l HorizontalScaling Successful 15m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-verticalscaling-x5mfy0 VerticalScaling Successful 39m -NAME TYPE STATUS AGE -Kafkaopsrequest.ops.kubedb.com/kafka-prod-verticalscaling-i0kr1l VerticalScaling Successful 2s ``` After Ops Request becomes `Successful`, We can validate the changes by checking the one of the pod, ```bash -$ kubectl get pod -n demo Kafka-prod-broker-0 -o json | jq '.spec.containers[0].resources' +$ kubectl get pod -n demo es-gitops-0 -o json | jq '.spec.containers[0].resources' { "limits": { - "memory": "1536Mi" + "cpu": "500m", + "memory": "1Gi" }, "requests": { - "cpu": "500m", - "memory": "1536Mi" + "cpu": "100m", + "memory": "1Gi" } } -``` -### Scale Kafka Replicas -Update the `Kafka.yaml` with the following, -```yaml -apiVersion: gitops.kubedb.com/v1alpha1 -kind: Kafka -metadata: - name: kafka-prod - namespace: demo -spec: - version: 3.9.0 - topology: - broker: - podTemplate: - spec: - containers: - - name: kafka - resources: - limits: - memory: 1540Mi - requests: - cpu: 500m - memory: 1536Mi - replicas: 3 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 1Gi - storageClassName: local-path - controller: - podTemplate: - spec: - containers: - - name: kafka - resources: - limits: - memory: 1540Mi - requests: - cpu: 500m - memory: 1536Mi - replicas: 3 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 1Gi - storageClassName: local-path - storageType: Durable - deletionPolicy: WipeOut ``` -Update the `replicas` to `3`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. -Now, `gitops` operator will detect the replica changes and create a `HorizontalScaling` KafkaOpsRequest to update the `Kafka` database replicas. List the resources created by `gitops` operator in the `demo` namespace. - -```bash -$ kubectl get kf,kafka,kfops -n demo -NAME TYPE VERSION STATUS AGE -kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 22h -NAME AGE -kafka.gitops.kubedb.com/kafka-prod 22h +### Expand Elasticsearch Volume -NAME TYPE STATUS AGE -kafkaopsrequest.ops.kubedb.com/kafka-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m -kafkaopsrequest.ops.kubedb.com/kafka-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s -``` - -After Ops Request becomes `Successful`, We can validate the changes by checking the number of pods, -```bash -$ kubectl get pod -n demo -l 'app.kubernetes.io/instance=kafka-prod' -NAME READY STATUS RESTARTS AGE -kafka-prod-broker-0 1/1 Running 0 34m -kafka-prod-broker-1 1/1 Running 0 33m -kafka-prod-broker-2 1/1 Running 0 33m -kafka-prod-controller-0 1/1 Running 0 32m -kafka-prod-controller-1 1/1 Running 0 31m -kafka-prod-controller-2 1/1 Running 0 31m -``` - -We can also scale down the replicas by updating the `replicas` fields. - -### Expand Kafka Volume - -Update the `Kafka.yaml` with the following, +Update the `Elasticsearch.yaml` with the following, ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Kafka +kind: Elasticsearch metadata: - name: kafka-prod + name: es-gitops namespace: demo spec: - version: 3.9.0 - topology: - broker: - podTemplate: - spec: - containers: - - name: kafka - resources: - limits: - memory: 1536Mi - requests: - cpu: 500m - memory: 1536Mi - replicas: 2 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 2Gi - storageClassName: Standard - controller: - podTemplate: - spec: - containers: - - name: kafka - resources: - limits: - memory: 1536Mi - requests: - cpu: 500m - memory: 1536Mi - replicas: 2 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 2Gi - storageClassName: Standard + version: xpack-8.2.3 + enableSSL: true + replicas: 3 + podTemplate: + spec: + containers: + - name: elasticsearch + resources: + limits: + cpu: 500m + memory: 1Gi + requests: + cpu: 100m + memory: 1Gi storageType: Durable + storage: + storageClassName: longhorn + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi deletionPolicy: WipeOut ``` -Update the `storage.resources.requests.storage` to `2Gi`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. +Update the `storage.resources.requests.storage` to `2Gi`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. -Now, `gitops` operator will detect the volume changes and create a `VolumeExpansion` KafkaOpsRequest to update the `Kafka` database volume. List the resources created by `gitops` operator in the `demo` namespace. +Now, `gitops` operator will detect the volume changes and create a `VolumeExpansion` ElasticsearchOpsRequest to update the `Elasticsearch` database volume. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ kubectl get kf,kafka,kfops -n demo -NAME TYPE VERSION STATUS AGE -kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 23m +$ kubectl get es,esops -n demo +NAME VERSION STATUS AGE +elasticsearch.kubedb.com/es-gitops xpack-8.2.3 Ready 3h1m -NAME AGE -kafka.gitops.kubedb.com/kafka-prod 23m +NAME TYPE STATUS AGE +elasticsearchopsrequest.ops.kubedb.com/es-gitops-horizontalscaling-32p116 HorizontalScaling Successful 157m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-verticalscaling-x5mfy0 VerticalScaling Successful 157m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-volumeexpansion-sata37 VolumeExpansion Successful 38m -NAME TYPE STATUS AGE -kafkaopsrequest.ops.kubedb.com/kafka-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m -kafkaopsrequest.ops.kubedb.com/kafka-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s -kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 19m ``` After Ops Request becomes `Successful`, We can validate the changes by checking the pvc size, ```bash -$ kubectl get pvc -n demo -l 'app.kubernetes.io/instance=kafka-prod' -NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE -kafka-prod-data-kafka-prod-broker-0 Bound pvc-2afd4835-5686-492b-be93-c6e040e0a6c6 2Gi RWO Standard 3h39m -kafka-prod-data-kafka-prod-broker-1 Bound pvc-aaf994cc-6b04-4c37-80d5-5e966dad8487 2Gi RWO Standard 3h39m -kafka-prod-data-kafka-prod-controller-0 Bound pvc-82d2b233-203d-4df2-a0fd-ecedbc0825b7 2Gi RWO Standard 3h39m -kafka-prod-data-kafka-prod-controller-1 Bound pvc-91852c29-ab1a-48ad-9255-a0b15d5a7515 2Gi RWO Standard 3h39m - +$ kubectl get pod -n demo -l 'app.kubernetes.io/instance=es-gitops' +NAME READY STATUS RESTARTS AGE +es-gitops-0 1/1 Running 0 36m +es-gitops-1 1/1 Running 0 16m +es-gitops-2 1/1 Running 0 15m ``` -## Reconfigure Kafka +## Reconfigure Elasticsearch At first, we will create a secret containing `user.conf` file with required configuration settings. -To know more about this configuration file, check [here](/docs/guides/kafka/configuration/kafka-combined.md) +To know more about this configuration file, check [here](/docs/guides/elasticsearch/configuration/combined-cluster/index.md) ```yaml apiVersion: v1 +stringData: + user.conf: | + max_connections=200 + shared_buffers=256MB kind: Secret metadata: - name: new-kf-combined-custom-config + name: es-configuration namespace: demo -stringData: - server.properties: |- - log.retention.hours=125 +type: Opaque ``` -Now, we will add this file to `kubedb /kf-configuration.yaml`. +Now, we will add this file to `kubedb /es-configuration.yaml`. ```bash $ tree . ├── kubedb -│ ├── kf-configuration.yaml -│ └── Kafka.yaml +│ ├── es-configuration.yaml +│ └── Elasticsearch.yaml 1 directories, 2 files ``` -Update the `Kafka.yaml` with the following, +Update the `Elasticsearch.yaml` with the following, ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Kafka +kind: Elasticsearch metadata: - name: kafka-prod + name: es-gitops namespace: demo spec: - configSecret: - name: new-kf-combined-custom-config - version: 3.9.0 - topology: - broker: - podTemplate: - spec: - containers: - - name: kafka - resources: - limits: - memory: 1536Mi - requests: - cpu: 500m - memory: 1536Mi - replicas: 2 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 2Gi - storageClassName: Standard - controller: - podTemplate: - spec: - containers: - - name: kafka - resources: - limits: - memory: 1536Mi - requests: - cpu: 500m - memory: 1536Mi - replicas: 2 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 2Gi - storageClassName: Standard + version: xpack-8.2.3 + enableSSL: true + replicas: 3 + configuration: + secretName: es-configuration + podTemplate: + spec: + containers: + - name: elasticsearch + resources: + limits: + cpu: 500m + memory: 1Gi + requests: + cpu: 100m + memory: 1Gi storageType: Durable + storage: + storageClassName: longhorn + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi deletionPolicy: WipeOut ``` -Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. +Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. -Now, `gitops` operator will detect the configuration changes and create a `Reconfigure` KafkaOpsRequest to update the `Kafka` database configuration. List the resources created by `gitops` operator in the `demo` namespace. +Now, `gitops` operator will detect the configuration changes and create a `Reconfigure` ElasticsearchOpsRequest to update the `Elasticsearch` database configuration. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ kubectl get kf,kafka,kfops -n demo -NAME TYPE VERSION STATUS AGE -kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 74m - -NAME AGE -kafka.gitops.kubedb.com/kafka-prod 74m - -NAME TYPE STATUS AGE -kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfigure-ukj41o Reconfigure Successful 24m -kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 70m +$ kubectl get es,esops -n demo +NAME VERSION STATUS AGE +elasticsearch.kubedb.com/es-gitops xpack-8.2.3 Ready 3h53m +NAME TYPE STATUS AGE +elasticsearchopsrequest.ops.kubedb.com/es-gitops-horizontalscaling-32p116 HorizontalScaling Successful 3h29m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-reconfigure-wj5qyx Reconfigure Successful 3m42s +elasticsearchopsrequest.ops.kubedb.com/es-gitops-verticalscaling-lvh38k VerticalScaling Successful 99m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-volumeexpansion-sata37 VolumeExpansion Successful 90m ``` -> We can also reconfigure the parameters creating another secret and reference the secret in the `configSecret` field. Also you can remove the `configSecret` field to use the default parameters. - -### Rotate Kafka Auth +### Rotate Elasticsearch Auth To do that, create a `kubernetes.io/basic-auth` type k8s secret with the new username and password. -We will do that using gitops, create the file `kubedb /kf-auth.yaml` with the following content, +We will generate a secret named `es-rotate-auth` with the following content, ```bash -kubectl create secret generic kf-rotate-auth -n demo \ ---type=kubernetes.io/basic-auth \ ---from-literal=username=kafka \ ---from-literal=password=kafka-secret -secret/kf-rotate-auth created +$ kubectl create secret generic es-rotate-auth -n demo \ + --type=kubernetes.io/basic-auth \ + --from-literal=username=elastic \ + --from-literal=password=elasticsearch-secret +secret/es-rotate-auth created ``` -Update the `Kafka.yaml` with the following, +Update the `Elasticsearch.yaml` with the following, ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Kafka +kind: Elasticsearch metadata: - name: kafka-prod + name: es-gitops namespace: demo spec: + version: xpack-8.2.3 + enableSSL: true + replicas: 3 authSecret: kind: Secret - name: kf-rotate-auth - configSecret: - name: new-kf-combined-custom-config - version: 3.9.0 - topology: - broker: - podTemplate: - spec: - containers: - - name: kafka - resources: - limits: - memory: 1536Mi - requests: - cpu: 500m - memory: 1536Mi - replicas: 2 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 2Gi - storageClassName: Standard - controller: - podTemplate: - spec: - containers: - - name: kafka - resources: - limits: - memory: 1536Mi - requests: - cpu: 500m - memory: 1536Mi - replicas: 2 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 2Gi - storageClassName: Standard + name: es-rotate-auth + configuration: + secretName: es-configuration + podTemplate: + spec: + containers: + - name: elasticsearch + resources: + limits: + cpu: 500m + memory: 1Gi + requests: + cpu: 100m + memory: 1Gi storageType: Durable + storage: + storageClassName: longhorn + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi deletionPolicy: WipeOut ``` -Change the `authSecret` field to `kf-rotate-auth`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. +Change the `authSecret` field to `es-rotate-auth`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. -Now, `gitops` operator will detect the auth changes and create a `RotateAuth` KafkaOpsRequest to update the `Kafka` database auth. List the resources created by `gitops` operator in the `demo` namespace. +Now, `gitops` operator will detect the auth changes and create a `RotateAuth` ElasticsearchOpsRequest to update the `Elasticsearch` database auth. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ kubectl get kf,kafka,kfops -n demo -NAME TYPE VERSION STATUS AGE -kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 7m11s - -NAME AGE -kafka.gitops.kubedb.com/kafka-prod 7m11s - -NAME TYPE STATUS AGE -kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfigure-ukj41o Reconfigure Successful 17h -kafkaopsrequest.ops.kubedb.com/kafka-prod-rotate-auth-43ris8 RotateAuth Successful 28m -kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 17h +$ kubectl get es,esops -n demo +NAME VERSION STATUS AGE +elasticsearch.kubedb.com/es-gitops xpack-9.1.9 Ready 18m +NAME TYPE STATUS AGE +elasticsearchopsrequest.ops.kubedb.com/es-gitops-horizontalscaling-wiezxp HorizontalScaling Successful 13m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-reconfigure-i9puam Reconfigure Successful 13m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-rotate-auth-xisot8 RotateAuth Successful 4m38s +elasticsearchopsrequest.ops.kubedb.com/es-gitops-verticalscaling-7dyz6e VerticalScaling Successful 13m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-volumeexpansion-pptef3 VolumeExpansion Successful 13m ``` @@ -585,7 +471,7 @@ kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansi We can add, rotate or remove TLS configuration using `gitops`. -To add tls, we are going to create an example `Issuer` that will be used to enable SSL/TLS in Kafka. Alternatively, you can follow this [cert-manager tutorial](https://cert-manager.io/docs/configuration/ca/) to create your own `Issuer`. +To add tls, we are going to create an example `Issuer` that will be used to enable SSL/TLS in Elasticsearch. Alternatively, you can follow this [cert-manager tutorial](https://cert-manager.io/docs/configuration/ca/) to create your own `Issuer`. - Start off by generating a ca certificates using openssl. @@ -601,42 +487,42 @@ writing new private key to './ca.key' - Now we are going to create a ca-secret using the certificate files that we have just generated. ```bash -$ kubectl create secret tls kafka-ca \ +$ kubectl create secret tls Elasticsearch-ca \ --cert=ca.crt \ --key=ca.key \ --namespace=demo -secret/Kafka-ca created +secret/Elasticsearch-ca created ``` -Now, Let's create an `Issuer` using the `Kafka-ca` secret that we have just created. The `YAML` file looks like this: +Now, Let's create an `Issuer` using the `Elasticsearch-ca` secret that we have just created. The `YAML` file looks like this: ```yaml apiVersion: cert-manager.io/v1 kind: Issuer metadata: - name: kf-issuer + name: es-issuer namespace: demo spec: ca: - secretName: kafka-ca + secretName: Elasticsearch-ca ``` -Let's add that to our `kubedb /kf-issuer.yaml` file. File structure will look like this, +Let's add that to our `kubedb /es-issuer.yaml` file. File structure will look like this, ```bash $ tree . ├── kubedb -│ ├── kf-configuration.yaml -│ ├── kf-issuer.yaml -│ └── Kafka.yaml +│ ├── es-configuration.yaml +│ ├── es-issuer.yaml +│ └── Elasticsearch.yaml 1 directories, 4 files ``` -Update the `Kafka.yaml` with the following, +Update the `Elasticsearch.yaml` with the following, ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Kafka +kind: Elasticsearch metadata: - name: kafka-prod + name: es-gitops namespace: demo spec: version: 3.9.0 @@ -645,7 +531,7 @@ spec: podTemplate: spec: containers: - - name: kafka + - name: Elasticsearch resources: limits: memory: 1536Mi @@ -664,7 +550,7 @@ spec: podTemplate: spec: containers: - - name: kafka + - name: Elasticsearch resources: limits: memory: 1536Mi @@ -683,204 +569,175 @@ spec: deletionPolicy: WipeOut ``` -Add `sslMode` and `tls` fields in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. +Add `sslMode` and `tls` fields in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. -Now, `gitops` operator will detect the tls changes and create a `ReconfigureTLS` KafkaOpsRequest to update the `Kafka` database tls. List the resources created by `gitops` operator in the `demo` namespace. +Now, `gitops` operator will detect the tls changes and create a `ReconfigureTLS` ElasticsearchOpsRequest to update the `Elasticsearch` database tls. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ kubectl get kf,kafka,kfops,pods -n demo +$ kubectl get es,Elasticsearch,esops,pods -n demo NAME TYPE VERSION STATUS AGE -kafka.kubedb.com/kafka-prod kubedb.com/v1 3.9.0 Ready 41m +Elasticsearch.kubedb.com/es-gitops kubedb.com/v1 3.9.0 Ready 41m NAME AGE -kafka.gitops.kubedb.com/kafka-prod 75m +Elasticsearch.gitops.kubedb.com/es-gitops 75m NAME TYPE STATUS AGE -kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfigure-ukj41o Reconfigure Successful 5d18h -kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 9m18s -kafkaopsrequest.ops.kubedb.com/kafka-prod-rotate-auth-43ris8 RotateAuth Successful 5d1h -kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d19h +Elasticsearchopsrequest.ops.kubedb.com/es-gitops-reconfigure-ukj41o Reconfigure Successful 5d18h +Elasticsearchopsrequest.ops.kubedb.com/es-gitops-reconfiguretls-r4mx7v ReconfigureTLS Successful 9m18s +Elasticsearchopsrequest.ops.kubedb.com/es-gitops-rotate-auth-43ris8 RotateAuth Successful 5d1h +Elasticsearchopsrequest.ops.kubedb.com/es-gitops-volumeexpansion-41xthr VolumeExpansion Successful 5d19h ``` -> We can also rotate the certificates updating `.spec.tls.certificates` field. Also you can remove the `.spec.tls` field to remove tls for Kafka. +> We can also rotate the certificates updating `.spec.tls.certificates` field. Also you can remove the `.spec.tls` field to remove tls for Elasticsearch. ### Update Version -List Kafka versions using `kubectl get Kafkaversion` and choose desired version that is compatible for upgrade from current version. Check the version constraints and ops request [here](/docs/guides/kafka/update-version/update-version.md). +List Elasticsearch versions using `kubectl get Elasticsearchversion` and choose desired version that is compatible for upgrade from current version. Check the version constraints and ops request [here](/docs/guides/elasticsearch/update-version/update-version.md). Let's choose `4.0.0` in this example. -Update the `Kafka.yaml` with the following, +Update the `Elasticsearch.yaml` with the following, ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Kafka +kind: Elasticsearch metadata: - name: kafka-prod + name: es-gitops namespace: demo spec: - version: 4.0.0 - topology: - broker: - podTemplate: - spec: - containers: - - name: kafka - resources: - limits: - memory: 1536Mi - requests: - cpu: 500m - memory: 1536Mi - replicas: 2 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 2Gi - storageClassName: Standard - controller: - podTemplate: - spec: - containers: - - name: kafka - resources: - limits: - memory: 1536Mi - requests: - cpu: 500m - memory: 1536Mi - replicas: 2 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 2Gi - storageClassName: Standard + version: xpack-9.2.3 + replicas: 3 + authSecret: + kind: Secret + name: es-rotate-auth + configuration: + secretName: es-configuration + podTemplate: + spec: + containers: + - name: elasticsearch + resources: + limits: + cpu: 500m + memory: 1Gi + requests: + cpu: 100m + memory: 1Gi storageType: Durable + storage: + storageClassName: longhorn + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi deletionPolicy: WipeOut ``` -Update the `version` field to `17.4`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. +Update the `version` field to `17.4`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. -Now, `gitops` operator will detect the version changes and create a `VersionUpdate` KafkaOpsRequest to update the `Kafka` database version. List the resources created by `gitops` operator in the `demo` namespace. +Now, `gitops` operator will detect the version changes and create a `VersionUpdate` ElasticsearchOpsRequest to update the `Elasticsearch` database version. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ kubectl get kf,kafka,kfops -n demo -NAME TYPE VERSION STATUS AGE -kafka.kubedb.com/kafka-prod kubedb.com/v1 4.0.0 Ready 3h47m +$ kubectl get es,Elasticsearch,esops -n demo +NAME VERSION STATUS AGE +elasticsearch.kubedb.com/es-gitops xpack-9.2.3 Ready 64m -NAME AGE -kafka.gitops.kubedb.com/kafka-prod 3h47m +NAME AGE +elasticsearch.gitops.kubedb.com/es-gitops 64m + +NAME TYPE STATUS AGE +elasticsearchopsrequest.ops.kubedb.com/es-gitops-horizontalscaling-os8fon HorizontalScaling Successful 58m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-reconfigure-hrbrs1 Reconfigure Successful 58m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-rotate-auth-s0tkka RotateAuth Successful 58m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-versionupdate-bn3rk4 UpdateVersion Successful 31m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-verticalscaling-vhzkxl VerticalScaling Successful 58m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-volumeexpansion-bzqd4r VolumeExpansion Successful 58m -NAME TYPE STATUS AGE -kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfigure-ukj41o Reconfigure Successful 5d22h -kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 4h16m -kafkaopsrequest.ops.kubedb.com/kafka-prod-rotate-auth-43ris8 RotateAuth Successful 5d6h -kafkaopsrequest.ops.kubedb.com/kafka-prod-versionupdate-wyn2dp UpdateVersion Successful 3h51m -kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d23h ``` -Now, we are going to verify whether the `Kafka`, `PetSet` and it's `Pod` have updated with new image. Let's check, +Now, we are going to verify whether the `Elasticsearch`, `PetSet` and it's `Pod` have updated with new image. Let's check, ```bash -$ kubectl get Kafka -n demo kafka-prod -o=jsonpath='{.spec.version}{"\n"}' -4.0.0 +$ kubectl get Elasticsearch -n demo es-gitops -o=jsonpath='{.spec.version}{"\n"}' +xpack-9.2.3 -$ kubectl get petset -n demo kafka-prod-broker -o=jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' -ghcr.io/appscode-images/kafka:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 +$ kubectl get petset -n demo es-gitops -o=jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/elastic:9.2.3@sha256:714f35c53d333fef7e673079d05ce80440caf2c20ca3dc9b3e366728527760bb -$ kubectl get pod -n demo kafka-prod-broker-0 -o=jsonpath='{.spec.containers[0].image}{"\n"}' -ghcr.io/appscode-images/kafka:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 +$ kubectl get pod -n demo es-gitops-0 -o=jsonpath='{.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/elastic:9.2.3@sha256:714f35c53d333fef7e673079d05ce80440caf2c20ca3dc9b3e366728527760bb ``` ### Enable Monitoring If you already don't have a Prometheus server running, deploy one following tutorial from [here](https://github.com/appscode/third-party-tools/blob/master/monitoring/prometheus/operator/README.md#deploy-prometheus-server). -Update the `Kafka.yaml` with the following, +Update the `Elasticsearch.yaml` with the following, ```yaml apiVersion: gitops.kubedb.com/v1alpha1 -kind: Kafka +kind: Elasticsearch metadata: - name: kafka-prod + name: es-gitops namespace: demo spec: - version: 4.0.0 - topology: - broker: - podTemplate: - spec: - containers: - - name: kafka - resources: - limits: - memory: 1536Mi - requests: - cpu: 500m - memory: 1536Mi - replicas: 2 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 2Gi - storageClassName: Standard - controller: - podTemplate: - spec: - containers: - - name: kafka - resources: - limits: - memory: 1536Mi - requests: - cpu: 500m - memory: 1536Mi - replicas: 2 - storage: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 2Gi - storageClassName: Standard + version: xpack-9.2.3 + replicas: 3 + authSecret: + kind: Secret + name: es-rotate-auth + configuration: + secretName: es-configuration + podTemplate: + spec: + containers: + - name: elasticsearch + resources: + limits: + cpu: 500m + memory: 1Gi + requests: + cpu: 100m + memory: 1Gi storageType: Durable + storage: + storageClassName: longhorn + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + deletionPolicy: WipeOut monitor: agent: prometheus.io/operator prometheus: - exporter: - port: 9091 serviceMonitor: labels: release: prometheus interval: 10s - deletionPolicy: WipeOut ``` -Add `monitor` field in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Kafka` CR is updated in your cluster. +Add `monitor` field in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. -Now, `gitops` operator will detect the monitoring changes and create a `Restart` KafkaOpsRequest to add the `Kafka` database monitoring. List the resources created by `gitops` operator in the `demo` namespace. +Now, `gitops` operator will detect the monitoring changes and create a `Restart` ElasticsearchOpsRequest to add the `Elasticsearch` database monitoring. List the resources created by `gitops` operator in the `demo` namespace. ```bash -$ kubectl get Kafkaes.gitops.kubedb.com,Kafkaes.kubedb.com,Kafkaopsrequest -n demo -NAME TYPE VERSION STATUS AGE -kafka.kubedb.com/kafka-prod kubedb.com/v1 4.0.0 Ready 5h12m +$ kubectl get Elasticsearches.gitops.kubedb.com,Elasticsearches.kubedb.com,Elasticsearchopsrequest -n demo +NAME AGE +elasticsearch.gitops.kubedb.com/es-gitops 106m -NAME AGE -kafka.gitops.kubedb.com/kafka-prod 5h12m +NAME VERSION STATUS AGE +elasticsearch.kubedb.com/es-gitops xpack-9.2.3 Ready 106m -NAME TYPE STATUS AGE -kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfigure-ukj41o Reconfigure Successful 6d -kafkaopsrequest.ops.kubedb.com/kafka-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 5h42m -kafkaopsrequest.ops.kubedb.com/kafka-prod-restart-ljpqih Restart Successful 3m51s -kafkaopsrequest.ops.kubedb.com/kafka-prod-rotate-auth-43ris8 RotateAuth Successful 5d7h -kafkaopsrequest.ops.kubedb.com/kafka-prod-versionupdate-wyn2dp UpdateVersion Successful 5h16m -kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansion Successful 6d +NAME TYPE STATUS AGE +elasticsearchopsrequest.ops.kubedb.com/es-gitops-horizontalscaling-os8fon HorizontalScaling Successful 101m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-reconfigure-hrbrs1 Reconfigure Successful 101m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-restart-7v0e29 Restart Successful 15m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-rotate-auth-s0tkka RotateAuth Successful 101m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-versionupdate-bn3rk4 UpdateVersion Successful 73m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-verticalscaling-vhzkxl VerticalScaling Successful 101m +elasticsearchopsrequest.ops.kubedb.com/es-gitops-volumeexpansion-bzqd4r VolumeExpansion Successful 101m ``` @@ -899,10 +756,10 @@ There are some other fields that will trigger `Restart` ops request. ## Next Steps -[//]: # (- Learn Kafka [GitOps](/docs/guides/kafka/concepts/Kafka-gitops.md)) -- Learn Kafka Scaling - - [Horizontal Scaling](/docs/guides/kafka/scaling/horizontal-scaling/combined.md) - - [Vertical Scaling](/docs/guides/kafka/scaling/vertical-scaling/combined.md) -- Learn Version Update Ops Request and Constraints [here](/docs/guides/kafka/update-version/overview.md) -- Monitor your KafkaQL database with KubeDB using [built-in Prometheus](/docs/guides/kafka/monitoring/using-builtin-prometheus.md). -- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). +[//]: # (- Learn Elasticsearch [GitOps](/docs/guides/elasticsearch/concepts/Elasticsearch-gitops.md)) +- Learn Elasticsearch Scaling + - [Horizontal Scaling](/docs/guides/elasticsearch/scaling/horizontal/combined.md) + - [Vertical Scaling](/docs/guides/elasticsearch/scaling/vertical/combined.md) +- Learn Version Update Ops Request and Constraints [here](/docs/guides/elasticsearch/update-version/overview.md) +- Monitor your ElasticsearchQL database with KubeDB using [built-in Prometheus](/docs/guides/elasticsearch/monitoring/using-builtin-prometheus.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md) \ No newline at end of file diff --git a/docs/guides/elasticsearch/gitops/overview.md b/docs/guides/elasticsearch/gitops/overview.md index cf959a479..bfa01c2ff 100644 --- a/docs/guides/elasticsearch/gitops/overview.md +++ b/docs/guides/elasticsearch/gitops/overview.md @@ -1,11 +1,11 @@ --- -title: Kafka Gitops Overview -description: Kafka Gitops Overview +title: Elasticsearch Gitops Overview +description: Elasticsearch Gitops Overview menu: docs_{{ .version }}: - identifier: kf-gitops-overview + identifier: es-gitops-overview name: Overview - parent: kf-gitops + parent: es-gitops weight: 10 menu_name: docs_{{ .version }} section_menu_id: guides @@ -13,20 +13,20 @@ section_menu_id: guides > New to KubeDB? Please start [here](/docs/README.md). -# GitOps Overview for Kafka +# GitOps Overview for Elasticsearch -This guide will give you an overview of how KubeDB `gitops` operator works with Kafka databases using the `gitops.kubedb.com/v1alpha1` API. It will help you understand the GitOps workflow for -managing Kafka databases in Kubernetes. +This guide will give you an overview of how KubeDB `gitops` operator works with Elasticsearch databases using the `gitops.kubedb.com/v1alpha1` API. It will help you understand the GitOps workflow for +managing Elasticsearch databases in Kubernetes. ## Before You Begin - You should be familiar with the following `KubeDB` concepts: - - [Kafka](/docs/guides/kafka/concepts/kafka.md) - - [KafkaOpsRequest](/docs/guides/kafka/concepts/kafkaopsrequest.md) + - [Elasticsearch](/docs/guides/elasticsearch/concepts/elasticsearch/index.md) + - [ElasticsearchOpsRequest](/docs/guides/elasticsearch/concepts/elasticsearch-ops-request/index.md) -## Workflow GitOps with Kafka +## Workflow GitOps with Elasticsearch The following diagram shows how the `KubeDB` GitOps Operator used to sync with your database. Open the image in a new tab to see the enlarged version. @@ -35,16 +35,16 @@ The following diagram shows how the `KubeDB` GitOps Operator used to sync with y GitOps Flow -
Fig: GitOps process of Kafka
+
Fig: GitOps process of Elasticsearch
-1. **Define GitOps Kafka**: Create Custom Resource (CR) of kind `Kafka` using the `gitops.kubedb.com/v1alpha1` API. +1. **Define GitOps Elasticsearch**: Create Custom Resource (CR) of kind `Elasticsearch` using the `gitops.kubedb.com/v1alpha1` API. 2. **Store in Git**: Push the CR to a Git repository. 3. **Automated Deployment**: Use a GitOps tool (like `ArgoCD` or `FluxCD`) to monitor the Git repository and synchronize the state of the Kubernetes cluster with the desired state defined in Git. -4. **Create Database**: The GitOps operator creates a corresponding KubeDB Kafka CR in the Kubernetes cluster to deploy the database. -5. **Handle Updates**: When you update the KafkaGitOps CR, the operator generates an Ops Request to safely apply the update(e.g. `VerticalScaling`, `HorizontalScaling`, `VolumeExapnsion`, `Reconfigure`, `RotateAuth`, `ReconfigureTLS`, `VersionUpdate`, ans `Restart`. +4. **Create Database**: The GitOps operator creates a corresponding KubeDB Elasticsearch CR in the Kubernetes cluster to deploy the database. +5. **Handle Updates**: When you update the ElasticsearchGitOps CR, the operator generates an Ops Request to safely apply the update(e.g. `VerticalScaling`, `HorizontalScaling`, `VolumeExapnsion`, `Reconfigure`, `RotateAuth`, `ReconfigureTLS`, `VersionUpdate`, ans `Restart`. -This flow makes managing Kafka databases efficient, reliable, and fully integrated with GitOps practices. +This flow makes managing Elasticsearch databases efficient, reliable, and fully integrated with GitOps practices. -In the next doc, we are going to show a step by step guide on running Kafka using GitOps. +In the next doc, we are going to show a step by step guide on running Elasticsearch using GitOps. diff --git a/docs/guides/kafka/gitops/gitops.md b/docs/guides/kafka/gitops/gitops.md index e5b5b2d96..881efb1dd 100644 --- a/docs/guides/kafka/gitops/gitops.md +++ b/docs/guides/kafka/gitops/gitops.md @@ -491,7 +491,7 @@ kafkaopsrequest.ops.kubedb.com/kafka-prod-volumeexpansion-41xthr VolumeExpansi To do that, create a `kubernetes.io/basic-auth` type k8s secret with the new username and password. -We will do that using gitops, create the file `kubedb /kf-auth.yaml` with the following content, +We will create a secret named `kf-rotate-auth ` with the following content, ```bash kubectl create secret generic kf-rotate-auth -n demo \ @@ -515,8 +515,8 @@ spec: authSecret: kind: Secret name: kf-rotate-auth - configSecret: - name: new-kf-combined-custom-config + configuration: + secretName: new-kf-combined-custom-config version: 3.9.0 topology: broker: diff --git a/docs/guides/mariadb/gitops/_index.md b/docs/guides/mariadb/gitops/_index.md new file mode 100644 index 000000000..cc82b5f6b --- /dev/null +++ b/docs/guides/mariadb/gitops/_index.md @@ -0,0 +1,10 @@ +--- +title: Elasticsearch Gitops +menu: + docs_{{ .version }}: + identifier: es-gitops + name: Elasticsearch Gitops + parent: es-elasticsearch-guides + weight: 70 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/mariadb/gitops/gitops.md b/docs/guides/mariadb/gitops/gitops.md new file mode 100644 index 000000000..ec2fdd8f5 --- /dev/null +++ b/docs/guides/mariadb/gitops/gitops.md @@ -0,0 +1,910 @@ +--- +title: MariaDB Gitops Overview +description: MariaDB Gitops Overview +menu: + docs_{{ .version }}: + identifier: es-gitops-overview + name: Overview + parent: es-gitops + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + + + +> New to KubeDB? Please start [here](/docs/README.md). + +# GitOps MariaDB using KubeDB GitOps Operator + +This guide will show you how to use `KubeDB` GitOps operator to create MariaDB database and manage updates using GitOps workflow. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Install `KubeDB` operator in your cluster following the steps [here](/docs/setup/README.md). Pass `--set kubedb-crd-manager.installGitOpsCRDs=true` in the kubedb installation process to enable `GitOps` operator. + +- You need to install GitOps tools like `ArgoCD` or `FluxCD` and configure with your Git Repository to monitor the Git repository and synchronize the state of the Kubernetes cluster with the desired state defined in Git. + + ```bash + $ kubectl create ns monitoring + namespace/monitoring created + + $ kubectl create ns demo + namespace/demo created + ``` +> Note: YAML files used in this tutorial are stored in [docs/examples/MariaDB](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/MariaDB) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +We are going to use `ArgoCD` in this tutorial. You can install `ArgoCD` in your cluster by following the steps [here](https://argo-cd.readthedocs.io/en/stable/getting_started/). Also, you need to install `argocd` CLI in your local machine. You can install `argocd` CLI by following the steps [here](https://argo-cd.readthedocs.io/en/stable/cli_installation/). + +## Creating Apps via CLI + +### For Public Repository +```bash +argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace +``` + +### For Private Repository +#### Using HTTPS +```bash +argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace --username --password +``` + +#### Using SSH +```bash +argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace --ssh-private-key-path ~/.ssh/id_rsa +``` + +## Create MariaDB Database using GitOps + +### Create a MariaDB GitOps CR +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: MariaDB +metadata: + name: mariadb-gitops + namespace: demo +spec: + version: "10.5.23" + storageType: Durable + storage: + storageClassName: longhorn + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut +``` + +Create a directory like below, +```bash +$ tree . +├── kubedb + └── MariaDB.yaml +1 directories, 1 files +``` + +Now commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `MariaDB` CR is created in your cluster. + +Our `gitops` operator will create an actual `MariaDB` database CR in the cluster. List the resources created by `gitops` operator in the `demo` namespace. + + +```bash +$ kubectl get MariaDB.gitops.kubedb.com,MariaDB.kubedb.com -n demo +NAME AGE +mariadb.gitops.kubedb.com/mariadb-gitops 2m + +NAME VERSION STATUS AGE +mariadb.kubedb.com/mariadb-gitops 10.5.23 Ready 2m + +``` + +List the resources created by `kubedb` operator created for `kubedb.com/v1` MariaDB. + +```bash +$ kubectl get petset,pod,secret,service,appbinding -n demo -l 'app.kubernetes.io/instance=mariadb-gitops' +NAME AGE +petset.apps.k8s.appscode.com/mariadb-gitops 2m45s + +NAME READY STATUS RESTARTS AGE +pod/mariadb-gitops-0 1/1 Running 0 2m45s + +NAME TYPE DATA AGE +secret/mariadb-gitops-auth kubernetes.io/basic-auth 2 2m47s + +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +service/mariadb-gitops ClusterIP 10.43.132.16 3306/TCP 2m47s +service/mariadb-gitops-pods ClusterIP None 3306/TCP 2m47s + +NAME TYPE VERSION AGE +appbinding.appcatalog.appscode.com/mariadb-gitops kubedb.com/mariadb 10.5.23 2m45s + +``` + +## Update MariaDB Database using GitOps + +### Scale MariaDB Database Resources + +```shell +$ kubectl get pod -n demo mariadb-gitops-0 -o json | jq '.spec.containers[].resources' +{ + "limits": { + "memory": "1Gi" + }, + "requests": { + "cpu": "500m", + "memory": "1Gi" + } +} + +``` +Update the `MariaDB.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: MariaDB +metadata: + name: MariaDB-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + controller: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + storageType: Durable + deletionPolicy: WipeOut + ``` + +Resource Requests and Limits are updated to `700m` CPU and `2Gi` Memory. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `MariaDB` CR is updated in your cluster. + +Now, `gitops` operator will detect the resource changes and create a `MariaDBOpsRequest` to update the `MariaDB` database. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ $ kubectl get kf,MariaDB,kfops -n demo +NAME TYPE VERSION STATUS AGE +MariaDB.kubedb.com/MariaDB-prod kubedb.com/v1 3.9.0 Ready 22h + +NAME AGE +MariaDB.gitops.kubedb.com/MariaDB-prod 22h + +NAME TYPE STATUS AGE +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-verticalscaling-i0kr1l VerticalScaling Successful 2s +``` + +After Ops Request becomes `Successful`, We can validate the changes by checking the one of the pod, +```bash +$ kubectl get pod -n demo MariaDB-prod-broker-0 -o json | jq '.spec.containers[0].resources' +{ + "limits": { + "memory": "1536Mi" + }, + "requests": { + "cpu": "500m", + "memory": "1536Mi" + } +} +``` + +### Scale MariaDB Replicas +Update the `MariaDB.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: MariaDB +metadata: + name: MariaDB-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 3 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + controller: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 3 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + storageType: Durable + deletionPolicy: WipeOut +``` + +Update the `replicas` to `3`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `MariaDB` CR is updated in your cluster. +Now, `gitops` operator will detect the replica changes and create a `HorizontalScaling` MariaDBOpsRequest to update the `MariaDB` database replicas. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,MariaDB,kfops -n demo +NAME TYPE VERSION STATUS AGE +MariaDB.kubedb.com/MariaDB-prod kubedb.com/v1 3.9.0 Ready 22h + +NAME AGE +MariaDB.gitops.kubedb.com/MariaDB-prod 22h + +NAME TYPE STATUS AGE +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s +``` + +After Ops Request becomes `Successful`, We can validate the changes by checking the number of pods, +```bash +$ kubectl get pod -n demo -l 'app.kubernetes.io/instance=MariaDB-prod' +NAME READY STATUS RESTARTS AGE +MariaDB-prod-broker-0 1/1 Running 0 34m +MariaDB-prod-broker-1 1/1 Running 0 33m +MariaDB-prod-broker-2 1/1 Running 0 33m +MariaDB-prod-controller-0 1/1 Running 0 32m +MariaDB-prod-controller-1 1/1 Running 0 31m +MariaDB-prod-controller-2 1/1 Running 0 31m +``` + +We can also scale down the replicas by updating the `replicas` fields. + +### Expand MariaDB Volume + +Update the `MariaDB.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: MariaDB +metadata: + name: MariaDB-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Update the `storage.resources.requests.storage` to `2Gi`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `MariaDB` CR is updated in your cluster. + +Now, `gitops` operator will detect the volume changes and create a `VolumeExpansion` MariaDBOpsRequest to update the `MariaDB` database volume. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,MariaDB,kfops -n demo +NAME TYPE VERSION STATUS AGE +MariaDB.kubedb.com/MariaDB-prod kubedb.com/v1 3.9.0 Ready 23m + +NAME AGE +MariaDB.gitops.kubedb.com/MariaDB-prod 23m + +NAME TYPE STATUS AGE +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-volumeexpansion-41xthr VolumeExpansion Successful 19m +``` + +After Ops Request becomes `Successful`, We can validate the changes by checking the pvc size, +```bash +$ kubectl get pvc -n demo -l 'app.kubernetes.io/instance=MariaDB-prod' +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE +MariaDB-prod-data-MariaDB-prod-broker-0 Bound pvc-2afd4835-5686-492b-be93-c6e040e0a6c6 2Gi RWO Standard 3h39m +MariaDB-prod-data-MariaDB-prod-broker-1 Bound pvc-aaf994cc-6b04-4c37-80d5-5e966dad8487 2Gi RWO Standard 3h39m +MariaDB-prod-data-MariaDB-prod-controller-0 Bound pvc-82d2b233-203d-4df2-a0fd-ecedbc0825b7 2Gi RWO Standard 3h39m +MariaDB-prod-data-MariaDB-prod-controller-1 Bound pvc-91852c29-ab1a-48ad-9255-a0b15d5a7515 2Gi RWO Standard 3h39m + +``` + +## Reconfigure MariaDB + +At first, we will create a secret containing `user.conf` file with required configuration settings. +To know more about this configuration file, check [here](/docs/guides/MariaDB/configuration/MariaDB-combined.md) +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: new-kf-combined-custom-config + namespace: demo +stringData: + server.properties: |- + log.retention.hours=125 +``` + +Now, we will add this file to `kubedb /kf-configuration.yaml`. + +```bash +$ tree . +├── kubedb +│ ├── kf-configuration.yaml +│ └── MariaDB.yaml +1 directories, 2 files +``` + +Update the `MariaDB.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: MariaDB +metadata: + name: MariaDB-prod + namespace: demo +spec: + configSecret: + name: new-kf-combined-custom-config + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `MariaDB` CR is updated in your cluster. + +Now, `gitops` operator will detect the configuration changes and create a `Reconfigure` MariaDBOpsRequest to update the `MariaDB` database configuration. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,MariaDB,kfops -n demo +NAME TYPE VERSION STATUS AGE +MariaDB.kubedb.com/MariaDB-prod kubedb.com/v1 3.9.0 Ready 74m + +NAME AGE +MariaDB.gitops.kubedb.com/MariaDB-prod 74m + +NAME TYPE STATUS AGE +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-reconfigure-ukj41o Reconfigure Successful 24m +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-volumeexpansion-41xthr VolumeExpansion Successful 70m + +``` + + + +> We can also reconfigure the parameters creating another secret and reference the secret in the `configSecret` field. Also you can remove the `configSecret` field to use the default parameters. + +### Rotate MariaDB Auth + +To do that, create a `kubernetes.io/basic-auth` type k8s secret with the new username and password. + +We will do that using gitops, create the file `kubedb /kf-auth.yaml` with the following content, + +```bash +kubectl create secret generic kf-rotate-auth -n demo \ +--type=kubernetes.io/basic-auth \ +--from-literal=username=MariaDB \ +--from-literal=password=MariaDB-secret +secret/kf-rotate-auth created + +``` + + + +Update the `MariaDB.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: MariaDB +metadata: + name: MariaDB-prod + namespace: demo +spec: + authSecret: + kind: Secret + name: kf-rotate-auth + configSecret: + name: new-kf-combined-custom-config + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Change the `authSecret` field to `kf-rotate-auth`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `MariaDB` CR is updated in your cluster. + +Now, `gitops` operator will detect the auth changes and create a `RotateAuth` MariaDBOpsRequest to update the `MariaDB` database auth. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,MariaDB,kfops -n demo +NAME TYPE VERSION STATUS AGE +MariaDB.kubedb.com/MariaDB-prod kubedb.com/v1 3.9.0 Ready 7m11s + +NAME AGE +MariaDB.gitops.kubedb.com/MariaDB-prod 7m11s + +NAME TYPE STATUS AGE +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-reconfigure-ukj41o Reconfigure Successful 17h +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-rotate-auth-43ris8 RotateAuth Successful 28m +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-volumeexpansion-41xthr VolumeExpansion Successful 17h + +``` + + +### TLS configuration + +We can add, rotate or remove TLS configuration using `gitops`. + +To add tls, we are going to create an example `Issuer` that will be used to enable SSL/TLS in MariaDB. Alternatively, you can follow this [cert-manager tutorial](https://cert-manager.io/docs/configuration/ca/) to create your own `Issuer`. + +- Start off by generating a ca certificates using openssl. + +```bash +$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ca.key -out ./ca.crt -subj "/CN=ca/O=kubedb" +Generating a RSA private key +................+++++ +........................+++++ +writing new private key to './ca.key' +----- +``` + +- Now we are going to create a ca-secret using the certificate files that we have just generated. + +```bash +$ kubectl create secret tls MariaDB-ca \ + --cert=ca.crt \ + --key=ca.key \ + --namespace=demo +secret/MariaDB-ca created +``` + +Now, Let's create an `Issuer` using the `MariaDB-ca` secret that we have just created. The `YAML` file looks like this: + +```yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: kf-issuer + namespace: demo +spec: + ca: + secretName: MariaDB-ca +``` + +Let's add that to our `kubedb /kf-issuer.yaml` file. File structure will look like this, +```bash +$ tree . +├── kubedb +│ ├── kf-configuration.yaml +│ ├── kf-issuer.yaml +│ └── MariaDB.yaml +1 directories, 4 files +``` + +Update the `MariaDB.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: MariaDB +metadata: + name: MariaDB-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Add `sslMode` and `tls` fields in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `MariaDB` CR is updated in your cluster. + +Now, `gitops` operator will detect the tls changes and create a `ReconfigureTLS` MariaDBOpsRequest to update the `MariaDB` database tls. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,MariaDB,kfops,pods -n demo +NAME TYPE VERSION STATUS AGE +MariaDB.kubedb.com/MariaDB-prod kubedb.com/v1 3.9.0 Ready 41m + +NAME AGE +MariaDB.gitops.kubedb.com/MariaDB-prod 75m + +NAME TYPE STATUS AGE +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-reconfigure-ukj41o Reconfigure Successful 5d18h +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 9m18s +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-rotate-auth-43ris8 RotateAuth Successful 5d1h +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d19h + +``` + + +> We can also rotate the certificates updating `.spec.tls.certificates` field. Also you can remove the `.spec.tls` field to remove tls for MariaDB. + +### Update Version + +List MariaDB versions using `kubectl get MariaDBversion` and choose desired version that is compatible for upgrade from current version. Check the version constraints and ops request [here](/docs/guides/MariaDB/update-version/update-version.md). + +Let's choose `4.0.0` in this example. + +Update the `MariaDB.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: MariaDB +metadata: + name: MariaDB-prod + namespace: demo +spec: + version: 4.0.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Update the `version` field to `17.4`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `MariaDB` CR is updated in your cluster. + +Now, `gitops` operator will detect the version changes and create a `VersionUpdate` MariaDBOpsRequest to update the `MariaDB` database version. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,MariaDB,kfops -n demo +NAME TYPE VERSION STATUS AGE +MariaDB.kubedb.com/MariaDB-prod kubedb.com/v1 4.0.0 Ready 3h47m + +NAME AGE +MariaDB.gitops.kubedb.com/MariaDB-prod 3h47m + +NAME TYPE STATUS AGE +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-reconfigure-ukj41o Reconfigure Successful 5d22h +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 4h16m +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-rotate-auth-43ris8 RotateAuth Successful 5d6h +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-versionupdate-wyn2dp UpdateVersion Successful 3h51m +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d23h +``` + + +Now, we are going to verify whether the `MariaDB`, `PetSet` and it's `Pod` have updated with new image. Let's check, + +```bash +$ kubectl get MariaDB -n demo MariaDB-prod -o=jsonpath='{.spec.version}{"\n"}' +4.0.0 + +$ kubectl get petset -n demo MariaDB-prod-broker -o=jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/MariaDB:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 + +$ kubectl get pod -n demo MariaDB-prod-broker-0 -o=jsonpath='{.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/MariaDB:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 +``` + +### Enable Monitoring + +If you already don't have a Prometheus server running, deploy one following tutorial from [here](https://github.com/appscode/third-party-tools/blob/master/monitoring/prometheus/operator/README.md#deploy-prometheus-server). + +Update the `MariaDB.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: MariaDB +metadata: + name: MariaDB-prod + namespace: demo +spec: + version: 4.0.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: MariaDB + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + monitor: + agent: prometheus.io/operator + prometheus: + exporter: + port: 9091 + serviceMonitor: + labels: + release: prometheus + interval: 10s + deletionPolicy: WipeOut +``` + +Add `monitor` field in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `MariaDB` CR is updated in your cluster. + +Now, `gitops` operator will detect the monitoring changes and create a `Restart` MariaDBOpsRequest to add the `MariaDB` database monitoring. List the resources created by `gitops` operator in the `demo` namespace. +```bash +$ kubectl get MariaDBes.gitops.kubedb.com,MariaDBes.kubedb.com,MariaDBopsrequest -n demo +NAME TYPE VERSION STATUS AGE +MariaDB.kubedb.com/MariaDB-prod kubedb.com/v1 4.0.0 Ready 5h12m + +NAME AGE +MariaDB.gitops.kubedb.com/MariaDB-prod 5h12m + +NAME TYPE STATUS AGE +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-reconfigure-ukj41o Reconfigure Successful 6d +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 5h42m +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-restart-ljpqih Restart Successful 3m51s +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-rotate-auth-43ris8 RotateAuth Successful 5d7h +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-versionupdate-wyn2dp UpdateVersion Successful 5h16m +MariaDBopsrequest.ops.kubedb.com/MariaDB-prod-volumeexpansion-41xthr VolumeExpansion Successful 6d + +``` + +Verify the monitoring is enabled by checking the prometheus targets. + +There are some other fields that will trigger `Restart` ops request. +- `.spec.monitor` +- `.spec.spec.archiver` +- `.spec.remoteReplica` +- `spec.replication` +- `.spec.standbyMode` +- `.spec.streamingMode` +- `.spec.enforceGroup` +- `.spec.sslMode` etc. + + +## Next Steps + +[//]: # (- Learn MariaDB [GitOps](/docs/guides/MariaDB/concepts/MariaDB-gitops.md)) +- Learn MariaDB Scaling + - [Horizontal Scaling](/docs/guides/MariaDB/scaling/horizontal-scaling/combined.md) + - [Vertical Scaling](/docs/guides/MariaDB/scaling/vertical-scaling/combined.md) +- Learn Version Update Ops Request and Constraints [here](/docs/guides/MariaDB/update-version/overview.md) +- Monitor your MariaDBQL database with KubeDB using [built-in Prometheus](/docs/guides/MariaDB/monitoring/using-builtin-prometheus.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/mariadb/gitops/overview.md b/docs/guides/mariadb/gitops/overview.md new file mode 100644 index 000000000..80ab6a2fc --- /dev/null +++ b/docs/guides/mariadb/gitops/overview.md @@ -0,0 +1,50 @@ +--- +title: MariaDB Gitops Overview +description: MariaDB Gitops Overview +menu: + docs_{{ .version }}: + identifier: es-gitops-overview + name: Overview + parent: es-gitops + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# GitOps Overview for MariaDB + +This guide will give you an overview of how KubeDB `gitops` operator works with MariaDB databases using the `gitops.kubedb.com/v1alpha1` API. It will help you understand the GitOps workflow for +managing MariaDB databases in Kubernetes. + +## Before You Begin + +- You should be familiar with the following `KubeDB` concepts: + - [MariaDB](/docs/guides/mariadb/concepts/mariadb/index.md) + - [MariaDBOpsRequest](/docs/guides/mariadb/concepts/opsrequest/index.md) + + + +## Workflow GitOps with MariaDB + +The following diagram shows how the `KubeDB` GitOps Operator used to sync with your database. Open the image in a new tab to see the enlarged version. + + +
+ + GitOps Flow + +
Fig: GitOps process of MariaDB
+ +
+ +1. **Define GitOps MariaDB**: Create Custom Resource (CR) of kind `MariaDB` using the `gitops.kubedb.com/v1alpha1` API. +2. **Store in Git**: Push the CR to a Git repository. +3. **Automated Deployment**: Use a GitOps tool (like `ArgoCD` or `FluxCD`) to monitor the Git repository and synchronize the state of the Kubernetes cluster with the desired state defined in Git. +4. **Create Database**: The GitOps operator creates a corresponding KubeDB MariaDB CR in the Kubernetes cluster to deploy the database. +5. **Handle Updates**: When you update the MariaDBGitOps CR, the operator generates an Ops Request to safely apply the update(e.g. `VerticalScaling`, `HorizontalScaling`, `VolumeExapnsion`, `Reconfigure`, `RotateAuth`, `ReconfigureTLS`, `VersionUpdate`, ans `Restart`. + +This flow makes managing MariaDB databases efficient, reliable, and fully integrated with GitOps practices. + +In the next doc, we are going to show a step by step guide on running MariaDB using GitOps. diff --git a/docs/guides/mongodb/gitops/_index.md b/docs/guides/mongodb/gitops/_index.md new file mode 100644 index 000000000..cc82b5f6b --- /dev/null +++ b/docs/guides/mongodb/gitops/_index.md @@ -0,0 +1,10 @@ +--- +title: Elasticsearch Gitops +menu: + docs_{{ .version }}: + identifier: es-gitops + name: Elasticsearch Gitops + parent: es-elasticsearch-guides + weight: 70 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/mongodb/gitops/gitops.md b/docs/guides/mongodb/gitops/gitops.md new file mode 100644 index 000000000..332c4f29d --- /dev/null +++ b/docs/guides/mongodb/gitops/gitops.md @@ -0,0 +1,900 @@ +--- +title: Elasticsearch Gitops Overview +description: Elasticsearch Gitops Overview +menu: + docs_{{ .version }}: + identifier: es-gitops-overview + name: Overview + parent: es-gitops + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + + + +> New to KubeDB? Please start [here](/docs/README.md). + +# GitOps Elasticsearch using KubeDB GitOps Operator + +This guide will show you how to use `KubeDB` GitOps operator to create Elasticsearch database and manage updates using GitOps workflow. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Install `KubeDB` operator in your cluster following the steps [here](/docs/setup/README.md). Pass `--set kubedb-crd-manager.installGitOpsCRDs=true` in the kubedb installation process to enable `GitOps` operator. + +- You need to install GitOps tools like `ArgoCD` or `FluxCD` and configure with your Git Repository to monitor the Git repository and synchronize the state of the Kubernetes cluster with the desired state defined in Git. + + ```bash + $ kubectl create ns monitoring + namespace/monitoring created + + $ kubectl create ns demo + namespace/demo created + ``` +> Note: YAML files used in this tutorial are stored in [docs/examples/Elasticsearch](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/Elasticsearch) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +We are going to use `ArgoCD` in this tutorial. You can install `ArgoCD` in your cluster by following the steps [here](https://argo-cd.readthedocs.io/en/stable/getting_started/). Also, you need to install `argocd` CLI in your local machine. You can install `argocd` CLI by following the steps [here](https://argo-cd.readthedocs.io/en/stable/cli_installation/). + +## Creating Apps via CLI + +### For Public Repository +```bash +argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace +``` + +### For Private Repository +#### Using HTTPS +```bash +argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace --username --password +``` + +#### Using SSH +```bash +argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace --ssh-private-key-path ~/.ssh/id_rsa +``` + +## Create Elasticsearch Database using GitOps + +### Create a Elasticsearch GitOps CR +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: es-gitops + namespace: demo +spec: + version: xpack-8.2.3 + enableSSL: true + replicas: 2 + storageType: Durable + storage: + storageClassName: longhorn + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut +``` + +Create a directory like below, +```bash +$ tree . +├── kubedb + └── Elasticsearch.yaml +1 directories, 1 files +``` + +Now commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is created in your cluster. + +Our `gitops` operator will create an actual `Elasticsearch` database CR in the cluster. List the resources created by `gitops` operator in the `demo` namespace. + + +```bash +$ kubectl get Elasticsearch.gitops.kubedb.com,Elasticsearch.kubedb.com -n demo +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 2m56s + +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 3.9.0 Ready 2m56s +``` + +List the resources created by `kubedb` operator created for `kubedb.com/v1` Elasticsearch. + +```bash +$ kubectl get petset,pod,secret,service,appbinding -n demo -l 'app.kubernetes.io/instance=Elasticsearch-prod' +NAME AGE +petset.apps.k8s.appscode.com/Elasticsearch-prod-broker 4m49s +petset.apps.k8s.appscode.com/Elasticsearch-prod-controller 4m47s + +NAME READY STATUS RESTARTS AGE +pod/Elasticsearch-prod-broker-0 1/1 Running 0 4m48s +pod/Elasticsearch-prod-broker-1 1/1 Running 0 4m42s +pod/Elasticsearch-prod-controller-0 1/1 Running 0 4m47s +pod/Elasticsearch-prod-controller-1 1/1 Running 0 4m40s + +NAME TYPE DATA AGE +secret/Elasticsearch-prod-auth kubernetes.io/basic-auth 2 4m51s + +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +service/Elasticsearch-prod-pods ClusterIP None 9092/TCP,9093/TCP,29092/TCP 4m51s + +NAME TYPE VERSION AGE +appbinding.appcatalog.appscode.com/Elasticsearch-prod kubedb.com/Elasticsearch 3.9.0 4m47s +``` + +## Update Elasticsearch Database using GitOps + +### Scale Elasticsearch Database Resources + +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + storageType: Durable + deletionPolicy: WipeOut + ``` + +Resource Requests and Limits are updated to `700m` CPU and `2Gi` Memory. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. + +Now, `gitops` operator will detect the resource changes and create a `ElasticsearchOpsRequest` to update the `Elasticsearch` database. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ $ kubectl get kf,Elasticsearch,kfops -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 3.9.0 Ready 22h + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 22h + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-verticalscaling-i0kr1l VerticalScaling Successful 2s +``` + +After Ops Request becomes `Successful`, We can validate the changes by checking the one of the pod, +```bash +$ kubectl get pod -n demo Elasticsearch-prod-broker-0 -o json | jq '.spec.containers[0].resources' +{ + "limits": { + "memory": "1536Mi" + }, + "requests": { + "cpu": "500m", + "memory": "1536Mi" + } +} +``` + +### Scale Elasticsearch Replicas +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 3 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 3 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + storageType: Durable + deletionPolicy: WipeOut +``` + +Update the `replicas` to `3`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. +Now, `gitops` operator will detect the replica changes and create a `HorizontalScaling` ElasticsearchOpsRequest to update the `Elasticsearch` database replicas. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Elasticsearch,kfops -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 3.9.0 Ready 22h + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 22h + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s +``` + +After Ops Request becomes `Successful`, We can validate the changes by checking the number of pods, +```bash +$ kubectl get pod -n demo -l 'app.kubernetes.io/instance=Elasticsearch-prod' +NAME READY STATUS RESTARTS AGE +Elasticsearch-prod-broker-0 1/1 Running 0 34m +Elasticsearch-prod-broker-1 1/1 Running 0 33m +Elasticsearch-prod-broker-2 1/1 Running 0 33m +Elasticsearch-prod-controller-0 1/1 Running 0 32m +Elasticsearch-prod-controller-1 1/1 Running 0 31m +Elasticsearch-prod-controller-2 1/1 Running 0 31m +``` + +We can also scale down the replicas by updating the `replicas` fields. + +### Expand Elasticsearch Volume + +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Update the `storage.resources.requests.storage` to `2Gi`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. + +Now, `gitops` operator will detect the volume changes and create a `VolumeExpansion` ElasticsearchOpsRequest to update the `Elasticsearch` database volume. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Elasticsearch,kfops -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 3.9.0 Ready 23m + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 23m + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-volumeexpansion-41xthr VolumeExpansion Successful 19m +``` + +After Ops Request becomes `Successful`, We can validate the changes by checking the pvc size, +```bash +$ kubectl get pvc -n demo -l 'app.kubernetes.io/instance=Elasticsearch-prod' +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE +Elasticsearch-prod-data-Elasticsearch-prod-broker-0 Bound pvc-2afd4835-5686-492b-be93-c6e040e0a6c6 2Gi RWO Standard 3h39m +Elasticsearch-prod-data-Elasticsearch-prod-broker-1 Bound pvc-aaf994cc-6b04-4c37-80d5-5e966dad8487 2Gi RWO Standard 3h39m +Elasticsearch-prod-data-Elasticsearch-prod-controller-0 Bound pvc-82d2b233-203d-4df2-a0fd-ecedbc0825b7 2Gi RWO Standard 3h39m +Elasticsearch-prod-data-Elasticsearch-prod-controller-1 Bound pvc-91852c29-ab1a-48ad-9255-a0b15d5a7515 2Gi RWO Standard 3h39m + +``` + +## Reconfigure Elasticsearch + +At first, we will create a secret containing `user.conf` file with required configuration settings. +To know more about this configuration file, check [here](/docs/guides/Elasticsearch/configuration/Elasticsearch-combined.md) +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: new-kf-combined-custom-config + namespace: demo +stringData: + server.properties: |- + log.retention.hours=125 +``` + +Now, we will add this file to `kubedb /kf-configuration.yaml`. + +```bash +$ tree . +├── kubedb +│ ├── kf-configuration.yaml +│ └── Elasticsearch.yaml +1 directories, 2 files +``` + +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + configSecret: + name: new-kf-combined-custom-config + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. + +Now, `gitops` operator will detect the configuration changes and create a `Reconfigure` ElasticsearchOpsRequest to update the `Elasticsearch` database configuration. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Elasticsearch,kfops -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 3.9.0 Ready 74m + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 74m + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfigure-ukj41o Reconfigure Successful 24m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-volumeexpansion-41xthr VolumeExpansion Successful 70m + +``` + + + +> We can also reconfigure the parameters creating another secret and reference the secret in the `configSecret` field. Also you can remove the `configSecret` field to use the default parameters. + +### Rotate Elasticsearch Auth + +To do that, create a `kubernetes.io/basic-auth` type k8s secret with the new username and password. + +We will do that using gitops, create the file `kubedb /kf-auth.yaml` with the following content, + +```bash +kubectl create secret generic kf-rotate-auth -n demo \ +--type=kubernetes.io/basic-auth \ +--from-literal=username=Elasticsearch \ +--from-literal=password=Elasticsearch-secret +secret/kf-rotate-auth created + +``` + + + +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + authSecret: + kind: Secret + name: kf-rotate-auth + configSecret: + name: new-kf-combined-custom-config + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Change the `authSecret` field to `kf-rotate-auth`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. + +Now, `gitops` operator will detect the auth changes and create a `RotateAuth` ElasticsearchOpsRequest to update the `Elasticsearch` database auth. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Elasticsearch,kfops -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 3.9.0 Ready 7m11s + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 7m11s + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfigure-ukj41o Reconfigure Successful 17h +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-rotate-auth-43ris8 RotateAuth Successful 28m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-volumeexpansion-41xthr VolumeExpansion Successful 17h + +``` + + +### TLS configuration + +We can add, rotate or remove TLS configuration using `gitops`. + +To add tls, we are going to create an example `Issuer` that will be used to enable SSL/TLS in Elasticsearch. Alternatively, you can follow this [cert-manager tutorial](https://cert-manager.io/docs/configuration/ca/) to create your own `Issuer`. + +- Start off by generating a ca certificates using openssl. + +```bash +$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ca.key -out ./ca.crt -subj "/CN=ca/O=kubedb" +Generating a RSA private key +................+++++ +........................+++++ +writing new private key to './ca.key' +----- +``` + +- Now we are going to create a ca-secret using the certificate files that we have just generated. + +```bash +$ kubectl create secret tls Elasticsearch-ca \ + --cert=ca.crt \ + --key=ca.key \ + --namespace=demo +secret/Elasticsearch-ca created +``` + +Now, Let's create an `Issuer` using the `Elasticsearch-ca` secret that we have just created. The `YAML` file looks like this: + +```yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: kf-issuer + namespace: demo +spec: + ca: + secretName: Elasticsearch-ca +``` + +Let's add that to our `kubedb /kf-issuer.yaml` file. File structure will look like this, +```bash +$ tree . +├── kubedb +│ ├── kf-configuration.yaml +│ ├── kf-issuer.yaml +│ └── Elasticsearch.yaml +1 directories, 4 files +``` + +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Add `sslMode` and `tls` fields in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. + +Now, `gitops` operator will detect the tls changes and create a `ReconfigureTLS` ElasticsearchOpsRequest to update the `Elasticsearch` database tls. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Elasticsearch,kfops,pods -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 3.9.0 Ready 41m + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 75m + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfigure-ukj41o Reconfigure Successful 5d18h +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 9m18s +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-rotate-auth-43ris8 RotateAuth Successful 5d1h +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d19h + +``` + + +> We can also rotate the certificates updating `.spec.tls.certificates` field. Also you can remove the `.spec.tls` field to remove tls for Elasticsearch. + +### Update Version + +List Elasticsearch versions using `kubectl get Elasticsearchversion` and choose desired version that is compatible for upgrade from current version. Check the version constraints and ops request [here](/docs/guides/Elasticsearch/update-version/update-version.md). + +Let's choose `4.0.0` in this example. + +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + version: 4.0.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Update the `version` field to `17.4`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. + +Now, `gitops` operator will detect the version changes and create a `VersionUpdate` ElasticsearchOpsRequest to update the `Elasticsearch` database version. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Elasticsearch,kfops -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 4.0.0 Ready 3h47m + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 3h47m + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfigure-ukj41o Reconfigure Successful 5d22h +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 4h16m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-rotate-auth-43ris8 RotateAuth Successful 5d6h +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-versionupdate-wyn2dp UpdateVersion Successful 3h51m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d23h +``` + + +Now, we are going to verify whether the `Elasticsearch`, `PetSet` and it's `Pod` have updated with new image. Let's check, + +```bash +$ kubectl get Elasticsearch -n demo Elasticsearch-prod -o=jsonpath='{.spec.version}{"\n"}' +4.0.0 + +$ kubectl get petset -n demo Elasticsearch-prod-broker -o=jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/Elasticsearch:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 + +$ kubectl get pod -n demo Elasticsearch-prod-broker-0 -o=jsonpath='{.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/Elasticsearch:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 +``` + +### Enable Monitoring + +If you already don't have a Prometheus server running, deploy one following tutorial from [here](https://github.com/appscode/third-party-tools/blob/master/monitoring/prometheus/operator/README.md#deploy-prometheus-server). + +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + version: 4.0.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + monitor: + agent: prometheus.io/operator + prometheus: + exporter: + port: 9091 + serviceMonitor: + labels: + release: prometheus + interval: 10s + deletionPolicy: WipeOut +``` + +Add `monitor` field in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. + +Now, `gitops` operator will detect the monitoring changes and create a `Restart` ElasticsearchOpsRequest to add the `Elasticsearch` database monitoring. List the resources created by `gitops` operator in the `demo` namespace. +```bash +$ kubectl get Elasticsearches.gitops.kubedb.com,Elasticsearches.kubedb.com,Elasticsearchopsrequest -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 4.0.0 Ready 5h12m + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 5h12m + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfigure-ukj41o Reconfigure Successful 6d +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 5h42m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-restart-ljpqih Restart Successful 3m51s +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-rotate-auth-43ris8 RotateAuth Successful 5d7h +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-versionupdate-wyn2dp UpdateVersion Successful 5h16m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-volumeexpansion-41xthr VolumeExpansion Successful 6d + +``` + +Verify the monitoring is enabled by checking the prometheus targets. + +There are some other fields that will trigger `Restart` ops request. +- `.spec.monitor` +- `.spec.spec.archiver` +- `.spec.remoteReplica` +- `spec.replication` +- `.spec.standbyMode` +- `.spec.streamingMode` +- `.spec.enforceGroup` +- `.spec.sslMode` etc. + + +## Next Steps + +[//]: # (- Learn Elasticsearch [GitOps](/docs/guides/Elasticsearch/concepts/Elasticsearch-gitops.md)) +- Learn Elasticsearch Scaling + - [Horizontal Scaling](/docs/guides/Elasticsearch/scaling/horizontal-scaling/combined.md) + - [Vertical Scaling](/docs/guides/Elasticsearch/scaling/vertical-scaling/combined.md) +- Learn Version Update Ops Request and Constraints [here](/docs/guides/Elasticsearch/update-version/overview.md) +- Monitor your ElasticsearchQL database with KubeDB using [built-in Prometheus](/docs/guides/Elasticsearch/monitoring/using-builtin-prometheus.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/mongodb/gitops/overview.md b/docs/guides/mongodb/gitops/overview.md new file mode 100644 index 000000000..bfa01c2ff --- /dev/null +++ b/docs/guides/mongodb/gitops/overview.md @@ -0,0 +1,50 @@ +--- +title: Elasticsearch Gitops Overview +description: Elasticsearch Gitops Overview +menu: + docs_{{ .version }}: + identifier: es-gitops-overview + name: Overview + parent: es-gitops + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# GitOps Overview for Elasticsearch + +This guide will give you an overview of how KubeDB `gitops` operator works with Elasticsearch databases using the `gitops.kubedb.com/v1alpha1` API. It will help you understand the GitOps workflow for +managing Elasticsearch databases in Kubernetes. + +## Before You Begin + +- You should be familiar with the following `KubeDB` concepts: + - [Elasticsearch](/docs/guides/elasticsearch/concepts/elasticsearch/index.md) + - [ElasticsearchOpsRequest](/docs/guides/elasticsearch/concepts/elasticsearch-ops-request/index.md) + + + +## Workflow GitOps with Elasticsearch + +The following diagram shows how the `KubeDB` GitOps Operator used to sync with your database. Open the image in a new tab to see the enlarged version. + + +
+ + GitOps Flow + +
Fig: GitOps process of Elasticsearch
+ +
+ +1. **Define GitOps Elasticsearch**: Create Custom Resource (CR) of kind `Elasticsearch` using the `gitops.kubedb.com/v1alpha1` API. +2. **Store in Git**: Push the CR to a Git repository. +3. **Automated Deployment**: Use a GitOps tool (like `ArgoCD` or `FluxCD`) to monitor the Git repository and synchronize the state of the Kubernetes cluster with the desired state defined in Git. +4. **Create Database**: The GitOps operator creates a corresponding KubeDB Elasticsearch CR in the Kubernetes cluster to deploy the database. +5. **Handle Updates**: When you update the ElasticsearchGitOps CR, the operator generates an Ops Request to safely apply the update(e.g. `VerticalScaling`, `HorizontalScaling`, `VolumeExapnsion`, `Reconfigure`, `RotateAuth`, `ReconfigureTLS`, `VersionUpdate`, ans `Restart`. + +This flow makes managing Elasticsearch databases efficient, reliable, and fully integrated with GitOps practices. + +In the next doc, we are going to show a step by step guide on running Elasticsearch using GitOps. diff --git a/docs/guides/mssqlserver/gitops/_index.md b/docs/guides/mssqlserver/gitops/_index.md new file mode 100644 index 000000000..cc82b5f6b --- /dev/null +++ b/docs/guides/mssqlserver/gitops/_index.md @@ -0,0 +1,10 @@ +--- +title: Elasticsearch Gitops +menu: + docs_{{ .version }}: + identifier: es-gitops + name: Elasticsearch Gitops + parent: es-elasticsearch-guides + weight: 70 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/mssqlserver/gitops/gitops.md b/docs/guides/mssqlserver/gitops/gitops.md new file mode 100644 index 000000000..332c4f29d --- /dev/null +++ b/docs/guides/mssqlserver/gitops/gitops.md @@ -0,0 +1,900 @@ +--- +title: Elasticsearch Gitops Overview +description: Elasticsearch Gitops Overview +menu: + docs_{{ .version }}: + identifier: es-gitops-overview + name: Overview + parent: es-gitops + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + + + +> New to KubeDB? Please start [here](/docs/README.md). + +# GitOps Elasticsearch using KubeDB GitOps Operator + +This guide will show you how to use `KubeDB` GitOps operator to create Elasticsearch database and manage updates using GitOps workflow. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Install `KubeDB` operator in your cluster following the steps [here](/docs/setup/README.md). Pass `--set kubedb-crd-manager.installGitOpsCRDs=true` in the kubedb installation process to enable `GitOps` operator. + +- You need to install GitOps tools like `ArgoCD` or `FluxCD` and configure with your Git Repository to monitor the Git repository and synchronize the state of the Kubernetes cluster with the desired state defined in Git. + + ```bash + $ kubectl create ns monitoring + namespace/monitoring created + + $ kubectl create ns demo + namespace/demo created + ``` +> Note: YAML files used in this tutorial are stored in [docs/examples/Elasticsearch](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/Elasticsearch) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +We are going to use `ArgoCD` in this tutorial. You can install `ArgoCD` in your cluster by following the steps [here](https://argo-cd.readthedocs.io/en/stable/getting_started/). Also, you need to install `argocd` CLI in your local machine. You can install `argocd` CLI by following the steps [here](https://argo-cd.readthedocs.io/en/stable/cli_installation/). + +## Creating Apps via CLI + +### For Public Repository +```bash +argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace +``` + +### For Private Repository +#### Using HTTPS +```bash +argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace --username --password +``` + +#### Using SSH +```bash +argocd app create kubedb --repo --path kubedb --dest-server https://kubernetes.default.svc --dest-namespace --ssh-private-key-path ~/.ssh/id_rsa +``` + +## Create Elasticsearch Database using GitOps + +### Create a Elasticsearch GitOps CR +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: es-gitops + namespace: demo +spec: + version: xpack-8.2.3 + enableSSL: true + replicas: 2 + storageType: Durable + storage: + storageClassName: longhorn + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut +``` + +Create a directory like below, +```bash +$ tree . +├── kubedb + └── Elasticsearch.yaml +1 directories, 1 files +``` + +Now commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is created in your cluster. + +Our `gitops` operator will create an actual `Elasticsearch` database CR in the cluster. List the resources created by `gitops` operator in the `demo` namespace. + + +```bash +$ kubectl get Elasticsearch.gitops.kubedb.com,Elasticsearch.kubedb.com -n demo +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 2m56s + +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 3.9.0 Ready 2m56s +``` + +List the resources created by `kubedb` operator created for `kubedb.com/v1` Elasticsearch. + +```bash +$ kubectl get petset,pod,secret,service,appbinding -n demo -l 'app.kubernetes.io/instance=Elasticsearch-prod' +NAME AGE +petset.apps.k8s.appscode.com/Elasticsearch-prod-broker 4m49s +petset.apps.k8s.appscode.com/Elasticsearch-prod-controller 4m47s + +NAME READY STATUS RESTARTS AGE +pod/Elasticsearch-prod-broker-0 1/1 Running 0 4m48s +pod/Elasticsearch-prod-broker-1 1/1 Running 0 4m42s +pod/Elasticsearch-prod-controller-0 1/1 Running 0 4m47s +pod/Elasticsearch-prod-controller-1 1/1 Running 0 4m40s + +NAME TYPE DATA AGE +secret/Elasticsearch-prod-auth kubernetes.io/basic-auth 2 4m51s + +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +service/Elasticsearch-prod-pods ClusterIP None 9092/TCP,9093/TCP,29092/TCP 4m51s + +NAME TYPE VERSION AGE +appbinding.appcatalog.appscode.com/Elasticsearch-prod kubedb.com/Elasticsearch 3.9.0 4m47s +``` + +## Update Elasticsearch Database using GitOps + +### Scale Elasticsearch Database Resources + +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + storageType: Durable + deletionPolicy: WipeOut + ``` + +Resource Requests and Limits are updated to `700m` CPU and `2Gi` Memory. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. + +Now, `gitops` operator will detect the resource changes and create a `ElasticsearchOpsRequest` to update the `Elasticsearch` database. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ $ kubectl get kf,Elasticsearch,kfops -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 3.9.0 Ready 22h + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 22h + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-verticalscaling-i0kr1l VerticalScaling Successful 2s +``` + +After Ops Request becomes `Successful`, We can validate the changes by checking the one of the pod, +```bash +$ kubectl get pod -n demo Elasticsearch-prod-broker-0 -o json | jq '.spec.containers[0].resources' +{ + "limits": { + "memory": "1536Mi" + }, + "requests": { + "cpu": "500m", + "memory": "1536Mi" + } +} +``` + +### Scale Elasticsearch Replicas +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 3 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1540Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 3 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-path + storageType: Durable + deletionPolicy: WipeOut +``` + +Update the `replicas` to `3`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. +Now, `gitops` operator will detect the replica changes and create a `HorizontalScaling` ElasticsearchOpsRequest to update the `Elasticsearch` database replicas. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Elasticsearch,kfops -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 3.9.0 Ready 22h + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 22h + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s +``` + +After Ops Request becomes `Successful`, We can validate the changes by checking the number of pods, +```bash +$ kubectl get pod -n demo -l 'app.kubernetes.io/instance=Elasticsearch-prod' +NAME READY STATUS RESTARTS AGE +Elasticsearch-prod-broker-0 1/1 Running 0 34m +Elasticsearch-prod-broker-1 1/1 Running 0 33m +Elasticsearch-prod-broker-2 1/1 Running 0 33m +Elasticsearch-prod-controller-0 1/1 Running 0 32m +Elasticsearch-prod-controller-1 1/1 Running 0 31m +Elasticsearch-prod-controller-2 1/1 Running 0 31m +``` + +We can also scale down the replicas by updating the `replicas` fields. + +### Expand Elasticsearch Volume + +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Update the `storage.resources.requests.storage` to `2Gi`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. + +Now, `gitops` operator will detect the volume changes and create a `VolumeExpansion` ElasticsearchOpsRequest to update the `Elasticsearch` database volume. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Elasticsearch,kfops -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 3.9.0 Ready 23m + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 23m + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-horizontalscaling-j0wni6 HorizontalScaling Successful 13m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-verticalscaling-tfkvi8 VerticalScaling Successful 8m29s +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-volumeexpansion-41xthr VolumeExpansion Successful 19m +``` + +After Ops Request becomes `Successful`, We can validate the changes by checking the pvc size, +```bash +$ kubectl get pvc -n demo -l 'app.kubernetes.io/instance=Elasticsearch-prod' +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE +Elasticsearch-prod-data-Elasticsearch-prod-broker-0 Bound pvc-2afd4835-5686-492b-be93-c6e040e0a6c6 2Gi RWO Standard 3h39m +Elasticsearch-prod-data-Elasticsearch-prod-broker-1 Bound pvc-aaf994cc-6b04-4c37-80d5-5e966dad8487 2Gi RWO Standard 3h39m +Elasticsearch-prod-data-Elasticsearch-prod-controller-0 Bound pvc-82d2b233-203d-4df2-a0fd-ecedbc0825b7 2Gi RWO Standard 3h39m +Elasticsearch-prod-data-Elasticsearch-prod-controller-1 Bound pvc-91852c29-ab1a-48ad-9255-a0b15d5a7515 2Gi RWO Standard 3h39m + +``` + +## Reconfigure Elasticsearch + +At first, we will create a secret containing `user.conf` file with required configuration settings. +To know more about this configuration file, check [here](/docs/guides/Elasticsearch/configuration/Elasticsearch-combined.md) +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: new-kf-combined-custom-config + namespace: demo +stringData: + server.properties: |- + log.retention.hours=125 +``` + +Now, we will add this file to `kubedb /kf-configuration.yaml`. + +```bash +$ tree . +├── kubedb +│ ├── kf-configuration.yaml +│ └── Elasticsearch.yaml +1 directories, 2 files +``` + +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + configSecret: + name: new-kf-combined-custom-config + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. + +Now, `gitops` operator will detect the configuration changes and create a `Reconfigure` ElasticsearchOpsRequest to update the `Elasticsearch` database configuration. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Elasticsearch,kfops -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 3.9.0 Ready 74m + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 74m + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfigure-ukj41o Reconfigure Successful 24m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-volumeexpansion-41xthr VolumeExpansion Successful 70m + +``` + + + +> We can also reconfigure the parameters creating another secret and reference the secret in the `configSecret` field. Also you can remove the `configSecret` field to use the default parameters. + +### Rotate Elasticsearch Auth + +To do that, create a `kubernetes.io/basic-auth` type k8s secret with the new username and password. + +We will do that using gitops, create the file `kubedb /kf-auth.yaml` with the following content, + +```bash +kubectl create secret generic kf-rotate-auth -n demo \ +--type=kubernetes.io/basic-auth \ +--from-literal=username=Elasticsearch \ +--from-literal=password=Elasticsearch-secret +secret/kf-rotate-auth created + +``` + + + +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + authSecret: + kind: Secret + name: kf-rotate-auth + configSecret: + name: new-kf-combined-custom-config + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Change the `authSecret` field to `kf-rotate-auth`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. + +Now, `gitops` operator will detect the auth changes and create a `RotateAuth` ElasticsearchOpsRequest to update the `Elasticsearch` database auth. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Elasticsearch,kfops -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 3.9.0 Ready 7m11s + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 7m11s + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfigure-ukj41o Reconfigure Successful 17h +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-rotate-auth-43ris8 RotateAuth Successful 28m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-volumeexpansion-41xthr VolumeExpansion Successful 17h + +``` + + +### TLS configuration + +We can add, rotate or remove TLS configuration using `gitops`. + +To add tls, we are going to create an example `Issuer` that will be used to enable SSL/TLS in Elasticsearch. Alternatively, you can follow this [cert-manager tutorial](https://cert-manager.io/docs/configuration/ca/) to create your own `Issuer`. + +- Start off by generating a ca certificates using openssl. + +```bash +$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ca.key -out ./ca.crt -subj "/CN=ca/O=kubedb" +Generating a RSA private key +................+++++ +........................+++++ +writing new private key to './ca.key' +----- +``` + +- Now we are going to create a ca-secret using the certificate files that we have just generated. + +```bash +$ kubectl create secret tls Elasticsearch-ca \ + --cert=ca.crt \ + --key=ca.key \ + --namespace=demo +secret/Elasticsearch-ca created +``` + +Now, Let's create an `Issuer` using the `Elasticsearch-ca` secret that we have just created. The `YAML` file looks like this: + +```yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: kf-issuer + namespace: demo +spec: + ca: + secretName: Elasticsearch-ca +``` + +Let's add that to our `kubedb /kf-issuer.yaml` file. File structure will look like this, +```bash +$ tree . +├── kubedb +│ ├── kf-configuration.yaml +│ ├── kf-issuer.yaml +│ └── Elasticsearch.yaml +1 directories, 4 files +``` + +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + version: 3.9.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Add `sslMode` and `tls` fields in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. + +Now, `gitops` operator will detect the tls changes and create a `ReconfigureTLS` ElasticsearchOpsRequest to update the `Elasticsearch` database tls. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Elasticsearch,kfops,pods -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 3.9.0 Ready 41m + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 75m + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfigure-ukj41o Reconfigure Successful 5d18h +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 9m18s +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-rotate-auth-43ris8 RotateAuth Successful 5d1h +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d19h + +``` + + +> We can also rotate the certificates updating `.spec.tls.certificates` field. Also you can remove the `.spec.tls` field to remove tls for Elasticsearch. + +### Update Version + +List Elasticsearch versions using `kubectl get Elasticsearchversion` and choose desired version that is compatible for upgrade from current version. Check the version constraints and ops request [here](/docs/guides/Elasticsearch/update-version/update-version.md). + +Let's choose `4.0.0` in this example. + +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + version: 4.0.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Update the `version` field to `17.4`. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. + +Now, `gitops` operator will detect the version changes and create a `VersionUpdate` ElasticsearchOpsRequest to update the `Elasticsearch` database version. List the resources created by `gitops` operator in the `demo` namespace. + +```bash +$ kubectl get kf,Elasticsearch,kfops -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 4.0.0 Ready 3h47m + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 3h47m + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfigure-ukj41o Reconfigure Successful 5d22h +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 4h16m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-rotate-auth-43ris8 RotateAuth Successful 5d6h +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-versionupdate-wyn2dp UpdateVersion Successful 3h51m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-volumeexpansion-41xthr VolumeExpansion Successful 5d23h +``` + + +Now, we are going to verify whether the `Elasticsearch`, `PetSet` and it's `Pod` have updated with new image. Let's check, + +```bash +$ kubectl get Elasticsearch -n demo Elasticsearch-prod -o=jsonpath='{.spec.version}{"\n"}' +4.0.0 + +$ kubectl get petset -n demo Elasticsearch-prod-broker -o=jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/Elasticsearch:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 + +$ kubectl get pod -n demo Elasticsearch-prod-broker-0 -o=jsonpath='{.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/Elasticsearch:4.0.0@sha256:42a79fe8f14b00b1c76d135bbbaf7605b8c66f45cf3eb749c59138f6df288b31 +``` + +### Enable Monitoring + +If you already don't have a Prometheus server running, deploy one following tutorial from [here](https://github.com/appscode/third-party-tools/blob/master/monitoring/prometheus/operator/README.md#deploy-prometheus-server). + +Update the `Elasticsearch.yaml` with the following, +```yaml +apiVersion: gitops.kubedb.com/v1alpha1 +kind: Elasticsearch +metadata: + name: Elasticsearch-prod + namespace: demo +spec: + version: 4.0.0 + topology: + broker: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + controller: + podTemplate: + spec: + containers: + - name: Elasticsearch + resources: + limits: + memory: 1536Mi + requests: + cpu: 500m + memory: 1536Mi + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi + storageClassName: Standard + storageType: Durable + monitor: + agent: prometheus.io/operator + prometheus: + exporter: + port: 9091 + serviceMonitor: + labels: + release: prometheus + interval: 10s + deletionPolicy: WipeOut +``` + +Add `monitor` field in the spec. Commit the changes and push to your Git repository. Your repository is synced with `ArgoCD` and the `Elasticsearch` CR is updated in your cluster. + +Now, `gitops` operator will detect the monitoring changes and create a `Restart` ElasticsearchOpsRequest to add the `Elasticsearch` database monitoring. List the resources created by `gitops` operator in the `demo` namespace. +```bash +$ kubectl get Elasticsearches.gitops.kubedb.com,Elasticsearches.kubedb.com,Elasticsearchopsrequest -n demo +NAME TYPE VERSION STATUS AGE +Elasticsearch.kubedb.com/Elasticsearch-prod kubedb.com/v1 4.0.0 Ready 5h12m + +NAME AGE +Elasticsearch.gitops.kubedb.com/Elasticsearch-prod 5h12m + +NAME TYPE STATUS AGE +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfigure-ukj41o Reconfigure Successful 6d +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-reconfiguretls-r4mx7v ReconfigureTLS Successful 5h42m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-restart-ljpqih Restart Successful 3m51s +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-rotate-auth-43ris8 RotateAuth Successful 5d7h +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-versionupdate-wyn2dp UpdateVersion Successful 5h16m +Elasticsearchopsrequest.ops.kubedb.com/Elasticsearch-prod-volumeexpansion-41xthr VolumeExpansion Successful 6d + +``` + +Verify the monitoring is enabled by checking the prometheus targets. + +There are some other fields that will trigger `Restart` ops request. +- `.spec.monitor` +- `.spec.spec.archiver` +- `.spec.remoteReplica` +- `spec.replication` +- `.spec.standbyMode` +- `.spec.streamingMode` +- `.spec.enforceGroup` +- `.spec.sslMode` etc. + + +## Next Steps + +[//]: # (- Learn Elasticsearch [GitOps](/docs/guides/Elasticsearch/concepts/Elasticsearch-gitops.md)) +- Learn Elasticsearch Scaling + - [Horizontal Scaling](/docs/guides/Elasticsearch/scaling/horizontal-scaling/combined.md) + - [Vertical Scaling](/docs/guides/Elasticsearch/scaling/vertical-scaling/combined.md) +- Learn Version Update Ops Request and Constraints [here](/docs/guides/Elasticsearch/update-version/overview.md) +- Monitor your ElasticsearchQL database with KubeDB using [built-in Prometheus](/docs/guides/Elasticsearch/monitoring/using-builtin-prometheus.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/mssqlserver/gitops/overview.md b/docs/guides/mssqlserver/gitops/overview.md new file mode 100644 index 000000000..bfa01c2ff --- /dev/null +++ b/docs/guides/mssqlserver/gitops/overview.md @@ -0,0 +1,50 @@ +--- +title: Elasticsearch Gitops Overview +description: Elasticsearch Gitops Overview +menu: + docs_{{ .version }}: + identifier: es-gitops-overview + name: Overview + parent: es-gitops + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# GitOps Overview for Elasticsearch + +This guide will give you an overview of how KubeDB `gitops` operator works with Elasticsearch databases using the `gitops.kubedb.com/v1alpha1` API. It will help you understand the GitOps workflow for +managing Elasticsearch databases in Kubernetes. + +## Before You Begin + +- You should be familiar with the following `KubeDB` concepts: + - [Elasticsearch](/docs/guides/elasticsearch/concepts/elasticsearch/index.md) + - [ElasticsearchOpsRequest](/docs/guides/elasticsearch/concepts/elasticsearch-ops-request/index.md) + + + +## Workflow GitOps with Elasticsearch + +The following diagram shows how the `KubeDB` GitOps Operator used to sync with your database. Open the image in a new tab to see the enlarged version. + + +
+ + GitOps Flow + +
Fig: GitOps process of Elasticsearch
+ +
+ +1. **Define GitOps Elasticsearch**: Create Custom Resource (CR) of kind `Elasticsearch` using the `gitops.kubedb.com/v1alpha1` API. +2. **Store in Git**: Push the CR to a Git repository. +3. **Automated Deployment**: Use a GitOps tool (like `ArgoCD` or `FluxCD`) to monitor the Git repository and synchronize the state of the Kubernetes cluster with the desired state defined in Git. +4. **Create Database**: The GitOps operator creates a corresponding KubeDB Elasticsearch CR in the Kubernetes cluster to deploy the database. +5. **Handle Updates**: When you update the ElasticsearchGitOps CR, the operator generates an Ops Request to safely apply the update(e.g. `VerticalScaling`, `HorizontalScaling`, `VolumeExapnsion`, `Reconfigure`, `RotateAuth`, `ReconfigureTLS`, `VersionUpdate`, ans `Restart`. + +This flow makes managing Elasticsearch databases efficient, reliable, and fully integrated with GitOps practices. + +In the next doc, we are going to show a step by step guide on running Elasticsearch using GitOps.