Splunk Operator: DeploymentServer CDR

[Thnai]

Please select the type of request

Enhancement

Tell us more

Describe the request
Addition of a DeploymentServer CDR to be able to deploy a complete SVA C3 setup in k8s

Expected features
Beeing able to manage forwarder and their apps from within a splunkoperator hosted splunk cluster on k8s
support for horizontal scaling of the DeploymentServer
Integrated ?AppFramework? to remotely (S3) store all needed apps and serverclasses

[Pixelshifter]

I've looked through the docs and openapi spec but I could not find the answer to my question: Does the Splunk operator provide the possibility to deploy a Deployment Server in k8s with which you can deploy apps on Forwarders? If not, is this issue stale or is this being worked on? We would like this functionality.

[hampusstrom]

Since splunk has not bothered to answer this multiple years later: #1589 should enable the management of etc/deployment-apps using this operator through the AppFramework

However, you will still need to deal with your own serverclasses configuration using spec.defaults_url, spec.defaults or as an app deployed to the local scope containing your serverclass.conf to a Standalone resource.

To be perfectly clear: THIS DOES NOT WORK in the current version of splunk-operator (3.0.0). #1589 must be merged first.

---
apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: MyImprovisedDeploymentServer
  namespace: splunk-operator
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  appRepo:
    defaults:
      volumeName: volume_app_repo
      scope: local 
   appSources:
     - name: MyDeploymentApps
        location: deployment-apps # the "directory path" within the bucket, lets pretend the app my_outputs_app.spl is here.
        # this is what's new in #1589. Right now the Standalone resource only supports the "local" scope, i.e. etc/apps
        scope: cluster # goes into etc/deployment-apps

     - name: MyLocalAppsToInstallOnDS
        location: LocalDSApps
        scope: local # goes into etc/apps

    volumes:
      - name: volume_app_repo
        path: indexdata-s2-bucket/standaloneNodes/s1data/
        endpoint: https://s3-us-west-2.amazonaws.com
        secretRef: s3-secret

  defaults: |-
    splunk:
      conf:
        - key: serverclass
           value:
             content:

               'serverClass:MyOutputsServerClass:app:my_outputs_app':
                 restartSplunkd: true
                 stateOnClient: enabled

               'serverClass:MyOutputsServerClass':
                 machineTypesFilter: windows-x64
                 whitelist.0 = *

Until then you could consider managing your etc/deployment-apps completely externally to the operator.
Something like making a NFS share and simply mounting it into etc/deployment-apps using spec.volumes.
And then changing the path in which the deployment server looks for apps i.e. "repositoryLocation" in serverclass.conf to point to the mountPath (/mnt/mydeploymentapps in the below example).

something like:

---
apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: MyImprovisedDeploymentServerWithout #1589
  namespace: splunk-operator
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  volumes:
  - name: mydeploymentapps    
      nfs:
        server: my-nfs-server.example.com
        path: /my-deployment-apps-nfs-share
        readOnly: true
  defaults: |-
    splunk:
      conf:
        - key: serverclass
           value:
             content:
               global:
                 # the name of the volume as specified above
                 # see https://splunk.github.io/splunk-operator/CustomResources.html#:~:text=volumes,-Volume
                 repositoryLocation: /mnt/mydeploymentapps 

               'serverClass:MyOutputsServerClass:app:my_outputs_app':
                 restartSplunkd: true
                 stateOnClient: enabled

               'serverClass:MyOutputsServerClass':
                 machineTypesFilter: windows-x64
                 whitelist.0 = *

The above example is completely untested but it should point anyone who's looking into using this operator to host deployment servers right now in the right direction.

Now that deploymentservers actually have the possibility to be clustered, I am surprised that the focus is not on dedicated DeploymentServer CRD:s as suggested in this issue.
https://help.splunk.com/en/splunk-enterprise/administer/update-your-deployment/10.0/configure-the-agent-management-system/implement-an-agent-management-cluster

Especially since writing serverclass.conf by hand is horrible, doing it in yaml is somehow even worse.
It could really do with a nice abstraction layer through the CRD.

Personally I would love to see something like below where the contents of spec.serverClasses gets templated into etc/system/local/serverclass.conf.

apiVersion: enterprise.splunk.com/vX
kind: DeploymentServer
metadata:
  name: ExampleDeploymentServerCRD
  namespace: splunk-operator
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  # sets up deployment server clustering with 3 members https://help.splunk.com/en/splunk-enterprise/administer/update-your-deployment/10.0/configure-the-agent-management-system/implement-an-agent-management-cluster
  # will require a shared ReadWriteMany PVC
  replicas: 3 
  serverClasses:
       # for each serverclass template a [serverClass:<spec.serverClasses[].name>]
    - name: MyWindowsServerClass
       stateOnClient: enabled
       restartSplunkd: true 
       allowlist: ["*"]
       blocklist: []
       machineTypesFilter: ["windows-x64"]
       # references to individual appSources in the AppFramework's spec.appRepo.appSources[]
       appSourcesRef:
         # for each app included in each appSourcesRef a [serverClass:<spec.serverClasses[].name>:app:<spec.serverClasses[].appSourcesRef[].appName>] block in serverclass.conf
         - name: MyDeploymentApp
            appName: SomeApp
         - name: MyWindowsApp
           # the name of the app in side the <volume_app_repo(bucket)/mywindowsapps(directory), im guessing this would require the appframework to unpack and extract the names of apps before doing the transfer. 
            appName: Splunk_TA_Windows 
            stateOnClient: disabled # overwrite on a appLevel for each app in the appSource
         - name: MyOtherAppInMyWindowsApps
            appName: Splunk_TA_microsoft_sysmon
  appRepo:
    defaults:
      volumeName: volume_app_repo
      scope: local 
   appSources:
     - name: MyDeploymentApps
        location: deployment-apps 
        scope: cluster 
     
     - name: MyWindowsApps
        location: mywindowsapps
        scope: cluster

     - name: MyLocalAppsToInstallOnDS
        location: LocalDSApps
        scope: local