feat: Added support for external-secrets (#28)

* feat: Added support for external-secrets

* fix: Update stripe setup to append secret to secrets manager instead of kubernetes

* fix: change script modes to executable

Author: bmonkman

scripts/check.sh

@@ -1,26 +1,25 @@
 #!/bin/bash
 
 # This script runs only when billingEnabled = "yes", invoked from makefile
-# modify the kubernetes application secret and appends STRIPE_API_SECRET_KEY
-# the deployment by default will pick up all key-value pairs as env-vars from the secret
+# It will modify the application secret in Secrets Manager and append STRIPE_API_SECRET_KEY
+# This will be synced to k8s and the deployment will pick up all key-value pairs as env vars from the secret
 
-if [[ "$ENVIRONMENT" == "" ]]; then
-  echo "Must specify \$ENVIRONMENT to create stripe secret ">&2; exit 1;
-elif [[ "$ENVIRONMENT" == "stage" ]]; then
-PUBLISHABLE_API_KEY=$stagingStripePublicApiKey
-SECRET_API_KEY=$stagingStripeSecretApiKey
+if [[ "$ENVIRONMENT" == "stage" ]]; then
+  PUBLISHABLE_API_KEY=$stagingStripePublicApiKey
+  SECRET_API_KEY=$stagingStripeSecretApiKey
 elif [[ "$ENVIRONMENT" == "prod" ]]; then
-PUBLISHABLE_API_KEY=$productionStripePublicApiKey
-SECRET_API_KEY=$productionStripeSecretApiKey
+  PUBLISHABLE_API_KEY=$productionStripePublicApiKey
+  SECRET_API_KEY=$productionStripeSecretApiKey
+else
+  echo 'Must specify $ENVIRONMENT (stage/prod) to create stripe secret'>&2; exit 1;
 fi
 
-CLUSTER_NAME=${PROJECT_NAME}-${ENVIRONMENT}-${REGION}
-NAMESPACE=${PROJECT_NAME}
-
+SECRET_NAME=${PROJECT_NAME}/kubernetes/${ENVIRONMENT}/${PROJECT_NAME}
 BASE64_TOKEN=$(printf ${SECRET_API_KEY} | base64)
-## Modify existing application secret to have stripe api key
-kubectl --context $CLUSTER_NAME -n $NAMESPACE get secret ${PROJECT_NAME} -o json | \
-  jq --arg STRIPE_API_SECRET_KEY $BASE64_TOKEN '.data["STRIPE_API_SECRET_KEY"]=$STRIPE_API_SECRET_KEY' \
-  | kubectl apply -f -
+
+# Modify existing application secret to add stripe api key
+UPDATED_SECRET=$(aws secretsmanager get-secret-value --region ${REGION} --secret=${SECRET_NAME} --query "SecretString" --output text | \
+  jq --arg STRIPE_API_SECRET_KEY ${BASE64_TOKEN} '.STRIPE_API_SECRET_KEY=$STRIPE_API_SECRET_KEY')
+aws secretsmanager update-secret --secret-id=${SECRET_NAME} --secret-string="${UPDATED_SECRET}"
 
 sh ${PROJECT_DIR}/scripts/stripe-example-setup.sh

scripts/gha-setup.sh

@@ -13,15 +13,15 @@ kubectl -n <% .Name %> get pods
 You can update the resource limits in the [kubernetes/base/deployment.yml][base-deployment], and control fine-grain customizations based on environment and specific deployments such as Scaling out your production replicas from the [overlays configurations][env-prod]
 
 ### Dev Environment
-This project is set up with a local/cloud hybrid dev environment. This means you can do fast local development of a single service, even if that service depends on other resources in your cluster. 
+This project is set up with a local/cloud hybrid dev environment. This means you can do fast local development of a single service, even if that service depends on other resources in your cluster.
 Make a change to your service, run it, and you can immediately see the new service in action in a real environment. You can also use any tools like your local IDE, debugger, etc. to test/debug/edit/run your service.
 
-Usually when developing you would run the service locally with a local database and any other dependencies running either locally or in containers using `docker-compose`, `minikube`, etc. 
+Usually when developing you would run the service locally with a local database and any other dependencies running either locally or in containers using `docker-compose`, `minikube`, etc.
 Now your service will have access to any dependencies within a namespace running in the EKS cluster, with access to resources there.
-[Telepresence](https://telepresence.io) is used to provide this functionality. 
+[Telepresence](https://telepresence.io) is used to provide this functionality.
 
  Development workflow:
- 
+
   1. Run `start-dev-env.sh` - You will be dropped into a shell that is the same as your local machine, but works as if it were running inside a pod in your k8s cluster
   2. Change code and run the server - As you run your local server, using local code, it will have access to remote dependencies, and will be sent traffic by the load balancer
   3. Test on your cloud environment with real dependencies - `https://<your name>-<% index .Params `stagingBackendSubdomain` %><% index .Params `stagingHostRoot` %>`
@@ -61,8 +61,21 @@ By default it requires `[lint, unit-test]` to be passing to allow Pull requests
 <% end %>
 
 ## Database credentials
-Your application is assumed[(ref)][base-deployment-secret] to rely on a database(RDS), In your Kubernetes
-application namespace, an application specific user has been created for you and hooked up to the application already.
+Your application is assumed to rely on a database. An application-specific user has been created for you by the `/scripts/create-db-user.sh` script in the infrastructure project.
+A Kubernetes secret will exist in your application namespace (<% .Name %>) that will provide these credentials to your application.
+
+## Secrets
+Along with the database credentials, any other secrets that need to be provided to the application can be managed in AWS Secrets Manager.
+Secrets have been created for each environment called `<% .Name %>/kubernetes/<environment>/<% .Name %>` which contain a list of environment variables that will be synced with the kubernetes secret in your namespace via a tool called [external-secrets](https://github.com/external-secrets/kubernetes-external-secrets)
+Any secrets managed by `external-secrets` will be synced to kubernetes every 15 seconds. Keep in mind that any changes must be made in Secrets Manager, as any that are made to the secret on the kubernetes side will be overwritten.
+You can see the `external-secrets` configuration in [kubernetes/overlays/staging/external-secret.yml](./kubernetes/overlays/staging/external-secret.yml) (this is the one for staging)
+
+To work with the secret in AWS you can use the web interface or the cli tool:
+```
+ aws secretsmanager get-secret-value --secret=<% .Name %>/kubernetes/stage/<% .Name %>
+```
+
+The intent is that the last part of the secret name is the component of your application this secret is for. For example: if you were adding a new billing service, the secret might be called `<% .Name %>/kubernetes/stage/billing`
 
 ## Cron Jobs
 An example cron job is specified in [kubernetes/base/cronjob.yml][base-cronjob].
@@ -136,6 +149,5 @@ The deployment only requires the environment variables:
 <!-- Links -->
 [base-cronjob]: ./kubernetes/base/cronjob.yml
 [base-deployment]: ./kubernetes/base/deployment.yml
-[base-deployment-secret]: ./kubernetes/base/deployment.yml#L49-58
 [env-prod]: ./kubernetes/overlays/production/deployment.yml
 [circleci-details]: ./.circleci/README.md

scripts/required-bins.sh

@@ -0,0 +1,11 @@
+# This will pull the contents of a secret from AWS Secret Manager and put it into a Kubernetes secret.
+# Any changes made to the AWS secret should be automatically synced over.
+# An AWS secret for the project should have been created automatically by /scripts/create-db-user.sh in the infrastructure project
+apiVersion: kubernetes-client.io/v1
+kind: ExternalSecret
+metadata:
+  name: <% .Name %>
+spec:
+  backendType: secretsManager
+  dataFrom:
+  - <% .Name %>/kubernetes/stage/<% .Name %>

scripts/setup-stripe-secrets.sh

@@ -7,6 +7,7 @@ patchesStrategicMerge:
 resources:
 - ../../base
 - ingress.yml
+- external-secret.yml
 
 configMapGenerator:
 - name: <% .Name %>-config

templates/README.md

@@ -0,0 +1,11 @@
+# This will pull the contents of a secret from AWS Secret Manager and put it into a Kubernetes secret.
+# Any changes made to the AWS secret should be automatically synced over.
+# An AWS secret for the project should have been created automatically by /scripts/create-db-user.sh in the infrastructure project
+apiVersion: kubernetes-client.io/v1
+kind: ExternalSecret
+metadata:
+  name: <% .Name %>
+spec:
+  backendType: secretsManager
+  dataFrom:
+  - <% .Name %>/kubernetes/prod/<% .Name %>

templates/kubernetes/overlays/dev/external-secret.yml

@@ -7,6 +7,7 @@ patchesStrategicMerge:
 <% end %>
 resources:
 - ../../base
+- external-secret.yml
 <%if eq (index .Params `userAuth`) "yes" %>## userAuth enabled - Oathkeeper proxies to backend instead of ingress
 # - ingress.yml
 <% else %>- ingress.yml<% end %>

templates/kubernetes/overlays/dev/kustomization.yml

@@ -0,0 +1,11 @@
+# This will pull the contents of a secret from AWS Secret Manager and put it into a Kubernetes secret.
+# Any changes made to the AWS secret should be automatically synced over.
+# An AWS secret for the project should have been created automatically by /scripts/create-db-user.sh in the infrastructure project
+apiVersion: kubernetes-client.io/v1
+kind: ExternalSecret
+metadata:
+  name: <% .Name %>
+spec:
+  backendType: secretsManager
+  dataFrom:
+  - <% .Name %>/kubernetes/stage/<% .Name %>