Multiple Cluster Support

[icereval]

Allow reuse of a single AWS account while allowing environments to deploy multiple clusters on a common network stack with separate Kubernetes versions.

Sample: https://github.com/icereval/kubit-multi-sample

e.g.

prod/terraform.tfvars

domain_names = { external = "k8s.example.com", internal = "prod.example.local" }
environment_name = "prod"

prod/clusters.tf

module "cluster1" {
  source = "./cluster1"
  assets_bucket_name = "${var.assets_bucket_name}"
  ...
}

...

test/terraform.tfvars

domain_names = { external = "test.example.com", internal = "test.example.local" }
environment_name = "test"

test/clusters.tf

module "cluster1" {
  source = "./cluster1"
  ...
}

module "cluster2" {
  source = "./cluster2"
  ...
}

[mgoodness]

@icereval have you confirmed that the KubernetesCluster tag isn't required on the resources from which you'd remove it? Looks like route tables, subnets, and internet gateway. If the AWS cloud provider in K8s doesn't need them, 👍 for getting rid of unnecessary tags.

Looks like the bulk of the changes are around prepending the environment to the cluster name (i.e. test-kubit). Couldn't you effectively achieve that today by setting the cluster name to test-kubit?

[icereval]

Agreed on all points, testing last night it was clear I was misreading the tag filter logic.

Will address this evening, thanks for the feedback.

[icereval]

I've cleaned up as many references as I could to environment_name and I believe the ones I've left are justified. (removed environment_name all together)

Removed KubernetesCluster tags from RouteTables and Internet Gateway as far as I can tell there is no need but could certainly be enlightened... Spun up a cluster and tested ELB and Ingress, all seems to work great.

KOPS Ingress Controller 👍 :

kubectl --namespace kube-system apply --filename "https://raw.githubusercontent.com/kubernetes/kops/master/addons/ingress-nginx/v1.4.0.yaml"

Test Echo Server - https://github.com/kubernetes/kops/tree/master/addons/ingress-nginx

I'm still working on an example app which should flush out any further deficiencies in the network stack modules, hope to get some time tomorrow.

[mgoodness]

Was reviewing the K8s 1.6 feature tracking sheet and discovered an issue that may affect your work. Looks like SIG-AWS plans to change tags such that resources can be shared among multiple clusters.

kubernetes/enhancements#173

Wait and see?

[icereval]

Certainly would help with sharing core subnets/route tables, though you can have 200 of them per vpc so its not terrible to make a few extra.

Mainly I'd imagine most want to avoid duplicating costly (and limited) NAT Gateways when using private subnets.

A sample implementation: https://github.com/icereval/kubit-multi-sample