EKS support for prombench by weastel · Pull Request #384 · prometheus/test-infra

weastel · 2020-06-17T20:11:54Z

This PR incorporates the following changes:-

Add Support for rangeVars
EKS commands for creation/deletion of clusters, creation/deletion of nodegroups and resource apply/delete
Temporary documentations for EKS

geekodour

nice work! :) some changes and questions.

infra/README.md

infra/infra.go

prombench/Makefile

prombench/README.md

pkg/provider/provider.go

weastel · 2020-07-06T16:31:03Z

@geekodour Can you please review it again?

weastel · 2020-07-07T20:34:53Z

@geekodour Docmentation added.

geekodour

Great stuff, nice improvements :) Thanks! ✨

Some questions and change suggestions.

prombench/README.md

prombench/manifests/cluster_gke.yaml

infra/infra.go

prombench/Makefile

prombench/docs/eks.md

infra/infra.go

pkg/provider/eks/eks.go

geekodour

Good progress! thanks for the work 😃

Added some suggestions, questions and ideas. :)

prombench/Makefile

prombench/README.md

prombench/manifests/cluster_gke.yaml

prombench/Makefile

prombench/docs/eks.md

prombench/Makefile

geekodour

We're almost nearing finishing this PR, I think a couple of more iterations :) Nice work 👍

Some suggestion, questions and changes.

prombench/Makefile

prombench/docs/eks.md

geekodour · 2020-07-31T05:06:00Z

pkg/provider/eks/eks.go

+// NewEKSClient sets the EKS client used when performing the GKE requests.
+func (c *EKS) NewEKSClient(*kingpin.ParseContext) error {
+	if c.AuthFilename != "" {
+	} else if c.AuthFilename = os.Getenv("AWS_APPLICATION_CREDENTIALS"); c.AuthFilename == "" {


Why do we need to check for AWS_APPLICATION_CREDENTIALS ?

Actually this seems like a good idea: https://docs.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets#using-encrypted-secrets-in-a-workflow

But we'll still be passing the base64 version of the toml credentials file in Github Actions, so we still need to see if the auth data is base64 encoded and create a file to pass to NewSharedCredentials.

test-infra/pkg/provider/gke/gke.go

Lines 92 to 103 in e7f8c93

// Check if auth data is base64 encoded and decode it.

encoded, err := regexp.MatchString("^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==)?$", c.Auth)

if err != nil {

return err

}

if encoded {

auth, err := base64.StdEncoding.DecodeString(c.Auth)

if err != nil {

return errors.Wrap(err, "could not decode auth data")

}

c.Auth = string(auth)

}

prombench/Makefile

pkg/provider/eks/eks.go

geekodour

Thanks for adding the changes! Added some comments to make this PR smaller by extracting out the GKE specific changes to a separate PR :)

infra/README.md

infra/infra.go

prombench/Makefile

prombench/README.md

prombench/Makefile

prombench/docs/eks.md

weastel · 2020-08-15T19:41:00Z

@geekodour Added info command for eks

Waiting for #425 to be merged 😝

pkg/provider/eks/eks.go

geekodour

Looks like this is shaping up nicely, some minor documentation nits :)

prombench/Makefile

prombench/docs/eks.md

geekodour

Nice, almost LGTM :) Few questions and minor changes. 🎇 Please update the PR description with all the new changes and remove the changes that are no longer part of the PR. Also mention about automated subnet ids will add support of funcbench etc.

@krasi-georgiev It'll be awesome if you can take a look at this one :)

pkg/provider/eks/eks.go

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

geekodour · 2020-09-05T15:36:48Z

Merged. 🎉🎊 Thanks for all your work on this PR @weastel !

Now need to rebase #421 to add funcbench support :)

This change adds EKS commands for the creation/deletion of clusters, nodegroups, and resources similar to the existing GKE commands in infra. The documentation for deploying prombench for EKS is also added. The changes in this commit are not sufficient for running funcbench on EKS. Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

weastel force-pushed the eks_support_for_prombench branch 2 times, most recently from e70eca2 to f13f7ea Compare June 19, 2020 09:47

weastel marked this pull request as ready for review June 25, 2020 11:22

geekodour reviewed Jun 26, 2020

View reviewed changes

weastel force-pushed the eks_support_for_prombench branch from 57db3bc to 8aa54a9 Compare July 6, 2020 16:28

geekodour reviewed Jul 12, 2020

View reviewed changes

infra/infra.go Outdated Show resolved Hide resolved

geekodour mentioned this pull request Jul 19, 2020

Refactor Infra, pkg/provider and GKE provider #413

Merged

1 task

geekodour reviewed Jul 19, 2020

View reviewed changes

pkg/provider/eks/eks.go Outdated Show resolved Hide resolved

weastel requested a review from geekodour July 24, 2020 21:20

geekodour reviewed Jul 26, 2020

View reviewed changes

weastel force-pushed the eks_support_for_prombench branch 2 times, most recently from 212bf71 to 118eb65 Compare July 29, 2020 17:08

geekodour reviewed Jul 31, 2020

View reviewed changes

weastel force-pushed the eks_support_for_prombench branch from c1120bc to 5c1e310 Compare August 3, 2020 23:56

geekodour reviewed Aug 4, 2020

View reviewed changes

geekodour reviewed Aug 5, 2020

View reviewed changes

prombench/docs/eks.md Outdated Show resolved Hide resolved

geekodour mentioned this pull request Aug 15, 2020

Re-arrange directory structure and docs for GKE to support multiple Providers #425

Merged

geekodour reviewed Aug 16, 2020

View reviewed changes

pkg/provider/eks/eks.go Outdated Show resolved Hide resolved

weastel force-pushed the eks_support_for_prombench branch 2 times, most recently from 7a302e8 to cbee5c1 Compare August 20, 2020 22:46

geekodour reviewed Aug 21, 2020

View reviewed changes

prombench/Makefile Outdated Show resolved Hide resolved

prombench/Makefile Outdated Show resolved Hide resolved

prombench/docs/eks.md Show resolved Hide resolved

prombench/docs/eks.md Outdated Show resolved Hide resolved

prombench/docs/eks.md Outdated Show resolved Hide resolved

weastel mentioned this pull request Aug 23, 2020

Update prombench workflows to incorporate new naming convention for eks prometheus/prometheus#7838

Merged

weastel requested a review from geekodour August 24, 2020 01:47

geekodour reviewed Aug 24, 2020

View reviewed changes

pkg/provider/eks/eks.go Outdated Show resolved Hide resolved

pkg/provider/eks/eks.go Outdated Show resolved Hide resolved

pkg/provider/eks/eks.go Outdated Show resolved Hide resolved

weastel requested a review from geekodour August 25, 2020 08:59

weastel force-pushed the eks_support_for_prombench branch from 8adf862 to 2615217 Compare August 25, 2020 09:05

weastel added 25 commits September 4, 2020 00:24

Add readme.md changes

7ade54c

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Fix NewK8sProvider

1876850

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Change instance type of eks so it matches gke configurations

fd57eed

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Fix PR to solve test based issues

604e893

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Chnage as per review

f22d10a

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Run genflagdocs.sh to pass test

f9157f7

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Add documentation for eks

b7cbe4c

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Make makefile more readable

2bb46e5

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

remove rangeVars and change documentation for the same and minor fixes

f30a228

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Run docs cmd

2b291e6

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Fix go mod and update eks.go file

0db7285

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Update documentations

ed35226

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

CHnage toml based credentials to yaml based

9a2afdd

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

CHnage toml based credentials to yaml based

1ad5384

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Fix formatting issues

c9e343f

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Add docs

4238719

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Add info command for eks and documentational changes

b6ce4e7

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Remove unnecessary debugging

0032526

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Rebased with respect to gke specific and added info command

e882504

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Fix rebased errors

6872fe1

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Update docs changes and remove uneccessary debugging

77c7248

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Add documentational chages

91eaacd

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Change logging for log.Fatal to fmt.Error

aa8fd1d

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Add aws iam based authentication

3dfc2ae

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

Add SERVICEACCOUNT_CLIENT_EMAIL as example@example.com by default

44870e4

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

weastel force-pushed the eks_support_for_prombench branch from 0783336 to 44870e4 Compare September 3, 2020 19:02

Display hostname for eks ingress service

a13d384

Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>

geekodour merged commit c162e41 into prometheus:master Sep 5, 2020

	// Check if auth data is base64 encoded and decode it.
	encoded, err := regexp.MatchString("^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{3}=\|[A-Za-z0-9+/]{2}==)?$", c.Auth)
	if err != nil {
	return err
	}
	if encoded {
	auth, err := base64.StdEncoding.DecodeString(c.Auth)
	if err != nil {
	return errors.Wrap(err, "could not decode auth data")
	}
	c.Auth = string(auth)
	}

Conversation

weastel commented Jun 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

geekodour left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

weastel commented Jul 6, 2020

Uh oh!

weastel commented Jul 7, 2020

Uh oh!

geekodour left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

geekodour left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

geekodour left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

geekodour Jul 31, 2020

Choose a reason for hiding this comment

Uh oh!

geekodour Jul 31, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

geekodour left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

weastel commented Aug 15, 2020

Uh oh!

Uh oh!

geekodour left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

weastel commented Jun 17, 2020 •

edited

Loading

geekodour left a comment •

edited

Loading