Merge pull request #2 from aws-samples/feat-register-event

feat: Update solution to register models using CloudWatch events

Author: brightsparc

OPERATIONS.md

@@ -4,8 +4,7 @@ Having created the A/B Testing Deployment Pipeline, this operations manual provi
 
 ## A/B Testing for Machine Learning models
 
-Successful A/B Testing for machine learning models requires measuring how effective predictions are against end users.  
-It is important to be able to identify users consistently and be able to attribute success actions against the model predictions back to users.
+Successful A/B Testing for machine learning models requires measuring how effective predictions are against end users.   It is important to be able to identify users consistently and be able to attribute success actions against the model predictions back to users.
 
 ### Conversion Metrics
 
@@ -46,7 +45,11 @@ The configuration is stored in the CodeCommit source repository by stage name eg
 * `epsilon` - The epsilon parameter used by the `EpsilonGreedy` strategy.
 * `warmup` - The number of invocations to warm up before applying the strategy.
 
-In addition to the above, you must specify the `champion` and `challenger` model variants for the deployment.
+In addition to the above, you must specify the `champion` and `challenger` model variants for the deployment.  
+
+These will be loaded from the two Model Package Groups in the registry that include the project name and suffixed with `champion` or `challenger` for example project name `ab-testing-pipeline` these model package groups in the sample notebook:
+
+![\[Model Registry\]](docs/ab-testing-pipeline-model-registry.png)
 
 **Latest Approved Versions**
 

README.md

@@ -146,7 +146,7 @@ Follow are a list of context values that are provided in the `cdk.json`, which c
 |---------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------|
 | `api_name`                | The API Gateway Name                                                                                                                                            | "ab-testing"                       |
 | `stage_name`              | The stage namespace for resource and API Gateway path                                                                                                           | "dev"                              |
-| `endpoint_filter`         | A prefix to filter which Amazon SageMaker endpoints the API can invoke                                                                                          | "*"                                |
+| `endpoint_prefix`         | A prefix to filter which Amazon SageMaker endpoints the API can invoked.                                                                                        | ""                                 |
 | `api_lambda_memory`       | The [lambda memory](https://docs.aws.amazon.com/lambda/latest/dg/configuration-memory.html) allocation for API endpoint.                                        | 768                                |
 | `api_lambda_timeout`      | The lambda timeout for the API endpoint.                                                                                                                        | 10                                 |
 | `metrics_lambda_memory`   | The [lambda memory](https://docs.aws.amazon.com/lambda/latest/dg/configuration-memory.html) allocated for metrics processing Lambda                             | 768                                |
@@ -164,9 +164,7 @@ Run the following command to deploy the API and testing infrastructure, optional
 cdk deploy ab-testing-api
 ```
 
-This stack will ask you to confirm any changes, and output the `RegisterLambda` which you will provide to the MLOps Project, and the `ApiEndpoint` which you will provide to the A/B Testing sample notebook.
-
-Amazon SageMaker Studio projects will be granted access to invoke the Register Lambda, so if you are seeing errors running the above command ensure you have [Enable SageMaker project templates for Studio users](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-projects-studio-updates.html).
+This stack will ask you to confirm any changes, and output the `ApiEndpoint` which you will provide to the A/B Testing sample notebook.
 
 ## Create the SageMaker MLOps Project Template
 
@@ -276,7 +274,6 @@ On the Create project page, SageMaker templates is chosen by default. This optio
   - The project name must have 32 characters or fewer.
 10. In the Project template parameters, provide the *Repository Name** you created previously eg:
   - For **StageName**, enter `dev` 
-  - For **RegisterLambda**, enter the `RegisterLambda` output from the `ab-testing-api` stack
   - For **CodeCommitSeedBucket**, enter the `CodeCommitSeedBucket` output from the `ab-testiing-service-catalog` stack
   - For **CodeCommitSeedKey**, enter the `CodeCommitSeedKey` output from the `ab-testiing-service-catalog` stack
 11. Choose Create project.

ab-testing-pipeline.yml

@@ -23,10 +23,6 @@ Parameters:
     Type: String
     Description: The optional s3 seed key
     MinLength: 1
-  RegisterLambda:
-    Type: String
-    Description: The AWS Lambda to invoke when registering this model
-    MinLength: 1
 Resources:
   CodeRepo:
     Type: AWS::CodeCommit::Repository
@@ -55,8 +51,6 @@ Resources:
         - Key: sagemaker:project-name
           Value:
             Ref: SageMakerProjectName
-    Metadata:
-      aws:cdk:path: ab-testing-pipeline/CodeRepo
   CdkBuild455F642E:
     Type: AWS::CodeBuild::Project
     Properties:
@@ -131,76 +125,6 @@ Resources:
             - Ref: SageMakerProjectName
             - -cdk-
             - Ref: StageName
-    Metadata:
-      aws:cdk:path: ab-testing-pipeline/CdkBuild/Resource
-  RegisterBuild8D2EA4DC:
-    Type: AWS::CodeBuild::Project
-    Properties:
-      Artifacts:
-        Type: CODEPIPELINE
-      Environment:
-        ComputeType: BUILD_GENERAL1_SMALL
-        EnvironmentVariables:
-          - Name: SAGEMAKER_PROJECT_NAME
-            Type: PLAINTEXT
-            Value:
-              Ref: SageMakerProjectName
-          - Name: STAGE_NAME
-            Type: PLAINTEXT
-            Value:
-              Ref: StageName
-          - Name: REGISTER_LAMBDA
-            Type: PLAINTEXT
-            Value:
-              Ref: RegisterLambda
-        Image: aws/codebuild/standard:1.0
-        ImagePullCredentialsType: CODEBUILD
-        PrivilegedMode: false
-        Type: LINUX_CONTAINER
-      ServiceRole:
-        Fn::Join:
-          - ""
-          - - "arn:"
-            - Ref: AWS::Partition
-            - ":iam::"
-            - Ref: AWS::AccountId
-            - :role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
-      Source:
-        BuildSpec: >-
-          {
-            "version": "0.2",
-            "phases": {
-              "build": {
-                "commands": [
-                  "python register.py > output.txt"
-                ]
-              }
-            },
-            "artifacts": {
-              "files": [
-                "output.txt"
-              ]
-            },
-            "environment": {
-              "buildImage": {
-                "type": "LINUX_CONTAINER",
-                "defaultComputeType": "BUILD_GENERAL1_SMALL",
-                "imageId": "aws/codebuild/amazonlinux2-x86_64-standard:3.0",
-                "imagePullPrincipalType": "CODEBUILD"
-              }
-            }
-          }
-        Type: CODEPIPELINE
-      EncryptionKey: alias/aws/s3
-      Name:
-        Fn::Join:
-          - ""
-          - - sagemaker-
-            - Ref: SageMakerProjectName
-            - -register-
-            - Ref: StageName
-    Metadata:
-      aws:cdk:path: ab-testing-pipeline/RegisterBuild/Resource
   S3Artifact80610462:
     Type: AWS::S3::Bucket
     Properties:
@@ -215,8 +139,6 @@ Resources:
             - Ref: AWS::Region
     UpdateReplacePolicy: Delete
     DeletionPolicy: Delete
-    Metadata:
-      aws:cdk:path: ab-testing-pipeline/S3Artifact/Resource
   PipelineC660917D:
     Type: AWS::CodePipeline::Pipeline
     Properties:
@@ -292,22 +214,6 @@ Resources:
               Name: SageMaker_CFN_Deploy
               RunOrder: 1
           Name: Deploy
-        - Actions:
-            - ActionTypeId:
-                Category: Build
-                Owner: AWS
-                Provider: CodeBuild
-                Version: "1"
-              Configuration:
-                ProjectName:
-                  Ref: RegisterBuild8D2EA4DC
-              InputArtifacts:
-                - Name: Artifact_Source_CodeCommit_Source
-              Name: Register_Build
-              OutputArtifacts:
-                - Name: Artifact_Register_Register_Build
-              RunOrder: 1
-          Name: Register
       ArtifactStore:
         Location:
           Ref: S3Artifact80610462
@@ -319,8 +225,6 @@ Resources:
             - Ref: SageMakerProjectName
             - -pipeline-
             - Ref: StageName
-    Metadata:
-      aws:cdk:path: ab-testing-pipeline/Pipeline/Resource
   DeployRule0F8E909D:
     Type: AWS::Events::Rule
     Properties:
@@ -369,8 +273,6 @@ Resources:
                 - ":iam::"
                 - Ref: AWS::AccountId
                 - :role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
-    Metadata:
-      aws:cdk:path: ab-testing-pipeline/DeployRule/Resource
   CodeRule663E3DC0:
     Type: AWS::Events::Rule
     Properties:
@@ -430,14 +332,10 @@ Resources:
                 - ":iam::"
                 - Ref: AWS::AccountId
                 - :role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
-    Metadata:
-      aws:cdk:path: ab-testing-pipeline/CodeRule/Resource
   CDKMetadata:
     Type: AWS::CDK::Metadata
     Properties:
       Analytics: v2:deflate64:H4sIAAAAAAAAE0WN0YrCMBBFv8X3OCIVln0T/YFSvyBOR3ZskynJRCkh/742Lfh0L3M5Z47we4Lj7mzfcY/9cMgogSDf1OJgrg/f2mAdKQXTUZQUkMxVfNSQUJf9e334z9CzsvhiFl1m6yC3MjLOVVVbMSg9oTjHClUwSWSVMNfhnnjsPxRPNLKnNsiT1k9bLSY2kC8JB6rnta3WaaO+fAW3Xgy9yGuE3KWxLkuWUkw765/4QwM/0OyekXkfkld2BN2a//2ekvEmAQAA
-    Metadata:
-      aws:cdk:path: ab-testing-pipeline/CDKMetadata/Default
     Condition: CDKMetadataAvailable
 Conditions:
   CDKMetadataAvailable:

cdk.json

@@ -11,7 +11,7 @@
     "log_level": "INFO",
     "api_name": "ab-testing",
     "stage_name": "dev",
-    "endpoint_filter": "*",
+    "endpoint_prefix": "",
     "api_lambda_memory": 768,
     "api_lambda_timeout": 60,
     "metrics_lambda_memory": 768,

deployment_pipeline/README.md

@@ -10,16 +10,18 @@ This deployment pipeline contains a few stages.
 1. **Source**: Pull the latest deployment configuration from AWS CodeCommit repository.
 1. **Build**: AWS CodeBuild job to create the AWS CloudFormation template for deploying the endpoint.
     - Query the Amazon SageMaker project to get the top approved models.
-    - Use the AWS CDK to create a CFN stack with multiple endpoint variants.
-    - Create a `register.json` file that contains the target SageMaker endpoint and A/B testing strategy.
-2. **Deploy**: Run the AWS CloudFormation stack to create/update the SageMaker endpoint.
-3. **Register**: Call the RegisterAPI for the endpoint to create/clear the A/B testing metrics.
+    - Use the AWS CDK to create a CFN stack to deploy multi-variant SageMaker Endpoint.
+2. **Deploy**: Run the AWS CloudFormation stack to create/update the SageMaker endpoint, tagged with properties based on configuration:
+    - `ab-testing:enabled` equals `true`
+    - `ab-testing:strategy` is one `WeightedSampling`, `EpslionGreedy`, `UCB1` or `ThompsonSampling`.
+    - `ab-testing:epsilon` is parameters for `EpslionGreedy` strategy, defaults to `0.1`.
+    - `ab-testing:warmup` the number of invocations to warmup with `WeightedSampling` strategy, defaults to `0`.
 
 ![\[AWS CodePipeline\]](../docs/ab-testing-pipeline-code-pipeline.png)
 
 ## Testing
 
-Once you have created a SageMaker Project, you can test the **Build** and **Register** stages locally by setting some environment variables, and running the commands found in the `buildspec` defined in the pipeline.
+Once you have created a SageMaker Project, you can test the **Build** stage and **Register** events locally by setting some environment variables.
 
 ### Build Stage
 
@@ -32,12 +34,11 @@ export STAGE_NAME="dev"
 cdk synth
 ```
 
-### Register Stage
+### Register
 
 Export the environment variable for the `REGISTER_LAMBDA` created as part of the `ab-testing-api` stack, then run `register.py` file.
 
 ```
 export REGISTER_LAMBDA="<<register_lambda>>"
 python register.py
 ```
-

deployment_pipeline/app.py

@@ -42,6 +42,13 @@
 with open(f"{stage_name}-config.json", "r") as f:
     j = json.load(f)
     deployment_config = DeploymentConfig(**j)
+    # Append tags for ab-testing
+    tags += [
+        core.CfnTag(key="ab-testing:enabled", value="true"),
+        core.CfnTag(key="ab-testing:strategy", value=deployment_config.strategy),
+        core.CfnTag(key="ab-testing:epsilon", value=str(deployment_config.epsilon)),
+        core.CfnTag(key="ab-testing:warmup", value=str(deployment_config.warmup)),
+    ]
 
 sagemaker = SageMakerStack(
     app,

deployment_pipeline/infra/deployment_config.py

@@ -42,6 +42,7 @@ def __init__(
         instance_type: str = "ml.t2.medium",
         strategy: str = "ThompsonSampling",
         warmup: int = 0,
+        epsilon: float = 0.1,
     ):
         self.stage_name = stage_name
         # Provide either the challenger variant count, or specific champion/challenger config
@@ -74,4 +75,5 @@ def __init__(
             self.challenger_variant_config = None
         self.strategy = strategy
         self.warmup = warmup
+        self.epsilon = epsilon
         super().__init__(instance_count, instance_type)

deployment_pipeline/infra/model_registry.py

@@ -59,7 +59,7 @@ def get_latest_approved_packages(
             # Return error if no packages found
             if len(model_packages) == 0 and creation_time_after is None:
                 error_message = (
-                    f"No latest packages found for: {model_package_group_name}"
+                    f"No approved packages found for: {model_package_group_name}"
                 )
                 logger.error(error_message)
                 raise Exception(error_message)
@@ -121,9 +121,7 @@ def get_versioned_approved_packages(
 
             # Return error if no packages found
             if len(model_packages) == 0:
-                error_message = (
-                    f"No versioned packages found for: {model_package_group_name}"
-                )
+                error_message = f"No approved packages found for: {model_package_group_name} and versions: {model_package_versions}"
                 logger.error(error_message)
                 raise Exception(error_message)
 

deployment_pipeline/prod-config.json

@@ -1,7 +1,8 @@
 {
     "stage_name": "prod",
-    "strategy": "ThompsonSampling",
+    "strategy": "EpsilonGreedy",
     "warmup": 100,
+    "epsilon": 0.1,
     "instance_count": 2,
     "instance_type": "ml.c5.large",
     "champion_variant_config": {

deployment_pipeline/register.py

@@ -24,7 +24,20 @@
 # Get the config and include with endpoint to register this model
 with open(f"{stage_name}-config.json", "r") as f:
     j = json.load(f)
-    event = json.dumps({"endpoint_name": endpoint_name, **j})
+    event = json.dumps({
+        'source': 'aws.sagemaker',
+        'detail-type': 'SageMaker Endpoint State Change',
+        'detail': {
+            'EndpointName': endpoint_name,
+            'EndpointStatus': 'IN_SERVICE',
+            'Tags': {
+                'ab-testing:enabled': 'true',
+                'ab-testing:strategy': j.get('strategy', 'ThompsonSampling'),
+                'ab-testing:epsilon': str(j.get('epsilon', 0.1)),
+                'ab-testing:warmup': str(j.get('warmup', 0)),
+            }
+        }
+    })
     response = lambda_client.invoke(
         FunctionName=register_lambda,
         InvocationType="RequestResponse",