Pre-Postprocessing feature seems to not work anymore.

[flacout]

Bug Description
Hello,
I built the image from the latest commit: 6a51a60
I pushed it in ECR and deployed a multi-model Sagemaker endpoint. The endpoint is working well for a simple model, but when I try to upload a model that contain code/inference.py, the new model cannot load the input and output handlers and therefore return an input error when I call the model.

I tried to built a previous version: commit: 3bab56e "change: update MME Pre/Post-Processing model and script paths (#153)" when the feature of pre-postprocessing was first introduce to the container, and this time it works perfectly. I can deploy models with code/inference.py, install new pip packages with the code/requirement.txt and add local python modules in the code/lib folder.

It seems to me that this feature of having models with independent pre-postprocessing capability was lost somehow in the latest commits of this repository, is this something intentional, or it was an error of manipulation, or am I missing something?

To reproduce
The model directory structure I used looks like this, following the readme guidelines:

|--[model_version_number]
    |--assets
    |--variables
    |--saved_model.pb
|--code
    |--lib
        |--external_module
    |--inference.py

For the inference.py I used the implement of handler instead of the input_handler and output_handler pair.

Thanks
Fabrice

[jinpengqi]

Hey,

Could you try to use model dir like this and see if it worked:

|--[model1]
  |--[model_version_number]
      |--assets
      |--variables
      |--saved_model.pb
|--code
    |--inference.py
    |--requirements.txt

Thanks

[richardimms]

Changing the model directory from OP to the suggested:

|--[model_version_number]
    |--assets
    |--variables
    |--saved_model.pb
|--code
    |--lib
        |--external_module
    |--inference.py

to this:

|--[model1]
  |--[model_version_number]
      |--assets
      |--variables
      |--saved_model.pb
|--code
    |--inference.py
    |--requirements.txt

Results in: No versions of servable {id} found under base path /opt/ml/models/{id}/model

[flacout]

I confirm @richardimms comment.
Deploying a model with structure:

|--[model1]
  |--[model_version_number]
      |--assets
      |--variables
      |--saved_model.pb
|--code
    |--inference.py
    |--requirements.txt

Lead to an error after invoking the endpoint:

response = runtime_sm_client.invoke_endpoint(EndpointName='multi-model-endpoint',
                                    ContentType='application/json',
                                    TargetModel='model1.tar.gz',
                                    Body=json.dumps(data))

Error:
Could not find base path /opt/ml/models/55dfd27a9d81013319c5f5d01600e748/model for servable 55dfd27a9d81013319c5f5d01600e748

[maheshpatil2021]

Any updates on this issue. This is a blocker for using Multi-Model containers in production.

[richardimms]

Seconded, any update on this @jinpengqi?

[jinpengqi]

Sry for the late. Let me help ping related oncall for a dive deeper here.

[maheshpatil2021]

Any updates? Would be good to get an ETA so we can plan accordingly. @jinpengqi

[jinpengqi]

Hey,

I have cut a tt to related team for this issue, they will take a look and give updates here.

[richardimms]

@jinpengqi any further update on this?

[hongshanli23]

@flacout and @richardimms can you post the code you used to create the model.

[richardimms]

Hello @hsl89 do you mean the actual saved model from Tensorflow, or the code to construct the model package we upload to S3?

If it's the saved model from Tensorflow then I just use the export_saved_model function provided by the estimator here: https://github.com/tensorflow/estimator/blob/master/tensorflow_estimator/python/estimator/estimator.py#L659-L736

The serving function is based on TensorFlow Ranking and is: https://github.com/tensorflow/ranking/blob/e0c009e378ac3b3a79fe260e90b20a93aca6901f/tensorflow_ranking/python/data.py#L1091-L1135

Sorry I can't share exact snippets, but it follows those two patterns.

It's probably worth mentioning, I can use Sagemaker single model endpoint with a primary container, using the same model.

[flacout]

@flacout and @richardimms can you post the code you used to create the model.

@hsl89
Here is the code to create the multi-model endpoint on sagemaker:

import sagemaker
import boto3
import json
import numpy as np

#create the model
sm_client = boto3.client('sagemaker')
role = sagemaker.get_execution_role()

image_name = # image created from "sagemaker-tensorflow-serving-container" repository
container = { 'Image':        image_name,
              'ModelDataUrl': 's3://somebucket/multi-models/',
              'Mode':         'MultiModel'
            }
response = sm_client.create_model(
              ModelName        = 'multi-model',
              ExecutionRoleArn = role,
              Containers       = [container])

# create endpoint config
response = sm_client.create_endpoint_config(
    EndpointConfigName = "multi-model-config",
    ProductionVariants=[{
        'InstanceType':        'ml.m4.xlarge',
        'InitialInstanceCount': 1,
        'ModelName':            'multi-model',
        'VariantName':          'AllTraffic'}])

# create endpoint
response = sm_client.create_endpoint(
              EndpointName       = 'multi-model-endpoint',
              EndpointConfigName = "multi-model-config")

Here is the code to package and deploy one tensorflow model:

# train a tensorflow model
model = # some keras model

# save model
model_version = '1'
export_dir = 'export/Servo/' + model_version
tf.saved_model.save(model, export_dir)

# Create archive
model_name = "model_1.tar.gz"
with tarfile.open(model_name, mode='w:gz') as archive:
    archive.add(f'export/Servo/{model_version}', arcname=model_version, recursive=True)
    archive.add(f'inference.py', arcname="code/inference.py")
    archive.add(f'postprocessing.py', arcname="code/lib/local_module.py")

# upload model-archive to s3
s3.upload_file(Filename=model_name, 
               Bucket='bucket_name', 
               Key=os.path.join("multi_model_folder", model_name))


# test the model is deployed
runtime_sm_client = boto3.client('runtime.sagemaker')
endpoint_name = 'tf-multi-model-endpoint'
signal = [1,2,3,4,4]
data = {"instances": signal.tolist()}
response = runtime_sm_client.invoke_endpoint(EndpointName=endpoint_name,
                                    ContentType='application/json',
                                    TargetModel=model_name,
                                    Body=json.dumps(data))

Here is the inference.py code:

def handler(data, context):
    if context.request_content_type == 'application/json':
        input_data = data.read().decode('utf-8')
        input_data = json.loads(input_data)
        
        processed_input, signal = _process_input(input_data)
        response = requests.post(context.rest_uri, data=processed_input)
    
    else:
        raise ValueError('{{"error": "unsupported content type {}"}}'.format(
            context.request_content_type or "unknown"))
    
    return _process_output(input_data, signal, response, context)


def _process_input(input_data):
    #do preprocessing
    signal = input_data['signal']
    data = {"instances": signal}

    return json.dumps(data), signal 
    

def _process_output(input_data, signal, response, context):
    if response.status_code != 200:
        raise ValueError(response.content.decode('utf-8'))
    
    response_content_type = context.accept_header
    prediction = response.content.decode('utf-8')
    prediction = np.array(json.loads(prediction)["predictions"]) 
    # do postprocessing
    return json.dumps({"predictions": output}), response_content_type

Let me know if you need some clarification about the code.
I hope the problem can be resolved promptly.
Thanks

[hongshanli23]

Hey @richardimms, I was looking for the code you used to create the multi-model endpoint. the code provided by @flacout is what I am looking for. will post updates once I found problems or workaround