Merge pull request #3 from lambda-feedback/579-adding-preview-command

579 adding preview command

Author: Loumstar

.gitignore

@@ -130,3 +130,6 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+# VSCode
+.vscode

Dockerfile

@@ -24,5 +24,5 @@ COPY handler.py ./app/
 COPY tests/*.py ./app/tests/
 COPY tools/*.py ./app/tools/
 
-ENV REQUEST_SCHEMA_URL https://raw.githubusercontent.com/lambda-feedback/request-response-schemas/master/request.json
-ENV RESPONSE_SCHEMA_URL https://raw.githubusercontent.com/lambda-feedback/request-response-schemas/master/responsev2.json
+# Request-response-schemas repo/branch to use for validation
+ENV SCHEMAS_URL=https://raw.githubusercontent.com/lambda-feedback/request-response-schemas/579-adding-preview-command

README.md

@@ -1,20 +1,25 @@
 # BaseEvalutionFunctionLayer
-Base docker image for evaluation functions coded in python. This layer cannot function alone, it needs to be extended in a specific way by evaluation function it supports. 
 
-This layer encompases all the behaviour that is universal to all evaluation functions: 
+Base docker image for evaluation functions coded in python. This layer cannot function alone, it needs to be extended in a specific way by the evaluation function it supports.
+
+This layer implements the behaviour common to all evaluation functions:
+
 - Request and response schema validation
 - Unit testing setup
 - Function commands:
-  - `eval`: calls the function in the user-defined `evaluation.py` file
+  - `eval`: calls the evaluation function in the user-defined `evaluation.py` file.
+  - `preview`: calls the preview function in the user-defined `preview.py` file.
   - `healthcheck`: runs all unittests for schema testing as well as user-defined tests in `evaluation_tests.py`.
-  - `docs`: returns the `docs.md` user-defined file. 
+  - `docs`: returns the `docs.md` user-defined file.
 
-*Note: user-defined files are those provided by the evaluation function code meant to extend this layer*
+_Note: user-defined files are those provided by the evaluation function code meant to extend this layer_
 
 ## Behaviour and Usage
-Commands as passed in 'command' header from each request. By default (if not header is present), the function will run the `eval` command. 
+
+Commands as passed in 'command' header from each request. By default (if no header is present), the function will run the `eval` command.
 
 ## Requirements from the superseding layer
+
 This function makes references to files and functions which don't exist yet in this layer - those need to be provided by the superseding layer. They're shown here in the way a dockerfile might be extending it.
 
 ```dockerfile
@@ -41,8 +46,8 @@ RUN chmod 755 $(find . -type d)
 CMD [ "/app/app.handler" ]
 ```
 
-
 ### Operating Container Structure
+
 Since this is only just a base layer for eval functions, the repo's file structure won't match the file structure inside the built image, which can get confusing at times. This is what the `/app/` directory (where all our data is contained) will look like for an operational function:
 
 ```
@@ -55,20 +60,25 @@ Since this is only just a base layer for eval functions, the repo's file structu
    |  |____requests.py
    |  |____responses.py
    |  |____handling.py
+   |  |____commands.py
+   |  |____docs.py
+   |  |____parse.py
    |____tools
    |  |______init__.py
+   |  |____commands.py
    |  |____validate.py
    |  |____docs.py
    |  |____parse.py
    |  |____healthcheck.py
+   |  |____utils.py
    |____docs.md
    |____handler.py
    |____evaluation_tests.py
    |____evaluation.py
 ```
 
-
 ## Dev Notes
+
 Can run the following command to look around the container of a running function
 
 ```bash
@@ -80,5 +90,3 @@ From a container which exposes port 8080 to the real port 9000, requests can be
 ```
 http://localhost:9000/2015-03-31/functions/function/invocations
 ```
-
-

__init__.py

@@ -1 +1 @@
-from .handler import handler
+from .handler import handler  # noqa

handler.py

@@ -1,213 +1,77 @@
-from .evaluation import evaluation_function
-
-from .tools import docs, parse
-from .tools.healthcheck import healthcheck
-from .tools import validate as v
-
 from evaluation_function_utils.errors import EvaluationException
-"""
-    Command Handler Functions.
-"""
-
-
-def handle_unknown_command(command):
-    """ 
-    Function to create the response when the command is unknown.
-    ---
-    This function does not handle any of the request body so it is neither parsed or
-    validated against a schema. Instead, a simple message is returned telling the 
-    requestor that the command isn't allowed.
-    """
-    return {"error": {"message": f"Unknown command '{command}'."}}
 
+from .tools import commands, docs, validate
+from .tools.parse import ParseError
+from .tools.utils import ErrorResponse, HandlerResponse, JsonType, Response
+from .tools.validate import ResBodyValidators, ValidationError
 
-def handle_healthcheck_command():
-    """
-    Function to create the response when commanded to perform a healthcheck.
-    ---
-    This function does not handle any of the request body so it is neither parsed or
-    validated against a schema.
-    """
-    return {"command": "healthcheck", "result": healthcheck()}
 
+def handle_command(event: JsonType, command: str) -> HandlerResponse:
+    """Switch case for handling different command options.
 
-def handle_eval_command(event):
-    """
-    Function to create the response when commanded to evaluate an answer.
-    ---
-    This function attempts to parse the request body, performs schema validation and
-    attempts to run the evaluation function on the given parameters.
+    Args:
+        event (JsonType): The AWS Lambda event recieved by the handler.
+        command (str): The name of the function to invoke.
 
-    If any of these fail, a message is returned and an error field is passed if more
-    information can be provided.
+    Returns:
+        HandlerResponse: The response object returned by the handler.
     """
-    body, parse_error = parse.parse_body(event)
-
-    if parse_error:
-        return {"error": parse_error}
+    # No validation of the doc commands.
+    if command in ("docs-dev", "docs"):
+        return docs.dev()
 
-    request_error = v.validate_request(body)
-
-    if request_error:
-        return {"error": request_error}
+    elif command == "docs-user":
+        return docs.user()
 
-    response = body["response"]
-    answer = body["answer"]
-    params = body.get("params", dict())
+    elif command in ("eval", "grade"):
+        response = commands.evaluate(event)
+        validator = ResBodyValidators.EVALUATION
 
-    try:
-        result = evaluation_function(response, answer, params)
+    elif command == "preview":
+        response = commands.preview(event)
+        validator = ResBodyValidators.PREVIEW
 
-    # Catch the custom EvaluationException (from evaluation_function_utils) first
-    except EvaluationException as e:
-        return {"error": e.error_dict}
-
-    except Exception as e:
-        return {
-            "error": {
-                "message":
-                "An exception was raised while executing the evaluation function.",
-                "detail": str(e) if str(e) != "" else repr(e)
-            }
-        }
-
-    # If a list of "cases" wasn't provided, we don't have any other way to get feedback
-    cases = params.get("cases", [])
-    if len(cases) == 0:
-        return {"command": "eval", "result": result}
-
-    # Determine what feedback to provide based on cases
-    matched_case, warnings = feedback_from_cases(response, params, cases)
-    if matched_case:
-        result["feedback"] = matched_case["feedback"]
-        result["matched_case"] = matched_case["id"]
-
-        # Override is_correct provided by the original block by the case 'mark'
-        if "mark" in matched_case:
-            result["is_correct"] = bool(int(matched_case["mark"]))
-
-    # Add warnings out output if any were encountered
-    if len(warnings) != 0:
-        result["warnings"] = warnings
-
-    return {"command": "eval", "result": result}
-
-
-def feedback_from_cases(response, params, cases):
-    """ 
-    Attempt to find the correct feedback from a list of cases.
-    Returns a matched 'case' (the full object), and optional list of warnings
-    """
+    elif command == "healthcheck":
+        response = commands.healthcheck()
+        validator = ResBodyValidators.HEALTHCHECK
+    else:
+        response = Response(
+            error=ErrorResponse(message=f"Unknown command '{command}'.")
+        )
+        validator = ResBodyValidators.EVALUATION
 
-    # A list of "cases" was provided, try matching to each of them
-    matches = []
-    warnings = []
-    eval_function_feedback = []
-    for i, case in enumerate(cases):
-        # Validate the case block has an answer and feedback
-        if 'answer' not in case:
-            warnings += [{"case": i, "message": "Missing answer field"}]
-            continue
-
-        if 'feedback' not in case:
-            warnings += [{"case": i, "message": "Missing feedback field"}]
-            continue
-
-        # Merge current evaluation params with any specified in case
-        case_params = case.get('params', {})
-
-        # Run the evaluation function based on this case's answer
-        try:
-            res = evaluation_function(response, case.get('answer'), {
-                **params,
-                **case_params
-            })
-
-        except EvaluationException as e:
-            warnings += [{"case": i, **e.error_dict}]
-            continue
-
-        except Exception as e:
-            warnings += [{
-                "case": i,
-                "message":
-                "An exception was raised while executing the evaluation function.",
-                "detail": str(e) if str(e) != "" else repr(e)
-            }]
-            continue
-
-        # Function should always return an 'is_correct' if no errors were raised
-        if not 'is_correct' in res:
-            warnings += [{
-                "case": i,
-                "message": "is_correct missing from function output"
-            }]
-            continue
-
-        # This case matches the response, add it's index to the list of matches
-        if res.get('is_correct') == True:
-            matches += [i]
-            eval_function_feedback += [res.get("feedback","")]
-
-    if len(matches) == 0:
-        return None, warnings
-
-    # Select the matched case
-    matched_case = cases[matches[0]]
-    matched_case['id'] = matches[0]
-    if not matched_case["params"].get("override_eval_feedback",False):
-        separator = "<br />" if len(eval_function_feedback[0]) > 0 else ""
-        matched_case.update({"feedback": matched_case.get("feedback","")\
-                                        +separator\
-                                        +eval_function_feedback[0]})
-
-    if len(matches) == 1:
-        # warnings += [{"case": matches[0]}]
-        return matched_case, warnings
+    validate.body(response, validator)
 
-    else:
-        s = ', '.join([str(m) for m in matches])
-        warnings += [{
-            "message":
-            f"Cases {s} were matched. Only the first one's feedback was returned"
-        }]
-        return matched_case, warnings
+    return response
 
 
-"""
-    Main Handler Function
-"""
+def handler(event: JsonType, _: JsonType = {}) -> HandlerResponse:
+    """Main function invoked by AWS Lambda to handle incoming requests.
 
+    Args:
+        event (JsonType): The AWS Lambda event received by the gateway.
 
-def handler(event, context={}):
-    """
-    Main function invoked by AWS Lambda to handle incoming requests.
-    ---
-    This function invokes the handler function for that particular command and returns
-    the result. It also performs validation on the response body to make sure it follows
-    the schema set out in the request-response-schema repo.
+    Returns:
+        HandlerResponse: The response to return back to the requestor.
     """
     headers = event.get("headers", dict())
     command = headers.get("command", "eval")
 
-    if command == "healthcheck":
-        response = handle_healthcheck_command()
-
-    elif command == "eval" or command == "grade":  # Remove once all funcs update to V2
-        response = handle_eval_command(event)
-
-    elif command == "docs-dev" or command == "docs":
-        return docs.send_dev_docs()
-
-    elif command == "docs-user":
-        return docs.send_user_docs()
+    try:
+        return handle_command(event, command)
 
-    else:
-        response = handle_unknown_command(command)
+    except (ParseError, ValidationError) as e:
+        error = ErrorResponse(message=e.message, detail=e.error_thrown)
 
-    response_error = v.validate_response(response)
+    except EvaluationException as e:
+        error = e.error_dict
 
-    if response_error:
-        return {"error": response_error}
+    # Catch-all for any unexpected errors.
+    except Exception as e:
+        error = ErrorResponse(
+            message="An exception was raised while "
+            "executing the evaluation function.",
+            detail=(str(e) if str(e) != "" else repr(e)),
+        )
 
-    return response
+    return Response(error=error)

requirements.txt

@@ -1,3 +1,8 @@
 jsonschema
 requests
-evaluation-function-utils
+evaluation-function-utils
+typing_extensions
+black
+autopep8
+pytest
+python-dotenv

tests/__init__.py

@@ -1,2 +0,0 @@
-from . import requests
-from . import responses