Skip to content

Commit 6e81595

Browse files
KarlLundengaardKarlLundengaard
authored
Merge pull request #16 from lambda-feedback/b544-eval=function-schema-403
move schemas into base layer
2 parents 5a50431 + 5d6aaf3 commit 6e81595

File tree

11 files changed

+566
-21
lines changed

11 files changed

+566
-21
lines changed

‎Dockerfile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,6 @@ COPY __init__.py ./app/
2626
COPY handler.py ./app/
2727
COPY tests/*.py ./app/tests/
2828
COPY tools/*.py ./app/tools/
29+
COPY schemas/ ./app/schemas/
2930

30-
# Request-response-schemas repo/branch to use for validation
31-
ENV SCHEMAS_URL=https://raw.githubusercontent.com/lambda-feedback/request-response-schemas/master
31+
ENV SCHEMA_DIR=/app/app/schemas/

‎schemas/readme.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Grading Function Request/Response Schema
2+
3+
## Introduction
4+
5+
This repo holds the schemas used for the grading function request and response bodies. This is used to check that:
6+
7+
- All requests from the frontend have the correct JSON structure so they can be interpreted by a grading function.
8+
- All grading function responses have the correct JSON structure so they can be interpeted by the frontend to show the correct answer and any feedback.
9+
10+
## Structure (v1)
11+
12+
### Request
13+
14+
The request body can have three fields:
15+
16+
- `response`, which contains the answers provided by the student. This can be anything: a string, boolean, number, array or object.
17+
- `answer`, which contains the correct answer held in the problem set database. This can also be anything.
18+
- `params`, which contains extra arguments for determining whether the student's answer is correct.
19+
20+
Only `response` and `answer` are required. Any extra fields will cause the validation to fail.
21+
22+
### Response
23+
24+
The response body can have three fields:
25+
26+
- `command`, which can only be either `"healthcheck"` or `"grade"`.
27+
28+
`"healthcheck"` is used only by the backend to check the grading function is alive and working correctly. `"grade"` is used for carrying out a grading request.
29+
30+
- `result`, which contains the results from the healthcheck or grading request. If the results are from a grading request, this is the dictionary that is returned from `grading_function()`.
31+
32+
- If the command is `grade`, then `results` must be an object and can have two fields:
33+
34+
- `is_correct`, which must be a boolean.
35+
- `feedback`, which contains any feedback about the student's answer. This can be an object, array or string.
36+
37+
Only `is_correct` is required.
38+
39+
- If the command is `healthcheck`, then `results` must contain:
40+
41+
- `tests_passed`, which must be a boolean.
42+
- `successes`, which contains the names of tests that passed and the time it took.
43+
- `failures`, which contains the names of tests that failed.
44+
- `errors`, which contains the names of tests that resulted in an error.
45+
46+
All fields are required.
47+
48+
- `error`, which contains information about any error that was thrown from a request. This can have two fields:
49+
50+
- `message`, i.e. a human-readable explanation of the cause. This must be a string.
51+
52+
- `error_thrown`, which contains more details about the error, in particular if the validation, parsing or the grading function caused an error. This must be an object although its content isn't validated.
53+
54+
Only `message` is required.
55+
56+
Both `command` and `result` are required, unless an error is thrown, in which only `error` can exist. Any extra fields will cause the validation to fail.
57+
58+
## Contact
59+
60+
For questions about these schemas or you would like to discuss possible changes, please [email Peter Johnson](mailto:peter.johnson@imperial.ac.uk).

‎schemas/request.json‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
{
2+
"title": "JSON schema for a grading request.",
3+
"description": "This schema is used to check whether a request to a grading function fits the basic structure.",
4+
"properties": {
5+
"response": {
6+
"not": {
7+
"type": "null"
8+
}
9+
},
10+
"answer": {
11+
"not": {
12+
"type": "null"
13+
}
14+
},
15+
"params": {"type": "object"}
16+
},
17+
"additionalProperties": false,
18+
"required": ["response", "answer"]
19+
}

‎schemas/request/eval.json‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
{
2+
"title": "JSON schema for a grading request.",
3+
"description": "This schema is used to check whether a request to a grading function fits the basic structure.",
4+
"properties": {
5+
"response": {
6+
"not": {
7+
"type": "null"
8+
}
9+
},
10+
"answer": {
11+
"not": {
12+
"type": "null"
13+
}
14+
},
15+
"params": {"type": "object"}
16+
},
17+
"additionalProperties": false,
18+
"required": ["response", "answer"]
19+
}

‎schemas/request/preview.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"title": "JSON schema for a grading request.",
3+
"description": "This schema is used to check whether a request to a grading function fits the basic structure.",
4+
"properties": {
5+
"response": {
6+
"not": {
7+
"type": "null"
8+
}
9+
},
10+
"params": {"type": "object"}
11+
},
12+
"additionalProperties": false,
13+
"required": ["response"]
14+
}

‎schemas/response.json‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
{
2+
"title": "JSON schema for the response from a grading function.",
3+
"description": "This schema is used to check whether a response from a grading function fits the basic structure.",
4+
"properties": {
5+
"command": {
6+
"type": "string",
7+
"enum": ["grade", "healthcheck"]
8+
},
9+
"result": {
10+
"type": "object",
11+
"properties": {
12+
"is_correct": {
13+
"type": "boolean"
14+
},
15+
"feedback": {
16+
"type": ["object", "array", "string"]
17+
},
18+
"tests_passed": {
19+
"type": "boolean"
20+
},
21+
"successes": {
22+
"type": "array"
23+
},
24+
"failures": {
25+
"type": "array"
26+
},
27+
"errors": {
28+
"type": "array"
29+
}
30+
},
31+
"additionalProperties": true
32+
},
33+
"error": {
34+
"type": "object",
35+
"properties": {
36+
"message": {
37+
"type": "string"
38+
},
39+
"error_thrown": {
40+
"type": ["object", "string"]
41+
}
42+
},
43+
"additionalProperties": true,
44+
"required": ["message"]
45+
}
46+
},
47+
"additionalProperties": false,
48+
"allOf": [
49+
{
50+
"if": {
51+
"required": ["result"]
52+
},
53+
"then": {
54+
"required": ["command"]
55+
},
56+
"else": {
57+
"required": ["error"]
58+
}
59+
},
60+
{
61+
"if": {
62+
"properties": {
63+
"command": {
64+
"const": "grade"
65+
}
66+
}
67+
},
68+
"then": {
69+
"properties": {
70+
"result": {
71+
"required": ["is_correct"]
72+
}
73+
},
74+
"anyOf": [
75+
{"required": ["result"]},
76+
{"required": ["error"]}
77+
]
78+
}
79+
},
80+
{
81+
"if": {
82+
"properties": {
83+
"command": {
84+
"const": "healthcheck"
85+
}
86+
}
87+
},
88+
"then": {
89+
"properties": {
90+
"result": {
91+
"required": ["tests_passed", "successes", "failures", "errors"]
92+
}
93+
},
94+
"anyOf": [
95+
{"required": ["result"]},
96+
{"required": ["error"]}
97+
]
98+
}
99+
}
100+
]
101+
}

‎schemas/response/eval.json‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
{
2+
"title": "JSON schema for the response from a grading function.",
3+
"description": "This schema is used to check whether a response from a grading function fits the basic structure.",
4+
"properties": {
5+
"command": {
6+
"const": "eval"
7+
},
8+
"result": {
9+
"type": "object",
10+
"properties": {
11+
"is_correct": {
12+
"type": "boolean"
13+
},
14+
"feedback": {
15+
"type": [
16+
"object",
17+
"array",
18+
"string"
19+
]
20+
}
21+
},
22+
"additionalProperties": true,
23+
"required": ["is_correct"]
24+
},
25+
"error": {
26+
"type": "object",
27+
"properties": {
28+
"message": {
29+
"type": "string"
30+
},
31+
"error_thrown": {
32+
"type": [
33+
"object",
34+
"string"
35+
]
36+
}
37+
},
38+
"additionalProperties": true,
39+
"required": [
40+
"message"
41+
]
42+
}
43+
},
44+
"additionalProperties": false,
45+
"allOf": [
46+
{
47+
"if": {
48+
"required": [
49+
"result"
50+
]
51+
},
52+
"then": {
53+
"required": [
54+
"command"
55+
]
56+
},
57+
"else": {
58+
"required": [
59+
"error"
60+
]
61+
}
62+
}
63+
]
64+
}

‎schemas/response/healthcheck.json‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
{
2+
"title": "JSON schema for the response from a healthcheck function.",
3+
"description": "This schema is used to check whether a response from a healthcheck function fits the basic structure.",
4+
"properties": {
5+
"command": {
6+
"const": "healthcheck"
7+
},
8+
"result": {
9+
"type": "object",
10+
"properties": {
11+
"tests_passed": {
12+
"type": "boolean"
13+
},
14+
"successes": {
15+
"type": "array"
16+
},
17+
"failures": {
18+
"type": "array"
19+
},
20+
"errors": {
21+
"type": "array"
22+
}
23+
},
24+
"additionalProperties": false,
25+
"required": [
26+
"tests_passed",
27+
"successes",
28+
"failures",
29+
"errors"
30+
]
31+
},
32+
"error": {
33+
"type": "object",
34+
"properties": {
35+
"message": {
36+
"type": "string"
37+
},
38+
"error_thrown": {
39+
"type": [
40+
"object",
41+
"string"
42+
]
43+
}
44+
},
45+
"additionalProperties": true,
46+
"required": [
47+
"message"
48+
]
49+
}
50+
},
51+
"additionalProperties": false,
52+
"allOf": [
53+
{
54+
"if": {
55+
"required": [
56+
"result"
57+
]
58+
},
59+
"then": {
60+
"required": [
61+
"command"
62+
]
63+
},
64+
"else": {
65+
"required": [
66+
"error"
67+
]
68+
}
69+
}
70+
]
71+
}

0 commit comments

Comments
 (0)