Support for function test with YAML files

tests/function/README.md

@@ -0,0 +1,298 @@
+# Function tests for the zhmcclient
+
+This directory contains YAML files that define test cases for testing the
+zhmcclient by simulating the HMC interactions. This allows to test all layers
+of the zhmcclient library from its public API down to the transport layer for
+communicating with the HMC. Specifically, the test driver replaces the
+functionality of the Python "requests" module used by the zhmcclient.
+
+At this point, the test concept does not yet support HMC notifications, so
+it can only be used for zhmcclient API functions that do not use the
+notification protocol (i.e. STOMP). This is verified by failing a test case
+that uses such a zhmcclient API call.
+
+This concept allows end2end-like testing without actually using an HMC.
+It also does not depend on the zhmcclient mock support (which is quite limited).
+
+Each YAML file defines a list of test cases.
+
+Each test case defines one call to a zhmcclient API function, with:
+
+* The zhmcclient API function to be tested and the parameters to be used
+
+* A list of HMC interactions resulting from that API function, each with:
+
+  - Expected HMC request payload generated by the zhmcclient code
+  - HMC response payload to be passed back to the zhmcclient code
+
+* The expected result of the zhmcclient API function, including any exceptions
+
+There is no setup for these tests, and no state of the simulated HMC or of the
+zhmcclient session needs to be kept or defined in the test case. The behavior is
+entirely determined by the HMC responses defined in the test case. For example,
+a test case that performs a "Get Properties" operation on a non-existing resource
+is defined by setting the HMC response payload in the test case to the
+corresponding error.
+
+The format of the test case files is defined in JSON schema
+[schemas/test_case_file.json](schemas/test_case_file.json).
+
+## Example test case file
+
+The following YAML is an example test case file that defines a single test case
+that performs a successful "List CPCs" operation:
+
+    - name: list_cpcs
+      description: "Test the 'List CPCs' operation"
+      skip: false
+      no_debug_details: false
+      zhmcclient_api_call:
+          target:
+            - class: zhmcclient.Client
+          attribute: cpcs
+          method: list
+          parameters:
+            - pull_full_properties: false
+      zhmcclient_api_result:
+          return:
+            type: list
+            items:
+              - type: zhmcclient.Cpc
+                attributes:
+                  - name: CPC1
+                  - dpm_enabled: true
+                  - full_properties: false
+                  - properties:
+                  - class
+                  - name: CPC1
+                  - status: active
+              - type: zhmcclient.Cpc
+                properties:
+                  - name: CPC2
+                  - status: active
+      hmc_interactions:
+        - http_request:
+              verb: POST
+              url: http://acme.com:80/cimom
+              headers:
+                  CIMOperation: MethodCall
+                  CIMMethod: GetInstance
+                  CIMObject: root/cimv2
+              data: >
+                  <?xml version="1.0" encoding="utf-8" ?>
+                  ... (payload)
+          http_response:
+              status: 200
+              headers:
+                  CIMOperation: MethodResponse
+              data: >
+                  <?xml version="1.0" encoding="utf-8" ?>
+                  ... (payload)
+
+
+## Description of the test case file format
+
+### Top-level elements
+
+The top-level elements in the test case YAML are:
+
+* `name`:
+  A name for the test case that is used in error messages.
+
+* `description`:
+  A short description of the test case.
+
+* `skip`
+  If this element exists, the test is bypassed.  This is a way to temporarily
+  bypass tests that are causing problems without hiding or removing the code.
+  It should only be used in rare cases. Please document any use of this
+  options. If the command line option to run a single test is set
+  this element is ignored.
+
+* `zhmcclient_api_call`:
+  A specification of the PyWBEM client function to test, and the input
+  arguments for its invocation.
+
+* `zhmcclient_api_result`:
+  A specification of the expected result of the PyWBEM client function that is
+  being tested. It is possible to specify expected CIM status codes, other
+  Python exceptions, or the resulting object in case of success.
+
+* `hmc_interactions`:
+  A list of dict items, each of which has the following properties:
+
+  * `http_request`:
+    A specification of the expected HTTP request the PyWBEM client produced
+    for the client function that is being tested. This includes the first line
+    of the HTTP request, any HTTP headers, and the CIM body data (the CIM-XML)
+
+  * `http_response`:
+    The HTTP response for the PyWBEM client function that will be handed back
+    to the client function that is being tested.
+    It is possible to specify successful eresponses, error responses, and even
+    inconsistent or illegal responses in order to verify how the client handles
+    those.
+
+### Elements in `zhmcclient_api_call` element
+
+* `url`, `creds`, `namespace`, `timeout`:
+  The same-named arguments of pywbem.WBEMConnection()
+
+* `debug`:
+  Boolean indicating whether the PyWBEM client enables debug mode.
+
+* `enable_stats`:
+  Boolean indicating whether the PyWBEM client enables gathering statistical
+  information on operations.
+
+* `operation`:
+  A specification of the WBEMConnection method (= CIM operation) to be
+  invoked. Its child elements are:
+
+  * `pywbem_method`:
+    Name of the Python method of pywbem.WBEMConnection as a string
+    (e.g. "GetInstance").
+
+  * arguments for that Python method. Each element has the argument's name.
+
+    If its Python type is boolean, string, or numerical, the element's
+    value is directly the desired argument value.
+
+    Otherwise, the element has child elements that specify how the Python
+    object is constructed, as follows:
+
+    * `pywbem_object`:
+      Name of the Python type to construct (i.e. the constructor), as a string.
+      (e.g. "CIMInstanceName").
+
+    * arguments for that constructor. Each element has the argument's name. This
+      can be nested at arbitraty depth, see the description of the arguments one
+      level up.
+
+### Elements in `zhmcclient_api_result` element
+
+* `cim_status`:
+  The numerical expected CIM status code of the operation.
+  This is optional; if not specified, it defaults to 0 (=Success).
+
+* `exception`:
+  The name of the Python exception that is expectd to be raised.
+  This is optional; if not specified, it defaults to None (=no exception is
+  raised).
+
+* `request_len`:
+  Defines expected length of the request. If this element exists, the
+  value is tested against the last_req_len field of the connection. If not
+  specified the test is bypassed.
+
+* `reply_len`:
+  Defines expected length of the response. If this element exists, the
+  value is tested against the last_reply_len field of the connection. If not
+  specified the test is bypassed.
+
+* `result`:
+  A specification of the expected result (= return value) of the operation,
+  implying that the operation succeeded when this is not the result to one
+  of the pull operations (Open..., Pull...). Use the `pullresult` element to
+  define the results of the pull operations.
+
+  If the Python type of the result is boolean, string, or numerical, the
+  element's value is directly the desired argument value.
+
+  Otherwise, the element has child elements that specify how the Python
+  object is constructed, as follows:
+
+  * `pywbem_object`:
+    Name of the Python type to construct (i.e. the constructor), as a string.
+    (e.g. "CIMInstanceName").
+
+  * arguments for that constructor. Each element has the argument's name. This
+    can be nested at arbitraty depth, see the description of the arguments one
+    level up.
+
+* `pullresult`:
+  A specification of the expected result (=return values) of the operation
+  implying that the operation succeeded for any of the pull operation requests.
+  This is required because, unlike the original operations, the pull operations
+  return a tuple of information for every response including (context for the
+  enumeration session, end_of_sequence indicator, and the array of instances
+  that were included in the response by the server).
+
+  The return is a named tuple with the  element names defined as follows:
+  TBD
+
+  The subelements of pullresult are:
+
+  * `context` Specified either as a string or `null` if no context is to
+    be returned to the user
+
+  * `eos` Boolean `True` or `False` indicating whether this was the
+    final response of an enumeration session. To be correct the `context`
+    must be `null` if the `eos` is `True`.
+
+  * `instances` or `paths` depending on whether instances or instance names
+    are being returned. The chile element to this element specifies the
+    Python object to be constructed in the same manner as the child elements
+    of `result`.
+
+### Elements in `http_request` element
+
+* `verb`:
+  The expected HTTP verb / method the PyWBEM client issues (e.g. "POST").
+
+* `url`:
+  The expected URL the PyWBEM client targets the HTTP request to
+  (e.g. "http://acme.com:80/cimom").
+  Note that this is the url specified as an argument to pywbem.WBEMConnection,
+  appended with "/cimom".
+
+* `headers`:
+  The expected HTTP header fields in the HTTP request. Only the onese specified
+  here are verified, others may be present and will not be verified.
+
+  The name of the element is the header field name, and its value is the
+  header field value. Header field names are treated case insensitively.
+
+* `data`:
+  The expected CIM-XML payload of the request. When comparing the actual CIM-XML
+  to the expected CIM-XML, whitespace in between XML elements (tags) and
+  attributes are being ignored.
+
+### Elements in `http_response` element
+
+* `exception`:
+  The specification of an expected exception at the socket level, during sending
+  of the HTTP request, by specifying the name of the static method of the
+  `test_client.CallBack` class.
+
+  This is optional; if not specified, it defaults to None (=no exception is
+  raised). If specified, the other elements will be ignored.
+
+  The following method names can be specified:
+
+  * `socket_ssl`
+    A `socket.sslerror` exception with arbitrary error code.
+
+  * `socket_104`
+    A `socket.error` exception with error code 104.
+
+  * `socket_32`
+    A `socket.error` exception with error code 32.
+
+* `status`:
+  The numerical HTTP status in the HTTP response handed back to the PyWBEM
+  client (e.g. 200).
+
+* `headers`:
+  The HTTP header fields in the HTTP response handed back to the PyWBEM
+  client.
+
+  TBD: Does HTTPretty have any standard header fields it adds?
+
+  The name of the element is the header field name, and its value is the
+  header field value. Header field names are treated case insensitively.
+
+* `data`:
+  The CIM-XML payload in the HTTP response handed back to the PyWBEM
+  client. The CIM-XML is handed back as resulting from the specified
+  YAML, including any whitespace.