Validator and Audit

CHANGELOG.rst

@@ -182,7 +182,7 @@ Release 5.0.2
 * Provide guidance on how to deal with specific error conditions `#81 <https://github.com/Project-Counter/cop5/issues/81>`_ [sushi]
 * Update description for Data_Type Journal `#25 <https://github.com/Project-Counter/cop5/issues/25>`_ [documentation]
 * Clarify required file formats for tabular COUNTER reports `#33 <https://github.com/Project-Counter/cop5/issues/33>`_ [documentation]
-* Update recommendations and add error level information for the Validation Tool `#39 <https://github.com/Project-Counter/cop5/issues/39>`_ [documentation]
+* Update recommendations and add error level information for the COUNTER Validator `#39 <https://github.com/Project-Counter/cop5/issues/39>`_ [documentation]
 * Clarify rules for using custom Exceptions `#53 <https://github.com/Project-Counter/cop5/issues/53>`_ [documentation]
 * Use Requested in place of Optional for columns/elements only included when requested `#55 <https://github.com/Project-Counter/cop5/issues/55>`_ [documentation]
 * Add note to appendixes that in case of conflicts the main document takes precedence `#94 <https://github.com/Project-Counter/cop5/issues/94>`_ [documentation]

source/03-specifications/03-counter-report-common-attributes-and-elements.rst

@@ -382,7 +382,7 @@ Table 3.p (below): List of Data_Type Values
        PR_P1
 
    * - Unspecified
-     - Content that cannot be classified by any of the other Data_Types due to lack of sufficient information. Note that report providers are expected to make all reasonable efforts to classify the content. Using Unspecified will give rise to a Warning in the Validation Tool.
+     - Content that cannot be classified by any of the other Data_Types due to lack of sufficient information. Note that report providers are expected to make all reasonable efforts to classify the content. Using Unspecified will give rise to a Warning in the COUNTER Validator.
      - A&I_Database\ |br|\ |lb|
        Aggregated_Full_Content\ |br|\ |lb|
        Data_Repository\ |br|\ |lb|
@@ -972,4 +972,4 @@ The value for an element might be missing or unknown, for example a title might
 
 For clarity, values such as “unknown”, “n/a” or “-” MUST NOT be used.
 
-If a non-empty value is required for an element and the value is empty or the element is omitted, the COUNTER Validation Tool reports a (Critical) Error which would cause the report to fail an audit. If Title, Item or Publisher is empty or Data_Type Unspecified is used, the COUNTER Validation Tool reports a Warning which might affect the result of an audit. See :numref:`validation-tool` for details on the error levels used by the Validation Tool.
+If a non-empty value is required for an element and the value is empty or the element is omitted, the COUNTER COUNTER Validator reports a (Critical) Error which would cause the report to fail an audit. If Title, Item or Publisher is empty or Data_Type Unspecified is used, the COUNTER COUNTER Validator reports a Warning which might affect the result of an audit. See :numref:`validation-tool` for details on the error levels used by the COUNTER Validator.

source/09-audit/01-the-audit-process.rst

@@ -29,8 +29,8 @@ Stages of the Audit Process
 
 Audits follow a five-stage process.
 
-* **Stage one** - pre-flight preparation. Report providers SHOULD run all of their COUNTER reports (JSON and tabular) through the COUNTER Validation Tool. Report providers SHOULD also seek feedback from large report consumers (e.g. JUSP) in respect of issues they have identified with the reports. Report providers SHOULD fix any problems that are discovered and re-run the Validation Tool before proceeding to Stage two.
-* **Stage two** - audit initiation. Auditor and report provider agree on the scope of the audit, which MUST include all COUNTER Reports and Standard Views of COUNTER Reports required by the Code of Practice but SHOULD include all COUNTER Reports and Standard Views of COUNTER Reports offered by the report provider. COUNTER’s Executive Director will adjudicate if the auditor and report provider cannot agree on the audit scope. Report providers MUST provide pre-flight documentation to the auditor, using the template provided in :ref:`Appendix E <appendix-e>`. If Stage one has been followed, report providers MUST provide the Validation Tool results to the auditor. 
+* **Stage one** - pre-flight preparation. Report providers SHOULD run all of their COUNTER reports (JSON and tabular) through the COUNTER COUNTER Validator. Report providers SHOULD also seek feedback from large report consumers (e.g. JUSP) in respect of issues they have identified with the reports. Report providers SHOULD fix any problems that are discovered and re-run the COUNTER Validator before proceeding to Stage two.
+* **Stage two** - audit initiation. Auditor and report provider agree on the scope of the audit, which MUST include all COUNTER Reports and Standard Views of COUNTER Reports required by the Code of Practice but SHOULD include all COUNTER Reports and Standard Views of COUNTER Reports offered by the report provider. COUNTER’s Executive Director will adjudicate if the auditor and report provider cannot agree on the audit scope. Report providers MUST provide pre-flight documentation to the auditor, using the template provided in :ref:`Appendix E <appendix-e>`. If Stage one has been followed, report providers MUST provide the COUNTER Validator results to the auditor. 
 * **Stage three** - seeding, audit begins. In this stage the auditors perform actions on the audited platforms in line with :ref:`Appendix E <appendix-e>`. Seeding takes a full calendar month.
 * **Stage four** - report reconciliation. The auditors reconcile the COUNTER reports generated by the report provider with the seeding actions they undertook in Stage three. The outputs from this stage may be an Interim Report or a Pass Report, which MUST be shared with the COUNTER Executive Director. If an Interim Report is issued, 
 

source/09-audit/03-timelines.rst

@@ -26,19 +26,19 @@ Table 9.a (below): Audit Timelines
 
    * - Seeding
      - 1 calendar month
-     - February
+     - Seeding actions in February
 
    * - Usage processed, reports available
      - Up to 28 days from end of seeding month
-     - March
+     - Reports available by 28 March
 
    * - Report reconciliation
      - 4 to 6 weeks
-     - April
+     - Reconciliation during April
 
    * - Fixes after interim report
      - Up to 3 months
-     - May to July
+     - Interim report issued end April, fixes due by end July
 
    * - Report reconciliation after fixes
      - 1 month
@@ -50,12 +50,13 @@ Extensions
 
 Report providers may request extensions to the usual timelines under the following circumstances.
 
-* **Delayed seeding** - where a report provider is able to demonstrate that they have run the COUNTER Validation Tool and are in the process of fixing problems identified by the tool before commencing an audit, they may apply for up to three months delay in the seeding process.
-* **Extension to the fix period** - where an Interim Report has identified significant problems with COUNTER reports, the report provider may apply for an extension of the fix period from three to up to six months.
+* **Delayed seeding** - where a report provider is able to demonstrate that they have run the COUNTER COUNTER Validator and are in the process of fixing problems identified by the tool before commencing an audit, they may apply for up to three months delay in the seeding process.
+* **Extension to the fix period** - where an Interim Report has identified significant problems with COUNTER reports, the report provider may apply for an extension of the fix period from the standard three to a maximum of six months (i.e. an additional three months).
 
 Applications for extensions MUST be sent to both the Executive Director (code@countermetrics.org) and the auditor, and will be considered on a case-by-case basis. Only one extension of each type will be granted for any audit, and all extensions will be tracked in the Registry.
 
-There may be limited, unforeseen circumstances in which an additional extension may be granted. In these cases the report provider and auditor MUST explain the extenuating circumstances in a short report to the COUNTER Executive Committee. If the additional extension is granted, the report provider will not be required to start seeding an audit until three months after a successful audit pass.
+There may be limited, unforeseen circumstances in which an additional extension may be granted. In these cases the report provider and auditor MUST explain the extenuating circumstances in a short report to the COUNTER Executive Committee. Note that being unable to prioritise fixes in the development queue does not qualify as an unforeseen circumstance. If the additional extension is granted, the report provider will not be required to start seeding an audit until three months after a successful audit pass.
+
 
 Alternate year audits
 """""""""""""""""""""

source/09-audit/04-counter-validation-tool.rst

@@ -4,12 +4,12 @@
 
 .. _validation-tool:
 
-COUNTER Validation Tool
+COUNTER COUNTER Validator
 -----------------------
 
-The `COUNTER Validation Tool <https://www.countermetrics.org/code-of-practice/tools/>`_ allows report providers and auditors to quickly perform compliance checks related to format and consistency of both JSON and tabular reports. Report providers SHOULD use this free tool to check their reports and COUNTER API implementation and fix issues detected by the tool before they begin the audit. It is recommended to use the tool not just when preparing for the audit, but to integrate the testing in the regular QA processes or at least test regularly to make sure the reports stay compliant. We recommend testing reports using the Validation Tool once every four months at a minimum, and monthly where possible. 
+The `COUNTER COUNTER Validator <https://www.countermetrics.org/code-of-practice/tools/>`_ allows report providers and auditors to quickly perform compliance checks related to format and consistency of both JSON and tabular reports. Report providers SHOULD use this free tool to check their reports and COUNTER API implementation and fix issues detected by the tool before they begin the audit. It is recommended to use the tool not just when preparing for the audit, but to integrate the testing in the regular QA processes or at least test regularly to make sure the reports stay compliant. We recommend testing reports using the COUNTER Validator once every four months at a minimum, and monthly where possible. 
 
-The COUNTER Validation Tool uses the following error levels to report issues with different severity:
+The COUNTER COUNTER Validator uses the following error levels to report issues with different severity:
 
 * **Fatal error** - Validation was aborted because an unrecoverable error was encountered, for example a missing or invalid Reporting_Period. The fatal error MUST be fixed before the report can be fully validated.
 * **Critical error** - Validation has detected a serious error that MUST be fixed, for example inconsistent data like a title with more Unique_Item_Requests than Total_Item_Requests or missing data. A critical error indicates that there could be a serious issue that would cause the audit to fail.

source/09-audit/05-inclusion-in-registry.rst

@@ -7,7 +7,7 @@ Application for inclusion in the COUNTER Registry
 
 The Registry of report providers and their platforms for which COUNTER-compliant usage reports are available is maintained by COUNTER and publicly available at `Registry <https://registry.countermetrics.org/>`_.
 
-Report providers may apply to the Executive Director (code@countermetrics.org) for their products to be included on the Registry. Report providers will have to provide proof of initial compliance by including the results of COUNTER Validation Tool tests showing compliance for each reports, including testing both the upload of the tabular reports and COUNTER API (formerly sushi) harvesting of the same reports. Upon receipt of the application and proof of compliance, report providers MUST allow at least one large report consumer (e.g. JUSP) to evaluate their usage reports.
+Report providers may apply to the Executive Director (code@countermetrics.org) for their products to be included on the Registry. Report providers will have to provide proof of initial compliance by including the results of COUNTER COUNTER Validator tests showing compliance for each reports, including testing both the upload of the tabular reports and COUNTER API (formerly sushi) harvesting of the same reports. Upon receipt of the application and proof of compliance, report providers MUST allow at least one large report consumer (e.g. JUSP) to evaluate their usage reports.
 
 When the reports are deemed to comply with the COUNTER Code of Practice, the report provider will be asked to sign a Declaration of COUNTER Compliance (:ref:`Appendix C <appendix-c>`), to be sent to the Executive Director, after which the report provider and its platforms will be added to the Registry. These report providers may then use the designation “COUNTER Compliance Pending”.
 

source/appendices/a-glossary-of-terms.rst

@@ -249,7 +249,7 @@ Note: The main Code of Practice document takes precedence in the case of any con
      - Status of a vendor who is currently not compliant but whose audit is in progress or scheduled.
      -
 
-   * - COUNTER Report Validation Tool
+   * - COUNTER Report COUNTER Validator
      - An online tool to validate COUNTER reports in JSON and tabular format.
      -
 
@@ -601,7 +601,7 @@ Note: The main Code of Practice document takes precedence in the case of any con
      -
 
    * - ISBN (International Standard Book Number)
-     - A unique standard identifier (ISO 2108) used to identify monographic publications (books). The format ISBN-13, with hyphens, is required. E-ISBN is the expected value, with print ISBNs provided only where E-ISBN is not available. Inclusion of an ISBN-10, or an incorrectly formatted ISBN-13, will be flagged as an error by the Validation Tool.
+     - A unique standard identifier (ISO 2108) used to identify monographic publications (books). The format ISBN-13, with hyphens, is required. E-ISBN is the expected value, with print ISBNs provided only where E-ISBN is not available. Inclusion of an ISBN-10, or an incorrectly formatted ISBN-13, will be flagged as an error by the COUNTER Validator.
      -
 
    * - ISBN-13
@@ -953,8 +953,8 @@ Note: The main Code of Practice document takes precedence in the case of any con
      - An umbrella term. Includes publishers, aggregators and others who directly provide access to content, as well as organizations that provide specialist reporting services on behalf of one or more organizations.
      - Science Direct, Clarivate, JSTOR, ScholarlyIQ
 
-   * - Report validation tool
-     - See COUNTER Report Validation Tool.
+   * - Report COUNTER Validator
+     - See COUNTER Report COUNTER Validator.
      -
 
    * - Reporting period\ |br|\ |lb|

source/appendices/e-audit-requirements-and-tests/02-general-tests.rst

@@ -8,10 +8,10 @@ E.2 General Audit Tests
 This section of the appendix outlines tests that MUST be run during all audits, and which are not specific to a particular COUNTER Report.
 
 
-E.2.1 Validation Tool
-"""""""""""""""""""""
+E.2.1 COUNTER Validator
+"""""""""""""""""""""""
 
-Once reports are available for testing (i.e. up to 28 days after the end of the seeding month), auditors MUST run the COUNTER Reports and Standard Views of COUNTER Reports that are within the scope of the audit through the Validation Tool. This test must include all attributes report providers are required to support, including attributes that are included only if called for.
+Once reports are available for testing (i.e. up to 28 days after the end of the seeding month), auditors MUST test the report provider's COUNTER API (sushi) configuration as well as the COUNTER Reports and Standard Views of COUNTER Reports that are within the scope of the audit using the COUNTER Validator. This test must include all attributes report providers are required to support, including attributes that are included only if called for.
 
 Where report providers have elected to follow the pre-flight preparation step outlined in :numref:`audit` of the Code of Practice, this audit test should not result in any errors.