Performance instrumentation for individual and overall constraint processing

[aj-stein-gsa]

User Story

As a developer of Metaschema-enabled software, models, and data, I would like performance instrumentation to measure individual constraints and overall model and external constraint processing to determine hotspots, performance bottlenecks, and areas for improvement.

Goals

• Determine micro and macro-level performance of this library and dependent tools
• Analyze performance to add, change, or remove features and a specific implementation to meet ongoing needs of Metaschema users for heterogenous use cases (e.g. OSCAL and others)

Dependencies

No response

Acceptance Criteria

• All website and readme documentation affected by the changes in this issue have been updated.
• A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR.
• The CI-CD build process runs without any reported errors on the PR. This can be confirmed by reviewing that all checks have passed in the PR.

Revisions

No response

[aj-stein-gsa]

@wandmagic I created this issue based upon our discussion in standup today about more precise perf counters.

[wandmagic]

This would be really handy especially as we scale up ssp data size

[david-waltermire]

How should this work? We need to get to some form of a spec we can implement.

[aj-stein-gsa]

How should this work? We need to get to some form of a spec we can implement.

Agreed. I was looking into instrumentation systems for Java applications.

[wandmagic]

we could just have a flag for --instrumentation in the CLI (or a flag to turn it off --skip-instrumentation)
and then in the sarif output, a custom prop with how much processing time was used during this rule:

  "version": "2.1.0",
  "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
  "runs": [{
    "tool": {
      "driver": {
        "name": "oscal-cli",
        "rules": [{
          "id": "{uuid}",
          "name": "resolves-to-valid-content",
          "properties": {
            "executionTimeMs": 324
          }
        }]
      }
    },
    "results": [
      // ... regular results
    ]
  }]
}

[aj-stein-gsa]

Before extending SARIF data and reinventing the wheel, one very rough (not so granular) data source we could tap into (but current do not) is the JUnit/Surefire reports we could store, but do not, in GitHub or elsewhere given we use that plugin with Maven.

That said, it only tells us what the macro-level "I ran this test that calls of this other code across modules in one or more function calls," and nothing more granular, like I said. I have been researching this on and off all morning and found nothing very compelling about time measurement and profiling, but I will have to read up on this area.

That said, if we could find a way outside of m-j and oscal-cli code to actually annotate code with time runs in SARIF since we know what code paths are used for tests, and annotate function calls that exceeds a thresshold or deserve investigation, we may be onto something I think no one else is doing (open source or on the inner source proprietary side, I'll have to ask; no one has ever hinted they do something like that, so we would be trendsetters).

[david-waltermire]

Before extending SARIF data and reinventing the wheel, one very rough (not so granular) data source we could tap into (but current do not) is the JUnit/Surefire reports we could store, but do not, in GitHub or elsewhere given we use that plugin with Maven.

Adding this to the SARIF output would be straight forward if we want to do this.

That said, it only tells us what the macro-level "I ran this test that calls of this other code across modules in one or more function calls," and nothing more granular, like I said. I have been researching this on and off all morning and found nothing very compelling about time measurement and profiling, but I will have to read up on this area.

That said, if we could find a way outside of m-j and oscal-cli code to actually annotate code with time runs in SARIF since we know what code paths are used for tests, and annotate function calls that exceeds a thresshold or deserve investigation, we may be onto something I think no one else is doing (open source or on the inner source proprietary side, I'll have to ask; no one has ever hinted they do something like that, so we would be trendsetters).

I am trying to pin down what such a feature would look like. What would be the user experience for something like this?