Quality control - use web services to check data quality

[smalers]

Now that web services are enabled for historical station data, TSTool could be used to perform quality control, to ensure that the data are OK for use in modeling and analysis. TSTool command files could be developed to check for out of range values and when detected, the State or data source agency could be notified. The following is an example of a likely bad data point in evaporation data (1.6 in in a day). I suppose this could be possible but it seems unlikely. This level of analysis goes beyond data access and could help many users of the data.

It should be possible to determine thresholds for basic checks and staff with basic skills could probably do the work.

[smalers]

Here is another example, this time for DWR data. Note the high value and there are some zero values. I don't recall the reservoir being completely drained but I could be wrong.

[smalers]

Here is another example. Checking for negative values could be done with a simple SQL query by data type. TSTool could also be used, especially if other checks are done.

I verified that web services is returning a negative value, perhaps because missing value -999 is being added to total before averaging. Note that TSTool originally used -999 for missing value based on modeling conventions and is still the default in some cases but I have moved to NaN and continue to do so. Internally, null is also handled and is recognized in JSON whereas NaN is not in the JSON standard. The -999 value was selected because it was used by the USGS. The internal code allows setting a missing value indicator for a time series, which is checked to ensure that missing data value is not considered an actual value.

    {
      "stationNum": 1650,
      "abbrev": "MICDCPCO",
      "usgsSiteId": "06746000",
      "measType": "Streamflow",
      "calYear": 1935,
      "calMonNum": 8,
      "minQCfs": -999.00,
      "maxQCfs": 19.00,
      "avgQCfs": -25.82,
      "totalQAf": -1587.59,
      "measCount": 31,
      "dataSource": "DWR",
      "modified": null
    },