Skip to content

fix: Add code to get_usgs_CDA to allow converting smaller interval data to store as 15 minute data. #228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tsressin wants to merge 7 commits into
base: main
Choose a base branch
from
+50 −8
Open

fix: Add code to get_usgs_CDA to allow converting smaller interval data to store as 15 minute data. #228

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
463b1d4
Copied code block to convert data to 15 minutes prior to storing from…
tsressin Jul 6, 2026
a168ba4
Updated comment to better reflect the purpose of the 15 minute conver…
tsressin Jul 6, 2026
85a3db1
Reformatted code to standard.
tsressin Jul 6, 2026
20cc807
Test new yaml
tsressin Jul 6, 2026
ead1155
Poetry checks passed.
tsressin Jul 6, 2026
a07ec81
Revert "Test new yaml"
tsressin Jul 6, 2026
77794e0
Fix getusgs_cda
tsressin Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions .pre-commit-config.yaml
View file
Open in desktop

@krowvin krowvin Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks to resolve the issue you had with poetry trying to run locally. Good!

Original file line number Diff line number Diff line change
Expand Up @@ -3,26 +3,30 @@ repos:
hooks:
- id: black
name: black
entry: poetry run black

@msweier msweier Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want a separate PR if the yaml needs to be changed?

@krowvin krowvin Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say so, I had started one to fix the issue with the formatting library versions being different based on version of python running.

I.e. your local vs CI versions

But this was also blocking the local tests from running.

Should we pick this out into its own PR? @msweier

@msweier msweier Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me!

@tsressin tsressin Jul 6, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to change the yaml in order to pass the commit checks, otherwise it could not find poetry.

I don't know if this is an everybody problem or just a me problem.

language: system
entry: black
language: python
additional_dependencies: [black]
types: [file, python]
exclude: ^cwmscli/_generated/ownership_data\.py$

- id: isort
name: isort
entry: poetry run isort
language: system
entry: isort
language: python
additional_dependencies: [isort]
types: [file, python]
exclude: ^cwmscli/_generated/ownership_data\.py$

- id: yamlfix
name: yamlfix
entry: poetry run yamlfix
language: system
entry: yamlfix
language: python
additional_dependencies: [yamlfix]
types: [file, yaml]

- id: ownership-sync
name: generated ownership files
entry: poetry run python scripts/sync_ownership.py --check
language: system
entry: python scripts/sync_ownership.py --check
language: python
additional_dependencies: [poetry]
pass_filenames: false
28 changes: 28 additions & 0 deletions .pre-commit-config.yaml.org
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
repos:

@msweier msweier Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This .pre-cmmit-file.cong.yaml.org is a copy of the original and probably was included by accident.

- repo: local
hooks:
- id: black
name: black
entry: poetry run black
language: system
types: [file, python]
exclude: ^cwmscli/_generated/ownership_data\.py$

- id: isort
name: isort
entry: poetry run isort
language: system
types: [file, python]
exclude: ^cwmscli/_generated/ownership_data\.py$

- id: yamlfix
name: yamlfix
entry: poetry run yamlfix
language: system
types: [file, yaml]

- id: ownership-sync
name: generated ownership files
entry: poetry run python scripts/sync_ownership.py --check
language: system
pass_filenames: false
10 changes: 10 additions & 0 deletions cwmscli/usgs/getusgs_cda.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,16 @@ def CWMS_writeData(USGS_ts, USGS_data, USGS_data_method, days_back):
office = row["office-id"]
values["quality-code"] = 0

# 15 minute conversion to allow storing smaller interval data (eg. 5 minute) as 15 minute data

@msweier msweier Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did this site start as 15 minute and went to 5 minute? Another thought is to use an irregular timeseries ~15Minute which would let you keep them in the same timeseries and preserving all the data. But if you want to subsample to 15 minutes, this works too!

@tsressin tsressin Jul 6, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some sites that went from 15 to 5 and some that have always been 5. For my purposes, 5 minutes is overkill and no need to fill up the database unnecessarily. So, I have always only stored 15 minute data for these sites. Previous versions of getusgs (non-cda and cda) allowed for this, but it was somehow missed in the cwms-cli version of getusgs. I just want to maintain my existing processing.

if ts_id.split(".")[3] == "15Minutes":
values_dt = values.copy()
values_dt["date-time"] = pd.to_datetime(
values_dt["date-time"]
)
values_dt.set_index("date-time", inplace=True)
values15 = values_dt.resample("15min").first()
values = values15.reset_index()

# write values to CWMS database
try:
data = cwms.timeseries_df_to_json(
Expand Down
Loading