Skip to content

Comments

Add initial ADPUPA BUFR processor + integration test#183

Open
yuvraajnarula wants to merge 7 commits intoopenclimatefix:mainfrom
yuvraajnarula:170/bufr_process
Open

Add initial ADPUPA BUFR processor + integration test#183
yuvraajnarula wants to merge 7 commits intoopenclimatefix:mainfrom
yuvraajnarula:170/bufr_process

Conversation

@yuvraajnarula
Copy link
Contributor

Pull Request

Description

This PR introduces the minimal version of the BUFR → NNJA-AI converter, focusing only on ADPUPA, as suggested in the review discussion. (#177 was getting hard to track and was in a slump)

The goal is to establish a clean decoding pipeline and ensure that a decoded BUFR observation matches the corresponding NNJA-AI Parquet representation.

  1. BUFR_Process (ADPUPA-only)
  • A minimal BUFR decoder using ecCodes:
  • Reads BUFR messages
  • Extracts core ADPUPA header metadata
  • Extracts all level sequences (mandatory, significant temp, significant wind, tropopause, max wind)
  • Returns an adpupa_obs dataclass matching the NNJA-AI schema

This forms the foundation for later CrIS / IASI / AMSU / ATMS modules.
2. ADPUPA dataclasses

  • These represent the structure of NNJA-AI ADPUPA Parquet files:
    -- GeoPoint
    -- adpupa_level
    -- adpupa_obs

The final Parquet writer will consume these in a later PR.
Fixes #170

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce.
Please also list any relevant details for your test configuration

  • Yes

If your changes affect data processing, have you plotted any changes? i.e. have you done a quick sanity check?

  • Yes

Checklist:

  • My code follows OCF's coding style guidelines
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked my code and corrected any misspellings

@yuvraajnarula
Copy link
Contributor Author

hey @jacobbieker
needed a little help, if you don't mind. I was trying to work with this implementation around this adpupa bufr. Which one would you suggest at the moment?

@yuvraajnarula
Copy link
Contributor Author

@jacobbieker
the parent dir since https://nomads.ncep.noaa.gov/pub/data/nccf/com/obsproc/prod/
the data is limited to only a few days in hand

@jacobbieker
Copy link
Member

@jacobbieker the parent dir since https://nomads.ncep.noaa.gov/pub/data/nccf/com/obsproc/prod/ the data is limited to only a few days in hand

Hey sorry for the delay,

Yes, the NOMADS only has a few days of data, the archive of everything, and where there will be a corresponding NNJA-AI Parquet file to compare against, would be here: https://noaa-reanalyses-pds.s3.amazonaws.com/index.html#observations/reanalysis/ where you can find another adpupa bufr that won't disappear.

@yuvraajnarula
Copy link
Contributor Author

thank you for your help

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUFR processor to read NOMADS BUFR files into NNJA-AI format

2 participants