Incomplete BGC variables

[btupper]

@oj713 wrote

I noted that a handful of the biogeochemical variables we're using for the jellyfish species are missing in the present-day Copernicus forecast: no3, po4, and si. Looks like they're included in the "cmems_mod_glo_bgc-nut_anfc_0.25deg_P1D-m" file, which we're not pulling for the ANALYSISFORECAST copernicus set (Copernicus documentation, page 12).

[btupper]

Yah, it looks like we don't archive any of the cmems_mod_glo_bgc-nut_anfc_0.25deg_P1D-m data. I'm not sure how that happened.

suppressPackageStartupMessages({
  library(copernicus)
  library(andreas)
  library(dplyr)
})

path = copernicus_path("world")
DB = read_database(path, multiple = TRUE)

vars = c("no3", "po4", "si")

n = DB |>
  filter(variable %in% vars) |>
  group_by(variable, id) |>
  count() |>
  rowwise() |>
  group_map(
    function(row, key){
      x = DB |>
        filter(variable == row$variable,
               id == row$id) |>
        arrange(date) |>
        slice(c(1, n()))
      
      row |> 
        mutate(first = pull(x, date)[1], 
              latest = pull(x, date)[2])
    }
  ) |>
  bind_rows()
print(n, n = nrow(n))

## # A tibble: 6 × 5
##   variable id                                        n first      latest    
##   <chr>    <chr>                                 <int> <date>     <date>    
## 1 no3      cmems_mod_glo_bgc_my_0.25deg_P1D-m    10957 1993-01-01 2022-12-31
## 2 no3      cmems_mod_glo_bgc_myint_0.25deg_P1D-m   790 2023-01-01 2025-02-28
## 3 po4      cmems_mod_glo_bgc_my_0.25deg_P1D-m    10957 1993-01-01 2022-12-31
## 4 po4      cmems_mod_glo_bgc_myint_0.25deg_P1D-m   790 2023-01-01 2025-02-28
## 5 si       cmems_mod_glo_bgc_my_0.25deg_P1D-m    10957 1993-01-01 2022-12-31
## 6 si       cmems_mod_glo_bgc_myint_0.25deg_P1D-m   790 2023-01-01 2025-02-28

[btupper]

@btupper modified /mnt/s1/projects/ecocast/coredata/copernicus/lut/GLOBAL_ANALYSISFORECAST_BGC_001_028.csv so the we now fetch and archive for no3, po4, and si in the cmems_mod_glo_bgc-nut_anfc_0.25deg_P1D-m product. These should be automatically added to the daily fecth going forward.

The question is... do we backfill (looks like to 2025-03-01) missing data?

[btupper]

Confirmed that the variables are now picked up. Here is from the log /mnt/s1/projects/ecocast/coredata/copernicus/log

INFO [2025-09-15 11:12:53] fetch_dataset: cmems_mod_glo_bgc-nut_anfc_0.25deg_P1D-m no3, po4, si
INFO [2025-09-15 11:12:53] fetch_dataset: 2025-09-14, 2025-09-15, 2025-09-16, 2025-09-17, 2025-09-18, 2025-09-19, 2025-09-20, 2025-09-21, 2025-09-22, 2025-09-23

The back filling question remains unanswered.

[SeascapeScience]

What's the missing time period that we would need to backfill?

[btupper]

Daily. It’s not lot of data. We probably should have s script to do that for any variable anyway. Ben Tupper Bigelow Laboratory for Ocean Sciences https://www.bigelow.org/ Tick forecast https://eco.bigelow.org/

…

On Mon, Sep 15, 2025 at 3:00 PM SeascapeScience ***@***.***> wrote: *SeascapeScience* left a comment (BigelowLab/andreas#8) <#8 (comment)> What's the missing time period that we would need to backfill? — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABG4SP5MC7AQ5NQBHSLACXL3S4EETAVCNFSM6AAAAACGRGMRLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEOJTGUYTONJYGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[SeascapeScience]

Daily for how many years?

[btupper]

Oh, only for the last six months. Ben Tupper Bigelow Laboratory for Ocean Sciences https://www.bigelow.org/ Tick forecast https://eco.bigelow.org/

…

On Mon, Sep 15, 2025 at 3:42 PM SeascapeScience ***@***.***> wrote: *SeascapeScience* left a comment (BigelowLab/andreas#8) <#8 (comment)> Daily for how many years? — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABG4SPZFDOU6BPYPRLK6QND3S4JDLAVCNFSM6AAAAACGRGMRLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEOJTGY3TEMJYGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[oj713]

Thanks for getting this sorted Ben! I don't need the backfilled data for my project, so I'm fine either way on the back-filling issue.