Skip to content

Add data for SpeXtra#32

Draft
teutoburg wants to merge 1 commit intomainfrom
fh/spextra-data
Draft

Add data for SpeXtra#32
teutoburg wants to merge 1 commit intomainfrom
fh/spextra-data

Conversation

@teutoburg
Copy link
Contributor

This is needed in various CI and RTD builds by ScopeSim et al., where it is currently downloaded every time. My understanding of ScopeSim_Data is to provide (some of) those things. This PR would bring that close to reality.

This will only work though if SpeXtra gets told where to look for these files, or becomes smart enough to do so. But including the files here is required to test that there, so yeah.

This adds around 5 MB, less than a 10 % increase in this repo.

This is needed in various CI and RTD builds by ScopeSim et al., where it is
currently downloaded every time. My understanding of ScopeSim_Data is to
provide (some of) those things. This PR would bring that close to reality.

This will only work though if SpeXtra gets told where to look for these
files, or becomes smart enough to do so. But including the files here is
required to test that there, so yeah.

This adds around 5 MB, less than a 10 % increase in this repo.
@teutoburg teutoburg self-assigned this Feb 23, 2026
@teutoburg teutoburg added the enhancement New feature or request label Feb 23, 2026
@teutoburg teutoburg marked this pull request as ready for review February 23, 2026 15:31
@teutoburg teutoburg requested a review from hugobuddel February 23, 2026 15:31
@hugobuddel
Copy link
Contributor

I don't know whether it makes sense to add the spextra library data to ScopeSim_Data. We already have some of the data in speXtra itself, so maybe add this data there too?

Or otherwise, move all that data here; that would make sense.

If we do want to store the data here, then we should do it the other way around: make this directory in ScopeSim_Data the download directory of speXtra (when ScopeSim_Data is installed). Then a P.R. with the necessary data will be created automatically when the ScopeSim_Data CI is ran.

@teutoburg
Copy link
Contributor Author

I don't know whether it makes sense to add the spextra library data to ScopeSim_Data. We already have some of the data in speXtra itself, so maybe add this data there too?

Only some SVO filter curves are bundled in the package. I think it's sensible to keep the package itself as slim as possible, i.e. not add more data there. The spextra repo also has a database dir, which is not bundled with the package. But that only contains the index files I think.

If we do want to store the data here, then we should do it the other way around: make this directory in ScopeSim_Data the download directory of speXtra (when ScopeSim_Data is installed). Then a P.R. with the necessary data will be created automatically when the ScopeSim_Data CI is ran.

Yeah that makes sense (I think). So instead add something like this (I'm not sure I can do this on Windows?) first and then modify the cache location in spextra? Sure, let's try that! I'm not sure how pooch (which we use for the download and caching in spextra) will react to changing cache location, but we'll see.

Then we also have a confirmation that the CI here actually works!

@hugobuddel
Copy link
Contributor

Yeah not sure how to do this properly with pooch. Because we should have a multi-tier setup in some way, with these considerations:

  • A file can exist in either the local cache of the user, or in ScopeSim_Data if it is installed, so both need to be checked. I'm not sure pooch can do that.
  • If ScopeSim_Data is installed, the downloaded data should probably only be cached there if it is ran through the CI, or by one of us locally, because normally it will be installed in site-packages directory and you don't want to download files to there usually.

@teutoburg
Copy link
Contributor Author

One simple (hacky? nah...) solution might be to first "manually" check ScopeSim_Data (via Path.exists() or something) and only if that's not there, try with pooch. Then have some condition that check if we're in the CI and in that case override the usual pooch cache dir with the ScopeSim_Data location. Not sure if that'll work but worth a try...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants