Alternate strategy for cross requesting epochs with wfcatalog database

[jschaeff]

For now, there is a background worker that creates on a daily basis another database in mongodb with connections to the metadata epochs as provided by ws station.

This has some drawbacks in a production system:

• if a worker fails, weird things can happer
• there is a 1 day lag on metadata changes
• the resulting database may be not well optimized for some requests

The expected work is just to understand the actual system and imagine better ways to do the job.

[NikolaosSokos]

Proposed plan:
Old: cron $\rightarrow$ mongosh $\rightarrow$ main.js (blind execution/hardcoded dates).
New: Docker $\rightarrow$ Python $\rightarrow$ PyMongo + Sentry (monitored/state-driven sync).

[jschaeff]

Another option would be to schedule refresh with celery jobs