Federator and routing inconsistent for network@NOA (CQ)

[helle-pedersen]

Hi,

another issue from the student lab work on obspy this morning: inconsistency between EIDA routing and federator. Don't know if it is a routing problem, a federator problem or a NOA problem.
As we tried only networks beginning in C, this problem might existe for other networks?
I just ran it again now (times below).

Cheers
Helle

Describe the problem
The students noticed that EIDA routing and federator didn't give the same number of networks. The difference is the CQ network that appears with the routing but not with the federator.

To Reproduce
With EIDA Routing client:
client = RoutingClient("eida-routing")
inventory = client.get_stations(network="C*")
print(inventory)
Gives networks C4, CA, CH, CK, CL, CP, CQ, CR (2x), CX, CZ
Inventory created at 2025-10-13T13:02:06.880598Z
Created by: ObsPy 1.4.2
https://www.obspy.org/
Sending institution: ObsPy FDSN Routing 1.4.2,RESIF-SI,SeisComP (GFZ,ICGC,NOA,ODC (6.8.4r3b2),RESIF-DC,SED)
Contains:
Networks (11):
C4, CA, CH, CK, CL, CP, CQ, CR (2x), CX, CZ
Stations (613):

---

With Federator
client = Client("EIDA")
inventory = client.get_stations(network="C*")
C4, CA, CH, CK, CL, CP, CR (2x), CX, CZ
print(inventory)
Inventory created at 2025-10-13T07:08:19.472273Z
Sending institution: EIDA-Federator (EIDA)
Contains:
Networks (10):
C4, CA, CH, CK, CL, CP, CR (2x), CX, CZ
Stations (591):

Client
Obspy

Add any other context about the problem here.

[petrrr]

@sheimers and @raciner

This issue is already a bit dated.
Could you have a look if this problem still persists and maybe solve it.

Some quick tests provide a somewhat mixed picture.

For the following query

• https://federator.orfeus-eu.org/fdsnws/station/1/query?level=network&network=CQ

I get a 504 Gateway Time-out. But this might be unrelated to this initial problem addressed here.

This one works for me:

• https://federator.orfeus-eu.org/fdsnws/station/1/query?level=station&network=CQ"

For

• https://federator.orfeus-eu.org/fdsnws/station/1/query?level=station&network=C*

I get 9 networks, the following networks: C4, CA, CH, CK, CL, CP, CQ, CX, CZ.

Which do not exactly match the report by @helle-pedersen. CQ is there, but CR is missing.

[helle-pedersen]

Thanks Peter for following up on this. It is interesting that the renewing the test gives yet another result.

I would suggest to use this simple example to figure out if there is some consistency issue, somewhere between federator, node, or the routing) that we could then systematically check for in the future. This could be part of future developments for EIDA i terms of consistency checks.

Good luck with the investigation!

Helle

[kaestli]

I see now that when asking https://federator.orfeus-eu.org/fdsnws/station/1/query?level=station&network=C* ,
i get (shortened):

<FDSNStationXML schemaVersion="1.1">
<Source>EIDA-Federator</Source>
<Sender>EIDA</Sender>
<Created>2025-12-11T10:55:07.060936</Created>
<Network code="CP" startDate="2018-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CK" startDate="1980-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CQ" startDate="2013-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="C4" startDate="2017-02-15T00:00:00" restrictedStatus="open"></Network>
<Network code="CA" startDate="1980-01-01T00:00:00" restrictedStatus="open"></Network>
<Network code="CH" startDate="1980-01-01T00:00:00" restrictedStatus="open"></Network>
<Network code="CX" startDate="1980-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CL" alternateCode="CRLNET" startDate="2000-01-01T00:00:00" endDate="2500-12-31T23:59:59" restrictedStatus="open"></Network>
<Network code="CZ" startDate="1973-01-01T00:00:00Z" restrictedStatus="open"></Network>
</FDSNStationXML>

... and I get it fast, i.e. from the frontend cache caching the entire compiled result for the exact URI. This cache lasts 1 hour (i think, Stefan to correct if wrong). - so sb. has asked this request within the last hour, and got an incomplete result (partial request to ODC for CR timing out)

If I submit the following request: https://federator.orfeus-eu.org/fdsnws/station/1/query?level=station&network=C*&startafter=1902-01-01T00:00:00
, i force a frontend cache miss (as nobody will have asked fort that exact startafter time within the last hour). Now the federator is expected to go back to the end nodes and collect the information with the new formal restriction. This takes a few seconds more and returns ... :

<FDSNStationXML schemaVersion="1.1">
<Source>EIDA-Federator</Source>
<Sender>EIDA</Sender>
<Created>2025-12-11T17:22:22.505906</Created>
<Network code="CP" startDate="2018-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CK" startDate="1980-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CQ" startDate="2013-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CA" startDate="1980-01-01T00:00:00" restrictedStatus="open"></Network>
<Network code="CR" startDate="2021-01-11T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CH" startDate="1980-01-01T00:00:00" restrictedStatus="open"></Network>
<Network code="C4" startDate="2017-02-15T00:00:00" restrictedStatus="open"></Network>
<Network code="CX" startDate="1980-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CZ" startDate="1973-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CL" alternateCode="CRLNET" startDate="2000-01-01T00:00:00" endDate="2500-12-31T23:59:59" restrictedStatus="open"></Network>
</FDSNStationXML>

CR included.
So ODC, from the point of view of the federator, was overloaded, timing out, or not reachable when the first, cache-filling request was created. Now, it is not any more (and, as we know, the fdsn standard does not allow to describe partial failure).

Looking at the EIDA availability report @szgrf.bgr.de, we see that ODC is, from the point of view of the federator, unavailable for some 7% of the requests. We will further investigate the cause (whether we use too many parallel connections, or whether those used just cannot cope with the load).