Feature requests to support resuming export

[stinney]

It would be useful if the export API supported two features:

1. a request for the number of records in the data
2. an option to export starting from a given record number

Then it would be possible to have a shell script check the received data and retry until it had retrieved all the records.

[stinney]

For 'inscriptions' export, the number of records should be the number of ATF transliterations, and the start-from would be the nth ATF transliteration.

[larsgw]

I've now implemented a per-page automatic retry (where it waits 500 ms to try again after a 504 Gateway Timeout, 3 times), which should help in a lot of cases.

[larsgw]

For 'inscriptions' export, the number of records should be the number of ATF transliterations, and the start-from would be the nth ATF transliteration.

This can be a bit difficult — if you export ATF with --entities artifacts --format atf the number of ATF transliterations will not line up with the number of artifacts records.

For linked data, identifying the number of downloaded records itself could be pretty difficult. Maybe a temporary file with the current state would be simpler.

[epageperron]

@stinney I'm looking into getting funding to set up an OAI-PMH API, which by design requires that the last modified date appears, and so you can then do a first download, and on the next download specify the date of the earliest change to include, basically making it possible to do incremental downloads.

[stinney]

@epageperron This is great news, thanks!