Paging without loading all data from the backend first

[ssteiner]

Hi

I'm dealing with large sets of data. The way the BsBaseGridRepository is written, it needs all records to be loaded in the Index and then does queries on that data.

If you're dealing with large sets of data, you wouldn't want to load all the data from your backend (be it a database.. or even worse, a remote server).. instead you'd want the first page of records and a total record count so the pager can be constructed. Then upon every paging operation, the next page is actually loaded from your backend.

What's the most effective way to go about that? I see a bunch of Counts in the BsBaseGridRepository - which has me worried that I need to override large swaths of standard functionality just to get to an effective paging that only loads data that I actually need.

Thanks
Stephan

[stefanprodan]

We are using the BsGrid for data-sets with millions of rows, the BsBaseGridRepository doesn't require for all the records to be loaded in memory. We use take/skip that does a count on all records and no offset that doesn't need to run count and is optimized for large data-sets. In both cases we never load all records.

[ssteiner]

Good to know that it works..

I guess my confusion stems from the fact that all those queries, the take/skip and the mapping all base on this.Query.. which is an IQueryable. If I had direct access to the database, I guess that would work. I've "inherited" a project though where the data is on a REST backend. And that way, there's no IQueryable to make queries on. While the backend supports paging itself and not only returns records, but also the total number of records for any query, I then wonder how to map that. The communication to the backend is also async (you know how invasive that is) which further complicates matters.

So, what would you recommend in that case? Build an IQueryable abstraction for the remote system? And if so.. do I need the whole shebang or can I get away with implementing only parts of it?