RemoteMediator has an initialize API which is guaranteed to complete before loading starts. You can override initialize to check for data staleness to do any setup you need (such as pruning old entries), though I’d personally be a bit surprised if only 1000 rows was causing significant performance issues.

For hot code paths such as these PagingSource queries, you can look to androidx.benchmark to figure out if it’s worth optimizing for.

Edit: You may want to consider simply clearing the table on a successful remote refresh if you don’t want to keep any of the old tweets on refresh as that’s the simplest approach. initialize is meant for the case where you need to do manual pruning / setup / staleness checks (e.g., you can check how old your data is and decide if you want to refresh or not), but simply clearing in remote refresh before inserting the new page works well because you’d expect to have to restart pagination anyway. Make sure to do the clear + insert in the same transaction so you don’t end up with two invalidates due to db updates!

Over the past days I have been getting comfortable with Android Architecture Components and I’ve been mostly interested in the Paging Library for handling and displaying lists of data from a remote source.
For the best UX it is better to load data first into the database, in this case Room then display it in a recyclerview using the pagingLibrary and adapters. My concern is this, lets say an api has 1000s of records or a twitter feed even and all this data has to pass through the db then to the UI, how to handle this storage on the device? Seeing as storing all the data on the device is a bad idea. Is there a way to remove ‘stale’ data or what is the recommended way of doing this?

Edit:
The android team has been working on V3 for the paging library here is a link to an article that might shed more light article here