jump to end of reports on init, better handle missing next_token#360
Draft
jump to end of reports on init, better handle missing next_token#360
next_token#360Conversation
Contributor
Author
|
while i'm here, i should probably add a test for report polling now that |
jesopo
commented
Aug 30, 2022
| * on the next endpoint call. If not, we're on the last page, | ||
| * which means we want to skip to the end of this page. | ||
| */ | ||
| from = response.next_token ?? this.from + response.event_reports.length; |
Contributor
Author
There was a problem hiding this comment.
this.from + response.event_reports.lengthisn't ideal because the from token is meant to be opaque, but i don't see a better way to do this. i've been in discussions with synapse devs to talk about how this endpoint could be better
Yoric
reviewed
Aug 30, 2022
| * the end of available reports, so we'll only consider reports | ||
| * that happened after we supported report polling. | ||
| */ | ||
| from = response.total; |
Contributor
There was a problem hiding this comment.
That sounds fishy. Isn't from supposed to be an opaque identifier?
| await this.mjolnir.logMessage(LogLevel.ERROR, "getAbuseReports", `failed to update progress: ${ex}`); | ||
| } | ||
| let from; | ||
| if (this.from === null) { |
Contributor
There was a problem hiding this comment.
You're right that we want to do something to avoid displaying tens of thousands of old reports the first time we start the Report Poller on the server.
I believe that we should rather:
- Upon startup of the
ReportPoller.- Load
fromfrom Mjölnir's account data. - If there is no
from
a. Fetch a cutout date from the configuration, or some default date if not specified (say August 1st 2022).
b. Start background enumerating all reports until the cutout date, without displaying them.
c. Use this to initializefrom. This may also give us a first few abuse reports that will need to be returned the first time we callgetAbuseReports.
d. Store this value offromin Mjölnir's account data.
- Load
- Whenever we call
getAbuseReports.- Wait until background enumeration is complete before returning any new reports.
- By definition, we have a
from(which may benullif the server has never encountered any abuse report). - Proceed as usual.
- Whenever we update
from, store it in Mjölnir's account data.
What do you think?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
in draft because this will still iterate the first page of reports when we first call the endpoint, even though we want to skip to the end