perf: Reparse the game database in parallel#346
Draft
Phantomical wants to merge 2 commits intoKSPModdingLibs:devfrom
Draft
perf: Reparse the game database in parallel#346Phantomical wants to merge 2 commits intoKSPModdingLibs:devfrom
Phantomical wants to merge 2 commits intoKSPModdingLibs:devfrom
Conversation
The config node uses a couple of mutable global statics in order to avoid allocations. This means that if multiple threads attempt to load a config node at the same time then they will stomp all over each other. This commit fixes this issue by making the relevant statics `[ThreadStatic]`, so that each thread gets its own version.
During startup FastLoader reloads the whole game database. This seems to take about 2s on my machine, though I have also see in vary quite a bit in other people's profiles. This PR parallelizes the directory traversal and config node parsing. The final result is that it now takes about 900ms to do. Tuning the parallelism for stock has actually been rather difficult. What I have here is that it recurses 3 directories deep, so: `GameData\ModName\InnerFolder`. This is probably undertuned for stock, since most of the work is in `GameData\Squad\Parts` but going deeper would likely result in too many tasks flying around (and, tbh, I expect that to already be the case on a large modded install already).
Collaborator
Author
|
Marking as draft until I get this performing like I expect. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR has two linked changes:
I have tuned the parallelism to go 3 levels deep from the root (e.g. each
GameData/Mod/SubFolderwill be done in parallel). This is enough to get some benefit on stock KSP. In my tests it takes about 900ms to process theSquad/Partsfolder which is likely to be a lower bound on how much improvement there will be.As it turns out, this is blocked from getting better performance with a modded folder by what looks to be contention on allocations when creating a modified stream reader, so the improvements seem to only really be about 0.5s in either case. I have a patch lying around that should address this, but I haven't tested it yet.
Stock Profiles
Before:
After:
And here's what the breakdown looks like internally
