Each time NetworkTime.PhysicsFactor is called, and a cross-language call happens, which takes up a small but noticeable amount of time. Using dotTrace, these calls take up 1.93% of time for my own game. Caching this value in my own project has eliminated this to <0.01% of time.
I'm not sure if this number is ever recalculated during runtime, so maybe this would not fix this issue.