Enforce a min_refresh_duration

[jacksontj]

Before this change a refresh of peers is run on a variety of scenarios--
including any time there is a topology change. In a moderately sized
cluster (200+ nodes) there is basically always a topology change
(connections closing, opening, etc.). Because of this constant state of
topology change the mesh ends up constantly updating the peer set
effectively in a loop.

This patch simply ensures that the refresh happens no more frequently
than initialInterval (2s) to avoid this CPU spinning.

I initially started looking into this because of an issue I ran into using protokube (kubernetes/kops#7427) -- but it seems that I'm not the only one (weaveworks/weave#3595)

[bboreham]

Thanks for the PR!

I wonder if it would be simpler to use a rate-limiter, like in #106 (call Wait() not Allow())?

[bboreham]

Another thought strikes me: we can turn off the "discovery" feature under Kubernetes, because we update the peer list from kube-utils every time a node is deleted, and a new node will connect to all existing nodes at the time it starts.

[jacksontj]

I wonder if it would be simpler to use a rate-limiter, like in #106 (call Wait() not Allow())?

We could use a rate-limiter if we wanted, this is basically the same effect -- so that would be an easy change to make

we can turn off the "discovery" feature under Kubernetes, because we update the peer list from kube-utils every time a node is deleted, and a new node will connect to all existing nodes at the time it starts.

I'm actually using this mesh directly (specifically in protokube, which is part of kops, which manages k8s -- so I'm already not using k8s).

[bboreham]

Yes, basically I am happier to think that golang.org/x/time/rate is battle-tested and covers odd cases better than a locally-coded version.

Ack on your specific use-case.

[bboreham]

After trying lots of different things in PRs, I no longer think the golang.org rate-limiter would be a good fit for this case.