Reduce thread usage

[panchr]

Summary

1. Remove extraneous sleep from ZookeeperWatcher thread
2. Use TimerSet to execute scheduled tasks. This primarily allows for multiplexing of threads in order to avoid having many "polling" threads which are primarily idle.
3. Remove on_expired_session handler for every Zookeeper(Poll)?Watcher and instead, only use one per ZK client (which is shared). This requires a slightly different callback, but works the same.

Note: I am not sure if this complexity is worth the gain. While this reduces the number of threads, the large portion of main memory usage comes from the additional dual-read watchers, which still exist. Perhaps just doing 1 is enough because it's simpler and less risky.

Testing

• CI
• Testing generated config in HAProxy

# haproxy.cfg contains the generated config from running on the `master` branch
# haproxy-refactored.cfg contains the generated config from this branch

$ egrep -o "server .* id" haproxy.cfg | sort > sorted-servers
$ egrep -o "server .* id" haproxy-refactored.cfg | sort > sorted-servers-refactored
$ diff -s sorted-servers sorted-servers-refactored                                                          
Files sorted-servers and sorted-servers-refactored are identical

• Threads are reduced. Purple is a "control" host running master, whereas the other two are running this branch of Synapse.
  

Misc

One clear benefit is that the CPU usage is reduced to somewhere between 33% and 50% of the original (again, purple is running master and other two are running this branch):

Reviewers

@austin-zhu @gmcatsf @Jason-Jian

[austin-zhu]

is this kill async or not?

[panchr]

I believe it is async from the docs

Begin an immediate shutdown. In-progress tasks will be allowed to complete but enqueued tasks will be dismissed and no new tasks will be accepted. Has no additional effect if the thread pool is not running.

[gmcatsf]

LGTM, can we add a test to verify the behavior when on_session_expired is actually invoked? I do not seem to find one in current rspec.

[panchr]

LGTM, can we add a test to verify the behavior when on_session_expired is actually invoked? I do not seem to find one in current rspec.

I added those tests now @gmcatsf

[gmcatsf]

There are conflicts with current master please resolve them to have a clean CI.