MongoDB runs DB instabilities during high load

[coderdj]

This has to do with the new authentication code. Somehow the master loses mongo connection after reporting an error.

[coderdj]

More info. When putting an error in the log I get a seg fault and the following in the master.log file:

2016-01-05T16:18:26 - [!ERROR!] MongoDB error checking command DB
2016-01-05T16:18:26 - [!ERROR!] DBClientBase::findN: transport error: master:27017 ns: run.$cmd query: { count: "daq_control", query: {} }
2016-01-05T16:18:26 - [!ERROR!] Failed to send log message to mongodb
2016-01-05T16:18:26 - [!ERROR!] Missed message: Dispatcher hasn't received a response from node reader2 for at least 30 seconds.


This happened after running at high rate for a while. It's a known issue that if the network to mongo is too slow inserts slow way down and we need a lot of time at the end of the run to read out accumulated data. This took longer than the 30 second timeout and the master tried to report the slave as dead.

A quick google indicates that the transport error can come when there's a general network failure. So it could be that the network was simply saturated and the mongo deployment was not accepting the connection (bad). Another cause can be an authentication failure. However since authentication is now always done with user/pass connection strings that doesn't seem likely. Other log messages were written without an issue during this run.

[coderdj]

I'm not sure if this is my fault, but I'm updating this issue to be a catch-all for runs DB problems related to strain on the DBs. Sometimes the error above comes, but more often we just get a seg fault in Mongo's SASLAuthentication library. Google does not provide an easy to find answer.

This really seems related to running at high rate or performing another resource-heavy operation, like a Mongodump. But here's the thing, the strain isn't on the runs DB at all, the resource-suckers are all on the buffer DBs.