prometheus http: superfluous response.WriteHeader #319
Description
Describe the bug
the node periodically stops working and gives an error
"http: superfluous response.WriteHeader"
To Reproduce
Steps to reproduce the behavior:
1 enable prometeus
2 request metrics every 3-5 sec
Expected behavior
Screenshots
Server :
- OS: Ubuntu 18.04
- Minter Version 1.1.3 - 1.1.5
Additional context
Mar 30 17:37:58 node-01-lab minterd[60007]: I[2020-03-30|17:37:58.652] Executed block module=state height=445440 validTxs=0 invalidTxs=0
Mar 30 17:37:58 node-01-lab minterd[60007]: I[2020-03-30|17:37:58.656] Updates to validators module=state updates=E43001EB90EA9CE73E63C93EC1F6CCDEF25CE3A2:12324051,E4656BE126AD3B0240FED35DC9218A680BD9D3A1:10746050,C8E0E3E3918BD6B7652AD526B2AA79E314BDBECC:9194007,37A2E1298F9BF90D64E177EEA09C1C2FF7526A03:6838495,C57F376FA419F7D3EFAF92546E2498D4E0CBF64A:5672001,8E36AD28D67B6C8E5A858EB89D8DADCE596ACF64:4314640,3DA17B6ADB8AD4E007C663B45C0592785CF529F5:3703563,E40B020EBF3B0F848054F7B552B377E2E84535D8:3390791,F314DA40E82365C30AAB918598D5BEA7C14ED18C:3170631,274A706B57D907B82BF2A8D09A2C4FC9CBA632A1:2822700,1FA57B4676E3BF71EDEB75AB8D721224F9B63139:2705289,2357D18EF1A7932D81D92997A2DC4EE4926A5167:2455321,6BD6070AD64ACC6AE6E9EBB57F8D1BED37924BA7:2347159,5B0D609F11C95EA6C8737B44098B45671B49C4D8:2144033,472241BE3DA8D2502EC42143207DF2BFBF6FB142:2053701,2F1643AC8F42A196E09D56604E96EABF40D31A7A:1986706,F8117B58FA10E888ADC41D7BD5BDDF69C4B679CE:1796873,48B4DCDF3CB995AF1A250BD9B9DAF0670BD8538F:1785527,1167B9CDA73ACF61DD4D547095A4EEA9E65FB57C:1736153,C56F4FCCE15670E95E747FCD35DE31C2FD165BF2:1399064,79F74D70EECEC249A22CB8F62C7DCBC6E6271272:1344410,76F198090EA6F028EFB2D0A4E3B8F17EBF9A481B:1284654,250AA5DDD60E04F9641880FCDC2C9F1B329B04AB:1113082,FE0366288EE5F77502977233B47AE9100EAC36AA:1079110,83ADD103E6E9958E731A2BC82B78A3D21D71E622:938978,2D8511E411B2DE903554642D62BC2AFB0E56901F:857231,FFFBFF90749719E024A4FE40C3574A3450F887EF:840604,D8C7836BC68DE509C4BA46DF9F0C8EE55A2DE4D4:739739,2EDE07A26576663CF724065B3631346112F3582F:735681,56C2F02DBC15E0E8EEEA06DB22BE110C128EDC9C:685901,3A310C4C18F96E1DB4C69CC0183BDF7868D25909:663410,881E474948E62A227728D157171ECE3D3B4D063F:553912,07945B7190495C2928C7882DE82EA6400B3A3158:537285,43760203B0A8F393256771497850AB8A48801B33:535722,CFBF359405F7DF2D56FDB981533628B5A5EA6B38:508972,AD4EA4CE2D4DCD7FF21A2EA7B303C515D9288ACC:508661,F3185036FC5C609AC240C64E161552FAC45A37EC:468739,B22B984386F6F474EBF711F1B4ADD8F5249553E2:445960,40E03221D1F80EC2E6D3AFEBFDF66C85802E20F6:363613,65B9E8D7CAD755F6FA48D1354E24C89FCC73661F:350103,622C70973D6DBEEA0EB4BDDA66CD4B2595532E17:326557,E766B0193777374B32E95B00A90C94D32F42AAD4:288620,BCAFB7535D8E808E1CCF12CD6BF39D7E5CBF20F1:269886,D44D9A6429BCB7A20897DB5408177FDC5B4DB5FC:269808,35B9BE575C9C427AFAB7141BBA1E0786BD029291:250599,61C35F269CCD2C6253D503BA49B2B339D71A48AD:204006,01959CC6CE0F1AC9A0E809D2DF23FC3AD2858920:192033,7C5CA666A60812AD69D5C0A772797BDAC4143E67:190108,D919FEB139AA85B875735C7EF395D22C24997EC4:160477,86805BF2E9BA3A71526C0A76A3536F805581C341:154085,6239FCBB81C8D1E8471A8A55E40D07CDB1A063B2:132127,4AED1026556A6ADA70E0296AFE92E3D3DE8A12F5:124131,EBEFB0AC72CA98CD7D12AF868A544614E33C5B2C:118663,E40398FAA915B0780576C99B44D715F9F6623930:97923,F1A19A8A8AF10A50AC86A70B2E0CD2CC91F830F6:46100,135AC0AC103089865E32E81621D6779E7235D235:32327
Mar 30 17:38:56 node-01-lab minterd[60007]: 2020/03/30 17:38:56 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
Mar 30 17:39:09 node-01-lab minterd[60007]: E[2020-03-30|17:39:09.208] Connection failed @ sendRoutine module=p2p peer=eb8d5dd2d0baf44e59859bed8a3b6591019e9f2b@xxxxxxxx:59218 conn=MConn{94.130.110.131:59218} err="pong timeout"
Mar 30 17:39:09 node-01-lab minterd[60007]: E[2020-03-30|17:39:09.208] Stopping peer for error module=p2p peer="Peer{MConn{xxxxxxxx:59218} eb8d5dd2d0baf44e59859bed8a3b6591019e9f2b in}" err="pong timeout"
Mar 30 17:39:15 node-01-lab minterd[60007]: E[2020-03-30|17:39:15.718] Connection failed @ sendRoutine module=p2p peer=b781f0e3e1bf44b15d71146332523995d1893bca@xxxxxxxx:57346 conn=MConn{95.216.114.103:57346} err="pong timeout"
Mar 30 17:39:15 node-01-lab minterd[60007]: E[2020-03-30|17:39:15.719] Stopping peer for error module=p2p peer="Peer{MConn{xxxxxxxx:57346} b781f0e3e1bf44b15d71146332523995d1893bca in}" err="pong timeout"
Mar 30 17:39:55 node-01-lab minterd[60007]: 2020/03/30 17:39:55 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
Mar 30 17:40:05 node-01-lab minterd[60007]: I[2020-03-30|17:40:05.518] Committed state module=state height=445440 txs=0 appHash=096730F7890F3AE129FC9AB03D1C7171748F8B24310DBA307476D97AEB18FC58
Mar 30 17:40:05 node-01-lab minterd[60007]: I[2020-03-30|17:40:05.522] Timed out module=consensus dur=-2m4.348303372s height=445441 round=0 step=RoundStepNewHeight
Mar 30 17:40:05 node-01-lab minterd[60007]: I[2020-03-30|17:40:05.522] Added to lastPrecommits: VoteSet{H:445440 R:0 T:2 +2/3:AF2DDDBA147A87AF826E007249B8ECCB1BA743F35F829AEFCE0452A353EE9595:1:7F6342EE2E87(0.7511158827566531) BA{56:x___x_xx_xxxxxxxxxxx___xx_x__xx__xxx__x__xxx_xxxxxxxxx_x} map[b781f0e3e1bf44b15d71146332523995d1893bca:AF2DDDBA147A87AF826E007249B8ECCB1BA743F35F829AEFCE0452A353EE9595:1:7F6342EE2E87]} module=consensus
Mar 30 17:40:05 node-01-lab minterd[60007]: I[2020-03-30|17:40:05.523] Added to lastPrecommits: VoteSet{H:445440 R:0 T:2 +2/3:AF2DDDBA147A87AF826E007249B8ECCB1BA743F35F829AEFCE0452A353EE9595:1:7F6342EE2E87(0.7514391528310052) BA{56:x__xx_xx_xxxxxxxxxxx___xx_x__xx__xxx__x__xxx_xxxxxxxxx_x} map[b781f0e3e1bf44b15d71146332523995d1893bca:AF2DDDBA147A87AF826E007249B8ECCB1BA743F35F829AEFCE0452A353EE9595:1:7F6342EE2E87]} module=consensus
Mar 30 17:40:05 node-01-lab minterd[60007]: I[2020-03-30|17:40:05.523] Added to lastPrecommits: VoteSet{H:445440 R:0 T:2 +2/3:AF2DDDBA147A87AF826E007249B8ECCB1BA743F35F829AEFCE0452A353EE9595:1:7F6342EE2E87(0.8433795139772882) BA{56:x__xx_xx_xxxxxxxxxxx___xx_x__xx__xxx__xx_xxx_xxxxxxxxx_x} map[b781f0e3e1bf44b15d71146332523995d1893bca:AF2DDDBA147A87AF826E007249B8ECCB1BA743F35F829AEFCE0452A353EE9595:1:7F6342EE2E87]} module=consensus
Mar 30 17:40:05 node-01-lab minterd[60007]: I[2020-03-30|17:40:05.524] Added to lastPrecommits: VoteSet{H:445440 R:0 T:2 +2/3:AF2DDDBA147A87AF826E007249B8ECCB1BA743F35F829AEFCE0452A353EE9595:1:7F6342EE2E87(0.8541704964592142) BA{56:x__xx_xx_xxxxxxxxxxx___xx_x__xx__xxx__xx_xxx_xxxxxxxxxxx} map[b781f0e3e1bf44b15d71146332523995d1893bca:AF2DDDBA147A87AF826E007249B8ECCB1BA743F35F829AEFCE0452A353EE9595:1:7F6342EE2E87]} module=consensus
Mar 30 17:40:05 node-01-lab minterd[60007]: I[2020-03-30|17:40:05.524] enterNewRound(445441/0). Current: 445441/0/RoundStepNewHeight module=consensus height=445441 round=0
Mar 30 17:40:05 node-01-lab minterd[60007]: I[2020-03-30|17:40:05.524] enterPropose(445441/0). Current: 445441/0/RoundStepNewRound module=consensus height=445441 round=0
Mar 30 17:40:05 node-01-lab minterd[60007]: I[2020-03-30|17:40:05.622] Stopping gossipVotesRoutine for peer module=consensus peer="Peer{MConn{xxxxxxxx:57346} b781f0e3e1bf44b15d71146332523995d1893bca in}"
Mar 30 17:40:05 node-01-lab minterd[60007]: I[2020-03-30|17:40:05.622] Stopping gossipDataRoutine for peer module=consensus peer="Peer{MConn{xxxxxxxx:59218} eb8d5dd2d0baf44e59859bed8a3b6591019e9f2b in}"
Mar 30 17:40:05 node-01-lab minterd[60007]: I[2020-03-30|17:40:05.622] Stopping gossipVotesRoutine for peer module=consensus peer="Peer{MConn{xxxxxxxx:59218} eb8d5dd2d0baf44e59859bed8a3b6591019e9f2b in}"
Mar 30 17:40:05 node-01-lab minterd[60007]: I[2020-03-30|17:40:05.622] Stopping gossipDataRoutine for peer module=consensus peer="Peer{MConn{xxxxxxxx:57346} b781f0e3e1bf44b15d71146332523995d1893bca in}"
Mar 30 17:40:07 node-01-lab minterd[60007]: I[2020-03-30|17:40:07.522] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{xxxxxxxx:59218} eb8d5dd2d0baf44e59859bed8a3b6591019e9f2b in}"
Mar 30 17:40:07 node-01-lab minterd[60007]: I[2020-03-30|17:40:07.522] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{xxxxxxxx:57346} b781f0e3e1bf44b15d71146332523995d1893bca in}"
Mar 30 17:40:07 node-01-lab minterd[60007]: I[2020-03-30|17:40:07.524] Timed out module=consensus dur=2s height=445441 round=0 step=RoundStepPropose
Mar 30 17:40:07 node-01-lab minterd[60007]: I[2020-03-30|17:40:07.524] enterPrevote(445441/0). Current: 445441/0/RoundStepPropose module=consensus
Mar 30 17:40:07 node-01-lab minterd[60007]: I[2020-03-30|17:40:07.524] enterPrevote: ProposalBlock is nil module=consensus height=445441 round=0
Mar 30 17:40:55 node-01-lab minterd[60007]: 2020/03/30 17:40:55 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
Mar 30 17:41:55 node-01-lab minterd[60007]: 2020/03/30 17:41:55 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
Mar 30 17:42:56 node-01-lab minterd[60007]: 2020/03/30 17:42:56 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
Mar 30 17:43:55 node-01-lab minterd[60007]: 2020/03/30 17:43:55 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
Mar 30 17:44:56 node-01-lab minterd[60007]: 2020/03/30 17:44:56 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
Mar 30 17:45:55 node-01-lab minterd[60007]: 2020/03/30 17:45:55 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
Mar 30 17:46:55 node-01-lab minterd[60007]: 2020/03/30 17:46:55 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
Mar 30 17:47:55 node-01-lab minterd[60007]: 2020/03/30 17:47:55 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
Mar 30 17:48:55 node-01-lab minterd[60007]: 2020/03/30 17:48:55 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
Mar 30 17:49:55 node-01-lab minterd[60007]: 2020/03/30 17:49:55 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
Mar 30 17:50:55 node-01-lab minterd[60007]: 2020/03/30 17:50:55 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)