feat: Add connection pooling and keep-alive #82
Conversation
This change introduces connection pooling, keep-alive, and HTTP/2 support to the website collector. It adds a new httpclient package to create a configurable http.Client and exposes the configuration options as command-line flags. It also adds connection reuse metrics to the output of the collect website command. Co-authored-by: Snider <631881+Snider@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello @Snider, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly upgrades the Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces connection pooling and keep-alive support, which is a great enhancement for the website collector. A new httpclient package is added to provide a configurable http.Client, and new flags are exposed in the collect_website command. The implementation is solid, especially the use of httptrace for metrics. I have a couple of suggestions for improvement: one regarding error handling for the new command-line flags, and another to make the HTTP transport initialization more robust by building upon Go's default transport.
| maxConnections, _ := cmd.Flags().GetInt("max-connections") | ||
| noKeepAlive, _ := cmd.Flags().GetBool("no-keepalive") | ||
| http1, _ := cmd.Flags().GetBool("http1") | ||
| idleTimeout, _ := cmd.Flags().GetDuration("idle-timeout") | ||
| maxIdle, _ := cmd.Flags().GetInt("max-idle") |
There was a problem hiding this comment.
The errors returned from parsing these new command-line flags are being ignored. If a user provides an invalid value for any of these flags (e.g., --max-connections=foo), the error will be discarded, and the variable will be assigned its zero value. This can lead to the application running with an unintended configuration without any warning. The errors should be checked and returned to the user, so they are aware of the issue.
maxConnections, err := cmd.Flags().GetInt("max-connections")
if err != nil {
return err
}
noKeepAlive, err := cmd.Flags().GetBool("no-keepalive")
if err != nil {
return err
}
http1, err := cmd.Flags().GetBool("http1")
if err != nil {
return err
}
idleTimeout, err := cmd.Flags().GetDuration("idle-timeout")
if err != nil {
return err
}
maxIdle, err := cmd.Flags().GetInt("max-idle")
if err != nil {
return err
}| transport := &http.Transport{ | ||
| Proxy: http.ProxyFromEnvironment, | ||
| DialContext: (&net.Dialer{ | ||
| Timeout: 30 * time.Second, | ||
| KeepAlive: 30 * time.Second, | ||
| }).DialContext, | ||
| MaxIdleConns: opts.MaxIdle, | ||
| IdleConnTimeout: opts.IdleTimeout, | ||
| TLSHandshakeTimeout: 10 * time.Second, | ||
| MaxConnsPerHost: opts.MaxPerHost, | ||
| DisableKeepAlives: opts.NoKeepAlive, | ||
| } |
There was a problem hiding this comment.
Instead of creating a new http.Transport from scratch, it's better practice to clone http.DefaultTransport. This ensures that you start with a known-good configuration with sensible defaults (like ForceAttemptHTTP2 for HTTP/2 support, which this PR aims to add) and then customize it. Re-implementing the defaults can lead to missing out on important settings or future improvements to DefaultTransport.
transport := http.DefaultTransport.(*http.Transport).Clone()
transport.MaxIdleConns = opts.MaxIdle
transport.IdleConnTimeout = opts.IdleTimeout
transport.MaxConnsPerHost = opts.MaxPerHost
transport.DisableKeepAlives = opts.NoKeepAlive
This change introduces connection pooling, keep-alive, and HTTP/2 support to the website collector. It adds a new httpclient package to create a configurable http.Client and exposes the configuration options as command-line flags. It also adds connection reuse metrics to the output of the collect website command.
Fixes #56
PR created automatically by Jules for task 16526616877032749391 started by @Snider