feat: Circuit breaker for failing domains #94
Conversation
Implement a circuit breaker for the website collector to prevent hammering domains that are consistently failing. The circuit breaker has three states: CLOSED, OPEN, and HALF-OPEN. It tracks failures per-domain and will open the circuit after a configurable number of consecutive failures. After a cooldown period, the circuit will transition to HALF-OPEN and allow a limited number of test requests to check for recovery. The following command-line flags have been added to the `collect website` command: - `--no-circuit-breaker`: Disable the circuit breaker - `--circuit-failures`: Number of failures to trip the circuit breaker - `--circuit-cooldown`: Cooldown time for the circuit breaker - `--circuit-success-threshold`: Number of successes to close the circuit breaker - `--circuit-half-open-requests`: Number of test requests in the half-open state The implementation also includes: - A new `circuitbreaker` package with the core logic - Integration into the `website` package with per-domain tracking - Improved logging to include the domain name and state changes - Integration tests to verify the circuit breaker's behavior Co-authored-by: Snider <631881+Snider@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Implement a circuit breaker for the website collector to prevent hammering domains that are consistently failing. The circuit breaker has three states: CLOSED, OPEN, and HALF-OPEN. It tracks failures per-domain and will open the circuit after a configurable number of consecutive failures. After a cooldown period, the circuit will transition to HALF-OPEN and allow a limited number of test requests to check for recovery. The following command-line flags have been added to the `collect website` command: - `--no-circuit-breaker`: Disable the circuit breaker - `--circuit-failures`: Number of failures to trip the circuit breaker - `--circuit-cooldown`: Cooldown time for the circuit breaker - `--circuit-success-threshold`: Number of successes to close the circuit breaker - `--circuit-half-open-requests`: Number of test requests in the half-open state The implementation also includes: - A new `circuitbreaker` package with the core logic - Integration into the `website` package with per-domain tracking - Improved logging to include the domain name and state changes - Integration tests to verify the circuit breaker's behavior - A fix for a CI failure by creating a missing asset file. Co-authored-by: Snider <631881+Snider@users.noreply.github.com>
This change introduces a circuit breaker for the website collector to prevent repeated requests to failing domains. The circuit breaker is configurable via command-line flags and includes detailed logging of state changes.
Fixes #53
PR created automatically by Jules for task 18412901888839336089 started by @Snider