fix(proxy): resolve circuit breaker state persistence and HalfOpen deadlock

This commit addresses several critical issues in the failover system:

**Circuit breaker state persistence (previous fix)**
- Promote ProviderRouter to ProxyState for cross-request state sharing
- Remove redundant router.rs module
- Fix 429 errors to be retryable (rate limiting should try other providers)

**Hot-update circuit breaker config**
- Add update_circuit_breaker_configs() to ProxyServer and ProxyService
- Connect update_circuit_breaker_config command to running circuit breakers
- Add reset_provider_circuit_breaker() for manual breaker reset

**Fix HalfOpen deadlock bug**
- Change half_open_requests from cumulative count to in-flight count
- Release quota in record_success()/record_failure() when in HalfOpen state
- Prevents permanent deadlock when success_threshold > 1

**Fix duplicate select_providers() call**
- Store providers list in RequestContext, pass to forward_with_retry()
- Avoid consuming HalfOpen quota twice per request
- Single call to select_providers() per request lifecycle

**Add per-provider retry with exponential backoff**
- Implement forward_with_provider_retry() with configurable max_retries
- Backoff delays: 100ms, 200ms, 400ms, etc.
This commit is contained in:
Jason
2025-12-13 22:47:49 +08:00
parent 5d424b1383
commit 5a5ca2a989
11 changed files with 347 additions and 141 deletions
+37
View File
@@ -681,4 +681,41 @@ impl ProxyService {
pub async fn is_running(&self) -> bool {
self.server.read().await.is_some()
}
/// 热更新熔断器配置
///
/// 如果代理服务器正在运行,将新配置应用到所有已创建的熔断器实例
pub async fn update_circuit_breaker_configs(
&self,
config: crate::proxy::CircuitBreakerConfig,
) -> Result<(), String> {
if let Some(server) = self.server.read().await.as_ref() {
server.update_circuit_breaker_configs(config).await;
log::info!("已热更新运行中的熔断器配置");
} else {
log::debug!("代理服务器未运行,熔断器配置将在下次启动时生效");
}
Ok(())
}
/// 重置指定 Provider 的熔断器
///
/// 如果代理服务器正在运行,立即重置内存中的熔断器状态
pub async fn reset_provider_circuit_breaker(
&self,
provider_id: &str,
app_type: &str,
) -> Result<(), String> {
if let Some(server) = self.server.read().await.as_ref() {
server
.reset_provider_circuit_breaker(provider_id, app_type)
.await;
log::info!(
"已重置 Provider {} (app: {}) 的熔断器",
provider_id,
app_type
);
}
Ok(())
}
}