Feat/auto failover switch (#440)

* feat(failover): add auto-failover master switch with proxy integration

- Add persistent auto_failover_enabled setting in database
- Add get/set_auto_failover_enabled commands
- Provider router respects master switch state
- Proxy shutdown automatically disables failover
- Enabling failover auto-starts proxy server
- Optimistic updates for failover queue toggle

* feat(proxy): persist proxy takeover state across app restarts

- Add proxy_takeover_{app_type} settings for per-app state tracking
- Restore proxy takeover state automatically on app startup
- Preserve state on normal exit, clear on manual stop
- Add stop_with_restore_keep_state method for graceful shutdown

* fix(proxy): set takeover state for all apps in start_with_takeover

* fix(windows): hide console window when checking CLI versions

Add CREATE_NO_WINDOW flag to prevent command prompt from flashing
when detecting claude/codex/gemini CLI versions on Windows.

* refactor(failover): make auto-failover toggle per-app independent

- Change setting key from 'auto_failover_enabled' to 'auto_failover_enabled_{app_type}'
- Update provider_router to check per-app failover setting
- When failover disabled, use current provider only; when enabled, use queue order
- Add unit tests for failover enabled/disabled behavior

* feat(failover): auto-switch to higher priority provider on recovery

- After circuit breaker reset, check if recovered provider has higher priority
- Automatically switch back if queue_order is lower (higher priority)
- Stream health check now resets circuit breaker on success/degraded

* chore: remove unused start_proxy_with_takeover command

- Remove command registration from lib.rs
- Add comment clarifying failover queue is preserved on proxy stop

* feat(ui): integrate failover controls into provider cards

- Add failover toggle button to provider card actions
- Show priority badge (P1, P2, ...) for queued providers
- Highlight active provider with green border in failover mode
- Sync drag-drop order with failover queue
- Move per-app failover toggle to FailoverQueueManager
- Simplify SettingsPage failover section

* test(providers): add mocks for failover hooks in ProviderList tests

* refactor(failover): merge failover_queue table into providers

- Add in_failover_queue field to providers table
- Remove standalone failover_queue table and related indexes
- Simplify queue ordering by reusing sort_index field
- Remove reorder_failover_queue and set_failover_item_enabled commands
- Update frontend to use simplified FailoverQueueItem type

* fix(database): ensure in_failover_queue column exists for v2 databases

Add column check in create_tables to handle existing v2 databases
that were created before the failover queue refactor.

* fix(ui): differentiate active provider border color by proxy mode

- Use green border/gradient when proxy takeover is active
- Use blue border/gradient in normal mode (no proxy)
- Improves visual distinction between proxy and non-proxy states

* fix(database): clear provider health record when removing from failover queue

When a provider is removed from the failover queue, its health monitoring
is no longer needed. This change ensures the health record is also deleted
from the database to prevent stale data.

* fix(failover): improve cache cleanup for provider health and circuit breaker

- Use removeQueries instead of invalidateQueries when stopping proxy to
  completely clear health and circuit breaker caches
- Clear provider health and circuit breaker caches when removing from
  failover queue
- Refresh failover queue after drag-sort since queue order depends on
  sort_index
- Only show health badge when provider is in failover queue

* style: apply prettier formatting to App.tsx and ProviderList.tsx

* fix(proxy): handle missing health records and clear health on proxy stop

- Return default healthy state when provider health record not found
- Add clear_provider_health_for_app to clear health for specific app
- Clear app health records when stopping proxy takeover

* fix(proxy): track actual provider used in forwarding for accurate logging

Introduce ForwardResult and ForwardError structs to return the actual
provider that handled the request. This ensures usage statistics and
error logs reflect the correct provider after failover.
This commit is contained in:
YoVinchen
2025-12-23 12:37:36 +08:00
committed by GitHub
parent bf570b6d2a
commit 1586451862
38 changed files with 972 additions and 810 deletions
+74 -8
View File
@@ -14,14 +14,6 @@ pub async fn start_proxy_server(
state.proxy_service.start().await
}
/// 启动代理服务器(带 Live 配置接管)
#[tauri::command]
pub async fn start_proxy_with_takeover(
state: tauri::State<'_, AppState>,
) -> Result<ProxyServerInfo, String> {
state.proxy_service.start_with_takeover().await
}
/// 停止代理服务器(恢复 Live 配置)
#[tauri::command]
pub async fn stop_proxy_with_restore(state: tauri::State<'_, AppState>) -> Result<(), String> {
@@ -111,8 +103,13 @@ pub async fn get_provider_health(
}
/// 重置熔断器
///
/// 重置后会检查是否应该切回队列中优先级更高的供应商:
/// 1. 检查自动故障转移是否开启
/// 2. 如果恢复的供应商在队列中优先级更高(queue_order 更小),则自动切换
#[tauri::command]
pub async fn reset_circuit_breaker(
app_handle: tauri::AppHandle,
state: tauri::State<'_, AppState>,
provider_id: String,
app_type: String,
@@ -129,6 +126,75 @@ pub async fn reset_circuit_breaker(
.reset_provider_circuit_breaker(&provider_id, &app_type)
.await?;
// 3. 检查是否应该切回优先级更高的供应商
let failover_key = format!("auto_failover_enabled_{app_type}");
let auto_failover_enabled = match db.get_setting(&failover_key) {
Ok(Some(value)) => value == "true",
Ok(None) => {
log::debug!(
"[{app_type}] Failover setting '{failover_key}' not found, defaulting to disabled"
);
false
}
Err(e) => {
log::error!(
"[{app_type}] Failed to read failover setting '{failover_key}': {e}, defaulting to disabled"
);
false
}
};
if auto_failover_enabled && state.proxy_service.is_running().await {
// 获取当前供应商 ID
let current_id = db
.get_current_provider(&app_type)
.map_err(|e| e.to_string())?;
if let Some(current_id) = current_id {
// 获取故障转移队列
let queue = db
.get_failover_queue(&app_type)
.map_err(|e| e.to_string())?;
// 找到恢复的供应商和当前供应商在队列中的位置(使用 sort_index
let restored_order = queue
.iter()
.find(|item| item.provider_id == provider_id)
.and_then(|item| item.sort_index);
let current_order = queue
.iter()
.find(|item| item.provider_id == current_id)
.and_then(|item| item.sort_index);
// 如果恢复的供应商优先级更高(sort_index 更小),则切换
if let (Some(restored), Some(current)) = (restored_order, current_order) {
if restored < current {
log::info!(
"[Recovery] 供应商 {provider_id} 已恢复且优先级更高 (P{restored} vs P{current}),自动切换"
);
// 获取供应商名称用于日志和事件
let provider_name = db
.get_all_providers(&app_type)
.ok()
.and_then(|providers| providers.get(&provider_id).map(|p| p.name.clone()))
.unwrap_or_else(|| provider_id.clone());
// 创建故障转移切换管理器并执行切换
let switch_manager =
crate::proxy::failover_switch::FailoverSwitchManager::new(db.clone());
if let Err(e) = switch_manager
.try_switch(Some(&app_handle), &app_type, &provider_id, &provider_name)
.await
{
log::error!("[Recovery] 自动切换失败: {e}");
}
}
}
}
}
Ok(())
}