feat: 新增 S3 兼容云存储同步 (#1351)

* Add S3 Cloud Sync design document

Design for adding AWS S3 as a new Cloud Sync backend alongside WebDAV.
Hybrid approach: extract shared sync protocol, add independent S3 transport.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add S3 cloud sync implementation design (reqwest + Sig V4)

Updated design based on 2026-03-06 draft: switches from rust-s3 crate
to hand-rolled AWS Sig V4 on existing reqwest for broader S3-compatible
service support (AWS, MinIO, R2, Alibaba OSS, Tencent COS, Huawei OBS).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add S3 cloud sync implementation plan (11 tasks, TDD)

Detailed step-by-step plan covering: sync_protocol extraction, S3 Sig V4
transport, settings, sync/auto-sync modules, Tauri commands, frontend
presets/dynamic form, and i18n.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* deps: add hmac crate for S3 Sig V4 signing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: extract sync_protocol.rs from webdav_sync.rs for shared use

Move transport-agnostic sync protocol logic (constants, types, snapshot
building, manifest validation, artifact verification, snapshot application,
utilities) into a new shared sync_protocol module so both WebDAV and the
upcoming S3 transport can reuse it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: use transport-neutral error keys in sync_protocol

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3 transport layer with AWS Sig V4 signing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3SyncSettings to AppSettings

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3 sync module with upload/download/fetch

Implements the S3 sync protocol layer (s3_sync.rs) that combines the
shared sync_protocol with the S3 transport. Mirrors the WebDAV sync
module structure with independent sync mutex, connection check,
upload, download, fetch_remote_info, and sync status persistence.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3 auto sync worker with debounce

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3 sync Tauri commands and auto sync worker startup

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3 sync TypeScript types and API layer

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3 sync i18n translations (en/zh/ja)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3 sync presets and dynamic form to sync settings

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: preserve HTTP scheme for S3 custom endpoints (MinIO support)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: add live S3 integration tests (env-var driven, --ignored)

Run with: S3_TEST_AK=... S3_TEST_SK=... S3_TEST_BUCKET=... cargo test --lib services::s3::integration_tests -- --ignored

Verifies test_connection, put_object, get_object, head_object, and 404
handling against a real S3 bucket using the project's own Sig V4 signing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: remove internal design docs before PR

* fix: wire S3 auto-sync to DB hook & sync UI state on async load

- P1: Add s3_auto_sync::notify_db_changed call in SQLite update_hook
  so S3 auto-sync worker receives DB change signals (was only wired
  for WebDAV, leaving S3 worker idle)

- P2: Add useEffect to update syncType selector when s3Config loads
  asynchronously, preventing stale "webdav" default for S3 users

* fix: satisfy clippy for s3 sync

* fix: address s3 sync review feedback

---------

Co-authored-by: Keith (via OpenClaw) <keithyt06@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Jason <farion1231@gmail.com>
This commit is contained in:
Keith Yu
2026-06-04 22:18:51 +08:00
committed by GitHub
parent ea95f39adf
commit 2a24da517f
26 changed files with 4393 additions and 795 deletions
+1
View File
@@ -749,6 +749,7 @@ dependencies = [
"dirs 5.0.1",
"flate2",
"futures",
"hmac",
"http",
"http-body",
"http-body-util",
+1
View File
@@ -77,6 +77,7 @@ indexmap = { version = "2", features = ["serde"] }
rust_decimal = "1.33"
uuid = { version = "1.11", features = ["v4"] }
sha2 = "0.10"
hmac = "0.12"
json5 = "0.4"
json-five = "0.3.1"
+2
View File
@@ -29,6 +29,7 @@ mod subscription;
mod sync_support;
mod lightweight;
mod s3_sync;
mod usage;
mod webdav_sync;
mod workspace;
@@ -61,6 +62,7 @@ pub use stream_check::*;
pub use subscription::*;
pub use lightweight::*;
pub use s3_sync::*;
pub use usage::*;
pub use webdav_sync::*;
pub use workspace::*;
+352
View File
@@ -0,0 +1,352 @@
#![allow(non_snake_case)]
use serde_json::{json, Value};
use tauri::State;
use crate::commands::sync_support::{
attach_warning, post_sync_warning_from_result, run_post_import_sync,
};
use crate::error::AppError;
use crate::services::s3_sync as s3_sync_service;
use crate::settings::{self, S3SyncSettings};
use crate::store::AppState;
fn persist_sync_error(settings: &mut S3SyncSettings, error: &AppError, source: &str) {
settings.status.last_error = Some(error.to_string());
settings.status.last_error_source = Some(source.to_string());
let _ = settings::update_s3_sync_status(settings.status.clone());
}
fn s3_not_configured_error() -> String {
AppError::localized(
"s3.sync.not_configured",
"未配置 S3 同步",
"S3 sync is not configured.",
)
.to_string()
}
fn s3_sync_disabled_error() -> String {
AppError::localized("s3.sync.disabled", "S3 同步未启用", "S3 sync is disabled.").to_string()
}
fn require_enabled_s3_settings() -> Result<S3SyncSettings, String> {
let settings = settings::get_s3_sync_settings().ok_or_else(s3_not_configured_error)?;
if !settings.enabled {
return Err(s3_sync_disabled_error());
}
Ok(settings)
}
fn resolve_secret_for_request(
mut incoming: S3SyncSettings,
existing: Option<S3SyncSettings>,
preserve_empty_secret: bool,
) -> S3SyncSettings {
if let Some(existing_settings) = existing {
if preserve_empty_secret && incoming.secret_access_key.is_empty() {
incoming.secret_access_key = existing_settings.secret_access_key;
}
}
incoming
}
#[cfg(test)]
fn s3_sync_mutex() -> &'static tokio::sync::Mutex<()> {
s3_sync_service::sync_mutex()
}
async fn run_with_s3_lock<T, Fut>(operation: Fut) -> Result<T, AppError>
where
Fut: std::future::Future<Output = Result<T, AppError>>,
{
s3_sync_service::run_with_sync_lock(operation).await
}
fn map_sync_result<T, F>(result: Result<T, AppError>, on_error: F) -> Result<T, String>
where
F: FnOnce(&AppError),
{
match result {
Ok(value) => Ok(value),
Err(err) => {
on_error(&err);
Err(err.to_string())
}
}
}
#[tauri::command]
pub async fn s3_test_connection(
settings: S3SyncSettings,
#[allow(non_snake_case)] preserveEmptyPassword: Option<bool>,
) -> Result<Value, String> {
let preserve_empty = preserveEmptyPassword.unwrap_or(true);
let resolved =
resolve_secret_for_request(settings, settings::get_s3_sync_settings(), preserve_empty);
s3_sync_service::check_connection(&resolved)
.await
.map_err(|e| e.to_string())?;
Ok(json!({
"success": true,
"message": "S3 connection ok"
}))
}
#[tauri::command]
pub async fn s3_sync_upload(state: State<'_, AppState>) -> Result<Value, String> {
let db = state.db.clone();
let mut settings = require_enabled_s3_settings()?;
let result = run_with_s3_lock(s3_sync_service::upload(&db, &mut settings)).await;
map_sync_result(result, |error| {
persist_sync_error(&mut settings, error, "manual")
})
}
#[tauri::command]
pub async fn s3_sync_download(state: State<'_, AppState>) -> Result<Value, String> {
let db = state.db.clone();
let db_for_sync = db.clone();
let mut settings = require_enabled_s3_settings()?;
let _auto_sync_suppression = crate::services::s3_auto_sync::AutoSyncSuppressionGuard::new();
let sync_result = run_with_s3_lock(s3_sync_service::download(&db, &mut settings)).await;
let mut result = map_sync_result(sync_result, |error| {
persist_sync_error(&mut settings, error, "manual")
})?;
// Post-download sync is best-effort: snapshot restore has already succeeded.
let warning = post_sync_warning_from_result(
tauri::async_runtime::spawn_blocking(move || run_post_import_sync(db_for_sync))
.await
.map_err(|e| e.to_string()),
);
if let Some(msg) = warning.as_ref() {
log::warn!("[S3] post-download sync warning: {msg}");
}
result = attach_warning(result, warning);
Ok(result)
}
#[tauri::command]
pub async fn s3_sync_save_settings(
settings: S3SyncSettings,
#[allow(non_snake_case)] passwordTouched: Option<bool>,
) -> Result<Value, String> {
let password_touched = passwordTouched.unwrap_or(false);
let existing = settings::get_s3_sync_settings();
let mut sync_settings =
resolve_secret_for_request(settings, existing.clone(), !password_touched);
// Preserve server-owned fields that the frontend does not manage
if let Some(existing_settings) = existing {
sync_settings.status = existing_settings.status;
}
sync_settings.normalize();
sync_settings.validate().map_err(|e| e.to_string())?;
settings::set_s3_sync_settings(Some(sync_settings)).map_err(|e| e.to_string())?;
Ok(json!({ "success": true }))
}
#[tauri::command]
pub async fn s3_sync_fetch_remote_info() -> Result<Value, String> {
let settings = require_enabled_s3_settings()?;
let info = s3_sync_service::fetch_remote_info(&settings)
.await
.map_err(|e| e.to_string())?;
Ok(info.unwrap_or(json!({ "empty": true })))
}
#[cfg(test)]
mod tests {
use super::{
map_sync_result, persist_sync_error, require_enabled_s3_settings,
resolve_secret_for_request, run_with_s3_lock, s3_sync_mutex,
};
use crate::error::AppError;
use crate::settings::{AppSettings, S3SyncSettings};
use serial_test::serial;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::Duration;
#[tokio::test]
async fn s3_sync_mutex_is_singleton() {
let a = s3_sync_mutex() as *const _;
let b = s3_sync_mutex() as *const _;
assert_eq!(a, b);
}
#[tokio::test]
#[serial]
async fn s3_sync_mutex_serializes_concurrent_access() {
let guard = s3_sync_mutex().lock().await;
let acquired = Arc::new(AtomicBool::new(false));
let acquired_bg = Arc::clone(&acquired);
let waiter = tokio::spawn(async move {
let _inner_guard = s3_sync_mutex().lock().await;
acquired_bg.store(true, Ordering::SeqCst);
});
tokio::time::sleep(Duration::from_millis(40)).await;
assert!(!acquired.load(Ordering::SeqCst));
drop(guard);
tokio::time::timeout(Duration::from_secs(1), waiter)
.await
.expect("background task should complete after lock release")
.expect("background task should not panic");
assert!(acquired.load(Ordering::SeqCst));
}
#[tokio::test]
#[serial]
async fn map_sync_result_runs_error_handler_after_lock_release() {
let result =
run_with_s3_lock(async { Err::<(), AppError>(AppError::Config("boom".to_string())) })
.await;
let mut lock_released = false;
let mapped = map_sync_result(result, |_| {
lock_released = s3_sync_mutex().try_lock().is_ok();
});
assert!(mapped.is_err());
assert!(lock_released);
}
#[test]
fn resolve_secret_for_request_preserves_existing_when_requested() {
let incoming = S3SyncSettings {
region: "us-east-1".to_string(),
bucket: "my-bucket".to_string(),
access_key_id: "AKID".to_string(),
secret_access_key: String::new(),
..S3SyncSettings::default()
};
let existing = Some(S3SyncSettings {
secret_access_key: "SECRET".to_string(),
..S3SyncSettings::default()
});
let resolved = resolve_secret_for_request(incoming, existing, true);
assert_eq!(resolved.secret_access_key, "SECRET");
}
#[test]
fn resolve_secret_for_request_allows_explicit_empty_secret() {
let incoming = S3SyncSettings {
region: "us-east-1".to_string(),
bucket: "my-bucket".to_string(),
access_key_id: "AKID".to_string(),
secret_access_key: String::new(),
..S3SyncSettings::default()
};
let existing = Some(S3SyncSettings {
secret_access_key: "SECRET".to_string(),
..S3SyncSettings::default()
});
let resolved = resolve_secret_for_request(incoming, existing, false);
assert!(resolved.secret_access_key.is_empty());
}
#[test]
#[serial]
fn persist_sync_error_updates_status_without_overwriting_credentials() {
let test_home = std::env::temp_dir().join("cc-switch-s3-sync-error-status-test");
let _ = std::fs::remove_dir_all(&test_home);
std::fs::create_dir_all(&test_home).expect("create test home");
std::env::set_var("CC_SWITCH_TEST_HOME", &test_home);
crate::settings::update_settings(AppSettings::default()).expect("reset settings");
let mut current = S3SyncSettings {
enabled: true,
region: "us-east-1".to_string(),
bucket: "my-bucket".to_string(),
access_key_id: "AKID".to_string(),
secret_access_key: "SECRET".to_string(),
remote_root: "cc-switch-sync".to_string(),
profile: "default".to_string(),
..S3SyncSettings::default()
};
crate::settings::set_s3_sync_settings(Some(current.clone())).expect("seed s3 settings");
persist_sync_error(
&mut current,
&crate::error::AppError::Config("boom".to_string()),
"manual",
);
let after = crate::settings::get_s3_sync_settings().expect("read s3 settings");
assert_eq!(after.region, "us-east-1");
assert_eq!(after.bucket, "my-bucket");
assert_eq!(after.access_key_id, "AKID");
assert_eq!(after.secret_access_key, "SECRET");
assert_eq!(after.remote_root, "cc-switch-sync");
assert_eq!(after.profile, "default");
assert!(
after
.status
.last_error
.as_deref()
.unwrap_or_default()
.contains("boom"),
"status error should be updated"
);
assert_eq!(after.status.last_error_source.as_deref(), Some("manual"));
}
#[test]
#[serial]
fn require_enabled_s3_settings_rejects_disabled_config() {
let test_home = std::env::temp_dir().join("cc-switch-s3-sync-enabled-disabled-test");
let _ = std::fs::remove_dir_all(&test_home);
std::fs::create_dir_all(&test_home).expect("create test home");
std::env::set_var("CC_SWITCH_TEST_HOME", &test_home);
crate::settings::update_settings(AppSettings::default()).expect("reset settings");
crate::settings::set_s3_sync_settings(Some(S3SyncSettings {
enabled: false,
region: "us-east-1".to_string(),
bucket: "my-bucket".to_string(),
access_key_id: "AKID".to_string(),
secret_access_key: "SECRET".to_string(),
..S3SyncSettings::default()
}))
.expect("seed disabled s3 settings");
let err = require_enabled_s3_settings().expect_err("disabled settings should fail");
assert!(
err.contains("disabled") || err.contains("未启用"),
"unexpected error: {err}"
);
}
#[test]
#[serial]
fn require_enabled_s3_settings_returns_settings_when_enabled() {
let test_home = std::env::temp_dir().join("cc-switch-s3-sync-enabled-ok-test");
let _ = std::fs::remove_dir_all(&test_home);
std::fs::create_dir_all(&test_home).expect("create test home");
std::env::set_var("CC_SWITCH_TEST_HOME", &test_home);
crate::settings::update_settings(AppSettings::default()).expect("reset settings");
crate::settings::set_s3_sync_settings(Some(S3SyncSettings {
enabled: true,
region: "us-east-1".to_string(),
bucket: "my-bucket".to_string(),
access_key_id: "AKID".to_string(),
secret_access_key: "SECRET".to_string(),
..S3SyncSettings::default()
}))
.expect("seed enabled s3 settings");
let settings = require_enabled_s3_settings().expect("enabled settings should be accepted");
assert!(settings.enabled);
assert_eq!(settings.region, "us-east-1");
}
}
+73 -1
View File
@@ -21,6 +21,20 @@ fn merge_settings_for_save(
}
_ => {}
}
match (&mut incoming.s3_sync, &existing.s3_sync) {
// incoming 没有 s3 → 保留现有
(None, _) => {
incoming.s3_sync = existing.s3_sync.clone();
}
// incoming 有 s3 但密钥为空,且现有有密钥 → 填回现有密钥
(Some(incoming_sync), Some(existing_sync))
if incoming_sync.secret_access_key.is_empty()
&& !existing_sync.secret_access_key.is_empty() =>
{
incoming_sync.secret_access_key = existing_sync.secret_access_key.clone();
}
_ => {}
}
if incoming.local_migrations.is_none() {
incoming.local_migrations = existing.local_migrations.clone();
} else if let (Some(incoming_migrations), Some(existing_migrations)) =
@@ -103,7 +117,7 @@ mod tests {
use super::merge_settings_for_save;
use crate::settings::{
AppSettings, CodexProviderTemplateMigration, CodexThirdPartyHistoryProviderBucketMigration,
LocalMigrations, WebDavSyncSettings,
LocalMigrations, S3SyncSettings, WebDavSyncSettings,
};
#[test]
@@ -226,6 +240,64 @@ mod tests {
);
}
#[test]
fn save_settings_should_preserve_existing_s3_when_payload_omits_it() {
let existing = AppSettings {
s3_sync: Some(S3SyncSettings {
bucket: "bucket".to_string(),
access_key_id: "ak".to_string(),
secret_access_key: "secret".to_string(),
..S3SyncSettings::default()
}),
..AppSettings::default()
};
let incoming = AppSettings::default();
let merged = merge_settings_for_save(incoming, &existing);
assert!(merged.s3_sync.is_some());
assert_eq!(
merged
.s3_sync
.as_ref()
.map(|v| v.secret_access_key.as_str()),
Some("secret")
);
}
#[test]
fn save_settings_should_preserve_s3_secret_when_incoming_has_empty_secret() {
let existing = AppSettings {
s3_sync: Some(S3SyncSettings {
bucket: "bucket".to_string(),
access_key_id: "ak".to_string(),
secret_access_key: "secret".to_string(),
..S3SyncSettings::default()
}),
..AppSettings::default()
};
let incoming = AppSettings {
s3_sync: Some(S3SyncSettings {
bucket: "bucket".to_string(),
access_key_id: "ak".to_string(),
secret_access_key: "".to_string(),
..S3SyncSettings::default()
}),
..AppSettings::default()
};
let merged = merge_settings_for_save(incoming, &existing);
assert_eq!(
merged
.s3_sync
.as_ref()
.map(|v| v.secret_access_key.as_str()),
Some("secret")
);
}
#[test]
fn save_settings_should_preserve_local_migrations_when_payload_omits_it() {
let existing = AppSettings {
+1
View File
@@ -82,6 +82,7 @@ fn register_db_change_hook(conn: &Connection) {
|action: Action, _database: &str, table: &str, _row_id: i64| match action {
Action::SQLITE_INSERT | Action::SQLITE_UPDATE | Action::SQLITE_DELETE => {
crate::services::webdav_auto_sync::notify_db_changed(table);
crate::services::s3_auto_sync::notify_db_changed(table);
}
_ => {}
},
+9
View File
@@ -865,6 +865,10 @@ pub fn run() {
app_state.db.clone(),
app.handle().clone(),
);
crate::services::s3_auto_sync::start_worker(
app_state.db.clone(),
app.handle().clone(),
);
// 将同一个实例注入到全局状态,避免重复创建导致的不一致
app.manage(app_state);
@@ -1198,6 +1202,11 @@ pub fn run() {
commands::webdav_sync_download,
commands::webdav_sync_save_settings,
commands::webdav_sync_fetch_remote_info,
commands::s3_test_connection,
commands::s3_sync_upload,
commands::s3_sync_download,
commands::s3_sync_save_settings,
commands::s3_sync_fetch_remote_info,
commands::save_file_dialog,
commands::open_file_dialog,
commands::open_zip_file_dialog,
+4
View File
@@ -10,6 +10,9 @@ pub mod omo;
pub mod prompt;
pub mod provider;
pub mod proxy;
pub mod s3;
pub mod s3_auto_sync;
pub mod s3_sync;
pub mod session_usage;
pub mod session_usage_codex;
pub mod session_usage_gemini;
@@ -18,6 +21,7 @@ pub mod speedtest;
pub mod sql_helpers;
pub mod stream_check;
pub mod subscription;
pub mod sync_protocol;
pub mod usage_cache;
pub mod usage_stats;
pub mod webdav;
+926
View File
@@ -0,0 +1,926 @@
//! S3 HTTP transport layer.
//!
//! Low-level HTTP primitives for S3 operations (PUT, GET, HEAD).
//! Implements AWS Signature Version 4 request signing.
//! The sync protocol logic lives in the upcoming `s3_sync` module.
use reqwest::StatusCode;
use std::time::Duration;
use url::Url;
use crate::error::AppError;
use crate::proxy::http_client;
use futures::StreamExt;
const DEFAULT_TIMEOUT_SECS: u64 = 30;
/// Timeout for large file transfers (PUT / GET of db.sql, skills.zip, etc.).
const TRANSFER_TIMEOUT_SECS: u64 = 300;
// ─── Credentials ─────────────────────────────────────────────
/// S3-compatible storage credentials.
pub(crate) struct S3Credentials {
pub access_key_id: String,
pub secret_access_key: String,
pub region: String,
pub bucket: String,
/// Custom endpoint host (e.g. `minio.example.com:9000`).
/// Empty string means AWS official endpoint.
pub endpoint: String,
}
// ─── URL construction ────────────────────────────────────────
/// Returns `true` for AWS official endpoints (empty or contains `amazonaws.com`).
fn is_aws_endpoint(endpoint: &str) -> bool {
endpoint.is_empty() || endpoint.contains("amazonaws.com")
}
/// Split an endpoint into its scheme and host-with-port parts.
///
/// Preserves the original scheme when one is provided, defaulting to `"https"`
/// when the endpoint is bare (no `://` prefix).
///
/// ```text
/// "http://minio:9000" → ("http", "minio:9000")
/// "https://storage.example.com" → ("https", "storage.example.com")
/// "minio:9000" → ("https", "minio:9000")
/// "storage.example.com" → ("https", "storage.example.com")
/// ```
fn split_scheme_host(endpoint: &str) -> (&str, &str) {
if let Some(rest) = endpoint.strip_prefix("http://") {
("http", rest.trim_end_matches('/'))
} else if let Some(rest) = endpoint.strip_prefix("https://") {
("https", rest.trim_end_matches('/'))
} else {
("https", endpoint.trim_end_matches('/'))
}
}
/// Build the full URL for an S3 object.
///
/// - AWS endpoints use virtual-hosted style: `https://{bucket}.s3.{region}.amazonaws.com/{key}`
/// - Custom endpoints use path style: `https://{endpoint}/{bucket}/{key}`
fn build_object_url(creds: &S3Credentials, key: &str) -> String {
let key = key.trim_start_matches('/');
if is_aws_endpoint(&creds.endpoint) {
format!(
"https://{}.s3.{}.amazonaws.com/{}",
creds.bucket, creds.region, key
)
} else {
let (scheme, host) = split_scheme_host(&creds.endpoint);
format!("{}://{}/{}/{}", scheme, host, creds.bucket, key)
}
}
/// Build the bucket-level URL (for HEAD bucket / test connection).
fn build_bucket_url(creds: &S3Credentials) -> String {
if is_aws_endpoint(&creds.endpoint) {
format!(
"https://{}.s3.{}.amazonaws.com/",
creds.bucket, creds.region
)
} else {
let (scheme, host) = split_scheme_host(&creds.endpoint);
format!("{}://{}/{}/", scheme, host, creds.bucket)
}
}
// ─── Cryptographic helpers ───────────────────────────────────
fn hmac_sha256(key: &[u8], data: &[u8]) -> Vec<u8> {
use hmac::{Hmac, Mac};
type HmacSha256 = Hmac<sha2::Sha256>;
let mut mac = HmacSha256::new_from_slice(key).expect("HMAC accepts any key length");
mac.update(data);
mac.finalize().into_bytes().to_vec()
}
fn sha256_hex(data: &[u8]) -> String {
use sha2::{Digest, Sha256};
format!("{:x}", Sha256::digest(data))
}
/// Percent-encode following AWS Sig V4 rules (RFC 3986 unreserved characters only).
fn uri_encode(input: &str, encode_slash: bool) -> String {
let mut out = String::with_capacity(input.len());
for byte in input.bytes() {
match byte {
b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => {
out.push(byte as char);
}
b'/' if !encode_slash => out.push('/'),
_ => {
use std::fmt::Write;
let _ = write!(out, "%{:02X}", byte);
}
}
}
out
}
// ─── AWS Signature V4 signing ────────────────────────────────
/// Sign an HTTP request using AWS Signature Version 4.
///
/// Mutates `headers` by adding `host`, `x-amz-date`, `x-amz-content-sha256`,
/// and the final `authorization` header.
fn sign_request(
method: &str,
url: &Url,
headers: &mut reqwest::header::HeaderMap,
body_hash: &str,
creds: &S3Credentials,
now: chrono::DateTime<chrono::Utc>,
) {
let timestamp = now.format("%Y%m%dT%H%M%SZ").to_string();
let datestamp = now.format("%Y%m%d").to_string();
// ── Step 1: Add required headers ──
let host_value = match url.port() {
Some(port) => format!("{}:{}", url.host_str().unwrap_or_default(), port),
None => url.host_str().unwrap_or_default().to_string(),
};
headers.insert("host", host_value.parse().unwrap());
headers.insert("x-amz-date", timestamp.parse().unwrap());
headers.insert("x-amz-content-sha256", body_hash.parse().unwrap());
// ── Step 2: Build canonical request ──
// Canonical URI (already percent-encoded by the url crate).
let canonical_uri = url.path();
let canonical_uri = if canonical_uri.is_empty() {
"/"
} else {
canonical_uri
};
// Canonical query string — sorted, re-encoded per Sig V4 rules.
let mut query_pairs: Vec<(String, String)> = url
.query_pairs()
.map(|(k, v)| (k.into_owned(), v.into_owned()))
.collect();
query_pairs.sort_by(|a, b| a.0.cmp(&b.0).then_with(|| a.1.cmp(&b.1)));
let canonical_query = if query_pairs.is_empty() {
String::new()
} else {
query_pairs
.iter()
.map(|(k, v)| format!("{}={}", uri_encode(k, true), uri_encode(v, true)))
.collect::<Vec<_>>()
.join("&")
};
// Canonical headers — sorted by lowercase name.
let mut header_names: Vec<String> = headers.keys().map(|k| k.as_str().to_lowercase()).collect();
header_names.sort();
header_names.dedup();
let canonical_headers: String = header_names
.iter()
.map(|name| {
let value = headers
.get(name.as_str())
.map(|v| v.to_str().unwrap_or("").trim().to_string())
.unwrap_or_default();
format!("{}:{}\n", name, value)
})
.collect();
let signed_headers = header_names.join(";");
let canonical_request = format!(
"{}\n{}\n{}\n{}\n{}\n{}",
method, canonical_uri, canonical_query, canonical_headers, signed_headers, body_hash
);
// ── Step 3: Build string to sign ──
let scope = format!("{}/{}/s3/aws4_request", datestamp, creds.region);
let string_to_sign = format!(
"AWS4-HMAC-SHA256\n{}\n{}\n{}",
timestamp,
scope,
sha256_hex(canonical_request.as_bytes())
);
// ── Step 4: Derive signing key ──
// HMAC(HMAC(HMAC(HMAC("AWS4"+secret, date), region), "s3"), "aws4_request")
let k_date = hmac_sha256(
format!("AWS4{}", creds.secret_access_key).as_bytes(),
datestamp.as_bytes(),
);
let k_region = hmac_sha256(&k_date, creds.region.as_bytes());
let k_service = hmac_sha256(&k_region, b"s3");
let k_signing = hmac_sha256(&k_service, b"aws4_request");
// ── Step 5: Compute signature ──
let sig_bytes = hmac_sha256(&k_signing, string_to_sign.as_bytes());
let signature: String = sig_bytes.iter().map(|b| format!("{:02x}", b)).collect();
// ── Step 6: Add Authorization header ──
let authorization = format!(
"AWS4-HMAC-SHA256 Credential={}/{}, SignedHeaders={}, Signature={}",
creds.access_key_id, scope, signed_headers, signature
);
headers.insert("authorization", authorization.parse().unwrap());
}
// ─── Error helpers ───────────────────────────────────────────
/// Redact a URL for safe inclusion in error messages (strips query parameters).
fn redact_url(raw: &str) -> String {
match Url::parse(raw) {
Ok(parsed) => {
let mut out = format!("{}://", parsed.scheme());
if let Some(host) = parsed.host_str() {
out.push_str(host);
}
if let Some(port) = parsed.port() {
out.push(':');
out.push_str(&port.to_string());
}
out.push_str(parsed.path());
out
}
Err(_) => raw.split('?').next().unwrap_or(raw).to_string(),
}
}
fn s3_transport_error(
key: &'static str,
op_zh: &str,
op_en: &str,
err: &reqwest::Error,
) -> AppError {
let (zh_reason, en_reason) = if err.is_timeout() {
("请求超时", "request timed out")
} else if err.is_connect() {
("连接失败", "connection failed")
} else if err.is_request() {
("请求构造失败", "request build failed")
} else {
("网络请求失败", "network request failed")
};
AppError::localized(
key,
format!("S3 {op_zh}失败({zh_reason}"),
format!("S3 {op_en} failed ({en_reason})"),
)
}
fn s3_status_error(op: &str, status: StatusCode, url: &str) -> AppError {
let safe_url = redact_url(url);
let mut zh = format!("S3 {op} 失败: {status} ({safe_url})");
let mut en = format!("S3 {op} failed: {status} ({safe_url})");
if matches!(status, StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN) {
zh.push_str("。请检查 Access Key ID 和 Secret Access Key。");
en.push_str(". Please verify your Access Key ID and Secret Access Key.");
} else if status == StatusCode::NOT_FOUND && op == "HEAD bucket" {
zh.push_str("。请检查存储桶名称和区域是否正确。");
en.push_str(". Please check the bucket name and region.");
}
AppError::localized("s3.http.status", zh, en)
}
fn response_too_large_error(url: &str, max_bytes: usize) -> AppError {
let max_mb = max_bytes / 1024 / 1024;
AppError::localized(
"s3.response_too_large",
format!("S3 响应体超过上限({} MB: {}", max_mb, redact_url(url)),
format!(
"S3 response body exceeds limit ({} MB): {}",
max_mb,
redact_url(url)
),
)
}
fn ensure_content_length_within_limit(
headers: &reqwest::header::HeaderMap,
max_bytes: usize,
url: &str,
) -> Result<(), AppError> {
let Some(cl) = headers.get(reqwest::header::CONTENT_LENGTH) else {
return Ok(());
};
let Ok(raw) = cl.to_str() else {
return Ok(());
};
let Ok(value) = raw.parse::<u64>() else {
return Ok(());
};
if value > max_bytes as u64 {
return Err(response_too_large_error(url, max_bytes));
}
Ok(())
}
// ─── Transport functions ─────────────────────────────────────
/// Test S3 connectivity by sending HEAD to the bucket root.
pub(crate) async fn test_connection(creds: &S3Credentials) -> Result<(), AppError> {
let url_str = build_bucket_url(creds);
let url = Url::parse(&url_str).map_err(|e| {
AppError::localized(
"s3.url.invalid",
format!("S3 URL 无效: {e}"),
format!("Invalid S3 URL: {e}"),
)
})?;
let client = http_client::get();
let body_hash = sha256_hex(b"");
let mut headers = reqwest::header::HeaderMap::new();
sign_request(
"HEAD",
&url,
&mut headers,
&body_hash,
creds,
chrono::Utc::now(),
);
let resp = client
.head(url.as_str())
.headers(headers)
.timeout(Duration::from_secs(DEFAULT_TIMEOUT_SECS))
.send()
.await
.map_err(|e| s3_transport_error("s3.connection_failed", "连接", "connection", &e))?;
if resp.status().is_success() {
return Ok(());
}
Err(s3_status_error("HEAD bucket", resp.status(), &url_str))
}
/// Upload bytes to an S3 object.
pub(crate) async fn put_object(
creds: &S3Credentials,
key: &str,
bytes: Vec<u8>,
content_type: &str,
) -> Result<(), AppError> {
let url_str = build_object_url(creds, key);
let url = Url::parse(&url_str).map_err(|e| {
AppError::localized(
"s3.url.invalid",
format!("S3 URL 无效: {e}"),
format!("Invalid S3 URL: {e}"),
)
})?;
let client = http_client::get();
let body_hash = sha256_hex(&bytes);
let mut headers = reqwest::header::HeaderMap::new();
headers.insert("content-type", content_type.parse().unwrap());
sign_request(
"PUT",
&url,
&mut headers,
&body_hash,
creds,
chrono::Utc::now(),
);
let resp = client
.put(url.as_str())
.headers(headers)
.body(bytes)
.timeout(Duration::from_secs(TRANSFER_TIMEOUT_SECS))
.send()
.await
.map_err(|e| s3_transport_error("s3.put_failed", "PUT 请求", "PUT request", &e))?;
if resp.status().is_success() {
return Ok(());
}
Err(s3_status_error("PUT", resp.status(), &url_str))
}
/// Download an S3 object. Returns `None` if the object does not exist (404).
///
/// On success returns `(body_bytes, optional_etag)`.
pub(crate) async fn get_object(
creds: &S3Credentials,
key: &str,
max_bytes: usize,
) -> Result<Option<(Vec<u8>, Option<String>)>, AppError> {
let url_str = build_object_url(creds, key);
let url = Url::parse(&url_str).map_err(|e| {
AppError::localized(
"s3.url.invalid",
format!("S3 URL 无效: {e}"),
format!("Invalid S3 URL: {e}"),
)
})?;
let client = http_client::get();
let body_hash = sha256_hex(b"");
let mut headers = reqwest::header::HeaderMap::new();
sign_request(
"GET",
&url,
&mut headers,
&body_hash,
creds,
chrono::Utc::now(),
);
let resp = client
.get(url.as_str())
.headers(headers)
.timeout(Duration::from_secs(TRANSFER_TIMEOUT_SECS))
.send()
.await
.map_err(|e| s3_transport_error("s3.get_failed", "GET 请求", "GET request", &e))?;
if resp.status() == StatusCode::NOT_FOUND {
return Ok(None);
}
if !resp.status().is_success() {
return Err(s3_status_error("GET", resp.status(), &url_str));
}
ensure_content_length_within_limit(resp.headers(), max_bytes, &url_str)?;
let etag = resp
.headers()
.get("etag")
.and_then(|v| v.to_str().ok())
.map(|s| s.to_string());
let mut bytes = Vec::new();
let mut stream = resp.bytes_stream();
while let Some(chunk) = stream.next().await {
let chunk = chunk.map_err(|e| {
AppError::localized(
"s3.response_read_failed",
format!("读取 S3 响应失败: {e}"),
format!("Failed to read S3 response: {e}"),
)
})?;
if bytes.len().saturating_add(chunk.len()) > max_bytes {
return Err(response_too_large_error(&url_str, max_bytes));
}
bytes.extend_from_slice(&chunk);
}
Ok(Some((bytes, etag)))
}
/// Retrieve the ETag of an S3 object via HEAD. Returns `None` on 404.
pub(crate) async fn head_object(
creds: &S3Credentials,
key: &str,
) -> Result<Option<String>, AppError> {
let url_str = build_object_url(creds, key);
let url = Url::parse(&url_str).map_err(|e| {
AppError::localized(
"s3.url.invalid",
format!("S3 URL 无效: {e}"),
format!("Invalid S3 URL: {e}"),
)
})?;
let client = http_client::get();
let body_hash = sha256_hex(b"");
let mut headers = reqwest::header::HeaderMap::new();
sign_request(
"HEAD",
&url,
&mut headers,
&body_hash,
creds,
chrono::Utc::now(),
);
let resp = client
.head(url.as_str())
.headers(headers)
.timeout(Duration::from_secs(DEFAULT_TIMEOUT_SECS))
.send()
.await
.map_err(|e| s3_transport_error("s3.head_failed", "HEAD 请求", "HEAD request", &e))?;
if resp.status() == StatusCode::NOT_FOUND {
return Ok(None);
}
if !resp.status().is_success() {
return Err(s3_status_error("HEAD", resp.status(), &url_str));
}
Ok(resp
.headers()
.get("etag")
.and_then(|v| v.to_str().ok())
.map(|s| s.to_string()))
}
// ─── Tests ───────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
use chrono::TimeZone;
// ── Crypto helpers ──
#[test]
fn sha256_hex_empty_body() {
assert_eq!(
sha256_hex(b""),
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
);
}
#[test]
fn sha256_hex_known_value() {
// From the AWS S3 documentation PUT Object example body.
assert_eq!(
sha256_hex(b"Welcome to Amazon S3."),
"44ce7dd67c959e0d3524ffac1771dfbba87d2b6b4b4e99e42034a8b803f8b072"
);
}
#[test]
fn hmac_sha256_rfc2104_test_vector() {
// HMAC-SHA256("key", "The quick brown fox jumps over the lazy dog")
let result = hmac_sha256(b"key", b"The quick brown fox jumps over the lazy dog");
let hex: String = result.iter().map(|b| format!("{:02x}", b)).collect();
assert_eq!(
hex,
"f7bc83f430538424b13298e6aa6fb143ef4d59a14946175997479dbc2d1a3cd8"
);
}
// ── URL construction: virtual-hosted vs path-style ──
#[test]
fn build_object_url_virtual_hosted_style_aws() {
let creds = test_creds("", "us-east-1", "mybucket");
assert_eq!(
build_object_url(&creds, "path/to/file.json"),
"https://mybucket.s3.us-east-1.amazonaws.com/path/to/file.json"
);
}
#[test]
fn build_object_url_virtual_hosted_explicit_aws_endpoint() {
let creds = test_creds(
"s3.ap-northeast-1.amazonaws.com",
"ap-northeast-1",
"mybucket",
);
assert_eq!(
build_object_url(&creds, "key.txt"),
"https://mybucket.s3.ap-northeast-1.amazonaws.com/key.txt"
);
}
#[test]
fn build_object_url_path_style_custom_endpoint() {
let creds = test_creds("minio.example.com:9000", "us-east-1", "mybucket");
assert_eq!(
build_object_url(&creds, "path/to/file.json"),
"https://minio.example.com:9000/mybucket/path/to/file.json"
);
}
#[test]
fn build_object_url_strips_leading_slash_from_key() {
let creds = test_creds("", "us-east-1", "b");
assert_eq!(
build_object_url(&creds, "/leading/slash.txt"),
"https://b.s3.us-east-1.amazonaws.com/leading/slash.txt"
);
}
#[test]
fn build_bucket_url_aws() {
let creds = test_creds("", "us-west-2", "testbucket");
assert_eq!(
build_bucket_url(&creds),
"https://testbucket.s3.us-west-2.amazonaws.com/"
);
}
#[test]
fn build_bucket_url_custom_endpoint() {
let creds = test_creds("storage.example.com", "us-east-1", "data");
assert_eq!(
build_bucket_url(&creds),
"https://storage.example.com/data/"
);
}
#[test]
fn build_object_url_endpoint_with_trailing_slash() {
let creds = test_creds("minio.local:9000/", "us-east-1", "b");
assert_eq!(
build_object_url(&creds, "k"),
"https://minio.local:9000/b/k"
);
}
#[test]
fn build_object_url_endpoint_with_scheme_prefix() {
let creds = test_creds("https://minio.local:9000", "us-east-1", "b");
assert_eq!(
build_object_url(&creds, "k"),
"https://minio.local:9000/b/k"
);
}
// ── HTTP scheme preservation (MinIO support) ──
#[test]
fn build_object_url_preserves_http_scheme() {
let creds = test_creds("http://minio:9000", "us-east-1", "mybucket");
let url = build_object_url(&creds, "path/to/file.json");
assert!(
url.starts_with("http://"),
"expected http:// scheme, got: {url}"
);
assert_eq!(url, "http://minio:9000/mybucket/path/to/file.json");
}
#[test]
fn build_object_url_preserves_https_scheme() {
let creds = test_creds("https://storage.example.com", "us-east-1", "mybucket");
let url = build_object_url(&creds, "path/to/file.json");
assert!(
url.starts_with("https://"),
"expected https:// scheme, got: {url}"
);
assert_eq!(
url,
"https://storage.example.com/mybucket/path/to/file.json"
);
}
#[test]
fn build_object_url_bare_endpoint_defaults_to_https() {
let creds = test_creds("minio:9000", "us-east-1", "mybucket");
let url = build_object_url(&creds, "path/to/file.json");
assert!(
url.starts_with("https://"),
"bare endpoint should default to https://, got: {url}"
);
assert_eq!(url, "https://minio:9000/mybucket/path/to/file.json");
}
#[test]
fn build_bucket_url_preserves_http_scheme() {
let creds = test_creds("http://minio:9000", "us-east-1", "data");
let url = build_bucket_url(&creds);
assert!(
url.starts_with("http://"),
"expected http:// scheme, got: {url}"
);
assert_eq!(url, "http://minio:9000/data/");
}
// ── Endpoint detection ──
#[test]
fn is_aws_endpoint_detection() {
assert!(is_aws_endpoint(""), "empty string = AWS");
assert!(is_aws_endpoint("s3.us-east-1.amazonaws.com"));
assert!(is_aws_endpoint("s3.amazonaws.com"));
assert!(!is_aws_endpoint("minio.example.com"));
assert!(!is_aws_endpoint("storage.googleapis.com"));
assert!(!is_aws_endpoint("r2.cloudflarestorage.com"));
}
// ── URI encoding ──
#[test]
fn uri_encode_preserves_unreserved_chars() {
assert_eq!(
uri_encode("hello-world_test.txt~v2", true),
"hello-world_test.txt~v2"
);
}
#[test]
fn uri_encode_encodes_spaces_and_special_chars() {
assert_eq!(uri_encode("hello world", true), "hello%20world");
assert_eq!(uri_encode("a+b=c&d", true), "a%2Bb%3Dc%26d");
}
#[test]
fn uri_encode_slash_handling() {
assert_eq!(uri_encode("path/to/file", false), "path/to/file");
assert_eq!(uri_encode("path/to/file", true), "path%2Fto%2Ffile");
}
// ── AWS Signature V4 signing ──
#[test]
fn sig_v4_signing_against_aws_test_vector() {
// Based on the AWS documentation example: GET /?lifecycle
// https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
let creds = S3Credentials {
access_key_id: "AKIAIOSFODNN7EXAMPLE".to_string(),
secret_access_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY".to_string(),
region: "us-east-1".to_string(),
bucket: "examplebucket".to_string(),
endpoint: String::new(),
};
let now = chrono::Utc.with_ymd_and_hms(2013, 5, 24, 0, 0, 0).unwrap();
let url = Url::parse("https://examplebucket.s3.amazonaws.com/?lifecycle").unwrap();
let body_hash = sha256_hex(b"");
let mut headers = reqwest::header::HeaderMap::new();
sign_request("GET", &url, &mut headers, &body_hash, &creds, now);
let auth = headers.get("authorization").unwrap().to_str().unwrap();
// Verify credential scope
assert!(
auth.starts_with(
"AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request"
),
"unexpected credential scope in: {auth}"
);
// Verify signed headers
assert!(
auth.contains("SignedHeaders=host;x-amz-content-sha256;x-amz-date"),
"unexpected signed headers in: {auth}"
);
// Verify exact signature from AWS documentation
assert!(
auth.contains(
"Signature=fea454ca298b7da1c68078a5d1bdbfbbe0d65c699e0f91ac7a200a0136783543"
),
"signature mismatch in: {auth}"
);
}
#[test]
fn sig_v4_includes_content_type_when_present() {
let creds = S3Credentials {
access_key_id: "TESTKEY".to_string(),
secret_access_key: "TESTSECRET".to_string(),
region: "us-east-1".to_string(),
bucket: "b".to_string(),
endpoint: String::new(),
};
let now = chrono::Utc.with_ymd_and_hms(2024, 1, 1, 0, 0, 0).unwrap();
let url = Url::parse("https://b.s3.us-east-1.amazonaws.com/key.json").unwrap();
let body_hash = sha256_hex(b"{}");
let mut headers = reqwest::header::HeaderMap::new();
headers.insert("content-type", "application/json".parse().unwrap());
sign_request("PUT", &url, &mut headers, &body_hash, &creds, now);
let auth = headers.get("authorization").unwrap().to_str().unwrap();
// content-type must appear in the signed headers
assert!(
auth.contains("content-type"),
"content-type should be in signed headers: {auth}"
);
}
#[test]
fn sig_v4_signing_key_derivation() {
// Verify the signing key derivation chain independently.
// Using the same AWS example credentials and date.
let secret = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY";
let datestamp = "20130524";
let region = "us-east-1";
let k_date = hmac_sha256(format!("AWS4{}", secret).as_bytes(), datestamp.as_bytes());
let k_region = hmac_sha256(&k_date, region.as_bytes());
let k_service = hmac_sha256(&k_region, b"s3");
let k_signing = hmac_sha256(&k_service, b"aws4_request");
// The signing key should be a 32-byte value (256 bits).
assert_eq!(k_signing.len(), 32);
// Verify it is deterministic — computing again yields the same result.
let k_date2 = hmac_sha256(format!("AWS4{}", secret).as_bytes(), datestamp.as_bytes());
let k_region2 = hmac_sha256(&k_date2, region.as_bytes());
let k_service2 = hmac_sha256(&k_region2, b"s3");
let k_signing2 = hmac_sha256(&k_service2, b"aws4_request");
assert_eq!(k_signing, k_signing2);
}
// ── Redact URL ──
#[test]
fn redact_url_strips_query_params() {
let r = redact_url(
"https://mybucket.s3.us-east-1.amazonaws.com/file.txt?X-Amz-Credential=AKID&X-Amz-Signature=abc",
);
assert!(!r.contains("AKID"));
assert!(!r.contains("abc"));
assert!(r.contains("mybucket.s3.us-east-1.amazonaws.com"));
assert!(r.contains("/file.txt"));
}
#[test]
fn redact_url_preserves_path() {
let r = redact_url("https://minio.local:9000/bucket/path/to/file.json");
assert_eq!(r, "https://minio.local:9000/bucket/path/to/file.json");
}
// ── Content-length check ──
#[test]
fn ensure_content_length_within_limit_rejects_oversized() {
use reqwest::header::{HeaderMap, HeaderValue, CONTENT_LENGTH};
let mut h = HeaderMap::new();
h.insert(CONTENT_LENGTH, HeaderValue::from_static("2048"));
assert!(ensure_content_length_within_limit(&h, 1024, "https://example.com").is_err());
}
#[test]
fn ensure_content_length_within_limit_accepts_within_bounds() {
use reqwest::header::{HeaderMap, HeaderValue, CONTENT_LENGTH};
let mut h = HeaderMap::new();
h.insert(CONTENT_LENGTH, HeaderValue::from_static("512"));
assert!(ensure_content_length_within_limit(&h, 1024, "https://example.com").is_ok());
let empty = HeaderMap::new();
assert!(ensure_content_length_within_limit(&empty, 1024, "https://example.com").is_ok());
}
// ── Helper ──
fn test_creds(endpoint: &str, region: &str, bucket: &str) -> S3Credentials {
S3Credentials {
access_key_id: String::new(),
secret_access_key: String::new(),
region: region.to_string(),
bucket: bucket.to_string(),
endpoint: endpoint.to_string(),
}
}
}
// ─── Live integration tests (run with --ignored) ─────────────
#[cfg(test)]
mod integration_tests {
use super::*;
fn test_creds() -> S3Credentials {
S3Credentials {
access_key_id: std::env::var("S3_TEST_AK").expect("S3_TEST_AK env required"),
secret_access_key: std::env::var("S3_TEST_SK").expect("S3_TEST_SK env required"),
region: std::env::var("S3_TEST_REGION").unwrap_or_else(|_| "us-east-1".to_string()),
bucket: std::env::var("S3_TEST_BUCKET").expect("S3_TEST_BUCKET env required"),
endpoint: std::env::var("S3_TEST_ENDPOINT").unwrap_or_default(),
}
}
#[tokio::test]
#[ignore]
async fn live_s3_connection() {
crate::proxy::http_client::init(None).ok();
let creds = test_creds();
let result = test_connection(&creds).await;
assert!(result.is_ok(), "Connection failed: {:?}", result.err());
println!("PASS: test_connection OK");
}
#[tokio::test]
#[ignore]
async fn live_s3_put_get_head_roundtrip() {
crate::proxy::http_client::init(None).ok();
let creds = test_creds();
let key = "cc-switch-sync/v2/default/_integration_test.json";
let data = br#"{"test":true,"ts":12345}"#;
// PUT
let r = put_object(&creds, key, data.to_vec(), "application/json").await;
assert!(r.is_ok(), "PUT failed: {:?}", r.err());
println!("PASS: put_object {} bytes", data.len());
// GET
let r = get_object(&creds, key, 1 << 20).await;
assert!(r.is_ok(), "GET failed: {:?}", r.err());
let (body, etag) = r.unwrap().expect("should exist");
assert_eq!(body, data);
println!("PASS: get_object {} bytes, etag={:?}", body.len(), etag);
// HEAD
let r = head_object(&creds, key).await;
assert!(r.is_ok(), "HEAD failed: {:?}", r.err());
assert!(r.unwrap().is_some());
println!("PASS: head_object OK");
// 404
let r = get_object(&creds, "cc-switch-sync/_no_such_key", 1024).await;
assert!(r.is_ok());
assert!(r.unwrap().is_none());
println!("PASS: get_object(404) returned None");
println!("ALL LIVE S3 TESTS PASSED");
}
}
+270
View File
@@ -0,0 +1,270 @@
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use std::sync::OnceLock;
use std::time::{Duration, Instant};
use serde_json::json;
use tauri::{AppHandle, Emitter};
use tokio::sync::mpsc::error::TrySendError;
use tokio::sync::mpsc::{channel, Receiver, Sender};
use crate::error::AppError;
use crate::services::s3_sync;
use crate::settings::{self, S3SyncSettings};
const AUTO_SYNC_DEBOUNCE_MS: u64 = 1000;
pub(crate) const MAX_AUTO_SYNC_WAIT_MS: u64 = 10_000;
static DB_CHANGE_TX: OnceLock<Sender<String>> = OnceLock::new();
static AUTO_SYNC_SUPPRESS_DEPTH: AtomicUsize = AtomicUsize::new(0);
pub(crate) struct AutoSyncSuppressionGuard;
impl AutoSyncSuppressionGuard {
pub fn new() -> Self {
AUTO_SYNC_SUPPRESS_DEPTH.fetch_add(1, Ordering::SeqCst);
Self
}
}
impl Drop for AutoSyncSuppressionGuard {
fn drop(&mut self) {
let _ =
AUTO_SYNC_SUPPRESS_DEPTH.fetch_update(Ordering::SeqCst, Ordering::SeqCst, |value| {
Some(value.saturating_sub(1))
});
}
}
pub(crate) fn is_auto_sync_suppressed() -> bool {
AUTO_SYNC_SUPPRESS_DEPTH.load(Ordering::SeqCst) > 0
}
pub fn should_trigger_for_table(table: &str) -> bool {
let normalized = table.trim().to_ascii_lowercase();
matches!(
normalized.as_str(),
"providers"
| "provider_endpoints"
| "mcp_servers"
| "prompts"
| "skills"
| "skill_repos"
| "settings"
| "proxy_config"
)
}
pub(crate) fn enqueue_change_signal(tx: &Sender<String>, table: &str) -> bool {
match tx.try_send(table.to_string()) {
Ok(()) => true,
Err(TrySendError::Full(_)) | Err(TrySendError::Closed(_)) => false,
}
}
pub(crate) fn auto_sync_wait_duration(started_at: Instant, now: Instant) -> Option<Duration> {
let max_wait = Duration::from_millis(MAX_AUTO_SYNC_WAIT_MS);
let debounce = Duration::from_millis(AUTO_SYNC_DEBOUNCE_MS);
let elapsed = now.saturating_duration_since(started_at);
if elapsed >= max_wait {
return None;
}
Some(debounce.min(max_wait - elapsed))
}
fn should_run_auto_sync(settings: Option<&S3SyncSettings>) -> bool {
let Some(sync) = settings else {
return false;
};
sync.enabled && sync.auto_sync
}
fn persist_auto_sync_error(settings: &mut S3SyncSettings, error: &AppError) {
settings.status.last_error = Some(error.to_string());
settings.status.last_error_source = Some("auto".to_string());
let _ = settings::update_s3_sync_status(settings.status.clone());
}
fn emit_auto_sync_status_updated(app: &AppHandle, status: &str, error: Option<&str>) {
let payload = match error {
Some(message) => json!({
"source": "auto",
"status": status,
"error": message,
}),
None => json!({
"source": "auto",
"status": status,
}),
};
if let Err(err) = app.emit("s3-sync-status-updated", payload) {
log::debug!("[S3] failed to emit sync status update event: {err}");
}
}
async fn run_auto_sync_upload(
db: &crate::database::Database,
app: &AppHandle,
) -> Result<(), AppError> {
let mut settings = settings::get_s3_sync_settings();
if !should_run_auto_sync(settings.as_ref()) {
return Ok(());
}
let mut sync_settings = match settings.take() {
Some(value) => value,
None => return Ok(()),
};
let result = s3_sync::run_with_sync_lock(s3_sync::upload(db, &mut sync_settings)).await;
match result {
Ok(_) => {
emit_auto_sync_status_updated(app, "success", None);
Ok(())
}
Err(err) => {
persist_auto_sync_error(&mut sync_settings, &err);
emit_auto_sync_status_updated(app, "error", Some(&err.to_string()));
Err(err)
}
}
}
pub fn notify_db_changed(table: &str) {
if is_auto_sync_suppressed() {
return;
}
if !should_trigger_for_table(table) {
return;
}
let Some(tx) = DB_CHANGE_TX.get() else {
return;
};
let _ = enqueue_change_signal(tx, table);
}
pub fn start_worker(db: Arc<crate::database::Database>, app: tauri::AppHandle) {
if DB_CHANGE_TX.get().is_some() {
return;
}
// Buffer size 1 is enough: we only need "dirty" signals, not every event.
let (tx, rx) = channel::<String>(1);
if DB_CHANGE_TX.set(tx).is_err() {
return;
}
tauri::async_runtime::spawn(async move {
run_worker_loop(db, rx, app).await;
});
}
async fn run_worker_loop(
db: Arc<crate::database::Database>,
mut rx: Receiver<String>,
app: tauri::AppHandle,
) {
while let Some(first_table) = rx.recv().await {
let started_at = Instant::now();
let mut merged_count = 1usize;
while let Some(wait_for) = auto_sync_wait_duration(started_at, Instant::now()) {
let timeout = tokio::time::timeout(wait_for, rx.recv()).await;
match timeout {
Ok(Some(_)) => merged_count += 1,
Ok(None) => return,
Err(_) => break,
}
}
log::debug!(
"[S3][AutoSync] Triggered by table={first_table}, merged_changes={merged_count}"
);
if let Err(err) = run_auto_sync_upload(&db, &app).await {
log::warn!("[S3][AutoSync] Upload failed: {err}");
}
}
}
#[cfg(test)]
mod tests {
use super::{
auto_sync_wait_duration, enqueue_change_signal, is_auto_sync_suppressed,
should_run_auto_sync, should_trigger_for_table, AutoSyncSuppressionGuard,
MAX_AUTO_SYNC_WAIT_MS,
};
use crate::settings::S3SyncSettings;
use std::time::{Duration, Instant};
use tokio::sync::mpsc::channel;
#[test]
fn should_trigger_sync_for_config_tables_only() {
assert!(should_trigger_for_table("providers"));
assert!(should_trigger_for_table("settings"));
assert!(!should_trigger_for_table("proxy_request_logs"));
assert!(!should_trigger_for_table("provider_health"));
}
#[test]
fn suppression_guard_enables_and_restores_state() {
assert!(!is_auto_sync_suppressed());
{
let _guard = AutoSyncSuppressionGuard::new();
assert!(is_auto_sync_suppressed());
}
assert!(!is_auto_sync_suppressed());
}
#[test]
fn max_wait_caps_flush_latency_for_continuous_events() {
let started = Instant::now();
let later = started + Duration::from_millis(MAX_AUTO_SYNC_WAIT_MS + 1);
assert!(auto_sync_wait_duration(started, later).is_none());
}
#[tokio::test]
async fn enqueue_change_signal_drops_when_channel_is_full() {
let (tx, _rx) = channel::<String>(1);
assert!(enqueue_change_signal(&tx, "providers"));
assert!(!enqueue_change_signal(&tx, "providers"));
}
#[test]
fn should_run_auto_sync_requires_enabled_and_auto_sync_flag() {
assert!(!should_run_auto_sync(None));
let disabled = S3SyncSettings {
enabled: false,
auto_sync: true,
..S3SyncSettings::default()
};
assert!(!should_run_auto_sync(Some(&disabled)));
let auto_sync_off = S3SyncSettings {
enabled: true,
auto_sync: false,
..S3SyncSettings::default()
};
assert!(!should_run_auto_sync(Some(&auto_sync_off)));
let enabled = S3SyncSettings {
enabled: true,
auto_sync: true,
..S3SyncSettings::default()
};
assert!(should_run_auto_sync(Some(&enabled)));
}
#[test]
fn service_layer_does_not_depend_on_commands_layer() {
let source = include_str!("s3_auto_sync.rs");
let needle = ["crate", "commands", ""].join("::");
assert!(
!source.contains(&needle),
"services layer should not depend on commands layer"
);
}
}
+319
View File
@@ -0,0 +1,319 @@
//! S3 v2 sync protocol layer.
//!
//! Implements manifest-based synchronization on top of the S3 transport
//! primitives in [`super::s3`]. Artifact set: `db.sql` + `skills.zip`.
use std::collections::BTreeMap;
use std::future::Future;
use std::sync::OnceLock;
use chrono::Utc;
use serde_json::Value;
use crate::error::AppError;
use crate::services::s3::{self, S3Credentials};
use crate::settings::{update_s3_sync_status, S3SyncSettings, WebDavSyncStatus};
use super::sync_protocol::{
apply_snapshot, build_local_snapshot, localized, persist_sync_success_best_effort, sha256_hex,
validate_artifact_size_limit, validate_manifest_compat, verify_artifact, ArtifactMeta,
RemoteLayout, SyncManifest, DB_COMPAT_VERSION, MAX_MANIFEST_BYTES, MAX_SYNC_ARTIFACT_BYTES,
PROTOCOL_VERSION, REMOTE_DB_SQL, REMOTE_MANIFEST, REMOTE_SKILLS_ZIP,
};
// ─── Sync lock ───────────────────────────────────────────────
pub fn sync_mutex() -> &'static tokio::sync::Mutex<()> {
static LOCK: OnceLock<tokio::sync::Mutex<()>> = OnceLock::new();
LOCK.get_or_init(|| tokio::sync::Mutex::new(()))
}
pub async fn run_with_sync_lock<T, Fut>(operation: Fut) -> Result<T, AppError>
where
Fut: Future<Output = Result<T, AppError>>,
{
let _guard = sync_mutex().lock().await;
operation.await
}
// ─── Public API ──────────────────────────────────────────────
/// Check S3 connectivity by issuing a HEAD request against the bucket.
pub async fn check_connection(settings: &S3SyncSettings) -> Result<(), AppError> {
settings.validate()?;
let creds = creds_for(settings);
s3::test_connection(&creds).await
}
/// Upload local snapshot (db + skills) to remote S3.
pub async fn upload(
db: &crate::database::Database,
settings: &mut S3SyncSettings,
) -> Result<Value, AppError> {
settings.validate()?;
let creds = creds_for(settings);
let snapshot = build_local_snapshot(db)?;
// Upload order: artifacts first, manifest last (best-effort consistency)
let db_key = s3_key(settings, REMOTE_DB_SQL);
s3::put_object(&creds, &db_key, snapshot.db_sql, "application/sql").await?;
let skills_key = s3_key(settings, REMOTE_SKILLS_ZIP);
s3::put_object(&creds, &skills_key, snapshot.skills_zip, "application/zip").await?;
let manifest_key = s3_key(settings, REMOTE_MANIFEST);
s3::put_object(
&creds,
&manifest_key,
snapshot.manifest_bytes,
"application/json",
)
.await?;
// Fetch etag (best-effort, don't fail the upload)
let etag = match s3::head_object(&creds, &manifest_key).await {
Ok(e) => e,
Err(e) => {
log::debug!("[S3] Failed to fetch ETag after upload: {e}");
None
}
};
let _persisted = persist_sync_success_best_effort(
settings,
snapshot.manifest_hash,
etag,
persist_sync_success,
);
Ok(serde_json::json!({ "status": "uploaded" }))
}
/// Download remote snapshot and apply to local database + skills.
pub async fn download(
db: &crate::database::Database,
settings: &mut S3SyncSettings,
) -> Result<Value, AppError> {
settings.validate()?;
let creds = creds_for(settings);
let manifest_key = s3_key(settings, REMOTE_MANIFEST);
let (manifest_bytes, etag) = s3::get_object(&creds, &manifest_key, MAX_MANIFEST_BYTES)
.await?
.ok_or_else(|| {
localized(
"s3.sync.remote_empty",
"远端没有可下载的同步数据",
"No downloadable sync data found on the remote.",
)
})?;
let manifest: SyncManifest =
serde_json::from_slice(&manifest_bytes).map_err(|e| AppError::Json {
path: REMOTE_MANIFEST.to_string(),
source: e,
})?;
validate_manifest_compat(&manifest, RemoteLayout::Current)?;
// Download and verify artifacts
let db_sql = download_and_verify(settings, &creds, REMOTE_DB_SQL, &manifest.artifacts).await?;
let skills_zip =
download_and_verify(settings, &creds, REMOTE_SKILLS_ZIP, &manifest.artifacts).await?;
// Apply snapshot
apply_snapshot(db, &db_sql, &skills_zip)?;
let manifest_hash = sha256_hex(&manifest_bytes);
let _persisted =
persist_sync_success_best_effort(settings, manifest_hash, etag, persist_sync_success);
Ok(serde_json::json!({ "status": "downloaded" }))
}
/// Fetch remote manifest info without downloading artifacts.
pub async fn fetch_remote_info(settings: &S3SyncSettings) -> Result<Option<Value>, AppError> {
settings.validate()?;
let creds = creds_for(settings);
let manifest_key = s3_key(settings, REMOTE_MANIFEST);
let Some((bytes, _)) = s3::get_object(&creds, &manifest_key, MAX_MANIFEST_BYTES).await? else {
return Ok(None);
};
let manifest: SyncManifest = serde_json::from_slice(&bytes).map_err(|e| AppError::Json {
path: REMOTE_MANIFEST.to_string(),
source: e,
})?;
let compatible = validate_manifest_compat(&manifest, RemoteLayout::Current).is_ok();
let payload = serde_json::json!({
"deviceName": manifest.device_name,
"createdAt": manifest.created_at,
"snapshotId": manifest.snapshot_id,
"version": manifest.version,
"protocolVersion": manifest.version,
"dbCompatVersion": manifest.db_compat_version,
"compatible": compatible,
"artifacts": manifest.artifacts.keys().collect::<Vec<_>>(),
"layout": RemoteLayout::Current.as_str(),
"remotePath": s3_dir_display(settings),
});
Ok(Some(payload))
}
// ─── Sync status persistence ─────────────────────────────────
fn persist_sync_success(
settings: &mut S3SyncSettings,
manifest_hash: String,
etag: Option<String>,
) -> Result<(), AppError> {
let status = WebDavSyncStatus {
last_sync_at: Some(Utc::now().timestamp()),
last_error: None,
last_error_source: None,
last_local_manifest_hash: Some(manifest_hash.clone()),
last_remote_manifest_hash: Some(manifest_hash),
last_remote_etag: etag,
};
settings.status = status.clone();
update_s3_sync_status(status)
}
// ─── Download & verify ───────────────────────────────────────
async fn download_and_verify(
settings: &S3SyncSettings,
creds: &S3Credentials,
artifact_name: &str,
artifacts: &BTreeMap<String, ArtifactMeta>,
) -> Result<Vec<u8>, AppError> {
let meta = artifacts.get(artifact_name).ok_or_else(|| {
localized(
"s3.sync.manifest_missing_artifact",
format!("manifest 中缺少 artifact: {artifact_name}"),
format!("Manifest missing artifact: {artifact_name}"),
)
})?;
validate_artifact_size_limit(artifact_name, meta.size)?;
let key = s3_key(settings, artifact_name);
let (bytes, _) = s3::get_object(creds, &key, MAX_SYNC_ARTIFACT_BYTES as usize)
.await?
.ok_or_else(|| {
localized(
"s3.sync.remote_missing_artifact",
format!("远端缺少 artifact 文件: {artifact_name}"),
format!("Remote artifact file missing: {artifact_name}"),
)
})?;
verify_artifact(&bytes, artifact_name, meta)?;
Ok(bytes)
}
// ─── S3 key helpers ──────────────────────────────────────────
/// Build the S3 object key for a given artifact.
///
/// Format: `{remote_root}/v{PROTOCOL_VERSION}/db-v{DB_COMPAT_VERSION}/{profile}/{artifact}`
/// Example: `cc-switch-sync/v2/db-v6/default/manifest.json`
fn s3_key(settings: &S3SyncSettings, artifact: &str) -> String {
format!(
"{}/v{}/db-v{}/{}/{}",
settings.remote_root, PROTOCOL_VERSION, DB_COMPAT_VERSION, settings.profile, artifact
)
}
fn s3_dir_display(settings: &S3SyncSettings) -> String {
format!(
"{}/v{}/db-v{}/{}",
settings.remote_root, PROTOCOL_VERSION, DB_COMPAT_VERSION, settings.profile
)
}
fn creds_for(settings: &S3SyncSettings) -> S3Credentials {
S3Credentials {
access_key_id: settings.access_key_id.clone(),
secret_access_key: settings.secret_access_key.clone(),
region: settings.region.clone(),
bucket: settings.bucket.clone(),
endpoint: settings.endpoint.clone(),
}
}
// ─── Tests ───────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
fn test_settings() -> S3SyncSettings {
S3SyncSettings {
remote_root: "cc-switch-sync".to_string(),
profile: "default".to_string(),
..S3SyncSettings::default()
}
}
#[test]
fn s3_key_uses_v2_and_correct_format() {
let settings = test_settings();
let key = s3_key(&settings, "manifest.json");
assert_eq!(key, "cc-switch-sync/v2/db-v6/default/manifest.json");
}
#[test]
fn s3_key_with_custom_profile() {
let settings = S3SyncSettings {
remote_root: "my-root".to_string(),
profile: "work".to_string(),
..S3SyncSettings::default()
};
assert_eq!(s3_key(&settings, "db.sql"), "my-root/v2/db-v6/work/db.sql");
}
#[test]
fn s3_key_matches_expected_pattern() {
let settings = test_settings();
let key = s3_key(&settings, "skills.zip");
// Should follow {remote_root}/v{version}/db-v{db}/{profile}/{artifact}
let parts: Vec<&str> = key.splitn(5, '/').collect();
assert_eq!(parts.len(), 5);
assert_eq!(parts[0], "cc-switch-sync");
assert_eq!(parts[1], "v2");
assert_eq!(parts[2], "db-v6");
assert_eq!(parts[3], "default");
assert_eq!(parts[4], "skills.zip");
}
#[test]
fn sync_mutex_is_singleton() {
let m1 = sync_mutex();
let m2 = sync_mutex();
assert!(
std::ptr::eq(m1, m2),
"sync_mutex must return the same instance"
);
}
#[test]
fn creds_for_maps_all_fields() {
let settings = S3SyncSettings {
access_key_id: "AKID".to_string(),
secret_access_key: "SECRET".to_string(),
region: "us-west-2".to_string(),
bucket: "my-bucket".to_string(),
endpoint: "minio.local:9000".to_string(),
..S3SyncSettings::default()
};
let creds = creds_for(&settings);
assert_eq!(creds.access_key_id, "AKID");
assert_eq!(creds.secret_access_key, "SECRET");
assert_eq!(creds.region, "us-west-2");
assert_eq!(creds.bucket, "my-bucket");
assert_eq!(creds.endpoint, "minio.local:9000");
}
}
+648
View File
@@ -0,0 +1,648 @@
//! Transport-agnostic sync protocol layer.
//!
//! Shared by WebDAV, S3, and future transports. Artifact set: `db.sql` + `skills.zip`.
use std::collections::BTreeMap;
use std::fs;
use std::process::Command;
use chrono::Utc;
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use tempfile::tempdir;
use crate::error::AppError;
// Re-export archive functions for use by transport layers.
pub(crate) use super::webdav_sync::archive::{
backup_current_skills, restore_skills_from_backup, restore_skills_zip, zip_skills_ssot,
};
// ─── Protocol constants ──────────────────────────────────────
/// Wire-format identifier stored in remote manifests.
/// Retains historic "webdav" naming for backward compatibility with existing remotes.
pub(crate) const PROTOCOL_FORMAT: &str = "cc-switch-webdav-sync";
pub(crate) const PROTOCOL_VERSION: u32 = 2;
pub(crate) const DB_COMPAT_VERSION: u32 = 6;
pub(crate) const LEGACY_DB_COMPAT_VERSION: u32 = 5;
pub(crate) const REMOTE_DB_SQL: &str = "db.sql";
pub(crate) const REMOTE_SKILLS_ZIP: &str = "skills.zip";
pub(crate) const REMOTE_MANIFEST: &str = "manifest.json";
pub(crate) const MAX_DEVICE_NAME_LEN: usize = 64;
pub(crate) const MAX_MANIFEST_BYTES: usize = 1024 * 1024;
pub(crate) const MAX_SYNC_ARTIFACT_BYTES: u64 = 512 * 1024 * 1024;
// ─── Error helpers ───────────────────────────────────────────
pub(crate) fn localized(
key: &'static str,
zh: impl Into<String>,
en: impl Into<String>,
) -> AppError {
AppError::localized(key, zh, en)
}
pub(crate) fn io_context_localized(
_key: &'static str,
zh: impl Into<String>,
en: impl Into<String>,
source: std::io::Error,
) -> AppError {
let zh_msg = zh.into();
let en_msg = en.into();
AppError::IoContext {
context: format!("{zh_msg} ({en_msg})"),
source,
}
}
// ─── Types ───────────────────────────────────────────────────
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub(crate) struct SyncManifest {
pub format: String,
pub version: u32,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub db_compat_version: Option<u32>,
pub device_name: String,
pub created_at: String,
pub artifacts: BTreeMap<String, ArtifactMeta>,
pub snapshot_id: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub(crate) struct ArtifactMeta {
pub sha256: String,
pub size: u64,
}
pub(crate) struct LocalSnapshot {
pub db_sql: Vec<u8>,
pub skills_zip: Vec<u8>,
pub manifest_bytes: Vec<u8>,
pub manifest_hash: String,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum RemoteLayout {
Current,
Legacy,
}
impl RemoteLayout {
pub(crate) fn as_str(self) -> &'static str {
match self {
Self::Current => "current",
Self::Legacy => "legacy",
}
}
}
// ─── Snapshot building ───────────────────────────────────────
pub(crate) fn build_local_snapshot(
db: &crate::database::Database,
) -> Result<LocalSnapshot, AppError> {
// Export database to SQL string
let sql_string = db.export_sql_string_for_sync()?;
let db_sql = sql_string.into_bytes();
// Pack skills into deterministic ZIP
let tmp = tempdir().map_err(|e| {
io_context_localized(
"sync.snapshot_tmpdir_failed",
"创建快照临时目录失败",
"Failed to create temporary directory for snapshot",
e,
)
})?;
let skills_zip_path = tmp.path().join(REMOTE_SKILLS_ZIP);
zip_skills_ssot(&skills_zip_path)?;
let skills_zip = fs::read(&skills_zip_path).map_err(|e| AppError::io(&skills_zip_path, e))?;
// Build artifact map and compute hashes
let mut artifacts = BTreeMap::new();
artifacts.insert(
REMOTE_DB_SQL.to_string(),
ArtifactMeta {
sha256: sha256_hex(&db_sql),
size: db_sql.len() as u64,
},
);
artifacts.insert(
REMOTE_SKILLS_ZIP.to_string(),
ArtifactMeta {
sha256: sha256_hex(&skills_zip),
size: skills_zip.len() as u64,
},
);
let snapshot_id = compute_snapshot_id(&artifacts);
let manifest = SyncManifest {
format: PROTOCOL_FORMAT.to_string(),
version: PROTOCOL_VERSION,
db_compat_version: Some(DB_COMPAT_VERSION),
device_name: detect_system_device_name().unwrap_or_else(|| "Unknown Device".to_string()),
created_at: Utc::now().to_rfc3339(),
artifacts,
snapshot_id,
};
let manifest_bytes =
serde_json::to_vec_pretty(&manifest).map_err(|e| AppError::JsonSerialize { source: e })?;
let manifest_hash = sha256_hex(&manifest_bytes);
Ok(LocalSnapshot {
db_sql,
skills_zip,
manifest_bytes,
manifest_hash,
})
}
// ─── Manifest handling ───────────────────────────────────────
/// Compute a deterministic snapshot identity from artifact hashes.
///
/// BTreeMap iteration order is sorted by key, ensuring stability.
pub(crate) fn compute_snapshot_id(artifacts: &BTreeMap<String, ArtifactMeta>) -> String {
let parts: Vec<String> = artifacts
.iter()
.map(|(name, meta)| format!("{}:{}", name, meta.sha256))
.collect();
sha256_hex(parts.join("|").as_bytes())
}
pub(crate) fn effective_db_compat_version(
manifest: &SyncManifest,
layout: RemoteLayout,
) -> Option<u32> {
manifest
.db_compat_version
.or_else(|| (layout == RemoteLayout::Legacy).then_some(LEGACY_DB_COMPAT_VERSION))
}
pub(crate) fn validate_manifest_compat(
manifest: &SyncManifest,
layout: RemoteLayout,
) -> Result<(), AppError> {
if manifest.format != PROTOCOL_FORMAT {
return Err(localized(
"sync.manifest_format_incompatible",
format!("远端 manifest 格式不兼容: {}", manifest.format),
format!(
"Remote manifest format is incompatible: {}",
manifest.format
),
));
}
if manifest.version != PROTOCOL_VERSION {
return Err(localized(
"sync.manifest_version_incompatible",
format!(
"远端 manifest 协议版本不兼容: v{} (本地 v{PROTOCOL_VERSION})",
manifest.version
),
format!(
"Remote manifest protocol version is incompatible: v{} (local v{PROTOCOL_VERSION})",
manifest.version
),
));
}
let Some(db_compat_version) = effective_db_compat_version(manifest, layout) else {
return Err(localized(
"sync.manifest_db_version_missing",
"远端 manifest 缺少数据库兼容版本",
"Remote manifest is missing the database compatibility version.",
));
};
match layout {
RemoteLayout::Current if db_compat_version != DB_COMPAT_VERSION => {
return Err(localized(
"sync.manifest_db_version_incompatible",
format!(
"远端数据库快照版本不兼容: db-v{db_compat_version} (本地 db-v{DB_COMPAT_VERSION})"
),
format!(
"Remote database snapshot version is incompatible: db-v{db_compat_version} (local db-v{DB_COMPAT_VERSION})"
),
));
}
RemoteLayout::Legacy if db_compat_version > DB_COMPAT_VERSION => {
return Err(localized(
"sync.manifest_db_version_incompatible",
format!(
"远端数据库快照版本不兼容: db-v{db_compat_version} (本地最高支持 db-v{DB_COMPAT_VERSION})"
),
format!(
"Remote database snapshot version is incompatible: db-v{db_compat_version} (local supports up to db-v{DB_COMPAT_VERSION})"
),
));
}
_ => {}
}
Ok(())
}
// ─── Artifact verification ───────────────────────────────────
pub(crate) fn validate_artifact_size_limit(artifact_name: &str, size: u64) -> Result<(), AppError> {
if size > MAX_SYNC_ARTIFACT_BYTES {
let max_mb = MAX_SYNC_ARTIFACT_BYTES / 1024 / 1024;
return Err(localized(
"sync.artifact_too_large",
format!("artifact {artifact_name} 超过下载上限({} MB", max_mb),
format!(
"Artifact {artifact_name} exceeds download limit ({} MB)",
max_mb
),
));
}
Ok(())
}
/// Verify that downloaded artifact bytes match the expected size and SHA-256 hash.
pub(crate) fn verify_artifact(
bytes: &[u8],
artifact_name: &str,
meta: &ArtifactMeta,
) -> Result<(), AppError> {
// Quick size check before expensive hash
if bytes.len() as u64 != meta.size {
return Err(localized(
"sync.artifact_size_mismatch",
format!(
"artifact {artifact_name} 大小不匹配 (expected: {}, got: {})",
meta.size,
bytes.len(),
),
format!(
"Artifact {artifact_name} size mismatch (expected: {}, got: {})",
meta.size,
bytes.len(),
),
));
}
let actual_hash = sha256_hex(bytes);
if actual_hash != meta.sha256 {
return Err(localized(
"sync.artifact_hash_mismatch",
format!(
"artifact {artifact_name} SHA256 校验失败 (expected: {}..., got: {}...)",
meta.sha256.get(..8).unwrap_or(&meta.sha256),
actual_hash.get(..8).unwrap_or(&actual_hash),
),
format!(
"Artifact {artifact_name} SHA256 verification failed (expected: {}..., got: {}...)",
meta.sha256.get(..8).unwrap_or(&meta.sha256),
actual_hash.get(..8).unwrap_or(&actual_hash),
),
));
}
Ok(())
}
// ─── Snapshot application ────────────────────────────────────
pub(crate) fn apply_snapshot(
db: &crate::database::Database,
db_sql: &[u8],
skills_zip: &[u8],
) -> Result<(), AppError> {
let sql_str = std::str::from_utf8(db_sql).map_err(|e| {
localized(
"sync.sql_not_utf8",
format!("SQL 非 UTF-8: {e}"),
format!("SQL is not valid UTF-8: {e}"),
)
})?;
let skills_backup = backup_current_skills()?;
// Replace skills first, then import database; roll back skills on DB failure.
restore_skills_zip(skills_zip)?;
if let Err(db_err) = db.import_sql_string_for_sync(sql_str) {
if let Err(rollback_err) = restore_skills_from_backup(&skills_backup) {
return Err(localized(
"sync.db_import_and_rollback_failed",
format!("导入数据库失败: {db_err}; 同时回滚 Skills 失败: {rollback_err}"),
format!(
"Database import failed: {db_err}; skills rollback also failed: {rollback_err}"
),
));
}
return Err(db_err);
}
Ok(())
}
// ─── Utilities ───────────────────────────────────────────────
pub(crate) fn sha256_hex(bytes: &[u8]) -> String {
let mut hasher = Sha256::new();
hasher.update(bytes);
format!("{:x}", hasher.finalize())
}
pub(crate) fn detect_system_device_name() -> Option<String> {
let env_name = ["CC_SWITCH_DEVICE_NAME", "COMPUTERNAME", "HOSTNAME"]
.iter()
.filter_map(|key| std::env::var(key).ok())
.find_map(|value| normalize_device_name(&value));
if env_name.is_some() {
return env_name;
}
let output = Command::new("hostname").output().ok()?;
if !output.status.success() {
return None;
}
let hostname = String::from_utf8(output.stdout).ok()?;
normalize_device_name(&hostname)
}
pub(crate) fn normalize_device_name(raw: &str) -> Option<String> {
let compact = raw
.chars()
.fold(String::with_capacity(raw.len()), |mut acc, ch| {
if ch.is_whitespace() {
acc.push(' ');
} else if !ch.is_control() {
acc.push(ch);
}
acc
});
let normalized = compact.split_whitespace().collect::<Vec<_>>().join(" ");
let trimmed = normalized.trim();
if trimmed.is_empty() {
return None;
}
let limited = trimmed
.chars()
.take(MAX_DEVICE_NAME_LEN)
.collect::<String>();
if limited.is_empty() {
None
} else {
Some(limited)
}
}
// ─── Sync status persistence ─────────────────────────────────
pub(crate) fn persist_sync_success_best_effort<S, F>(
settings: &mut S,
manifest_hash: String,
etag: Option<String>,
persist_fn: F,
) -> bool
where
F: FnOnce(&mut S, String, Option<String>) -> Result<(), AppError>,
{
match persist_fn(settings, manifest_hash, etag) {
Ok(()) => true,
Err(err) => {
log::warn!("[Sync] Persist sync status failed, keep operation success: {err}");
false
}
}
}
// ─── Tests ───────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
fn artifact(sha256: &str, size: u64) -> ArtifactMeta {
ArtifactMeta {
sha256: sha256.to_string(),
size,
}
}
#[test]
fn snapshot_id_is_stable() {
let mut artifacts = BTreeMap::new();
artifacts.insert("db.sql".to_string(), artifact("abc123", 100));
artifacts.insert("skills.zip".to_string(), artifact("def456", 200));
let id1 = compute_snapshot_id(&artifacts);
let id2 = compute_snapshot_id(&artifacts);
assert_eq!(id1, id2);
}
#[test]
fn snapshot_id_changes_with_artifacts() {
let mut a1 = BTreeMap::new();
a1.insert("db.sql".to_string(), artifact("hash-a", 1));
let mut a2 = BTreeMap::new();
a2.insert("db.sql".to_string(), artifact("hash-b", 1));
assert_ne!(compute_snapshot_id(&a1), compute_snapshot_id(&a2));
}
#[test]
fn sha256_hex_is_correct() {
let hash = sha256_hex(b"hello");
assert_eq!(
hash,
"2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"
);
}
#[test]
fn persist_best_effort_returns_true_on_success() {
let mut dummy = ();
let ok = persist_sync_success_best_effort(
&mut dummy,
"hash".to_string(),
Some("etag".to_string()),
|_settings, _hash, _etag| Ok(()),
);
assert!(ok);
}
#[test]
fn persist_best_effort_returns_false_on_error() {
let mut dummy = ();
let ok = persist_sync_success_best_effort(
&mut dummy,
"hash".to_string(),
None,
|_settings, _hash, _etag| Err(AppError::Config("boom".to_string())),
);
assert!(!ok);
}
fn manifest_with(format: &str, version: u32, db_compat_version: Option<u32>) -> SyncManifest {
let mut artifacts = BTreeMap::new();
artifacts.insert("db.sql".to_string(), artifact("abc", 1));
artifacts.insert("skills.zip".to_string(), artifact("def", 2));
SyncManifest {
format: format.to_string(),
version,
db_compat_version,
device_name: "My MacBook".to_string(),
created_at: "2026-02-12T00:00:00Z".to_string(),
artifacts,
snapshot_id: "snap-1".to_string(),
}
}
#[test]
fn validate_manifest_compat_accepts_supported_manifest() {
let manifest = manifest_with(PROTOCOL_FORMAT, PROTOCOL_VERSION, Some(DB_COMPAT_VERSION));
assert!(validate_manifest_compat(&manifest, RemoteLayout::Current).is_ok());
}
#[test]
fn validate_manifest_compat_rejects_wrong_format() {
let manifest = manifest_with("other-format", PROTOCOL_VERSION, Some(DB_COMPAT_VERSION));
assert!(validate_manifest_compat(&manifest, RemoteLayout::Current).is_err());
}
#[test]
fn validate_manifest_compat_rejects_wrong_version() {
let manifest = manifest_with(
PROTOCOL_FORMAT,
PROTOCOL_VERSION + 1,
Some(DB_COMPAT_VERSION),
);
assert!(validate_manifest_compat(&manifest, RemoteLayout::Current).is_err());
}
#[test]
fn validate_manifest_compat_accepts_legacy_manifest_without_db_compat() {
let manifest = manifest_with(PROTOCOL_FORMAT, PROTOCOL_VERSION, None);
assert!(validate_manifest_compat(&manifest, RemoteLayout::Legacy).is_ok());
}
#[test]
fn validate_manifest_compat_rejects_current_manifest_with_wrong_db_compat() {
let manifest = manifest_with(
PROTOCOL_FORMAT,
PROTOCOL_VERSION,
Some(LEGACY_DB_COMPAT_VERSION),
);
assert!(validate_manifest_compat(&manifest, RemoteLayout::Current).is_err());
}
#[test]
fn validate_manifest_compat_rejects_legacy_manifest_from_newer_db_generation() {
let manifest = manifest_with(
PROTOCOL_FORMAT,
PROTOCOL_VERSION,
Some(DB_COMPAT_VERSION + 1),
);
assert!(validate_manifest_compat(&manifest, RemoteLayout::Legacy).is_err());
}
#[test]
fn effective_db_compat_version_defaults_legacy_layout_to_v5() {
let manifest = manifest_with(PROTOCOL_FORMAT, PROTOCOL_VERSION, None);
assert_eq!(
effective_db_compat_version(&manifest, RemoteLayout::Legacy),
Some(LEGACY_DB_COMPAT_VERSION)
);
assert_eq!(
effective_db_compat_version(&manifest, RemoteLayout::Current),
None
);
}
#[test]
fn normalize_device_name_returns_none_for_blank_input() {
assert_eq!(normalize_device_name(" \n\t "), None);
}
#[test]
fn normalize_device_name_collapses_whitespace_and_drops_control_chars() {
assert_eq!(
normalize_device_name(" Mac\tBook \n Pro\u{0007} "),
Some("Mac Book Pro".to_string())
);
}
#[test]
fn normalize_device_name_truncates_to_max_len() {
let long = "a".repeat(80);
assert_eq!(normalize_device_name(&long).map(|s| s.len()), Some(64));
}
#[test]
fn manifest_serialization_uses_device_name_only() {
let manifest = manifest_with(PROTOCOL_FORMAT, PROTOCOL_VERSION, Some(DB_COMPAT_VERSION));
let value = serde_json::to_value(&manifest).expect("serialize manifest");
assert!(
value.get("deviceName").is_some(),
"manifest should contain deviceName"
);
assert_eq!(
value.get("dbCompatVersion").and_then(|v| v.as_u64()),
Some(DB_COMPAT_VERSION as u64)
);
assert!(
value.get("deviceId").is_none(),
"manifest should not contain deviceId"
);
}
#[test]
fn validate_artifact_size_limit_rejects_oversized_artifacts() {
let err = validate_artifact_size_limit("skills.zip", MAX_SYNC_ARTIFACT_BYTES + 1)
.expect_err("artifact larger than limit should be rejected");
assert!(
err.to_string().contains("too large") || err.to_string().contains("超过"),
"unexpected error: {err}"
);
}
#[test]
fn validate_artifact_size_limit_accepts_limit_boundary() {
assert!(validate_artifact_size_limit("skills.zip", MAX_SYNC_ARTIFACT_BYTES).is_ok());
}
#[test]
fn verify_artifact_rejects_size_mismatch() {
let meta = artifact("abc123", 100);
let bytes = vec![0u8; 50];
let err = verify_artifact(&bytes, "test.bin", &meta)
.expect_err("size mismatch should be rejected");
assert!(
err.to_string().contains("mismatch") || err.to_string().contains("不匹配"),
"unexpected error: {err}"
);
}
#[test]
fn verify_artifact_rejects_hash_mismatch() {
let meta = ArtifactMeta {
sha256: "0000000000000000000000000000000000000000000000000000000000000000".to_string(),
size: 5,
};
let bytes = b"hello";
let err = verify_artifact(bytes, "test.bin", &meta)
.expect_err("hash mismatch should be rejected");
assert!(
err.to_string().contains("verification failed") || err.to_string().contains("校验失败"),
"unexpected error: {err}"
);
}
#[test]
fn verify_artifact_accepts_matching_data() {
let data = b"hello";
let meta = ArtifactMeta {
sha256: sha256_hex(data),
size: data.len() as u64,
};
assert!(verify_artifact(data, "test.bin", &meta).is_ok());
}
}
+11 -560
View File
@@ -4,16 +4,11 @@
//! primitives in [`super::webdav`]. Artifact set: `db.sql` + `skills.zip`.
use std::collections::BTreeMap;
use std::fs;
use std::future::Future;
use std::process::Command;
use std::sync::OnceLock;
use chrono::Utc;
use serde::{Deserialize, Serialize};
use serde_json::Value;
use sha2::{Digest, Sha256};
use tempfile::tempdir;
use crate::error::AppError;
use crate::services::webdav::{
@@ -22,23 +17,17 @@ use crate::services::webdav::{
};
use crate::settings::{update_webdav_sync_status, WebDavSyncSettings, WebDavSyncStatus};
mod archive;
use archive::{
backup_current_skills, restore_skills_from_backup, restore_skills_zip, zip_skills_ssot,
use super::sync_protocol::{
apply_snapshot, build_local_snapshot, effective_db_compat_version, localized,
persist_sync_success_best_effort, sha256_hex, validate_artifact_size_limit,
validate_manifest_compat, verify_artifact, ArtifactMeta, RemoteLayout, SyncManifest,
DB_COMPAT_VERSION, MAX_MANIFEST_BYTES, MAX_SYNC_ARTIFACT_BYTES, PROTOCOL_VERSION,
REMOTE_DB_SQL, REMOTE_MANIFEST, REMOTE_SKILLS_ZIP,
};
// ─── Protocol constants ──────────────────────────────────────
pub(crate) mod archive;
const PROTOCOL_FORMAT: &str = "cc-switch-webdav-sync";
const PROTOCOL_VERSION: u32 = 2;
const DB_COMPAT_VERSION: u32 = 6;
const LEGACY_DB_COMPAT_VERSION: u32 = 5;
const REMOTE_DB_SQL: &str = "db.sql";
const REMOTE_SKILLS_ZIP: &str = "skills.zip";
const REMOTE_MANIFEST: &str = "manifest.json";
const MAX_DEVICE_NAME_LEN: usize = 64;
const MAX_MANIFEST_BYTES: usize = 1024 * 1024;
pub(super) const MAX_SYNC_ARTIFACT_BYTES: u64 = 512 * 1024 * 1024;
// ─── Sync lock ───────────────────────────────────────────────
pub fn sync_mutex() -> &'static tokio::sync::Mutex<()> {
static LOCK: OnceLock<tokio::sync::Mutex<()>> = OnceLock::new();
@@ -53,74 +42,12 @@ where
operation.await
}
fn localized(key: &'static str, zh: impl Into<String>, en: impl Into<String>) -> AppError {
AppError::localized(key, zh, en)
}
fn io_context_localized(
_key: &'static str,
zh: impl Into<String>,
en: impl Into<String>,
source: std::io::Error,
) -> AppError {
let zh_msg = zh.into();
let en_msg = en.into();
AppError::IoContext {
context: format!("{zh_msg} ({en_msg})"),
source,
}
}
// ─── Types ───────────────────────────────────────────────────
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
struct SyncManifest {
format: String,
version: u32,
#[serde(default, skip_serializing_if = "Option::is_none")]
db_compat_version: Option<u32>,
device_name: String,
created_at: String,
artifacts: BTreeMap<String, ArtifactMeta>,
snapshot_id: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
struct ArtifactMeta {
sha256: String,
size: u64,
}
struct LocalSnapshot {
db_sql: Vec<u8>,
skills_zip: Vec<u8>,
manifest_bytes: Vec<u8>,
manifest_hash: String,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum RemoteLayout {
Current,
Legacy,
}
impl RemoteLayout {
fn as_str(self) -> &'static str {
match self {
Self::Current => "current",
Self::Legacy => "legacy",
}
}
}
struct RemoteSnapshot {
layout: RemoteLayout,
manifest: SyncManifest,
manifest_bytes: Vec<u8>,
manifest_etag: Option<String>,
}
// ─── Public API ──────────────────────────────────────────────
/// Check WebDAV connectivity and ensure remote directory structure.
@@ -143,7 +70,7 @@ pub async fn upload(
let dir_segs = remote_dir_segments(settings, RemoteLayout::Current);
ensure_remote_directories(&settings.base_url, &dir_segs, &auth).await?;
let snapshot = build_local_snapshot(db, settings)?;
let snapshot = build_local_snapshot(db)?;
// Upload order: artifacts first, manifest last (best-effort consistency)
let db_url = remote_file_url(settings, RemoteLayout::Current, REMOTE_DB_SQL)?;
@@ -259,7 +186,7 @@ pub async fn fetch_remote_info(settings: &WebDavSyncSettings) -> Result<Option<V
Ok(Some(payload))
}
// ─── Sync status persistence (I3: deduplicated) ─────────────
// ─── Sync status persistence ─────────────────────────────────
fn persist_sync_success(
settings: &mut WebDavSyncSettings,
@@ -278,214 +205,6 @@ fn persist_sync_success(
update_webdav_sync_status(status)
}
fn persist_sync_success_best_effort<F>(
settings: &mut WebDavSyncSettings,
manifest_hash: String,
etag: Option<String>,
persist_fn: F,
) -> bool
where
F: FnOnce(&mut WebDavSyncSettings, String, Option<String>) -> Result<(), AppError>,
{
match persist_fn(settings, manifest_hash, etag) {
Ok(()) => true,
Err(err) => {
log::warn!("[WebDAV] Persist sync status failed, keep operation success: {err}");
false
}
}
}
// ─── Snapshot building ───────────────────────────────────────
fn build_local_snapshot(
db: &crate::database::Database,
_settings: &WebDavSyncSettings,
) -> Result<LocalSnapshot, AppError> {
// Export database to SQL string
let sql_string = db.export_sql_string_for_sync()?;
let db_sql = sql_string.into_bytes();
// Pack skills into deterministic ZIP
let tmp = tempdir().map_err(|e| {
io_context_localized(
"webdav.sync.snapshot_tmpdir_failed",
"创建 WebDAV 快照临时目录失败",
"Failed to create temporary directory for WebDAV snapshot",
e,
)
})?;
let skills_zip_path = tmp.path().join(REMOTE_SKILLS_ZIP);
zip_skills_ssot(&skills_zip_path)?;
let skills_zip = fs::read(&skills_zip_path).map_err(|e| AppError::io(&skills_zip_path, e))?;
// Build artifact map and compute hashes
let mut artifacts = BTreeMap::new();
artifacts.insert(
REMOTE_DB_SQL.to_string(),
ArtifactMeta {
sha256: sha256_hex(&db_sql),
size: db_sql.len() as u64,
},
);
artifacts.insert(
REMOTE_SKILLS_ZIP.to_string(),
ArtifactMeta {
sha256: sha256_hex(&skills_zip),
size: skills_zip.len() as u64,
},
);
let snapshot_id = compute_snapshot_id(&artifacts);
let manifest = SyncManifest {
format: PROTOCOL_FORMAT.to_string(),
version: PROTOCOL_VERSION,
db_compat_version: Some(DB_COMPAT_VERSION),
device_name: detect_system_device_name().unwrap_or_else(|| "Unknown Device".to_string()),
created_at: Utc::now().to_rfc3339(),
artifacts,
snapshot_id,
};
let manifest_bytes =
serde_json::to_vec_pretty(&manifest).map_err(|e| AppError::JsonSerialize { source: e })?;
let manifest_hash = sha256_hex(&manifest_bytes);
Ok(LocalSnapshot {
db_sql,
skills_zip,
manifest_bytes,
manifest_hash,
})
}
/// Compute a deterministic snapshot identity from artifact hashes.
///
/// BTreeMap iteration order is sorted by key, ensuring stability.
fn compute_snapshot_id(artifacts: &BTreeMap<String, ArtifactMeta>) -> String {
let parts: Vec<String> = artifacts
.iter()
.map(|(name, meta)| format!("{}:{}", name, meta.sha256))
.collect();
sha256_hex(parts.join("|").as_bytes())
}
fn sha256_hex(bytes: &[u8]) -> String {
let mut hasher = Sha256::new();
hasher.update(bytes);
format!("{:x}", hasher.finalize())
}
fn detect_system_device_name() -> Option<String> {
let env_name = ["CC_SWITCH_DEVICE_NAME", "COMPUTERNAME", "HOSTNAME"]
.iter()
.filter_map(|key| std::env::var(key).ok())
.find_map(|value| normalize_device_name(&value));
if env_name.is_some() {
return env_name;
}
let output = Command::new("hostname").output().ok()?;
if !output.status.success() {
return None;
}
let hostname = String::from_utf8(output.stdout).ok()?;
normalize_device_name(&hostname)
}
fn normalize_device_name(raw: &str) -> Option<String> {
let compact = raw
.chars()
.fold(String::with_capacity(raw.len()), |mut acc, ch| {
if ch.is_whitespace() {
acc.push(' ');
} else if !ch.is_control() {
acc.push(ch);
}
acc
});
let normalized = compact.split_whitespace().collect::<Vec<_>>().join(" ");
let trimmed = normalized.trim();
if trimmed.is_empty() {
return None;
}
let limited = trimmed
.chars()
.take(MAX_DEVICE_NAME_LEN)
.collect::<String>();
if limited.is_empty() {
None
} else {
Some(limited)
}
}
fn effective_db_compat_version(manifest: &SyncManifest, layout: RemoteLayout) -> Option<u32> {
manifest
.db_compat_version
.or_else(|| (layout == RemoteLayout::Legacy).then_some(LEGACY_DB_COMPAT_VERSION))
}
fn validate_manifest_compat(manifest: &SyncManifest, layout: RemoteLayout) -> Result<(), AppError> {
if manifest.format != PROTOCOL_FORMAT {
return Err(localized(
"webdav.sync.manifest_format_incompatible",
format!("远端 manifest 格式不兼容: {}", manifest.format),
format!(
"Remote manifest format is incompatible: {}",
manifest.format
),
));
}
if manifest.version != PROTOCOL_VERSION {
return Err(localized(
"webdav.sync.manifest_version_incompatible",
format!(
"远端 manifest 协议版本不兼容: v{} (本地 v{PROTOCOL_VERSION})",
manifest.version
),
format!(
"Remote manifest protocol version is incompatible: v{} (local v{PROTOCOL_VERSION})",
manifest.version
),
));
}
let Some(db_compat_version) = effective_db_compat_version(manifest, layout) else {
return Err(localized(
"webdav.sync.manifest_db_version_missing",
"远端 manifest 缺少数据库兼容版本",
"Remote manifest is missing the database compatibility version.",
));
};
match layout {
RemoteLayout::Current if db_compat_version != DB_COMPAT_VERSION => {
return Err(localized(
"webdav.sync.manifest_db_version_incompatible",
format!(
"远端数据库快照版本不兼容: db-v{db_compat_version} (本地 db-v{DB_COMPAT_VERSION})"
),
format!(
"Remote database snapshot version is incompatible: db-v{db_compat_version} (local db-v{DB_COMPAT_VERSION})"
),
));
}
RemoteLayout::Legacy if db_compat_version > DB_COMPAT_VERSION => {
return Err(localized(
"webdav.sync.manifest_db_version_incompatible",
format!(
"远端数据库快照版本不兼容: db-v{db_compat_version} (本地最高支持 db-v{DB_COMPAT_VERSION})"
),
format!(
"Remote database snapshot version is incompatible: db-v{db_compat_version} (local supports up to db-v{DB_COMPAT_VERSION})"
),
));
}
_ => {}
}
Ok(())
}
async fn find_remote_snapshot(
settings: &WebDavSyncSettings,
auth: &WebDavAuth,
@@ -521,7 +240,6 @@ async fn fetch_remote_snapshot(
manifest_etag,
}))
}
// ─── Download & verify ───────────────────────────────────────
async fn download_and_verify(
@@ -551,75 +269,10 @@ async fn download_and_verify(
)
})?;
// Quick size check before expensive hash
if bytes.len() as u64 != meta.size {
return Err(localized(
"webdav.sync.artifact_size_mismatch",
format!(
"artifact {artifact_name} 大小不匹配 (expected: {}, got: {})",
meta.size,
bytes.len(),
),
format!(
"Artifact {artifact_name} size mismatch (expected: {}, got: {})",
meta.size,
bytes.len(),
),
));
}
let actual_hash = sha256_hex(&bytes);
if actual_hash != meta.sha256 {
return Err(localized(
"webdav.sync.artifact_hash_mismatch",
format!(
"artifact {artifact_name} SHA256 校验失败 (expected: {}..., got: {}...)",
meta.sha256.get(..8).unwrap_or(&meta.sha256),
actual_hash.get(..8).unwrap_or(&actual_hash),
),
format!(
"Artifact {artifact_name} SHA256 verification failed (expected: {}..., got: {}...)",
meta.sha256.get(..8).unwrap_or(&meta.sha256),
actual_hash.get(..8).unwrap_or(&actual_hash),
),
));
}
verify_artifact(&bytes, artifact_name, meta)?;
Ok(bytes)
}
fn apply_snapshot(
db: &crate::database::Database,
db_sql: &[u8],
skills_zip: &[u8],
) -> Result<(), AppError> {
let sql_str = std::str::from_utf8(db_sql).map_err(|e| {
localized(
"webdav.sync.sql_not_utf8",
format!("SQL 非 UTF-8: {e}"),
format!("SQL is not valid UTF-8: {e}"),
)
})?;
let skills_backup = backup_current_skills()?;
// 先替换 skills,再导入数据库;若导入失败则回滚 skills,避免“半恢复”。
restore_skills_zip(skills_zip)?;
if let Err(db_err) = db.import_sql_string_for_sync(sql_str) {
if let Err(rollback_err) = restore_skills_from_backup(&skills_backup) {
return Err(localized(
"webdav.sync.db_import_and_rollback_failed",
format!("导入数据库失败: {db_err}; 同时回滚 Skills 失败: {rollback_err}"),
format!(
"Database import failed: {db_err}; skills rollback also failed: {rollback_err}"
),
));
}
return Err(db_err);
}
Ok(())
}
// ─── Remote path helpers ─────────────────────────────────────
fn remote_dir_segments(settings: &WebDavSyncSettings, layout: RemoteLayout) -> Vec<String> {
@@ -652,53 +305,12 @@ fn auth_for(settings: &WebDavSyncSettings) -> WebDavAuth {
auth_from_credentials(&settings.username, &settings.password)
}
fn validate_artifact_size_limit(artifact_name: &str, size: u64) -> Result<(), AppError> {
if size > MAX_SYNC_ARTIFACT_BYTES {
let max_mb = MAX_SYNC_ARTIFACT_BYTES / 1024 / 1024;
return Err(localized(
"webdav.sync.artifact_too_large",
format!("artifact {artifact_name} 超过下载上限({max_mb} MB"),
format!("Artifact {artifact_name} exceeds download limit ({max_mb} MB)"),
));
}
Ok(())
}
// ─── Tests ───────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
fn artifact(sha256: &str, size: u64) -> ArtifactMeta {
ArtifactMeta {
sha256: sha256.to_string(),
size,
}
}
#[test]
fn snapshot_id_is_stable() {
let mut artifacts = BTreeMap::new();
artifacts.insert("db.sql".to_string(), artifact("abc123", 100));
artifacts.insert("skills.zip".to_string(), artifact("def456", 200));
let id1 = compute_snapshot_id(&artifacts);
let id2 = compute_snapshot_id(&artifacts);
assert_eq!(id1, id2);
}
#[test]
fn snapshot_id_changes_with_artifacts() {
let mut a1 = BTreeMap::new();
a1.insert("db.sql".to_string(), artifact("hash-a", 1));
let mut a2 = BTreeMap::new();
a2.insert("db.sql".to_string(), artifact("hash-b", 1));
assert_ne!(compute_snapshot_id(&a1), compute_snapshot_id(&a2));
}
#[test]
fn remote_dir_segments_uses_current_layout() {
let settings = WebDavSyncSettings {
@@ -720,165 +332,4 @@ mod tests {
let segs = remote_dir_segments(&settings, RemoteLayout::Legacy);
assert_eq!(segs, vec!["cc-switch-sync", "v2", "default"]);
}
#[test]
fn sha256_hex_is_correct() {
let hash = sha256_hex(b"hello");
assert_eq!(
hash,
"2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"
);
}
#[test]
fn persist_best_effort_returns_true_on_success() {
let mut settings = WebDavSyncSettings::default();
let ok = persist_sync_success_best_effort(
&mut settings,
"hash".to_string(),
Some("etag".to_string()),
|_settings, _hash, _etag| Ok(()),
);
assert!(ok);
}
#[test]
fn persist_best_effort_returns_false_on_error() {
let mut settings = WebDavSyncSettings::default();
let ok = persist_sync_success_best_effort(
&mut settings,
"hash".to_string(),
None,
|_settings, _hash, _etag| Err(AppError::Config("boom".to_string())),
);
assert!(!ok);
}
fn manifest_with(format: &str, version: u32, db_compat_version: Option<u32>) -> SyncManifest {
let mut artifacts = BTreeMap::new();
artifacts.insert("db.sql".to_string(), artifact("abc", 1));
artifacts.insert("skills.zip".to_string(), artifact("def", 2));
SyncManifest {
format: format.to_string(),
version,
db_compat_version,
device_name: "My MacBook".to_string(),
created_at: "2026-02-12T00:00:00Z".to_string(),
artifacts,
snapshot_id: "snap-1".to_string(),
}
}
#[test]
fn validate_manifest_compat_accepts_supported_manifest() {
let manifest = manifest_with(PROTOCOL_FORMAT, PROTOCOL_VERSION, Some(DB_COMPAT_VERSION));
assert!(validate_manifest_compat(&manifest, RemoteLayout::Current).is_ok());
}
#[test]
fn validate_manifest_compat_rejects_wrong_format() {
let manifest = manifest_with("other-format", PROTOCOL_VERSION, Some(DB_COMPAT_VERSION));
assert!(validate_manifest_compat(&manifest, RemoteLayout::Current).is_err());
}
#[test]
fn validate_manifest_compat_rejects_wrong_version() {
let manifest = manifest_with(
PROTOCOL_FORMAT,
PROTOCOL_VERSION + 1,
Some(DB_COMPAT_VERSION),
);
assert!(validate_manifest_compat(&manifest, RemoteLayout::Current).is_err());
}
#[test]
fn validate_manifest_compat_accepts_legacy_manifest_without_db_compat() {
let manifest = manifest_with(PROTOCOL_FORMAT, PROTOCOL_VERSION, None);
assert!(validate_manifest_compat(&manifest, RemoteLayout::Legacy).is_ok());
}
#[test]
fn validate_manifest_compat_rejects_current_manifest_with_wrong_db_compat() {
let manifest = manifest_with(
PROTOCOL_FORMAT,
PROTOCOL_VERSION,
Some(LEGACY_DB_COMPAT_VERSION),
);
assert!(validate_manifest_compat(&manifest, RemoteLayout::Current).is_err());
}
#[test]
fn validate_manifest_compat_rejects_legacy_manifest_from_newer_db_generation() {
let manifest = manifest_with(
PROTOCOL_FORMAT,
PROTOCOL_VERSION,
Some(DB_COMPAT_VERSION + 1),
);
assert!(validate_manifest_compat(&manifest, RemoteLayout::Legacy).is_err());
}
#[test]
fn effective_db_compat_version_defaults_legacy_layout_to_v5() {
let manifest = manifest_with(PROTOCOL_FORMAT, PROTOCOL_VERSION, None);
assert_eq!(
effective_db_compat_version(&manifest, RemoteLayout::Legacy),
Some(LEGACY_DB_COMPAT_VERSION)
);
assert_eq!(
effective_db_compat_version(&manifest, RemoteLayout::Current),
None
);
}
#[test]
fn normalize_device_name_returns_none_for_blank_input() {
assert_eq!(normalize_device_name(" \n\t "), None);
}
#[test]
fn normalize_device_name_collapses_whitespace_and_drops_control_chars() {
assert_eq!(
normalize_device_name(" Mac\tBook \n Pro\u{0007} "),
Some("Mac Book Pro".to_string())
);
}
#[test]
fn normalize_device_name_truncates_to_max_len() {
let long = "a".repeat(80);
assert_eq!(normalize_device_name(&long).map(|s| s.len()), Some(64));
}
#[test]
fn manifest_serialization_uses_device_name_only() {
let manifest = manifest_with(PROTOCOL_FORMAT, PROTOCOL_VERSION, Some(DB_COMPAT_VERSION));
let value = serde_json::to_value(&manifest).expect("serialize manifest");
assert!(
value.get("deviceName").is_some(),
"manifest should contain deviceName"
);
assert_eq!(
value.get("dbCompatVersion").and_then(|v| v.as_u64()),
Some(DB_COMPAT_VERSION as u64)
);
assert!(
value.get("deviceId").is_none(),
"manifest should not contain deviceId"
);
}
#[test]
fn validate_artifact_size_limit_rejects_oversized_artifacts() {
let err = validate_artifact_size_limit("skills.zip", MAX_SYNC_ARTIFACT_BYTES + 1)
.expect_err("artifact larger than limit should be rejected");
assert!(
err.to_string().contains("too large") || err.to_string().contains("超过"),
"unexpected error: {err}"
);
}
#[test]
fn validate_artifact_size_limit_accepts_limit_boundary() {
assert!(validate_artifact_size_limit("skills.zip", MAX_SYNC_ARTIFACT_BYTES).is_ok());
}
}
@@ -10,19 +10,21 @@ use zip::DateTime;
use crate::error::AppError;
use crate::services::skill::SkillService;
use super::{io_context_localized, localized, MAX_SYNC_ARTIFACT_BYTES, REMOTE_SKILLS_ZIP};
use crate::services::sync_protocol::{
io_context_localized, localized, MAX_SYNC_ARTIFACT_BYTES, REMOTE_SKILLS_ZIP,
};
/// Maximum number of entries allowed in a zip archive.
const MAX_EXTRACT_ENTRIES: usize = 10_000;
pub(super) struct SkillsBackup {
pub(crate) struct SkillsBackup {
_tmp: TempDir,
backup_dir: PathBuf,
ssot_path: PathBuf,
existed: bool,
}
pub(super) fn zip_skills_ssot(dest_path: &Path) -> Result<(), AppError> {
pub(crate) fn zip_skills_ssot(dest_path: &Path) -> Result<(), AppError> {
let source = SkillService::get_ssot_dir().map_err(|e| {
localized(
"webdav.sync.skills_ssot_dir_failed",
@@ -63,7 +65,7 @@ pub(super) fn zip_skills_ssot(dest_path: &Path) -> Result<(), AppError> {
Ok(())
}
pub(super) fn restore_skills_zip(raw: &[u8]) -> Result<(), AppError> {
pub(crate) fn restore_skills_zip(raw: &[u8]) -> Result<(), AppError> {
let tmp = tempdir().map_err(|e| {
io_context_localized(
"webdav.sync.skills_extract_tmpdir_failed",
@@ -159,7 +161,7 @@ pub(super) fn restore_skills_zip(raw: &[u8]) -> Result<(), AppError> {
Ok(())
}
pub(super) fn backup_current_skills() -> Result<SkillsBackup, AppError> {
pub(crate) fn backup_current_skills() -> Result<SkillsBackup, AppError> {
let ssot = SkillService::get_ssot_dir().map_err(|e| {
localized(
"webdav.sync.skills_ssot_dir_failed",
@@ -190,7 +192,7 @@ pub(super) fn backup_current_skills() -> Result<SkillsBackup, AppError> {
})
}
pub(super) fn restore_skills_from_backup(backup: &SkillsBackup) -> Result<(), AppError> {
pub(crate) fn restore_skills_from_backup(backup: &SkillsBackup) -> Result<(), AppError> {
if backup.ssot_path.exists() {
fs::remove_dir_all(&backup.ssot_path).map_err(|e| AppError::io(&backup.ssot_path, e))?;
}
+135
View File
@@ -176,6 +176,106 @@ impl WebDavSyncSettings {
}
}
/// S3 同步设置
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct S3SyncSettings {
#[serde(default)]
pub enabled: bool,
#[serde(default)]
pub auto_sync: bool,
#[serde(default)]
pub region: String,
#[serde(default)]
pub bucket: String,
#[serde(default)]
pub access_key_id: String,
#[serde(default)]
pub secret_access_key: String,
#[serde(default)]
pub endpoint: String,
#[serde(default = "default_remote_root")]
pub remote_root: String,
#[serde(default = "default_profile")]
pub profile: String,
#[serde(default)]
pub status: WebDavSyncStatus,
}
impl Default for S3SyncSettings {
fn default() -> Self {
Self {
enabled: false,
auto_sync: false,
region: String::new(),
bucket: String::new(),
access_key_id: String::new(),
secret_access_key: String::new(),
endpoint: String::new(),
remote_root: default_remote_root(),
profile: default_profile(),
status: WebDavSyncStatus::default(),
}
}
}
impl S3SyncSettings {
pub fn validate(&self) -> Result<(), crate::error::AppError> {
if self.bucket.trim().is_empty() {
return Err(crate::error::AppError::localized(
"s3.bucket.required",
"S3 存储桶不能为空",
"S3 bucket is required.",
));
}
if self.region.trim().is_empty() {
return Err(crate::error::AppError::localized(
"s3.region.required",
"S3 区域不能为空",
"S3 region is required.",
));
}
if self.access_key_id.trim().is_empty() {
return Err(crate::error::AppError::localized(
"s3.access_key_id.required",
"S3 Access Key ID 不能为空",
"S3 Access Key ID is required.",
));
}
if self.secret_access_key.trim().is_empty() {
return Err(crate::error::AppError::localized(
"s3.secret_access_key.required",
"S3 Secret Access Key 不能为空",
"S3 Secret Access Key is required.",
));
}
Ok(())
}
pub fn normalize(&mut self) {
self.region = self.region.trim().to_string();
self.bucket = self.bucket.trim().to_string();
self.access_key_id = self.access_key_id.trim().to_string();
self.endpoint = self.endpoint.trim().to_string();
self.remote_root = self.remote_root.trim().to_string();
self.profile = self.profile.trim().to_string();
if self.remote_root.is_empty() {
self.remote_root = default_remote_root();
}
if self.profile.is_empty() {
self.profile = default_profile();
}
}
/// Returns true if all credential fields are blank (no config to persist).
fn is_empty(&self) -> bool {
self.bucket.is_empty()
&& self.region.is_empty()
&& self.access_key_id.is_empty()
&& self.secret_access_key.is_empty()
}
}
/// 本机自动迁移状态。
///
/// 这里记录的是本机启动时执行过的一次性迁移;标记不随数据库同步。
@@ -322,6 +422,10 @@ pub struct AppSettings {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub webdav_sync: Option<WebDavSyncSettings>,
// ===== S3 同步设置 =====
#[serde(default, skip_serializing_if = "Option::is_none")]
pub s3_sync: Option<S3SyncSettings>,
// ===== WebDAV 备份设置(旧版,保留向后兼容)=====
#[serde(default, skip_serializing_if = "Option::is_none")]
pub webdav_backup: Option<serde_json::Value>,
@@ -392,6 +496,7 @@ impl Default for AppSettings {
skill_sync_method: SyncMethod::default(),
skill_storage_location: SkillStorageLocation::default(),
webdav_sync: None,
s3_sync: None,
webdav_backup: None,
backup_interval_hours: None,
backup_retain_count: None,
@@ -467,6 +572,13 @@ impl AppSettings {
self.webdav_sync = None;
}
}
if let Some(s3) = &mut self.s3_sync {
s3.normalize();
if s3.is_empty() {
self.s3_sync = None;
}
}
}
fn load_from_file() -> Self {
@@ -570,6 +682,9 @@ pub fn get_settings_for_frontend() -> AppSettings {
if let Some(sync) = &mut settings.webdav_sync {
sync.password.clear();
}
if let Some(s3) = &mut settings.s3_sync {
s3.secret_access_key.clear();
}
settings.webdav_backup = None;
settings
}
@@ -882,6 +997,26 @@ pub fn update_webdav_sync_status(status: WebDavSyncStatus) -> Result<(), AppErro
})
}
// ===== S3 同步设置管理函数 =====
pub fn get_s3_sync_settings() -> Option<S3SyncSettings> {
settings_store().read().ok()?.s3_sync.clone()
}
pub fn set_s3_sync_settings(settings: Option<S3SyncSettings>) -> Result<(), AppError> {
mutate_settings(|current| {
current.s3_sync = settings;
})
}
pub fn update_s3_sync_status(status: WebDavSyncStatus) -> Result<(), AppError> {
mutate_settings(|current| {
if let Some(s3) = current.s3_sync.as_mut() {
s3.status = status;
}
})
}
#[cfg(test)]
mod tests {
use super::*;
+18 -2
View File
@@ -107,7 +107,7 @@ type View =
| "openclawAgents"
| "hermesMemory";
interface WebDavSyncStatusUpdatedPayload {
interface SyncStatusUpdatedPayload {
source?: string;
status?: string;
error?: string;
@@ -371,7 +371,7 @@ function App() {
}
});
useTauriEvent<WebDavSyncStatusUpdatedPayload | null | undefined>(
useTauriEvent<SyncStatusUpdatedPayload | null | undefined>(
"webdav-sync-status-updated",
async (payload) => {
const statusPayload = payload ?? {};
@@ -387,6 +387,22 @@ function App() {
},
);
useTauriEvent<SyncStatusUpdatedPayload | null | undefined>(
"s3-sync-status-updated",
async (payload) => {
const statusPayload = payload ?? {};
await queryClient.invalidateQueries({ queryKey: ["settings"] });
if (statusPayload.source !== "auto" || statusPayload.status !== "error") {
return;
}
toast.error(
t("settings.s3Sync.autoSyncFailedToast", {
error: statusPayload.error || t("common.unknown"),
}),
);
},
);
useTauriEvent<{ appType: string; providerName: string }>(
"proxy-official-warning",
(payload) => {
+1
View File
@@ -415,6 +415,7 @@ export function SettingsPage({
<AccordionContent className="px-6 pb-6 pt-4 border-t border-border/50">
<WebdavSyncSection
config={settings?.webdavSync}
s3Config={settings?.s3Sync}
settings={settings}
onAutoSave={handleAutoSave}
/>
File diff suppressed because it is too large Load Diff
+10 -4
View File
@@ -193,8 +193,11 @@ export function useSettings(): UseSettingsResult {
const sanitizedOpenclawDir = sanitizeDir(
mergedSettings.openclawConfigDir,
);
const { webdavSync: _ignoredWebdavSync, ...restSettings } =
mergedSettings;
const {
webdavSync: _ignoredWebdavSync,
s3Sync: _ignoredS3Sync,
...restSettings
} = mergedSettings;
const payload: Settings = {
...restSettings,
@@ -327,8 +330,11 @@ export function useSettings(): UseSettingsResult {
const previousGeminiDir = sanitizeDir(data?.geminiConfigDir);
const previousOpencodeDir = sanitizeDir(data?.opencodeConfigDir);
const previousOpenclawDir = sanitizeDir(data?.openclawConfigDir);
const { webdavSync: _ignoredWebdavSync, ...restSettings } =
mergedSettings;
const {
webdavSync: _ignoredWebdavSync,
s3Sync: _ignoredS3Sync,
...restSettings
} = mergedSettings;
const payload: Settings = {
...restSettings,
+101
View File
@@ -546,6 +546,107 @@
"confirm": "Confirm Upload"
}
},
"syncType": {
"label": "Sync Type",
"webdav": "WebDAV",
"s3": "S3 Compatible"
},
"s3Sync": {
"presets": {
"label": "Provider",
"awsS3": "AWS S3",
"awsS3Hint": "Use IAM Access Key with s3:PutObject/GetObject/HeadObject permissions. Region: us-east-1, ap-northeast-1, etc.",
"minio": "MinIO",
"minioHint": "Custom endpoint required, e.g. http://minio.local:9000. Region can be 'us-east-1'.",
"r2": "Cloudflare R2",
"r2Hint": "Endpoint: https://<account_id>.r2.cloudflarestorage.com. Region: 'auto'.",
"oss": "Alibaba Cloud OSS",
"ossHint": "Endpoint: https://oss-<region>.aliyuncs.com. Region: cn-hangzhou, cn-shanghai, etc.",
"cos": "Tencent Cloud COS",
"cosHint": "Endpoint: https://cos.<region>.myqcloud.com. Region: ap-guangzhou, ap-beijing, etc.",
"obs": "Huawei OBS",
"obsHint": "Endpoint: https://obs.<region>.myhuaweicloud.com. Region: cn-north-4, cn-east-3, etc.",
"custom": "S3 Compatible (Custom)",
"customHint": "Any S3-compatible service. Custom endpoint required."
},
"region": "Region",
"regionPlaceholder": "e.g. us-east-1",
"bucket": "Bucket",
"bucketPlaceholder": "e.g. my-cc-switch-bucket",
"accessKeyId": "Access Key ID",
"accessKeyIdPlaceholder": "e.g. AKIA...",
"secretAccessKey": "Secret Access Key",
"secretAccessKeyPlaceholder": "Leave unchanged to keep the existing secret",
"endpoint": "Endpoint (optional for AWS)",
"endpointHint": "Required for most S3-compatible services",
"endpointPlaceholder": "e.g. https://s3-compatible.example.com",
"remoteRoot": "Remote Root Directory",
"remoteRootDefault": "Default: cc-switch-sync",
"profile": "Sync Profile Name",
"profileDefault": "Default: default",
"autoSync": "Auto Sync",
"autoSyncHint": "When enabled, each database change triggers an automatic S3 upload.",
"enabled": "Enable S3 Sync",
"enabledHint": "Turn on S3 sync for manual and automatic cloud operations.",
"mutualExclusionTitle": "Switch Sync Type",
"mutualExclusionMessage": "Enabling S3 sync will disable the current WebDAV sync. Continue?",
"mutualExclusionMessageReverse": "Enabling WebDAV sync will disable the current S3 sync. Continue?",
"mutualExclusionFailed": "Failed to switch sync type: {{error}}",
"testConnection": "Test Connection",
"connectionOk": "S3 connection successful",
"test": "Test Connection",
"testing": "Testing...",
"testSuccess": "S3 connection successful",
"testFailed": "S3 connection failed: {{error}}",
"save": "Save Config",
"saving": "Saving...",
"saveFailed": "Failed to save config: {{error}}",
"saveAndTestSuccess": "Config saved, S3 connection OK",
"saveAndTestFailed": "Config saved, but S3 connection test failed: {{error}}",
"upload": "Upload to Cloud",
"uploading": "Uploading...",
"uploadSuccess": "Uploaded to S3",
"uploadFailed": "S3 upload failed: {{error}}",
"autoSyncFailedToast": "S3 auto sync failed: {{error}}",
"download": "Download from Cloud",
"downloading": "Downloading...",
"downloadSuccess": "Downloaded and restored from S3",
"downloadFailed": "S3 download failed: {{error}}",
"lastSync": "Last sync: {{time}}",
"autoSyncLastErrorTitle": "Last S3 auto sync failed",
"autoSyncLastErrorHint": "Please check network or S3 settings. Auto sync will retry on future changes.",
"missingBucket": "Please enter the S3 bucket",
"noRemoteData": "No sync data found in the S3 bucket",
"incompatibleVersion": "Remote data is incompatible (version {{version}}). This client supports protocol v2 / db-v6.",
"unsaved": "Unsaved",
"saved": "Saved",
"unsavedChanges": "Please save config first",
"saveBeforeSync": "Save configuration first to enable upload/download.",
"fetchingRemote": "Fetching remote info...",
"fetchRemoteFailed": "Failed to fetch remote info. Please check configuration and network.",
"notConfigured": "S3 sync is not configured",
"disabled": "S3 sync is disabled",
"confirmDownload": {
"title": "Restore from S3",
"deviceName": "Uploaded by",
"createdAt": "Uploaded at",
"artifacts": "Contents",
"warning": "This will overwrite all local data and skill configurations",
"confirm": "Confirm Restore"
},
"confirmUpload": {
"title": "Upload to S3",
"content": "The following will be synced to the S3 bucket:",
"dbItem": "Database (all provider configs and data)",
"skillsItem": "Skills (all custom skills)",
"targetPath": "Target object path",
"existingData": "Existing cloud data",
"deviceName": "Uploaded by",
"createdAt": "Uploaded at",
"warning": "This will overwrite existing sync data at the S3 path",
"confirm": "Confirm Upload"
}
},
"autoReload": "Data refreshed",
"languageOptionChinese": "简体中文",
"languageOptionTraditionalChinese": "繁體中文",
+101
View File
@@ -546,6 +546,107 @@
"confirm": "アップロードを実行"
}
},
"syncType": {
"label": "同期方式",
"webdav": "WebDAV",
"s3": "S3互換ストレージ"
},
"s3Sync": {
"presets": {
"label": "サービス",
"awsS3": "AWS S3",
"awsS3Hint": "IAM Access Keyを使用。s3:PutObject/GetObject/HeadObject権限が必要です。Region例: us-east-1, ap-northeast-1",
"minio": "MinIO",
"minioHint": "カスタムEndpointが必要です(例: http://minio.local:9000)。Regionは us-east-1 に設定可能",
"r2": "Cloudflare R2",
"r2Hint": "Endpoint: https://<account_id>.r2.cloudflarestorage.com、Regionは auto に設定",
"oss": "Alibaba Cloud OSS",
"ossHint": "Endpoint: https://oss-<region>.aliyuncs.com、Region例: cn-hangzhou, cn-shanghai",
"cos": "Tencent Cloud COS",
"cosHint": "Endpoint: https://cos.<region>.myqcloud.com、Region例: ap-guangzhou, ap-beijing",
"obs": "Huawei OBS",
"obsHint": "Endpoint: https://obs.<region>.myhuaweicloud.com、Region例: cn-north-4, cn-east-3",
"custom": "S3互換サービス(カスタム)",
"customHint": "任意のS3互換サービス。カスタムEndpointが必要です"
},
"region": "リージョン",
"regionPlaceholder": "例: us-east-1",
"bucket": "バケット",
"bucketPlaceholder": "例: my-cc-switch-bucket",
"accessKeyId": "Access Key ID",
"accessKeyIdPlaceholder": "例: AKIA...",
"secretAccessKey": "Secret Access Key",
"secretAccessKeyPlaceholder": "既存のシークレットを保持する場合は変更しないでください",
"endpoint": "エンドポイント(AWSは空欄可)",
"endpointHint": "多くの S3 互換サービスでは必須です",
"endpointPlaceholder": "例: https://s3-compatible.example.com",
"remoteRoot": "リモートルートディレクトリ",
"remoteRootDefault": "デフォルト: cc-switch-sync",
"profile": "同期プロファイル名",
"profileDefault": "デフォルト: default",
"autoSync": "自動同期",
"autoSyncHint": "有効にすると、データベース変更のたびに S3 へ自動アップロードします。",
"enabled": "S3 同期を有効化",
"enabledHint": "S3 のアップロード/ダウンロードと自動同期を有効にします。",
"mutualExclusionTitle": "同期方式の切り替え",
"mutualExclusionMessage": "S3同期を有効にすると、現在のWebDAV同期が無効になります。続行しますか?",
"mutualExclusionMessageReverse": "WebDAV同期を有効にすると、現在のS3同期が無効になります。続行しますか?",
"mutualExclusionFailed": "同期方式の切り替えに失敗しました:{{error}}",
"testConnection": "接続テスト",
"connectionOk": "S3接続成功",
"test": "接続テスト",
"testing": "テスト中...",
"testSuccess": "S3 接続成功",
"testFailed": "S3 接続失敗:{{error}}",
"save": "設定を保存",
"saving": "保存中...",
"saveFailed": "設定の保存に失敗しました:{{error}}",
"saveAndTestSuccess": "設定を保存しました。S3 接続正常です",
"saveAndTestFailed": "設定を保存しましたが、S3 接続テストに失敗しました:{{error}}",
"upload": "クラウドにアップロード",
"uploading": "アップロード中...",
"uploadSuccess": "S3 にアップロードしました",
"uploadFailed": "S3 アップロードに失敗しました:{{error}}",
"autoSyncFailedToast": "S3 自動同期に失敗しました:{{error}}",
"download": "クラウドからダウンロード",
"downloading": "ダウンロード中...",
"downloadSuccess": "S3 からダウンロード・復元しました",
"downloadFailed": "S3 ダウンロードに失敗しました:{{error}}",
"lastSync": "前回の同期:{{time}}",
"autoSyncLastErrorTitle": "前回の S3 自動同期に失敗しました",
"autoSyncLastErrorHint": "ネットワークまたは S3 設定を確認してください。次回の変更時に自動再試行されます。",
"missingBucket": "S3 バケットを入力してください",
"noRemoteData": "S3 バケットに同期データが見つかりません",
"incompatibleVersion": "リモートデータに互換性がありません(バージョン {{version}})。このクライアントは protocol v2 / db-v6 をサポートしています。",
"unsaved": "未保存",
"saved": "保存済み",
"unsavedChanges": "先に設定を保存してください",
"saveBeforeSync": "アップロード/ダウンロードを有効にするには、先に設定を保存してください。",
"fetchingRemote": "リモート情報を取得中...",
"fetchRemoteFailed": "リモート情報の取得に失敗しました。設定とネットワークを確認してください。",
"notConfigured": "S3同期が設定されていません",
"disabled": "S3同期が無効です",
"confirmDownload": {
"title": "S3 から復元",
"deviceName": "アップロード元",
"createdAt": "アップロード日時",
"artifacts": "内容",
"warning": "ローカルのすべてのデータとスキル設定が上書きされます",
"confirm": "復元を実行"
},
"confirmUpload": {
"title": "S3 にアップロード",
"content": "以下の内容を S3 バケットに同期します:",
"dbItem": "データベース(すべてのプロバイダー設定とデータ)",
"skillsItem": "スキル(すべてのカスタムスキル)",
"targetPath": "保存先オブジェクトパス",
"existingData": "クラウドの既存データ",
"deviceName": "アップロード元",
"createdAt": "アップロード日時",
"warning": "この S3 パスにある既存の同期データが上書きされます",
"confirm": "アップロードを実行"
}
},
"autoReload": "データを更新しました",
"languageOptionChinese": "简体中文",
"languageOptionTraditionalChinese": "繁體中文",
+101
View File
@@ -546,6 +546,107 @@
"confirm": "確認上傳"
}
},
"syncType": {
"label": "同步方式",
"webdav": "WebDAV",
"s3": "S3 相容儲存"
},
"s3Sync": {
"presets": {
"label": "服務商",
"awsS3": "AWS S3",
"awsS3Hint": "使用 IAM Access Key,需要 s3:PutObject/GetObject/HeadObject 權限。Region 範例:us-east-1, ap-northeast-1",
"minio": "MinIO",
"minioHint": "需要自訂 Endpoint,如 http://minio.local:9000。Region 可設為 us-east-1",
"r2": "Cloudflare R2",
"r2Hint": "Endpoint: https://<account_id>.r2.cloudflarestorage.comRegion 設為 auto",
"oss": "阿里雲 OSS",
"ossHint": "Endpoint: https://oss-<region>.aliyuncs.comRegion 範例:cn-hangzhou, cn-shanghai",
"cos": "騰訊雲 COS",
"cosHint": "Endpoint: https://cos.<region>.myqcloud.comRegion 範例:ap-guangzhou, ap-beijing",
"obs": "華為雲 OBS",
"obsHint": "Endpoint: https://obs.<region>.myhuaweicloud.comRegion 範例:cn-north-4, cn-east-3",
"custom": "S3 相容服務(自訂)",
"customHint": "任意 S3 相容服務,需要自訂 Endpoint"
},
"region": "區域 (Region)",
"regionPlaceholder": "例如 us-east-1",
"bucket": "儲存桶 (Bucket)",
"bucketPlaceholder": "例如 my-cc-switch-bucket",
"accessKeyId": "Access Key ID",
"accessKeyIdPlaceholder": "例如 AKIA...",
"secretAccessKey": "Secret Access Key",
"secretAccessKeyPlaceholder": "留空則保持現有金鑰",
"endpoint": "EndpointAWS 可留空)",
"endpointHint": "多數 S3 相容服務需要填寫",
"endpointPlaceholder": "例如 https://s3-compatible.example.com",
"remoteRoot": "遠端根目錄",
"remoteRootDefault": "預設: cc-switch-sync",
"profile": "同步設定檔名稱",
"profileDefault": "預設: default",
"autoSync": "自動同步",
"autoSyncHint": "開啟後每次資料庫變更都會自動上傳到 S3。",
"enabled": "啟用 S3 同步",
"enabledHint": "開啟後可使用 S3 上傳/下載和自動同步。",
"mutualExclusionTitle": "切換同步方式",
"mutualExclusionMessage": "啟用 S3 同步將停用目前的 WebDAV 同步,是否繼續?",
"mutualExclusionMessageReverse": "啟用 WebDAV 同步將停用目前的 S3 同步,是否繼續?",
"mutualExclusionFailed": "切換同步方式失敗:{{error}}",
"testConnection": "測試連線",
"connectionOk": "S3 連線成功",
"test": "測試連線",
"testing": "測試中...",
"testSuccess": "S3 連線成功",
"testFailed": "S3 連線失敗:{{error}}",
"save": "儲存設定",
"saving": "儲存中...",
"saveFailed": "儲存設定失敗:{{error}}",
"saveAndTestSuccess": "設定已儲存,S3 連線正常",
"saveAndTestFailed": "設定已儲存,但 S3 連線測試失敗:{{error}}",
"upload": "上傳至雲端",
"uploading": "上傳中...",
"uploadSuccess": "已上傳到 S3",
"uploadFailed": "S3 上傳失敗:{{error}}",
"autoSyncFailedToast": "S3 自動同步失敗:{{error}}",
"download": "從雲端下載",
"downloading": "下載中...",
"downloadSuccess": "已從 S3 下載並還原",
"downloadFailed": "S3 下載失敗:{{error}}",
"lastSync": "上次同步:{{time}}",
"autoSyncLastErrorTitle": "上次 S3 自動同步失敗",
"autoSyncLastErrorHint": "請檢查網路或 S3 設定,系統會在後續變更時繼續自動重試。",
"missingBucket": "請填寫 S3 儲存桶",
"noRemoteData": "S3 儲存桶中沒有找到同步資料",
"incompatibleVersion": "遠端資料版本不相容(版本 {{version}}),目前支援協議 v2 / db-v6",
"unsaved": "未儲存",
"saved": "已儲存",
"unsavedChanges": "請先儲存設定",
"saveBeforeSync": "請先儲存設定,再使用上傳/下載。",
"fetchingRemote": "取得遠端資訊...",
"fetchRemoteFailed": "取得遠端資訊失敗,請檢查設定和網路後重試。",
"notConfigured": "未設定 S3 同步",
"disabled": "S3 同步未啟用",
"confirmDownload": {
"title": "從 S3 還原",
"deviceName": "上傳裝置",
"createdAt": "上傳時間",
"artifacts": "包含內容",
"warning": "還原將覆寫本地所有資料和技能設定",
"confirm": "確認還原"
},
"confirmUpload": {
"title": "上傳到 S3",
"content": "將同步以下內容到 S3 儲存桶:",
"dbItem": "資料庫(所有 Provider 設定和資料)",
"skillsItem": "技能包(所有自訂技能)",
"targetPath": "目標物件路徑",
"existingData": "雲端已有資料",
"deviceName": "上傳裝置",
"createdAt": "上傳時間",
"warning": "將覆寫該 S3 路徑下已有的同步資料",
"confirm": "確認上傳"
}
},
"autoReload": "資料已重新整理",
"languageOptionChinese": "简体中文",
"languageOptionTraditionalChinese": "繁體中文",
+101
View File
@@ -546,6 +546,107 @@
"confirm": "确认上传"
}
},
"syncType": {
"label": "同步方式",
"webdav": "WebDAV",
"s3": "S3 兼容存储"
},
"s3Sync": {
"presets": {
"label": "服务商",
"awsS3": "AWS S3",
"awsS3Hint": "使用 IAM Access Key,需要 s3:PutObject/GetObject/HeadObject 权限。Region 示例:us-east-1, ap-northeast-1",
"minio": "MinIO",
"minioHint": "需要自定义 Endpoint,如 http://minio.local:9000。Region 可设为 us-east-1",
"r2": "Cloudflare R2",
"r2Hint": "Endpoint: https://<account_id>.r2.cloudflarestorage.comRegion 设为 auto",
"oss": "阿里云 OSS",
"ossHint": "Endpoint: https://oss-<region>.aliyuncs.comRegion 示例:cn-hangzhou, cn-shanghai",
"cos": "腾讯云 COS",
"cosHint": "Endpoint: https://cos.<region>.myqcloud.comRegion 示例:ap-guangzhou, ap-beijing",
"obs": "华为云 OBS",
"obsHint": "Endpoint: https://obs.<region>.myhuaweicloud.comRegion 示例:cn-north-4, cn-east-3",
"custom": "S3 兼容服务(自定义)",
"customHint": "任意 S3 兼容服务,需要自定义 Endpoint"
},
"region": "区域 (Region)",
"regionPlaceholder": "例如 us-east-1",
"bucket": "存储桶 (Bucket)",
"bucketPlaceholder": "例如 my-cc-switch-bucket",
"accessKeyId": "Access Key ID",
"accessKeyIdPlaceholder": "例如 AKIA...",
"secretAccessKey": "Secret Access Key",
"secretAccessKeyPlaceholder": "留空则保持现有密钥",
"endpoint": "EndpointAWS 可留空)",
"endpointHint": "多数 S3 兼容服务需要填写",
"endpointPlaceholder": "例如 https://s3-compatible.example.com",
"remoteRoot": "远程根目录",
"remoteRootDefault": "默认: cc-switch-sync",
"profile": "同步配置名",
"profileDefault": "默认: default",
"autoSync": "自动同步",
"autoSyncHint": "开启后每次数据库变更都会自动上传到 S3。",
"enabled": "启用 S3 同步",
"enabledHint": "开启后可使用 S3 上传/下载和自动同步。",
"mutualExclusionTitle": "切换同步方式",
"mutualExclusionMessage": "启用 S3 同步将禁用当前的 WebDAV 同步,是否继续?",
"mutualExclusionMessageReverse": "启用 WebDAV 同步将禁用当前的 S3 同步,是否继续?",
"mutualExclusionFailed": "切换同步方式失败:{{error}}",
"testConnection": "测试连接",
"connectionOk": "S3 连接成功",
"test": "测试连接",
"testing": "测试中...",
"testSuccess": "S3 连接成功",
"testFailed": "S3 连接失败:{{error}}",
"save": "保存配置",
"saving": "保存中...",
"saveFailed": "保存配置失败:{{error}}",
"saveAndTestSuccess": "配置已保存,S3 连接正常",
"saveAndTestFailed": "配置已保存,但 S3 连接测试失败:{{error}}",
"upload": "上传到云端",
"uploading": "上传中...",
"uploadSuccess": "已上传到 S3",
"uploadFailed": "S3 上传失败:{{error}}",
"autoSyncFailedToast": "S3 自动同步失败:{{error}}",
"download": "从云端下载",
"downloading": "下载中...",
"downloadSuccess": "已从 S3 下载并恢复",
"downloadFailed": "S3 下载失败:{{error}}",
"lastSync": "上次同步:{{time}}",
"autoSyncLastErrorTitle": "上次 S3 自动同步失败",
"autoSyncLastErrorHint": "请检查网络或 S3 配置,系统会在后续变更时继续自动重试。",
"missingBucket": "请填写 S3 存储桶",
"noRemoteData": "S3 存储桶中没有找到同步数据",
"incompatibleVersion": "远端数据版本不兼容(版本 {{version}}),当前支持协议 v2 / db-v6",
"unsaved": "未保存",
"saved": "已保存",
"unsavedChanges": "请先保存配置",
"saveBeforeSync": "请先保存配置,再使用上传/下载。",
"fetchingRemote": "获取远端信息...",
"fetchRemoteFailed": "获取远端信息失败,请检查配置和网络后重试。",
"notConfigured": "未配置 S3 同步",
"disabled": "S3 同步未启用",
"confirmDownload": {
"title": "从 S3 恢复",
"deviceName": "上传设备",
"createdAt": "上传时间",
"artifacts": "包含内容",
"warning": "恢复将覆盖本地所有数据和技能配置",
"confirm": "确认恢复"
},
"confirmUpload": {
"title": "上传到 S3",
"content": "将同步以下内容到 S3 存储桶:",
"dbItem": "数据库(所有 Provider 配置和数据)",
"skillsItem": "技能包(所有自定义技能)",
"targetPath": "目标对象路径",
"existingData": "云端已有数据",
"deviceName": "上传设备",
"createdAt": "上传时间",
"warning": "将覆盖该 S3 路径下已有的同步数据",
"confirm": "确认上传"
}
},
"autoReload": "数据已刷新",
"languageOptionChinese": "简体中文",
"languageOptionTraditionalChinese": "繁體中文",
+40 -1
View File
@@ -1,5 +1,10 @@
import { invoke } from "@tauri-apps/api/core";
import type { Settings, WebDavSyncSettings, RemoteSnapshotInfo } from "@/types";
import type {
Settings,
WebDavSyncSettings,
S3SyncSettings,
RemoteSnapshotInfo,
} from "@/types";
import type { AppId } from "./types";
export interface ConfigTransferResult {
@@ -142,6 +147,40 @@ export const settingsApi = {
return await invoke("webdav_sync_fetch_remote_info");
},
// ===== S3 Sync API =====
async s3TestConnection(
settings: S3SyncSettings,
preserveEmptyPassword = true,
): Promise<WebDavTestResult> {
return await invoke("s3_test_connection", {
settings,
preserveEmptyPassword,
});
},
async s3SyncUpload(): Promise<WebDavSyncResult> {
return await invoke("s3_sync_upload");
},
async s3SyncDownload(): Promise<WebDavSyncResult> {
return await invoke("s3_sync_download");
},
async s3SyncSaveSettings(
settings: S3SyncSettings,
passwordTouched: boolean,
): Promise<{ success: boolean }> {
return await invoke("s3_sync_save_settings", {
settings,
passwordTouched,
});
},
async s3SyncFetchRemoteInfo(): Promise<RemoteSnapshotInfo | { empty: true }> {
return await invoke("s3_sync_fetch_remote_info");
},
async syncCurrentProvidersLive(): Promise<void> {
const result = (await invoke("sync_current_providers_live")) as {
success?: boolean;
+17
View File
@@ -289,6 +289,20 @@ export interface WebDavSyncSettings {
status?: WebDavSyncStatus;
}
// S3 同步配置
export interface S3SyncSettings {
enabled?: boolean;
autoSync?: boolean;
region?: string;
bucket?: string;
accessKeyId?: string;
secretAccessKey?: string;
endpoint?: string;
remoteRoot?: string;
profile?: string;
status?: WebDavSyncStatus;
}
export type RemoteSnapshotLayout = "current" | "legacy";
// 远端快照信息(下载前预览)
@@ -382,6 +396,9 @@ export interface Settings {
// ===== WebDAV v2 同步设置 =====
webdavSync?: WebDavSyncSettings;
// ===== S3 同步设置 =====
s3Sync?: S3SyncSettings;
// ===== 备份策略设置 =====
// Auto-backup interval in hours (0=disabled, default 24)
backupIntervalHours?: number;
+16
View File
@@ -244,6 +244,22 @@ describe("App integration with MSW", () => {
await waitFor(() => {
expect(toastErrorMock).toHaveBeenCalled();
});
toastErrorMock.mockReset();
expect(() => {
emitTauriEvent("s3-sync-status-updated", null);
}).not.toThrow();
expect(toastErrorMock).not.toHaveBeenCalled();
emitTauriEvent("s3-sync-status-updated", {
source: "auto",
status: "error",
error: "s3 timeout",
});
await waitFor(() => {
expect(toastErrorMock).toHaveBeenCalled();
});
});
it("duplicates openclaw providers with a generated key that avoids live-only ids", async () => {