feat: 新增 S3 兼容云存储同步 (#1351)

* Add S3 Cloud Sync design document

Design for adding AWS S3 as a new Cloud Sync backend alongside WebDAV.
Hybrid approach: extract shared sync protocol, add independent S3 transport.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add S3 cloud sync implementation design (reqwest + Sig V4)

Updated design based on 2026-03-06 draft: switches from rust-s3 crate
to hand-rolled AWS Sig V4 on existing reqwest for broader S3-compatible
service support (AWS, MinIO, R2, Alibaba OSS, Tencent COS, Huawei OBS).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add S3 cloud sync implementation plan (11 tasks, TDD)

Detailed step-by-step plan covering: sync_protocol extraction, S3 Sig V4
transport, settings, sync/auto-sync modules, Tauri commands, frontend
presets/dynamic form, and i18n.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* deps: add hmac crate for S3 Sig V4 signing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: extract sync_protocol.rs from webdav_sync.rs for shared use

Move transport-agnostic sync protocol logic (constants, types, snapshot
building, manifest validation, artifact verification, snapshot application,
utilities) into a new shared sync_protocol module so both WebDAV and the
upcoming S3 transport can reuse it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: use transport-neutral error keys in sync_protocol

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3 transport layer with AWS Sig V4 signing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3SyncSettings to AppSettings

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3 sync module with upload/download/fetch

Implements the S3 sync protocol layer (s3_sync.rs) that combines the
shared sync_protocol with the S3 transport. Mirrors the WebDAV sync
module structure with independent sync mutex, connection check,
upload, download, fetch_remote_info, and sync status persistence.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3 auto sync worker with debounce

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3 sync Tauri commands and auto sync worker startup

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3 sync TypeScript types and API layer

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3 sync i18n translations (en/zh/ja)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add S3 sync presets and dynamic form to sync settings

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: preserve HTTP scheme for S3 custom endpoints (MinIO support)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: add live S3 integration tests (env-var driven, --ignored)

Run with: S3_TEST_AK=... S3_TEST_SK=... S3_TEST_BUCKET=... cargo test --lib services::s3::integration_tests -- --ignored

Verifies test_connection, put_object, get_object, head_object, and 404
handling against a real S3 bucket using the project's own Sig V4 signing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: remove internal design docs before PR

* fix: wire S3 auto-sync to DB hook & sync UI state on async load

- P1: Add s3_auto_sync::notify_db_changed call in SQLite update_hook
  so S3 auto-sync worker receives DB change signals (was only wired
  for WebDAV, leaving S3 worker idle)

- P2: Add useEffect to update syncType selector when s3Config loads
  asynchronously, preventing stale "webdav" default for S3 users

* fix: satisfy clippy for s3 sync

* fix: address s3 sync review feedback

---------

Co-authored-by: Keith (via OpenClaw) <keithyt06@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Jason <farion1231@gmail.com>
This commit is contained in:
Keith Yu
2026-06-04 22:18:51 +08:00
committed by GitHub
parent ea95f39adf
commit 2a24da517f
26 changed files with 4393 additions and 795 deletions
+648
View File
@@ -0,0 +1,648 @@
//! Transport-agnostic sync protocol layer.
//!
//! Shared by WebDAV, S3, and future transports. Artifact set: `db.sql` + `skills.zip`.
use std::collections::BTreeMap;
use std::fs;
use std::process::Command;
use chrono::Utc;
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use tempfile::tempdir;
use crate::error::AppError;
// Re-export archive functions for use by transport layers.
pub(crate) use super::webdav_sync::archive::{
backup_current_skills, restore_skills_from_backup, restore_skills_zip, zip_skills_ssot,
};
// ─── Protocol constants ──────────────────────────────────────
/// Wire-format identifier stored in remote manifests.
/// Retains historic "webdav" naming for backward compatibility with existing remotes.
pub(crate) const PROTOCOL_FORMAT: &str = "cc-switch-webdav-sync";
pub(crate) const PROTOCOL_VERSION: u32 = 2;
pub(crate) const DB_COMPAT_VERSION: u32 = 6;
pub(crate) const LEGACY_DB_COMPAT_VERSION: u32 = 5;
pub(crate) const REMOTE_DB_SQL: &str = "db.sql";
pub(crate) const REMOTE_SKILLS_ZIP: &str = "skills.zip";
pub(crate) const REMOTE_MANIFEST: &str = "manifest.json";
pub(crate) const MAX_DEVICE_NAME_LEN: usize = 64;
pub(crate) const MAX_MANIFEST_BYTES: usize = 1024 * 1024;
pub(crate) const MAX_SYNC_ARTIFACT_BYTES: u64 = 512 * 1024 * 1024;
// ─── Error helpers ───────────────────────────────────────────
pub(crate) fn localized(
key: &'static str,
zh: impl Into<String>,
en: impl Into<String>,
) -> AppError {
AppError::localized(key, zh, en)
}
pub(crate) fn io_context_localized(
_key: &'static str,
zh: impl Into<String>,
en: impl Into<String>,
source: std::io::Error,
) -> AppError {
let zh_msg = zh.into();
let en_msg = en.into();
AppError::IoContext {
context: format!("{zh_msg} ({en_msg})"),
source,
}
}
// ─── Types ───────────────────────────────────────────────────
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub(crate) struct SyncManifest {
pub format: String,
pub version: u32,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub db_compat_version: Option<u32>,
pub device_name: String,
pub created_at: String,
pub artifacts: BTreeMap<String, ArtifactMeta>,
pub snapshot_id: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub(crate) struct ArtifactMeta {
pub sha256: String,
pub size: u64,
}
pub(crate) struct LocalSnapshot {
pub db_sql: Vec<u8>,
pub skills_zip: Vec<u8>,
pub manifest_bytes: Vec<u8>,
pub manifest_hash: String,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum RemoteLayout {
Current,
Legacy,
}
impl RemoteLayout {
pub(crate) fn as_str(self) -> &'static str {
match self {
Self::Current => "current",
Self::Legacy => "legacy",
}
}
}
// ─── Snapshot building ───────────────────────────────────────
pub(crate) fn build_local_snapshot(
db: &crate::database::Database,
) -> Result<LocalSnapshot, AppError> {
// Export database to SQL string
let sql_string = db.export_sql_string_for_sync()?;
let db_sql = sql_string.into_bytes();
// Pack skills into deterministic ZIP
let tmp = tempdir().map_err(|e| {
io_context_localized(
"sync.snapshot_tmpdir_failed",
"创建快照临时目录失败",
"Failed to create temporary directory for snapshot",
e,
)
})?;
let skills_zip_path = tmp.path().join(REMOTE_SKILLS_ZIP);
zip_skills_ssot(&skills_zip_path)?;
let skills_zip = fs::read(&skills_zip_path).map_err(|e| AppError::io(&skills_zip_path, e))?;
// Build artifact map and compute hashes
let mut artifacts = BTreeMap::new();
artifacts.insert(
REMOTE_DB_SQL.to_string(),
ArtifactMeta {
sha256: sha256_hex(&db_sql),
size: db_sql.len() as u64,
},
);
artifacts.insert(
REMOTE_SKILLS_ZIP.to_string(),
ArtifactMeta {
sha256: sha256_hex(&skills_zip),
size: skills_zip.len() as u64,
},
);
let snapshot_id = compute_snapshot_id(&artifacts);
let manifest = SyncManifest {
format: PROTOCOL_FORMAT.to_string(),
version: PROTOCOL_VERSION,
db_compat_version: Some(DB_COMPAT_VERSION),
device_name: detect_system_device_name().unwrap_or_else(|| "Unknown Device".to_string()),
created_at: Utc::now().to_rfc3339(),
artifacts,
snapshot_id,
};
let manifest_bytes =
serde_json::to_vec_pretty(&manifest).map_err(|e| AppError::JsonSerialize { source: e })?;
let manifest_hash = sha256_hex(&manifest_bytes);
Ok(LocalSnapshot {
db_sql,
skills_zip,
manifest_bytes,
manifest_hash,
})
}
// ─── Manifest handling ───────────────────────────────────────
/// Compute a deterministic snapshot identity from artifact hashes.
///
/// BTreeMap iteration order is sorted by key, ensuring stability.
pub(crate) fn compute_snapshot_id(artifacts: &BTreeMap<String, ArtifactMeta>) -> String {
let parts: Vec<String> = artifacts
.iter()
.map(|(name, meta)| format!("{}:{}", name, meta.sha256))
.collect();
sha256_hex(parts.join("|").as_bytes())
}
pub(crate) fn effective_db_compat_version(
manifest: &SyncManifest,
layout: RemoteLayout,
) -> Option<u32> {
manifest
.db_compat_version
.or_else(|| (layout == RemoteLayout::Legacy).then_some(LEGACY_DB_COMPAT_VERSION))
}
pub(crate) fn validate_manifest_compat(
manifest: &SyncManifest,
layout: RemoteLayout,
) -> Result<(), AppError> {
if manifest.format != PROTOCOL_FORMAT {
return Err(localized(
"sync.manifest_format_incompatible",
format!("远端 manifest 格式不兼容: {}", manifest.format),
format!(
"Remote manifest format is incompatible: {}",
manifest.format
),
));
}
if manifest.version != PROTOCOL_VERSION {
return Err(localized(
"sync.manifest_version_incompatible",
format!(
"远端 manifest 协议版本不兼容: v{} (本地 v{PROTOCOL_VERSION})",
manifest.version
),
format!(
"Remote manifest protocol version is incompatible: v{} (local v{PROTOCOL_VERSION})",
manifest.version
),
));
}
let Some(db_compat_version) = effective_db_compat_version(manifest, layout) else {
return Err(localized(
"sync.manifest_db_version_missing",
"远端 manifest 缺少数据库兼容版本",
"Remote manifest is missing the database compatibility version.",
));
};
match layout {
RemoteLayout::Current if db_compat_version != DB_COMPAT_VERSION => {
return Err(localized(
"sync.manifest_db_version_incompatible",
format!(
"远端数据库快照版本不兼容: db-v{db_compat_version} (本地 db-v{DB_COMPAT_VERSION})"
),
format!(
"Remote database snapshot version is incompatible: db-v{db_compat_version} (local db-v{DB_COMPAT_VERSION})"
),
));
}
RemoteLayout::Legacy if db_compat_version > DB_COMPAT_VERSION => {
return Err(localized(
"sync.manifest_db_version_incompatible",
format!(
"远端数据库快照版本不兼容: db-v{db_compat_version} (本地最高支持 db-v{DB_COMPAT_VERSION})"
),
format!(
"Remote database snapshot version is incompatible: db-v{db_compat_version} (local supports up to db-v{DB_COMPAT_VERSION})"
),
));
}
_ => {}
}
Ok(())
}
// ─── Artifact verification ───────────────────────────────────
pub(crate) fn validate_artifact_size_limit(artifact_name: &str, size: u64) -> Result<(), AppError> {
if size > MAX_SYNC_ARTIFACT_BYTES {
let max_mb = MAX_SYNC_ARTIFACT_BYTES / 1024 / 1024;
return Err(localized(
"sync.artifact_too_large",
format!("artifact {artifact_name} 超过下载上限({} MB", max_mb),
format!(
"Artifact {artifact_name} exceeds download limit ({} MB)",
max_mb
),
));
}
Ok(())
}
/// Verify that downloaded artifact bytes match the expected size and SHA-256 hash.
pub(crate) fn verify_artifact(
bytes: &[u8],
artifact_name: &str,
meta: &ArtifactMeta,
) -> Result<(), AppError> {
// Quick size check before expensive hash
if bytes.len() as u64 != meta.size {
return Err(localized(
"sync.artifact_size_mismatch",
format!(
"artifact {artifact_name} 大小不匹配 (expected: {}, got: {})",
meta.size,
bytes.len(),
),
format!(
"Artifact {artifact_name} size mismatch (expected: {}, got: {})",
meta.size,
bytes.len(),
),
));
}
let actual_hash = sha256_hex(bytes);
if actual_hash != meta.sha256 {
return Err(localized(
"sync.artifact_hash_mismatch",
format!(
"artifact {artifact_name} SHA256 校验失败 (expected: {}..., got: {}...)",
meta.sha256.get(..8).unwrap_or(&meta.sha256),
actual_hash.get(..8).unwrap_or(&actual_hash),
),
format!(
"Artifact {artifact_name} SHA256 verification failed (expected: {}..., got: {}...)",
meta.sha256.get(..8).unwrap_or(&meta.sha256),
actual_hash.get(..8).unwrap_or(&actual_hash),
),
));
}
Ok(())
}
// ─── Snapshot application ────────────────────────────────────
pub(crate) fn apply_snapshot(
db: &crate::database::Database,
db_sql: &[u8],
skills_zip: &[u8],
) -> Result<(), AppError> {
let sql_str = std::str::from_utf8(db_sql).map_err(|e| {
localized(
"sync.sql_not_utf8",
format!("SQL 非 UTF-8: {e}"),
format!("SQL is not valid UTF-8: {e}"),
)
})?;
let skills_backup = backup_current_skills()?;
// Replace skills first, then import database; roll back skills on DB failure.
restore_skills_zip(skills_zip)?;
if let Err(db_err) = db.import_sql_string_for_sync(sql_str) {
if let Err(rollback_err) = restore_skills_from_backup(&skills_backup) {
return Err(localized(
"sync.db_import_and_rollback_failed",
format!("导入数据库失败: {db_err}; 同时回滚 Skills 失败: {rollback_err}"),
format!(
"Database import failed: {db_err}; skills rollback also failed: {rollback_err}"
),
));
}
return Err(db_err);
}
Ok(())
}
// ─── Utilities ───────────────────────────────────────────────
pub(crate) fn sha256_hex(bytes: &[u8]) -> String {
let mut hasher = Sha256::new();
hasher.update(bytes);
format!("{:x}", hasher.finalize())
}
pub(crate) fn detect_system_device_name() -> Option<String> {
let env_name = ["CC_SWITCH_DEVICE_NAME", "COMPUTERNAME", "HOSTNAME"]
.iter()
.filter_map(|key| std::env::var(key).ok())
.find_map(|value| normalize_device_name(&value));
if env_name.is_some() {
return env_name;
}
let output = Command::new("hostname").output().ok()?;
if !output.status.success() {
return None;
}
let hostname = String::from_utf8(output.stdout).ok()?;
normalize_device_name(&hostname)
}
pub(crate) fn normalize_device_name(raw: &str) -> Option<String> {
let compact = raw
.chars()
.fold(String::with_capacity(raw.len()), |mut acc, ch| {
if ch.is_whitespace() {
acc.push(' ');
} else if !ch.is_control() {
acc.push(ch);
}
acc
});
let normalized = compact.split_whitespace().collect::<Vec<_>>().join(" ");
let trimmed = normalized.trim();
if trimmed.is_empty() {
return None;
}
let limited = trimmed
.chars()
.take(MAX_DEVICE_NAME_LEN)
.collect::<String>();
if limited.is_empty() {
None
} else {
Some(limited)
}
}
// ─── Sync status persistence ─────────────────────────────────
pub(crate) fn persist_sync_success_best_effort<S, F>(
settings: &mut S,
manifest_hash: String,
etag: Option<String>,
persist_fn: F,
) -> bool
where
F: FnOnce(&mut S, String, Option<String>) -> Result<(), AppError>,
{
match persist_fn(settings, manifest_hash, etag) {
Ok(()) => true,
Err(err) => {
log::warn!("[Sync] Persist sync status failed, keep operation success: {err}");
false
}
}
}
// ─── Tests ───────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
fn artifact(sha256: &str, size: u64) -> ArtifactMeta {
ArtifactMeta {
sha256: sha256.to_string(),
size,
}
}
#[test]
fn snapshot_id_is_stable() {
let mut artifacts = BTreeMap::new();
artifacts.insert("db.sql".to_string(), artifact("abc123", 100));
artifacts.insert("skills.zip".to_string(), artifact("def456", 200));
let id1 = compute_snapshot_id(&artifacts);
let id2 = compute_snapshot_id(&artifacts);
assert_eq!(id1, id2);
}
#[test]
fn snapshot_id_changes_with_artifacts() {
let mut a1 = BTreeMap::new();
a1.insert("db.sql".to_string(), artifact("hash-a", 1));
let mut a2 = BTreeMap::new();
a2.insert("db.sql".to_string(), artifact("hash-b", 1));
assert_ne!(compute_snapshot_id(&a1), compute_snapshot_id(&a2));
}
#[test]
fn sha256_hex_is_correct() {
let hash = sha256_hex(b"hello");
assert_eq!(
hash,
"2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"
);
}
#[test]
fn persist_best_effort_returns_true_on_success() {
let mut dummy = ();
let ok = persist_sync_success_best_effort(
&mut dummy,
"hash".to_string(),
Some("etag".to_string()),
|_settings, _hash, _etag| Ok(()),
);
assert!(ok);
}
#[test]
fn persist_best_effort_returns_false_on_error() {
let mut dummy = ();
let ok = persist_sync_success_best_effort(
&mut dummy,
"hash".to_string(),
None,
|_settings, _hash, _etag| Err(AppError::Config("boom".to_string())),
);
assert!(!ok);
}
fn manifest_with(format: &str, version: u32, db_compat_version: Option<u32>) -> SyncManifest {
let mut artifacts = BTreeMap::new();
artifacts.insert("db.sql".to_string(), artifact("abc", 1));
artifacts.insert("skills.zip".to_string(), artifact("def", 2));
SyncManifest {
format: format.to_string(),
version,
db_compat_version,
device_name: "My MacBook".to_string(),
created_at: "2026-02-12T00:00:00Z".to_string(),
artifacts,
snapshot_id: "snap-1".to_string(),
}
}
#[test]
fn validate_manifest_compat_accepts_supported_manifest() {
let manifest = manifest_with(PROTOCOL_FORMAT, PROTOCOL_VERSION, Some(DB_COMPAT_VERSION));
assert!(validate_manifest_compat(&manifest, RemoteLayout::Current).is_ok());
}
#[test]
fn validate_manifest_compat_rejects_wrong_format() {
let manifest = manifest_with("other-format", PROTOCOL_VERSION, Some(DB_COMPAT_VERSION));
assert!(validate_manifest_compat(&manifest, RemoteLayout::Current).is_err());
}
#[test]
fn validate_manifest_compat_rejects_wrong_version() {
let manifest = manifest_with(
PROTOCOL_FORMAT,
PROTOCOL_VERSION + 1,
Some(DB_COMPAT_VERSION),
);
assert!(validate_manifest_compat(&manifest, RemoteLayout::Current).is_err());
}
#[test]
fn validate_manifest_compat_accepts_legacy_manifest_without_db_compat() {
let manifest = manifest_with(PROTOCOL_FORMAT, PROTOCOL_VERSION, None);
assert!(validate_manifest_compat(&manifest, RemoteLayout::Legacy).is_ok());
}
#[test]
fn validate_manifest_compat_rejects_current_manifest_with_wrong_db_compat() {
let manifest = manifest_with(
PROTOCOL_FORMAT,
PROTOCOL_VERSION,
Some(LEGACY_DB_COMPAT_VERSION),
);
assert!(validate_manifest_compat(&manifest, RemoteLayout::Current).is_err());
}
#[test]
fn validate_manifest_compat_rejects_legacy_manifest_from_newer_db_generation() {
let manifest = manifest_with(
PROTOCOL_FORMAT,
PROTOCOL_VERSION,
Some(DB_COMPAT_VERSION + 1),
);
assert!(validate_manifest_compat(&manifest, RemoteLayout::Legacy).is_err());
}
#[test]
fn effective_db_compat_version_defaults_legacy_layout_to_v5() {
let manifest = manifest_with(PROTOCOL_FORMAT, PROTOCOL_VERSION, None);
assert_eq!(
effective_db_compat_version(&manifest, RemoteLayout::Legacy),
Some(LEGACY_DB_COMPAT_VERSION)
);
assert_eq!(
effective_db_compat_version(&manifest, RemoteLayout::Current),
None
);
}
#[test]
fn normalize_device_name_returns_none_for_blank_input() {
assert_eq!(normalize_device_name(" \n\t "), None);
}
#[test]
fn normalize_device_name_collapses_whitespace_and_drops_control_chars() {
assert_eq!(
normalize_device_name(" Mac\tBook \n Pro\u{0007} "),
Some("Mac Book Pro".to_string())
);
}
#[test]
fn normalize_device_name_truncates_to_max_len() {
let long = "a".repeat(80);
assert_eq!(normalize_device_name(&long).map(|s| s.len()), Some(64));
}
#[test]
fn manifest_serialization_uses_device_name_only() {
let manifest = manifest_with(PROTOCOL_FORMAT, PROTOCOL_VERSION, Some(DB_COMPAT_VERSION));
let value = serde_json::to_value(&manifest).expect("serialize manifest");
assert!(
value.get("deviceName").is_some(),
"manifest should contain deviceName"
);
assert_eq!(
value.get("dbCompatVersion").and_then(|v| v.as_u64()),
Some(DB_COMPAT_VERSION as u64)
);
assert!(
value.get("deviceId").is_none(),
"manifest should not contain deviceId"
);
}
#[test]
fn validate_artifact_size_limit_rejects_oversized_artifacts() {
let err = validate_artifact_size_limit("skills.zip", MAX_SYNC_ARTIFACT_BYTES + 1)
.expect_err("artifact larger than limit should be rejected");
assert!(
err.to_string().contains("too large") || err.to_string().contains("超过"),
"unexpected error: {err}"
);
}
#[test]
fn validate_artifact_size_limit_accepts_limit_boundary() {
assert!(validate_artifact_size_limit("skills.zip", MAX_SYNC_ARTIFACT_BYTES).is_ok());
}
#[test]
fn verify_artifact_rejects_size_mismatch() {
let meta = artifact("abc123", 100);
let bytes = vec![0u8; 50];
let err = verify_artifact(&bytes, "test.bin", &meta)
.expect_err("size mismatch should be rejected");
assert!(
err.to_string().contains("mismatch") || err.to_string().contains("不匹配"),
"unexpected error: {err}"
);
}
#[test]
fn verify_artifact_rejects_hash_mismatch() {
let meta = ArtifactMeta {
sha256: "0000000000000000000000000000000000000000000000000000000000000000".to_string(),
size: 5,
};
let bytes = b"hello";
let err = verify_artifact(bytes, "test.bin", &meta)
.expect_err("hash mismatch should be rejected");
assert!(
err.to_string().contains("verification failed") || err.to_string().contains("校验失败"),
"unexpected error: {err}"
);
}
#[test]
fn verify_artifact_accepts_matching_data() {
let data = b"hello";
let meta = ArtifactMeta {
sha256: sha256_hex(data),
size: data.len() as u64,
};
assert!(verify_artifact(data, "test.bin", &meta).is_ok());
}
}