fix: handle UTF-8 multi-byte characters split across stream chunk boundaries (#1923)

* fix: handle UTF-8 multi-byte characters split across stream chunk boundaries

Replace String::from_utf8_lossy with append_utf8_safe in all four SSE
streaming paths. When a multi-byte UTF-8 character (e.g. Chinese, emoji)
is split across TCP chunk boundaries, from_utf8_lossy silently replaces
the incomplete halves with U+FFFD (�). This caused intermittent garbled
output in Claude Code when using the Copilot reverse proxy, because the
format conversion streams reconstruct SSE events from the corrupted buffer.

The new append_utf8_safe function preserves incomplete trailing bytes in
a remainder buffer and merges them with the next chunk before decoding,
ensuring characters are never split during UTF-8 conversion.

Fixes: intermittent U+FFFD replacement characters in Claude Code output
via Copilot proxy (not reproducible with direct Copilot connections like
opencode because they pass through raw bytes without format conversion).

* style: fix cargo fmt formatting in UTF-8 boundary tests

---------

Co-authored-by: Cod1ng <codingts@gmail.com>
Co-authored-by: encodets <encodets@gmail.com>
This commit is contained in:
Cod1ng
2026-04-08 10:02:35 +08:00
committed by GitHub
parent 34f16886a2
commit dc4524e960
5 changed files with 364 additions and 9 deletions
+2 -2
View File
@@ -568,6 +568,7 @@ pub fn create_logged_passthrough_stream(
) -> impl Stream<Item = Result<Bytes, std::io::Error>> + Send {
async_stream::stream! {
let mut buffer = String::new();
let mut utf8_remainder: Vec<u8> = Vec::new();
let mut collector = usage_collector;
let mut is_first_chunk = true;
@@ -619,8 +620,7 @@ pub fn create_logged_passthrough_stream(
);
}
is_first_chunk = false;
let text = String::from_utf8_lossy(&bytes);
buffer.push_str(&text);
crate::proxy::sse::append_utf8_safe(&mut buffer, &mut utf8_remainder, &bytes);
// 尝试解析并记录完整的 SSE 事件
while let Some(pos) = buffer.find("\n\n") {