Skip to content

fix: decrypt copy source per-frame across internal parts (InvalidTag on large multipart COPY)#104

Merged
ServerSideHannes merged 1 commit into
mainfrom
fix/copy-framed-internal-parts
Jul 1, 2026
Merged

fix: decrypt copy source per-frame across internal parts (InvalidTag on large multipart COPY)#104
ServerSideHannes merged 1 commit into
mainfrom
fix/copy-framed-internal-parts

Conversation

@ServerSideHannes

Copy link
Copy Markdown
Owner

Problem

Server-side COPY of large multipart-encrypted objects fails with cryptography.exceptions.InvalidTag. Observed in production on oceanio-dc2-scylladb-backups-v3 (1000MB SSTable backups): every copy of a multipart source whose internal parts exceed 8MB fails, and the pods OOMKill under the load.

File "/app/s3proxy/handlers/base.py", line 269, in _iter_multipart_plaintext
    part_plaintext = crypto.decrypt(ciphertext, dek)
cryptography.exceptions.InvalidTag

Root cause

_iter_multipart_plaintext (the copy source reader) decrypted each whole client part as a single AES-GCM seal:

ct_end = ct_offset + part.ciphertext_size - 1   # whole client part
part_plaintext = crypto.decrypt(ciphertext, dek)

But a client part is not one seal. It expands into multiple internal parts (separate S3 parts), and each internal part is a sequence of independent frames (nonce||ct||tag, FRAME_PLAINTEXT_SIZE = 8MB each). The failing request fetched a ~50MB client part (ciphertext_size: 52429136, 6 internal parts of 8.33MB, each >8MB → 2 frames) and authenticated it as one GCM message → InvalidTag.

The GET path was already fixed for this layout (get.py::_stream_internal_parts); the copy source-read path through _iter_multipart_plaintext was never updated. Existing tests missed it because they only build single-frame source parts (exactly 8MB, internal_parts=[]).

Fix

Walk internal_parts → frames (via crypto.ciphertext_frame_byte_sizes) and decrypt one frame at a time, with range trimming preserved. Legacy single-seal parts are the 1-frame case and read through unchanged. Bonus: peak memory for copy source reads drops from a whole ~50MB client part to O(frame) = 8MB, relieving the OOM/MEMORY_BACKPRESSURE symptom on these copies.

Verification

  • New TestIterMultipartPlaintextFramed reproduces the exact production InvalidTag against the old reader (fails pre-fix, passes post-fix), plus a range round-trip.
  • Full unit suite: 560 passed. Lint clean.

_iter_multipart_plaintext decrypted each whole client part as a single
AES-GCM seal. A client part expands into multiple internal parts (separate
S3 parts), each itself a sequence of independent frames, so any source
whose parts hold more than one frame (e.g. internal parts >8MB) failed to
COPY with cryptography.exceptions.InvalidTag.

Walk internal_parts -> frames and decrypt one frame at a time, matching the
GET reader. Also bounds copy source-read peak memory to O(frame) instead of
a whole client part.
@ServerSideHannes ServerSideHannes merged commit bc0bd36 into main Jul 1, 2026
4 checks passed
@ServerSideHannes ServerSideHannes deleted the fix/copy-framed-internal-parts branch July 1, 2026 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant