fix(client): read whole object when ParallelGet concurrency is 1#357
Conversation
A single worker gains nothing from chunking, so skip the ranged discovery request entirely and read the object in one revision-consistent request rather than serialising ranged GETs.
alecthomas
requested review from
jrobotham-square
and removed request for
a team
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e2c5fcc7d3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if concurrency == 1 { | ||
| return fullRead(ctx, c, key, dst) |
There was a problem hiding this comment.
When cache.ParallelGet is called with an *S3 cache and concurrency == 1, this new fast path switches from serial ranged Open calls to an unrestricted full Open. For large non-ranged S3 reads, S3.Open calls parallelGetReader (internal/cache/s3.go:246), which then uses s.config.DownloadConcurrency workers (internal/cache/s3_parallel_get.go:78, default 8 in internal/cache/s3.go:47-49), so callers that set ParallelGet(..., concurrency: 1) to throttle S3 traffic can still launch multiple concurrent S3 range GETs. Keep the serial ranged path for backends whose full Open is internally parallel, or provide an explicit single-stream/full-read path that preserves the requested concurrency limit.
Useful? React with 👍 / 👎.
1 participant
A single worker gains nothing from chunking, so skip the ranged
discovery request entirely and read the object in one
revision-consistent request rather than serialising ranged GETs.