Add ts dev proxy: a local MITM dev proxy serving production hostnames from staging#798
Open
aram356 wants to merge 48 commits into
Open
Add ts dev proxy: a local MITM dev proxy serving production hostnames from staging#798aram356 wants to merge 48 commits into
ts dev proxy: a local MITM dev proxy serving production hostnames from staging#798aram356 wants to merge 48 commits into
Conversation
- Compare r.from case-insensitively in RuleTable::first_match to enforce the lowercase invariant regardless of how Rule.from was built - Reject trailing-colon inputs (empty port string) as RuleError::Port in Authority::parse; add rejects_empty_or_missing_port test - Assert scheme_is_tls in rewrite_default_preserves_from_host_and_sets_sni_to_to and rewrite_host_uses_to_authority_with_port
Add ConfigError::BasicAuthFile variant with a path-carrying display message so file-not-found and permission errors are no longer reported as the misleading "--basic-auth must be USER:PASS" format error. The read failure now maps to BasicAuthFile; only parse failures map to BasicAuth. Includes a unit test that asserts the correct variant is returned for a missing file.
Replace the write-then-chmod pattern in CertAuthority::persist() with a single OpenOptions::create_new(true).mode(0o600) open, eliminating the window where the private key was briefly world/group-readable on disk.
…NNECT - Thread the raw buffered bytes through `RequestHead` so `blind_forward_http` can write the complete request head to the upstream before piping the rest of the socket bidirectionally (spec §8.4). Previously the head was discarded, sending a truncated/empty request. - Update `blind_forward_http` doc comment to reflect that it now replays the original head rather than falsely claiming it always did. - Add `unmatched_connect_off_loopback_is_refused_with_403` integration test. The proxy listener is bound on `127.0.0.1:0` (real socket) but `cfg.listen` is patched to `0.0.0.0:<port>` before being handed to `serve_on`, so `is_loopback` is computed as false while the test can still connect via loopback. Asserts that an unmatched `CONNECT` receives `403` and no tunnel is established.
…s-dev-proxy-spec # Conflicts: # Cargo.toml
Replace the dead `std::thread::park()` restore thread in `launch_safari`
with a file-based persist-and-recover scheme. Before applying the PAC
URL, write `<ca_dir>/safari-proxy-restore` capturing the network service
name and the prior auto-proxy URL (or an empty second line when
auto-proxy was off). Add `restore_system_proxy_if_pending(ca_dir)`,
which reads and deletes that file then runs the appropriate `networksetup`
command to put things back.
Wire it into `run()` in two places: at startup (crash recovery from a
previous hard-killed run) and in a `tokio::select!` on `ctrl_c()` (clean
exit). Also move the function-local `use std::time::{SystemTime,
UNIX_EPOCH}` out of `make_temp_dir` to the top-level imports per project
convention.
…-spec' into worktree-review-ts-dev-proxy-spec
…h, and service detection
…ell-quote restore URL
… trust revocation
…FROM validation - Abort `ca regenerate` when the old CA's keychain revocation can't be confirmed, so on-disk key material never outlives its OS trust. - Declare the CLI macOS-only via `compile_error!` on non-macOS targets (keychain, Safari, and networksetup are all macOS-specific), and gate the macOS-only helpers (`manual_restore_command`, `restore_auto_proxy`) while ungating `shell_quote` so the shared launch path compiles cleanly. - Shell-quote the keychain, cert, and profile paths in the `ca install` and Firefox `certutil` fallback instructions. - Validate rule FROM as a bare hostname before embedding it in the generated PAC, browser URL, and upstream Host header. - Ignore the workspace-excluded crate's `target/` directory. Spec, plan, and guide updated to match.
The macOS-only `compile_error!` lived inside the proxy module while all native deps stayed unconditional, so a build for the repo-default wasm32-wasip1 target failed first in tokio/ring/aws-lc-sys instead of with the intended "macOS only" error — an easy developer footgun (a plain `cargo check` in the crate inherits the wasm default from .cargo/config.toml). - Move every dependency (and dev-dependency) under `[target.'cfg(target_os = "macos")'.dependencies]` so unsupported targets build none of the native TLS/networking stack. - Lift the platform gate to the crate root: `compile_error!` in lib.rs and `#[cfg(target_os = "macos")]` on the command modules, the CLI types, and the binary entry point. The proxy module's own `compile_error!` is removed. - Gate the e2e test crate to macOS (its deps are now macOS-scoped). Now `cargo check --target wasm32-wasip1` emits exactly one error — the macOS-only message — with no dependency build attempts. Native build/test unchanged (31 unit + 6 e2e pass). Spec and plan updated to match.
Let `--to` target a bare IP and `--rewrite-host` carry the hostname that
endpoint expects. The flag now takes an optional value:
- omitted -> Host = FROM (default; unchanged)
- --rewrite-host -> Host = TO host (the prior bare-flag behavior)
- --rewrite-host <HOST> -> Host = <HOST> and TLS SNI = <HOST>
Connection still dials `--to`, so pointing at an IP works: the proxy presents
the explicit hostname for both SNI and Host while the socket goes to the IP.
Replaces `Rule.preserve_host: bool` with a `HostMode { PreserveFrom, UseTo,
Explicit }` enum threaded through `rewrite_for`; the explicit host is validated
as a hostname (new `ConfigError::InvalidRewriteHost`). Spec, plan, and guide
updated; adds tests for all three forms plus invalid-value rejection.
The `compile_error!` fired on a plain `cargo build` (no `--target`), which the repo's `.cargo/config.toml` resolves to `wasm32-wasip1` — so it tripped even on macOS and blocked the natural build command. Remove the gate. The deps and command modules stay scoped to `target_os = "macos"`, so unsupported targets build an empty shell instead of dragging tokio/ring/aws-lc-sys through an unsupported build. Build natively with an explicit `--target` (proper no-`--target` defaulting is separate, edgezero-aligned work). Also fix a stray `x` that slipped into a config test assertion, which broke the lib-test build. Spec and plan updated to match.
Replace the overloaded `--rewrite-host <HOST>` (which set both Host and SNI to reach an IP upstream) with two orthogonal knobs: - `--resolve HOST:IP` (repeatable, curl-style) pins where a hostname's upstream connection dials, leaving the SNI/Host derivation untouched. This is the self-contained way to reach a server by IP while keeping `--to` a hostname so the TLS SNI and certificate stay valid — no /etc/hosts edit. - `--rewrite-host` is a plain bool again: send `Host: TO` instead of `Host: FROM`. The SNI is always the TO host. Drops the `HostMode` enum in favor of `Rule.rewrite_host: bool` named and passed straight from the flag (no `preserve_host` inversion). Adds an e2e proving a non-resolvable TO host still reaches the upstream via --resolve. Spec and guide updated; the guide now leads with the --resolve flow.
The DNS pin only covered the matched-rule MITM upstream, so a CONNECT to a host that isn't a configured --from (a directly-hit host, a sub-resource) still went through the blind tunnel via real DNS and ignored the pin. Route all three connection paths (MITM, blind tunnel, plain-HTTP forward) through a shared `connect_upstream` helper that honors the pin case-insensitively. Also log each pin at startup (`--resolve pin: HOST -> IP`) so it's visible at the default info level — the per-connection summary shows the hostname (SNI/cert use it) even though the socket dials the pinned IP.
…n host Trusted Server derives request_host from X-Forwarded-Host (then Host) and anchors all first-party URL rewriting to it. The proxy now always sends X-Forwarded-Host: FROM — standard forward-proxy behavior — so TS emits production-host URLs (tsjs, GPT, DataDome, …) even when --rewrite-host sends Host: TO for an upstream that routes/validates on its own hostname. This decouples routing (Host) from the displayed first-party host: use --rewrite-host freely for host-validating upstreams without skewing the rewritten URLs onto TO. Adds an e2e asserting Host=TO + X-Forwarded-Host=FROM under --rewrite-host. Spec and guide updated (the earlier "avoid --rewrite-host for TS" guidance no longer applies).
aram356
changed the title
ts dev proxy: a local MITM dev proxy serving production hostnames from staging
- rewrite: match the port-stripped host with eq_ignore_ascii_case instead of allocating a lowercased copy per rule (FROM is already stored lowercase). - config: warn when a --resolve HOST matches no rule's TO host (a likely typo whose pin would otherwise silently never apply); still succeeds. Add a test. - config: document why is_valid_host rejects underscores. - browser: correct the Linux chrome_command comment (it doesn't fall back to chromium; the arm is unreached on the macOS-only build anyway).
Addresses two review findings on the dev proxy (the trusted-host design for --rewrite-host through the real adapter is tracked separately): - server: strip any inbound `Forwarded` before stamping `X-Forwarded-Host`. Trusted Server resolves the request host from `Forwarded` → `X-Forwarded-Host` → `Host`, so a client-supplied `Forwarded` would otherwise outrank the FROM host the proxy injects. Add a unit test. - docs: reconcile stale `X-Orig-Host`-only guidance in the spec/plan — the functional first-party-host header is `X-Forwarded-Host` (TS reads it for request_host); `X-Orig-Host` is an informational duplicate. Note the inbound-`Forwarded` strip in the spec/plan and the `RewriteOutcome` field doc.
This was referenced Jun 26, 2026
ChristianPavilonis
left a comment
ChristianPavilonis
left a comment
Collaborator
There was a problem hiding this comment.
Automated review: I reviewed the new trusted-server-cli dev proxy implementation, CA/browser orchestration, CI wiring, and the related docs. CI is currently green and I did not find a blocking issue that requires requesting changes, but I found several correctness/security edge cases that should be addressed before relying on this tool broadly.
…v-proxy-spec # Conflicts: # .github/workflows/test.yml # .gitignore
…fixes - ca regenerate: propagate old cert/key deletion failures (tolerate already-absent) instead of `.ok()`, so a failed delete can't leave the stale key silently in use after printing "regenerated" (P1). - config: reject --basic-auth/--basic-auth-file on a non-loopback --listen; a matched CONNECT is MITM'd even off loopback, so injected credentials would be exposed to any reachable client (P1). - server: stamp an authoritative `X-Forwarded-Proto: https` (the browser leg is always TLS) and drop spoofable `Fastly-SSL`, so `--upstream-plaintext` (or a spoofed header) can't downgrade the first-party scheme (P2). - browser (Firefox): initialise an empty `sql:` NSS DB before `certutil -A`, and import into it, instead of importing against an uninitialised profile DB that fails with SEC_ERROR_BAD_DATABASE (P2). - browser: derive the browser-connect address from `cfg.listen` (normalising wildcard binds to loopback, bracketing IPv6) and use it consistently in Chrome/Firefox/Safari/PAC and the manual hints, instead of hard-coding 127.0.0.1 — so a non-default --listen is honored (P2). Adds unit tests for each.
Collaborator
Author
|
Thanks for the review — all five findings addressed in
Unit tests added for each; CLI crate fmt/clippy/test green. |
ChristianPavilonis
left a comment
ChristianPavilonis
left a comment
Collaborator
There was a problem hiding this comment.
Automated review: I reviewed the current head of PR #798 (new trusted-server-cli dev proxy, CA/browser orchestration, tests/CI, and docs) against main. CI is green and I did not find a blocking correctness/security issue requiring REQUEST_CHANGES, but I found several medium-risk hardening/correctness issues worth addressing before broad use of this MITM dev tool.
…form - server: strip RFC 7230 hop-by-hop request headers and every header named in an inbound `Connection` token before stamping authoritative headers, so a client can't mark `X-Forwarded-Host`/`X-Forwarded-Proto`/`Authorization` as connection-specific and have a downstream intermediary drop them (P2). - browser: create throwaway browser profiles with `tempfile` (random name, 0700, O_EXCL) instead of a predictable timestamped path + `create_dir_all` that a local racer could pre-create; keep the `TempDir` alive until the browser exits (P2). - ca: re-secure existing CA material on the reload path (dir 0700, key 0600) before reusing it, so a drifted/restored group- or world-readable private key is not used indefinitely (P2). - browser: scope `ca_uninstall`'s `find`/`delete-certificate` to the same login keychain `ca_install` trusts into, so uninstall/regenerate act on exactly the trust location this tool manages (P2). - main: on non-macOS the `ts` binary now prints a clear unsupported-platform error and exits nonzero instead of silently succeeding as an empty no-op (P2). Adds unit tests for the header stripping and the CA re-secure-on-reload.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #824
Summary
Adds
ts dev proxy— a local TLS-terminating (MITM) developer proxy that serves a production publisher hostname from a dev/staging upstream by swapping the TLS SNI. A real browser shows the production domain in the address bar while a Compute/staging service answers, so you can exercise Trusted Server's host-anchored URL rewriting against real first-party behavior without DNS or/etc/hostschanges.What's included
New
tsdeveloper CLI (crates/trusted-server-cli, a native binary excluded from the wasm workspace):Proxy flags
--map FROM=TO(repeatable) or-f/--from+-t/--to HOST[:PORT]— the rewrite rule(s); rules are explicit (notrusted-server.tomlinference).--rewrite-host— sendHost: TOupstream (default keepsHost: FROMso core's URL rewriting stays anchored to the production host).--resolve HOST:IP(repeatable) — curl-style DNS pin; dials the given IP on every upstream path while keeping SNI/Hoston the hostname, so the cert still validates. Makes the proxy self-contained (no hosts-file edits).--basic-auth USER:PASS/--basic-auth-file PATH— injectAuthorizationwhen absent (clears a gated upstream).--listen ADDR(default127.0.0.1:18080),--launch chrome|firefox|safari,--insecure(accept self-signed upstreams),--allow-non-loopback.Example
--resolve staging.example.net:192.0.2.10dials192.0.2.10for the upstreamleg while leaving the TLS SNI and
Hostasstaging.example.net, so theupstream certificate still validates — the curl
--resolvemodel. Repeat theflag to pin multiple hosts. IPv6 targets work too (the value is split on the
first
:), e.g.--resolve staging.example.net:2001:db8::10.Architecture
TO) signed by a per-machine local CA; the decrypted stream is proxied request-by-request over keep-alive. Unmatched hosts are blind-tunnelled on loopback (never MITM'd), and aHostmatching no rule is refused421(no smuggling through the CONNECT-authority rule).ca installadds it to the login keychain;uninstall/regeneraterevoke prior trust before replacing key material)./proxy.pacroute; Chrome/Firefox/Safari launch helpers (Safari persists & restores system-proxy state across hard kills).X-Forwarded-Host/X-Orig-Host = FROMso the upstream can keep emitting first-party URLs on the production host even when--rewrite-hostsendsHost: TO.Safety
403(can't become an open proxy) unless--allow-non-loopback.cfg-scoped to macOS, so non-macOS targets compile to an empty shell with a clear message instead of a hardcompile_error!(which previously brokecargo buildunder the repo's wasm default target).Tests, CI, docs
tests/proxy_e2e.rs): MITM rewrite/forward, blind tunnel cert identity,--resolvepinning,--rewrite-hostForwarded-Host behavior, injected Basic auth, keep-alive multi-request,421/403guards.wasm32-wasip1by default).docs/guide/ts-dev-proxy.md(setup, trust, troubleshooting) + the design spec and implementation plan underdocs/superpowers/.Notes