Capture network traffic

Record HAR and streaming captures, including authenticated sessions and full request and response bodies.

Three commands in this repository capture traffic, and which one you want depends on how much control you need over the session:

chrome-to-har navigates to a URL and writes a HAR. Use it for one-shot, scriptable captures.
cdp captures while you drive the browser, by hand or from an agent. Use it when getting to the traffic requires logging in, clicking through, or otherwise doing something a URL alone will not express.
churl -har captures alongside a fetch, when the page content is what you are really after.

For the complete flag reference, see go doc ./cmd/chrome-to-har, go doc ./cmd/cdp, and go doc ./cmd/churl. This guide covers the workflows those flags add up to.

A single-page capture

chrome-to-har -url https://example.com -output example.har

Modern pages keep loading well after the load event fires, so a capture that stops too early is the usual reason a request you expected is missing. Against a page that fetches /api/items from JavaScript after load:

chrome-to-har -headless -url http://localhost:8099/ -output demo.har
jq -r '.log.entries[] | "\(.request.method) \(.request.url) \(.response.status)"' demo.har

GET http://localhost:8099/ 200

One entry: the XHR never made it. Waiting for something the fetch produces fixes it:

chrome-to-har -headless -url http://localhost:8099/ -wait-for '#done' -output demo2.har

GET http://localhost:8099/ 200
GET http://localhost:8099/api/items 200
GET http://localhost:8099/favicon.ico 404

-wait-for waits for a selector; -wait-stable waits for the network and DOM to go quiet, which is what you want when you do not know a selector to wait on. To keep the archive to what you care about, filter it with -urls and -omit, or stop requests from being made at all with -block.

(The fixture server used above is the one from the Quickstart.)

Streaming instead of archiving

A HAR is written at the end of the run, which is inconvenient for a long session and impossible for one that never ends. Both chrome-to-har -stream and cdp -harl emit one JSON object per entry as it arrives:

chrome-to-har -url https://example.com -stream -urls 'api\.example\.com'
cdp -harl -harl-file session.har.jsonl

cdp -harl writes to output.har.jsonl by default. Pass -harl-file - to stream to stdout, which is what you want when piping into jq.

cdp -har writes the file whether cdp launches the browser or attaches to one already listening on the debug port. -har-mode selects how much it records: enhanced, the default, captures roughly 3.4 KB per entry against simple’s 576 bytes for the same page. Reach for chrome-to-har when the HAR is the deliverable rather than a sanity check.

Authenticated sessions

This is the part that most often goes wrong. A browser cdp launches on its own has an empty profile: no cookies, no session, so no access to anything behind a login.

There are two ways to get an authenticated session, and they are not interchangeable:

Copy a profile. -use-profile copies the named Chrome profile, with its cookies, into a temporary directory and launches against the copy:

cdp -list-profiles
cdp -use-profile Default -har session.har -url https://example.com -interactive

The copy means your real profile is never written to, but also that anything requiring a live session — a token refreshed since the copy, a device challenge — may fail. Copying cookies out of Chrome’s database needs sqlite3 on your PATH when you use -cookie-domains.

Attach to a browser you already logged into. This keeps the real session, which is what you want for anything with a strict sign-in:

# Start the browser with debugging enabled, then log in normally.
cdp attach                      # prints attachable targets, or how to launch
cdp -remote-host localhost -remote-port 9222 -har session.har -shell

cdp attach with no browser listening prints the exact command to launch one.

Capturing bodies

By default a capture records metadata and headers. -full-capture also records complete request and response bodies, which is what you need when the answer is in a response payload rather than in the request log:

cdp -full-capture -harl -output-dir ./capture -shell

Bodies make captures large. -max-body-bytes caps each one; the default of 0 keeps them whole.

-output-dir organizes output by the navigated page’s domain rather than one flat file, and saved page sources land under that domain’s directory. Set -group-by-page=false to group by the domain each request went to instead.

-full-capture also captures WebRTC activity, controlled by -webrtc-capture: any of sdp, datachannel, and ice, or all, or none. It defaults to sdp,datachannel, because ICE is noisy and rarely what you are looking for.

Secrets

Captures of authenticated sessions contain credentials — that is what makes them useful and what makes them dangerous to pass around. HAR and source output is redacted by default. -no-scrub turns redaction off; think about where the file is going before you use it.

Reading a capture

A HAR is JSON, so jq is usually enough:

# Requests that failed
jq '.log.entries[] | select(.response.status >= 400) | .request.url' capture.har

# Everything sent to one host
jq -r '.log.entries[] | select(.request.url | contains("api.example.com"))
	| "\(.request.method) \(.request.url) \(.response.status)"' capture.har

# Headers on a specific request, when you need to replay it
jq '.log.entries[] | select(.request.url | contains("/download"))
	| .request.headers' capture.har

Streamed NDJSON is one entry per line, so the same filters work with jq -c over the stream, without waiting for the run to finish.

Next steps

Differential capture — comparing two captures of the same page.
How cdp works — why capture stops when the tool stops, and why the two authentication paths differ.
Troubleshooting — missing requests and logged-out sessions.