And Lo,
for the Earth was empty of Form,
and void.
And Darkness was all over the Face of the Deep.
And We said: 'Look at that fucker Dance.'
― David Foster Wallace, Infinite JestBack to Index
Early Bukowski is so weird and pretentious and sad.
Far from the image of Bukowski recognized in certain b-tier art consumer circles today: the tired but ultimately complacent and entertainingly vulgar, foul-mouthed, grotesuqe, post-sexual old fuck.
It is a frustrated and resentful young Buk, aggresively naive, lobbing trite and tired old observations about consumerism and superficiality at a society he sees himself as somehow, not belonging, if not so much as superior to. Criticizing, unsubtly justifying himself and his obvious misery instead of trying to merely 'convey' and 'express'
We see a man utterly, yet understandably, incapable of conceiving of a reader who is genuinely curious to read about who he is, what he sees and says and does, to get to know him as opposed to his thoughts and opinions and, strangely, words.
A sort of like, archetypal adolescent lamentation, representative of types that he would, himself, eventually come to describe as: 'writing about life as if they had a real angle on it.'
He, to put it rather mildly, bores. If his trick is being such a disastrous mess that you can not, for the life of you, look away, then clearly he isn't there yet, as he writes.
The aspiration to come out on top is still there, and it's so familiarly tedious and dull, another ambitious striver making ostensibly pointed remarks on the sourness of the proverbial grapes. He loathes and resents us for having and enjoying and taking for grant all these things which he will, eventually, come to confess a profound pain of deprivation and unmet need for. The deadthly-tired old toad now brooding, ruminating, reflecting on all those things he never had, will never have. The deep and festering wounds of rejection, isolation and pain ever wide open, never healed, nor closed or scarred. A Bukowski who, in due time, wearing his collosal gash on the proverbial sleeve, would gather small and rowdy crowds in small rooms translucent with smoke and loud with jeering and swear words, crowds that never missed a good laugh at a bodily-function type joke but would often seem to fail to perceive the endings of his rather more serious pieces and, like, forget to applaud, or respond, the way that failing to add the appropriate inflection at the end of a sentence can leave your interlocutor like, hanging, waiting for you to go on
Because, still, at that point, we see a man, who, like so many of us, still hopes and strives, kicks back and wants, grasps for it, frustrated and sad.
Ok so. A while back I'm chasing HTTP GETs that time out. Over loopback. Big server, tons of headroom, no network in the path, just curl talking to nginx on the same goddamn box. This should take, what, a millisecond. It was hitting my 10s timeout instead. Not every request. That's the part that gets you.
First thing you do, obviously, is check the error log. Nothing. Access log? Every line looks fine. Syslog? Nothing interesting. dmesg? Clean. No OOM, no kernel weirdness, the box isn't even breaking a sweat. So now you're nowhere, holding a server that times out for no reason it'll admit to.
So I did the dumbest possible thing. Wrote a little bash loop that just fires GETs forever and prints how many it's done since the last timeout. And then I sat there. Watching logs scroll in another terminal. Waiting to see if anything so much as twitched in syslog at the exact moment a request died.
If I have ever felt more like a monkey, I genuinely cannot tell you when.
And then it occurs to me, oh, there's a way to actually tie curl's timeouts to nginx's log lines. The server was already logging the User-Agent. So instead of printing the iteration number to my own terminal where it does nothing, I started shoving it into the User-Agent of every request:
i=0
while true; do
i=$((i+1))
curl -m 10 --user-agent "iter:$i" -s -o /dev/null \
-w "%{http_code} %{time_total} iter:$i\n" \
http://127.0.0.1/
done
Now a timeout from curl comes with a unique number I can grep for. So I wait. One fires. Grab the iteration off curl's output. grep the access log.
Status: 200.
Ok. Fine. Probably fat-fingered the grep. Wait for another one. grep.
200.
Every single timed-out request, logged 200. Every. Single. One. The server is sitting there convinced it served a request the client never got a byte of.
So I stopped, and stared into the abyss of the terminal for a while.
Once you stop staring it's not even that exotic. nginx logs a request when it decides the request is done, and "done", to nginx, means: I finished shoving the response into the kernel send buffer. That's the whole bar. It does not mean the client got anything. It does not mean the connection closed clean. There is no FIN-ACK anywhere in this picture. From nginx's point of view the thing ended the instant the last writev / sendfile / SSL_write came back with nothing left to send. Status 200, bytes_sent whatever, write the line, move on, next.
Whatever happens on the wire after that is somebody else's problem. And on loopback, apparently, somebody else had a real problem.
I think people read an access log like it's the receipt at a restaurant. 200, you got your food, enjoy. But it's not the receipt. It's the kitchen ticket. The kitchen is telling you the food went out. Whether it ever landed on your table is a completely different question, and the access log does not know the answer and is never going to.
Once I actually got this I stopped harassing nginx and went to look at the TCP state, which is, of course, where the answer was always going to be. Send buffer full. Nothing getting acked from the far end of loopback (yeah, loopback can do that, its own rabbit hole, its own bad evening). curl eventually gives up. nginx never even notices, because nginx left the building ages ago.
I'd love to tell you the next chapter, what was actually broken down in the kernel, but that's a different post and I'm honestly still not fully sure. This one's simpler. The point of this one is: stop trusting the access log.
If your access log says 200 and your client says it timed out, the client is right. Always. The access log is telling you what the server meant to do. It is not telling you what happened to the bytes.
Couple things I do differently now:
$bytes_sent and $request_length in the log format and actually look at them when timeouts get weird. bytes_sent equals the full content length? The server thinks it did its job. Short? Ok, now that's a clue.The lesson isn't really about nginx. nginx is doing the exact thing it says on the tin. The lesson is that every log line is fired by some specific line of code, some specific event, and if you don't know which event, you do not know what the line means. Miss that and you end up where I was, 11pm, staring at a terminal, asking a server that swears it succeeded why it's also the thing timing out my client.
Ask me how I know.
Back to IndexHot take, and almost nobody frames it this way: HTTP/3 is one of the cleanest dev-vs-ops fights I can think of, and everyone insists on talking about anything else. Head-of-line blocking. 0-RTT. Cool. Sure. Meanwhile the actual thing QUIC did, organizationally, is the transport layer got up, walked out of the kernel, and moved into your application binary — and the second it did that, it walked out of ops' hands and straight into the developers' lap.
For like thirty years the deal was simple and kind of beautiful. Devs write the app. Ops own the network. TCP lives in the kernel. Congestion control, retransmits, backoff, RTO, all of it, somebody else's problem. You tune a few sysctls, you bump the kernel when you want a new congestion controller, and the same one TCP stack serves every process on the box. One place to watch it. One place to fix it. ss, tcpdump, netstat, the /proc/net/* counters, nstat, a whole ecosystem of tools that all quietly assume the kernel is the source of truth about your connections. And it was.
QUIC throws the whole contract in the bin. It runs over UDP, and the entire reliable-transport state machine — streams, flow control, congestion control, loss detection, retransmits, acks, the lot — lives up in userspace, inside whatever library the app happened to link against. nginx-quic, ngtcp2, quiche, msquic, lsquic, the Go stdlib one, chromium's one. These are not the same code. They do not behave the same under loss. They do not expose the same counters. There is no ss for QUIC. The kernel sees a stream of UDP datagrams and shrugs at you.
That's a transfer of responsibility wearing a protocol upgrade as a disguise.
Concretely: something goes wrong at the transport layer in a QUIC deployment, and the reflex ops answer, "let me check the kernel, the NIC, the sysctls", gets you basically nowhere. The behaviour you care about is now a function of which library your devs picked, which version they pinned, what congestion controller it happens to implement, and what, if anything, it felt like logging. Dev team links one library, the CDN out front links another, and congrats, you've got two different transports that merely agree to interoperate on the wire. Every property you used to get for free, one stack, kernel-tuned, observable with normal tools, gone. And getting it back is a development project now, not an ops ticket.
This is also, I think, why a lot of shops quietly hate operating QUIC even while they love serving it. The exact thing that made TCP boring — that it was somebody else's code, in somebody else's address space, watchable with somebody else's tools — is precisely the thing QUIC hands back to you. What you get in exchange is the ability to ship transport changes without waiting on a kernel release, which is incredible if you are Google and miserable if you're four people in an ops channel trying to work out why p99 went sideways on a Tuesday.
None of this is me arguing against QUIC. I serve it, it's fine, it's good even. It's me saying QUIC isn't just a protocol change, it's an org-chart change — and the people who end up debugging it at 3am are, almost always, not the people who got to vote for it.
Back to IndexIf your nginx alerting hangs off $upstream_response_time because somebody once told you it's "how long the request took to be served", you are quietly underreporting a whole class of incident. Specifically: anything where requests pile up behind proxy_cache_lock.
The thing I had to learn the slow way: time spent waiting on the cache lock lands in $request_time but not in $upstream_response_time. They're not the same metric and they don't differ by some constant you can subtract back out. Under contention they drift apart by seconds, and the one that feels like the right thing to alert on is exactly the one hiding the problem.
proxy_cache_lock exists to coalesce parallel cache-fill requests. Ten clients ask for the same uncached object at once, you don't want ten fetches hammering your origin, you want one fetch and nine followers waiting on the result. The followers, while they wait, are blocked. That wait is real wall time. It's user-visible. It's in $request_time. It is not in $upstream_response_time, because $upstream_response_time measures the upstream call and nothing else: connect plus send plus read. Time the request spent pinned to a lock isn't part of that, and nginx is not going to go back and add it in for your convenience.
Single-worker nginx, cache-locked location, a lua origin that sleeps before it answers. Same cache key for every request so the lock is shared.
daemon off;
master_process on;
worker_processes 1;
error_log stderr;
events {}
http {
log_format cachetest
'$time_iso8601 conn=$connection req#$connection_requests '
'"$request" $status cache=$upstream_cache_status '
'rt=$request_time urt=$upstream_response_time';
proxy_cache_path /mnt/disk1/cd levels=1:2 keys_zone=normal:10m
max_size=10m inactive=1h use_temp_path=off;
upstream origin { server 127.0.0.1:8081; }
server {
listen 8080;
access_log /dev/stderr cachetest;
location / {
proxy_pass http://origin;
proxy_cache normal;
proxy_cache_key $uri;
proxy_cache_valid 200 1h;
proxy_cache_lock on;
proxy_cache_lock_timeout 2s;
proxy_cache_lock_age 1h;
}
}
server {
listen 127.0.0.1:8081;
location / {
content_by_lua_block {
ngx.sleep(3)
ngx.say("ok")
}
}
}
}
Drive it with five parallel curl calls at the same URL.
Origin sleeps 3s, proxy_cache_lock_timeout 2s. The leader goes upstream. The four followers sit on the lock for 2s, the lock times out (the leader still isn't done, it finishes at 3s), and now each follower goes upstream on its own.
conn=2 cache=MISS rt=3.010 urt=3.006 # leader
conn=4 cache=MISS rt=5.010 urt=3.006 # follower
conn=6 cache=MISS rt=5.010 urt=3.006
conn=7 cache=MISS rt=5.010 urt=3.006
conn=8 cache=MISS rt=5.010 urt=3.006
For each follower, rt - urt = 2.0s exactly. That's the lock wait, right there. urt reflects only the upstream call, identical for all five.
If your dashboard graphs urt p99, this incident is invisible. Every request "took 3 seconds upstream", which is technically true and operationally useless. The clients waited 5.
Now make the origin uncacheable. Same 3s sleep, but the response sets Cache-Control: no-store. Push the lock timeout high enough that it never fires (proxy_cache_lock_timeout 60s). The cache never populates, so no upstream call ever satisfies a follower, the lock just keeps getting handed down the line.
content_by_lua_block {
ngx.header["Cache-Control"] = "no-store"
ngx.sleep(3)
ngx.say("ok")
}
rt=3 urt=3 # leader
rt=6 urt=3 # waited 3s
rt=9 urt=3 # waited 6s
rt=12 urt=3
rt=15 urt=3
urt stays flat at 3s. rt climbs by the cumulative lock wait. Alert on the upstream metric and the box looks perfectly healthy: every upstream call takes 3 seconds, which is what it always takes. Meanwhile the fifth client waited fifteen.
The reason this trips people up, I think, is that "upstream response time" sounds like the longest, most far-away thing nginx waits on, and intuitively the longest thing should dominate the latency. But nginx isn't narrating, it's bookkeeping. $upstream_response_time is "time spent talking to the upstream server", measured from inside the connect/send/read calls and nowhere else. Time spent waiting on internal coordination, locks, queues, anything that isn't the upstream socket, doesn't show up. The name oversells it.
The fix is boring: alert on $request_time if you want to know what the client actually felt. Keep $upstream_response_time in the log too, so you can compute rt - urt and watch which component is moving. High rt with a flat urt across a pile of parallel requests for the same key is the fingerprint of cache-lock contention, and basically nothing else produces that exact shape.
Same lesson as the access-log post, slightly different shape: every metric is the answer to one specific question. Don't know the question, you don't know what you're looking at.
Back to IndexNobody reaches for parallel requests. Watch any engineer hit a flaky, intermittent problem — ops, nginx, backend, it doesn't matter — and they reach for curl in a loop. Sequential, every time. I have never once seen anyone fire requests in parallel unless they were already desperate and out of other ideas. So you get a room full of people running curls one after another, some passing and some failing, and not one of them can say why. The answer is almost always that two of the requests happened to overlap in time — accidental parallelism — and that overlap is the whole bug. Fire the same requests in parallel on purpose and the failure reproduces on the first try. The intuition about concurrency is so bad that the one tool which would catch the bug immediately is the tool nobody thinks to pick up.
And when you do finally reach for it, it bites you a second time: "parallel" keeps quietly collapsing back into "serial." You fire off N requests at once, feel clever, and a layer down they line up single file and you never see it. So the rest of this is less a tutorial than a list of where I keep getting bitten, and the flags that make it honest.
Simplest way to send the same request many times, at once:
curl -sS -Z --parallel-immediate \
https://example.com/ -o /dev/null \
https://example.com/ -o /dev/null \
https://example.com/ -o /dev/null
-Z (--parallel) means "do these concurrently instead of one after another." List the same URL N times and they go out together. I keep a little wrapper for exactly this — it repeats whatever URL you hand it seven times and prints each one's time, local port and speed, so you can watch them overlap.
Now the part that bites. Plain --parallel, given the chance, reuses one connection for all of them — so your seven "parallel" requests get multiplexed down a single socket and quietly take turns. Parallel on paper, serial on the wire. The fix is --parallel-immediate: open a fresh connection for each, right away, instead of waiting to reuse one. The note I once left myself:
--parallel # same connection, can serialize on you
--parallel-immediate # different connections, actually concurrent
The tell is local_port in the output: same port across all of them means they shared a connection; different ports means they really went out in parallel.
The lookalike that is NOT parallel: --next.
curl https://a/ --next https://b/
--next does chain several requests into one curl process, but it runs them one after another. Serial batching that reads like the real thing. If you wanted them at the same time you wanted -Z, not --next.
And if you'd rather curl not drive the parallelism at all — I gave GNU parallel an honest couple of tries and it drove me crazy, the quoting especially — two boring things that always work:
# xargs, 8 at a time
printf '%s\n' "${urls[@]}" | xargs -P 8 -I{} curl -sS {} -o /dev/null
# or just background them and wait
for u in "${urls[@]}"; do curl -sS "$u" -o /dev/null & done; wait
That's the whole toolkit. The flags are easy. The hard part is remembering that every "in parallel" you write has at least three places it can secretly become "in sequence" — a shared connection, a shared lock, one worker doing one thing at a time — and your intuition will not warn you about any of them. You just have to go look.
Back to IndexCode agents will tell you they wrote a thorough test. They reach for words like "comprehensive" and "exercises the happy path and edge cases". They're guessing. Coverage data is the one thing in the room that isn't guessing.
For an nginx fork I work on I keep a Claude Code skill called /coverage-audit. It rebuilds with gcov instrumentation, wipes stale .gcda, runs a named pytest target, then reads the .gcov output for the source files the test touched. For every uncovered line it picks a bucket:
*alloc returning NULLngx_array_push failing in postconfig#if (NGX_DEBUG)#if 0The first five aren't worth chasing. Mock them into green and all you've bought is a bigger number and zero extra confidence. The last three are the real gaps. Write a test that hits those.
The skill spits out something like:
foo.c: 90.62% of 192 lines
Testable:
- foo.c:175 — `return NGX_DECLINED` when feature off
→ add a test with `feature off;` in the server block.
- foo.c:93-94, 134-135 — `not_found = 1; return NGX_OK`
→ hit by plain-HTTP listener; add a non-SSL test.
Not worth chasing:
- Allocation failure: foo.c:393, 408, 422, 435
- Defensive guard: foo.c:241, 374, 454
Verdict: test exercises legitimate input combinations cleanly;
gaps are real branches plus fault-injection paths.
Run the audit in a different Claude session than the one that wrote the test. The auditor needs a different context, the .gcov files and the source, and it shouldn't be the same agent quietly invested in defending the thing it just wrote. Two sessions, one window each. The audit hands you a punch list, the implementer works through it. Same reason you don't let it mark its own homework.
Second discipline: review the commit graph, not just the files. Agents pile changes into whichever commit they currently care about. Earlier today my implementer squashed three test commits into one, fine, but it also dragged a Dockerfile change and a CI image-tag bump into the test commit, because those had been bundled with one of the tests at some point. Wrong home. They belonged in the earlier base-image commit. The fix:
$ git diff <squashed>~ <squashed> -- Dockerfile ci.yml > /tmp/base.patch $ git diff <squashed>~ <squashed> -- tests/ > /tmp/tests.patch $ git reset --hard <base-image-commit>~ $ git cherry-pick --no-commit <base-image-commit> $ git apply --index /tmp/base.patch $ git commit -F new-base-image-msg $ git apply --index /tmp/tests.patch $ git commit -F new-tests-msg $ git cherry-pick <core-wiring-commit>
An agent will run that whole sequence without a stumble the moment you tell it "the Dockerfile hunks belong in commit X". It will not get there on its own. Where a change belongs is a human judgment. Mechanically rewriting the graph to put it there is the agent's actual strength.
The pattern: agents produce, gcov audits, humans curate the graph.
Back to IndexThis site runs a JA4 module in nginx. Hit /ja34 and it reflects back the fingerprint nginx computed for your TLS/QUIC handshake, ciphers, extensions, ALPN, the lot, plus a pile of $ssl_* variables. JA4 fingerprints how a client says hello, and it's only worth anything if it's correct. "Looks plausible" is not correct. A fingerprint nobody can independently check is just a string the server made up.
So the question that actually matters: is the JA4 my nginx emits the same JA4 a known-good implementation would compute from the exact same bytes on the wire? You need a second opinion from something that didn't write the first one. Wireshark has had a JA4 implementation for a while, and it derives it from the raw ClientHello, completely independent of nginx. If nginx and Wireshark agree about the same handshake, I believe the number. That's the whole experiment: capture the handshake, decrypt it, read off Wireshark's JA4, compare.
The wrinkle: this is HTTP/3. The handshake I want to look at is a TLS 1.3 ClientHello riding inside QUIC CRYPTO frames, and QUIC encrypts its handshake. (Remember the earlier post about QUIC dragging the transport into userspace? Same bill, now due: there's no plaintext handshake left to sniff.) To see it I need the TLS secrets. curl writes them if you set SSLKEYLOGFILE, and tshark reads them via -o tls.keylog_file:… and derives the QUIC keys off them. Standard stuff.
$ export SSLKEYLOGFILE=~/sslkeylogfile.log
$ tshark -i wlp44s0 -o tls.keylog_file:$SSLKEYLOGFILE \
-f 'udp port 443' -Y quic -VThe QUIC Initial packets decrypted fine. They always will: their keys come deterministically off the Destination Connection ID, no secrets needed. But every Handshake and 1-RTT packet after that handed me:
[Expert Info (Warning/Decryption): Failed to create decryption
context: Secrets are not available]"Secrets are not available." Fine. Except I'd just cat'd the keylog and the secrets were sitting right goddamn there: CLIENT_HANDSHAKE_TRAFFIC_SECRET, SERVER_HANDSHAKE_TRAFFIC_SECRET, CLIENT_TRAFFIC_SECRET_0, the works, keyed by the same client random as the ClientHello in the capture. The secrets were not unavailable. They were on disk, in the exact file I'd pointed tshark at, correct.
First theory: a race. Live capture dissects packets as they land; curl only writes each secret once it derives it mid-handshake. So maybe tshark went looking for the handshake key, didn't find it yet, cached the miss, and never went back once curl finished writing. Plausible. The standard fix is to not decrypt live at all: capture to a file, let the handshake finish and the keylog fully populate, then dissect the file.
$ tshark -i wlp44s0 -f 'host 162.19.246.242 and udp port 443' \
-w /tmp/quic.pcapng
# ...curl --http3-only https://pwnrzclb.net/ja34 in another shell...
$ tshark -r /tmp/quic.pcapng -o tls.keylog_file:$SSLKEYLOGFILE -VSame failure. Identical. So it was never a race: the secrets were complete and on disk before this second tshark even started, and it still told me they weren't available. When the obvious explanation is wrong, stop theorising about the application and go ask the kernel what actually happened.
$ sudo dmesg | grep -i apparmor | grep tshark
apparmor="DENIED" operation="open" profile="tshark"
name="/home/.../sslkeylogfile.log" requested_mask="r" denied_mask="r"
apparmor="DENIED" operation="open" profile="tshark"
name="/home/.../.config/wireshark/preferences" requested_mask="r"There it is. This box runs a recent Ubuntu, and recent Ubuntu ships an enforcing AppArmor profile for tshark (Canonical added it in 2024, part of the unprivileged-userns-restrictions push). The profile pins tshark to a tight allowlist: its own binary, the Wireshark data dir, a tmp area, /proc/self/fd. Reading some arbitrary file under $HOME is not on it. The open() on my keylog returned EACCES before tshark read a single byte of it.
And here's the part that belongs in this blog specifically: tshark took a permission-denied on the keylog and reported it to me as "secrets are not available". Those are not the same thing. One means "this process cannot open the file you gave me". The other means "the keys for this connection aren't in the material I loaded". tshark collapsed the first into the second, and in doing so sent me chasing a race condition that never existed. Same disease as the access log that says 200 on a timeout, same disease as $upstream_response_time sitting flat while the client waits: the tool answers a question you didn't ask and lets you assume it answered yours. The Can't open your preferences file warning tshark prints on startup was this same denial, and I'd been filing it under "harmless noise" for months.
The fix is a local override. The shipped profile include if existss one, exactly so you don't have to touch the packaged file:
# /etc/apparmor.d/local/tshark
owner @{HOME}/sslkeylogfile.log r,
owner @{HOME}/.config/wireshark/{,**} r,$ sudo apparmor_parser --replace /etc/apparmor.d/tshark(If you go this way: capturing without sudo needs you in the wireshark group, and a shell that predates you being added to it won't have the group, so sg wireshark -c '…' or a fresh login. That one at least fails honestly.)
Override in place, re-dissect. Now the 1-RTT stream decrypts, and there in plaintext is the request that kicked the whole thing off:
HTTP3 ... HEADERS: GET /ja34
Header: :path: /ja34
TLSv1.3 ... Client Hello
[JA4: q13d0311h3_55b375c5d22e_a11bc413b5d6]And what nginx had independently reported for that exact connection at /ja34:
ja4: q13d0311h3_55b375c5d22e_a11bc413b5d6Byte-for-byte identical. That's the whole point of the exercise. Decoded, q13d0311h3 says: QUIC transport, TLS 1.3, SNI is a domain name, 3 cipher suites, 11 extensions, first ALPN h3. Then 55b375c5d22e is the truncated hash of the sorted cipher list, and a11bc413b5d6 the hash of the sorted extension and signature-algorithm list. Two implementations that share no code looked at the same ClientHello and spat out the same twelve hex digits on each side. The implementation is correct, and now I can say that instead of hoping it.
One useful discrepancy fell out of it. The ja3_string nginx emits and the one Wireshark emits do not match. Not because either is wrong: nginx hands back the extension list sorted (0-10-13-16-...-57) while Wireshark keeps wire order (57-10-22-23-...-16). Classic JA3 is order-sensitive by design, so a sorted "JA3" hashes differently from a canonical one. JA4 sorts ciphers and extensions as part of its spec, which is exactly why the JA4 agreed while the JA3 didn't. Lesson rhymes with every other post here: two fingerprints "matching" only means something once you know precisely what each side canonicalises before it hashes. Compare the wrong normal forms and you either miss a real difference or invent one that was never there.
The fingerprint was always right. The keylog was always complete. The only thing ever broken was a tool describing a permission error as a cryptographic one, and me believing the description instead of checking it.
Back to IndexClaude Code keeps its state in ~/.claude. Agent definitions, skills, per-agent persistent memory, settings. And in the same directory, the runtime churn: session transcripts, shell history, live credentials. If you only ever work on one machine this is a non-problem and you can stop reading.
I work on three. A work VM where most of the real work happens, the host workstation it runs on, and a secondary laptop. And here's the thing nobody warns you about: this is the dotfiles problem, the one we collectively solved fifteen years ago with a git repo and some symlinks, except now the files learn things. An agent on the VM spends an afternoon working out some gnarly corner of a system, which API endpoint lies, which service wants a flag nobody documents, and writes it into its persistent memory. The same agent on the host knows none of it. Same agent. Same definition, same name, same job. It just never got the memo, because the memo lives in a directory on a different machine. Basic shit, sharing what one agent learned with the same agent somewhere else, turns out to be a hard problem, and the configs fork underneath you the whole time.
The thing that actually set me off: a dashboard-analysis agent that had only ever lived on the work VM. I wanted it on the host, because the host is where I can talk to it, voice input, no SSH hop in the middle. The definition is a file. Copy a file, easy. Except an agent without its memory is a new hire wearing the old hire's badge. Everything it had learned about the dashboards, which panels lie, which datasource is the slow one, was in its memory directory, and that directory was about to fork into two divergent truths the second I copied it.
The naive fix is sitting right there: git init ~/.claude, push it somewhere, pull everywhere. It fails for two reasons. One obvious in hindsight, one obvious immediately.
The hindsight one: ~/.claude isn't a config directory, it's a live runtime directory that happens to contain config. Sessions append, history churns, caches come and go. Version that and every git status is a wall of noise, and noise is how you get hurt: you stop reading the wall, then one git add --all with a credentials file sitting in the tree and you've just pushed live tokens to a remote. Not hypothetical. That's the blast radius of one lazy command.
The immediate one: the machines aren't equal and shouldn't be. The work VM carries work agents, work skills, work credentials. None of that has any business on a personal laptop. "Sync everything" doesn't just sync state, it syncs liability, it dumps work secrets onto whichever machine pulls next. Different machines want different subsets, and a bare git repo has no opinion about subsets.
So, the actual attempts, in the order I made them.
One: on the primary machine, the repo IS the directory. On the work VM the dotfiles repo is checked out as ~/.claude, strict gitignore holding back the runtime churn, git-crypt on the secrets. Works great. Also only ever describes one machine's truth. Single-player solution wearing a multiplayer haircut.
Two: an install.sh that symlinks everything everywhere. The classic dotfiles move. Clone on a new machine, run the script, every artifact gets a symlink. I wrote it, looked at it, and realized running it on the laptop would dump work credentials and a dozen work-only agents onto a personal machine in one keystroke. So install.sh got a refusal guard and a policy in its place: setting up a machine needs per-machine reasoning about what it should carry, and a blind script is structurally incapable of reasoning about anything. Disabling your own installer feels like defeat. It isn't. Cheapest correct decision in the whole story.
Three: install-safe.sh, the universally-safe subset. If the full installer is banned, what's the largest set of links that's correct on every machine, no judgment required? Turns out small but real: the repo-steward agent, the editor config. Idempotent and non-destructive. It never clobbers; if a real file already sits at a target path it prints SKIP and moves on, and untangling that is explicitly a human's job. Boring by design. Boring is the feature.
Four: the memory-dir symlink. This is the actual win. Take the agent's persistent-memory directory in the live ~/.claude on the host and make it a symlink into the repo clone. Now when the agent learns something at runtime, it's writing into the repo working tree. The repo becomes a sync bus: runtime learnings turn into commits, the other machine pulls, and the same agent over there wakes up knowing things it learned somewhere it's never been. The agent's instructions tell it to sync at session boundaries, and the sync itself is handed off to a repo-steward agent that owns the git mechanics: fetch, rebase, push, and a check that the encrypted blobs are actually ciphertext before anything leaves the machine. The agent that learns is not the agent that pushes. Separation of duties, but for robots.
Five: git-crypt for everything secret-shaped. Secrets in the repo are ciphertext at rest, key carried out of band. A machine without the key can clone all day and hold nothing but noise. I applied this with more paranoia than strictly necessary: even archived voice-dictation transcripts get encrypted, on the theory that anything I said out loud near a microphone is not something I want grep-able on a forge.
Six: the machine-profiles design doc. The endgame: classify every artifact in the repo, every agent, skill, memory dir, script, secret, as work, personal, or both, and drive installation from per-profile manifests keyed on a MACHINE_TYPE variable. And the rule I care about most: when MACHINE_TYPE is unset, fail closed. Install nothing, rather than fall back to "everything", because "everything" is the original sin this whole thing exists to undo. Status: designed, written down, argued with, not implemented. The document is real. The machinery is vaporware.
What actually worked: the memory symlink as sync bus. Agent knowledge flows between machines now, which was the entire point. Disabling the blind installer before it bit me. The idempotent safe subset. git-crypt. And, underrated, writing the work/personal classification down before building anything, because the table turned out to be the hard part and the installer is just the table made executable.
What still bites. The hand-copied agent files from the pre-symlink era are now actively in the way: the safe installer wants to lay a symlink where a real copied file sits, and, correctly, refuses to clobber it, prints SKIP, leaves the migration to me. My own past shortcuts are blocking their own replacement, and the same guard that protects me from the installer also protects the stale copies from getting fixed. The manifest engine is still a design doc. And the hard part is wide open: two machines writing to the same agent's memory at once is a distributed-writes merge problem, and my current "solution" is discipline, sync at session boundaries, rebase, don't run the same agent in two places at once. Which is to say it's not a mechanism, it's a promise I make to myself. Promises scale notoriously well.
Lessons, for whatever they're worth:
The files learn things now. The least we can do is teach them to commute.
Back to Indexdual-boot HP laptop. Windows 11 Pro and Ubuntu, UEFI mode. the symptom never moved the whole way through: reboot, and it goes straight into Windows, no GRUB menu, no chance to pick Ubuntu. what changed every round was why, and each round disproved the round before it. took four to land on something that survives reboots in both directions, and the last one only worked after the third one bricked booting into Windows. this is the whole thing, start to finish, because the dead ends are the useful part.
facts that hold the whole way through, so i stop repeating them: Secure Boot is OFF (Confirm-SecureBootUEFI → False), so this is not an SBAT/dbx shim revocation. Fast Startup is OFF (HiberbootEnabled = 0), so it's not a hybrid-shutdown skip. the ESP is Disk 0 Partition 1 (\Device\HarddiskVolume1; under Linux it's /dev/nvme0n1p1, a FAT volume with fs-uuid 36AB-DB35), and GRUB's files, shimx64.efi, grubx64.efi, mmx64.efi, the 117-byte grub.cfg stub, are all present and correct at \EFI\ubuntu\. GRUB was never the broken thing. the firmware boot policy was.
first look at the firmware boot manager (bcdedit /enum {fwbootmgr}): displayorder was just {bootmgr} (Windows) followed by a generic USB entry. no Ubuntu/GRUB entry in NVRAM at all, and timeout 0, so the firmware launched the first thing instantly. a Windows update had re-asserted Windows Boot Manager as the top entry and dropped the Ubuntu one entirely.
fix: recreate the entry pointing at the shim and shove it to the front of the firmware order.
bcdedit /copy {bootmgr} /d "Ubuntu"
bcdedit /set {NEW-GUID} path \EFI\ubuntu\shimx64.efi
bcdedit /set {fwbootmgr} displayorder {NEW-GUID} /addfirst
GRUB menu came back. worked, confirmed. (the entry is a /copy of {bootmgr}, so it inherits a pile of irrelevant Windows-boot-manager sub-fields; for a firmware entry only device + path matter, the firmware just launches \EFI\ubuntu\shimx64.efi and the shim chainloads GRUB.) then it reverted.
next time it broke, Windows again, but now the order was already correct. Ubuntu first, device pointing at the real ESP, shimx64.efi present. nothing was "wrong" to re-fix. Secure Boot off, Fast Startup off, both ruled out. the likely trigger was a pair of security updates (KB5094126 / KB5094135) from a few days earlier, but the interesting part isn't the KB. it's that the firmware now showed the correct order and ignored it.
the decisive diagnostic for "is the firmware honoring NVRAM at all" is BootNext (a.k.a. bootsequence in bcdedit): a one-time EFI variable the firmware is contractually obliged to honor for exactly the next boot, independent of the regular order. set it at Ubuntu, re-assert the order, reboot once.
bcdedit /set {fwbootmgr} bootsequence {UBUNTU-GUID} :: BootNext, one-time
bcdedit /set {fwbootmgr} displayorder {UBUNTU-GUID} /addfirst
the logic of the test: if GRUB appears, the entry is fine and the firmware just won't keep the persistent order, annoying but fixable. if it still boots Windows, the firmware is ignoring NVRAM entirely and every bcdedit reorder in the world is wasted motion.
this is the round that names the real problem. the one-time BootNext override was gone, consumed by a boot, and that boot landed in Windows anyway. order correct, ESP intact, shim present, Secure Boot off, Fast Startup off, and the firmware had ignored a one-shot override aimed point-blank at Ubuntu.
conclusion: this HP firmware does not honor the OS-set UEFI boot order. At all. it reads NVRAM when you write it, bcdedit reports success, then re-asserts its own policy on the next boot and launches the hardcoded Windows path regardless. rounds 1 and 2 hadn't really "worked twice and reverted", they'd been coincidences. NVRAM reordering is a dead end on this box.
so stop fighting the order. the firmware always launches one path: \EFI\Microsoft\Boot\bootmgfw.efi. if you can't make the firmware point at GRUB, make that path be GRUB. back up the real Windows boot manager, then overwrite it:
mountvol S: /S
copy S:\EFI\Microsoft\Boot\bootmgfw.efi S:\EFI\Microsoft\Boot\bootmgfw.efi.bak :: pristine, 3,055,616 B
copy S:\EFI\ubuntu\grubx64.efi S:\EFI\Microsoft\Boot\bootmgfw.efi :: now 2,828,168 B
two things matter here. first, back up the genuine boot manager before you clobber it, verify it's actually a Microsoft PE binary and not an already-hijacked copy; don't trust a specific size, those shift across Windows builds. second, and the part that trips people: overwrite with grubx64.efi, not shimx64.efi. Secure Boot is off, so the signed shim buys nothing here, and a shim dropped into the Microsoft directory would look for grub next to itself (in \EFI\Microsoft\Boot\) and fail. grubx64.efi carries its compiled-in /EFI/ubuntu prefix, so no matter what path the firmware launches it from, it still finds the grub.cfg stub. reboot: firmware "boots Windows", gets GRUB, menu appears. looked like victory.
GRUB now appeared every time. but selecting "Windows Boot Manager" in the menu flashed once and dropped straight back to… the GRUB menu. booting into Windows was now impossible.
the punchline: os-prober had generated GRUB's Windows menuentry to chainload /EFI/Microsoft/Boot/bootmgfw.efi, but that file is now GRUB itself. "Boot Windows" therefore meant "load /EFI/ubuntu/grub.cfg again" → same menu → loop, forever. we built the booby trap ourselves, back in Round 3. the generated entry looked like this, and the chainloader line is the whole bug:
menuentry 'Windows Boot Manager (on /dev/nvme0n1p1)' ... 'osprober-efi-36AB-DB35' {
insmod part_gpt
insmod fat
search --no-floppy --fs-uuid --set=root 36AB-DB35
chainloader /EFI/Microsoft/Boot/bootmgfw.efi
}
the fast way to confirm "this file is not what its name says" is a byte-compare. two files that should differ sharing an md5 means one clobbered the other:
md5sum /boot/efi/EFI/Microsoft/Boot/bootmgfw.efi # b63c7618fb96d52520b42b8cc7b124f0
md5sum /boot/efi/EFI/ubuntu/grubx64.efi # b63c7618fb96d52520b42b8cc7b124f0
# identical — "bootmgfw.efi" is grub now.
the pristine backup from Round 3 is still sitting there (bootmgfw.efi.bak, 3,055,616 B, md5 895d6de079f42e848d87d59a937a4183), that's the real Windows manager. the resolution keeps the hijack (it's the only thing that makes GRUB appear on this firmware) but stops the chainloader from pointing at the lie. three moves:
1. give the real Windows manager a stable name of its own (leave .bak untouched as the archival copy):
sudo cp /boot/efi/EFI/Microsoft/Boot/bootmgfw.efi.bak \
/boot/efi/EFI/Microsoft/Boot/bootmgfw_real.efi
# verify: md5sum → 895d6de079f42e848d87d59a937a4183
2. hand-write a Windows entry that chainloads bootmgfw_real.efi instead of the hijacked path, in /etc/grub.d/40_custom:
menuentry 'Windows Boot Manager' --class windows --class os {
insmod part_gpt
insmod fat
search --no-floppy --fs-uuid --set=root 36AB-DB35
chainloader /EFI/Microsoft/Boot/bootmgfw_real.efi
}
3. disable os-prober so the next update-grub doesn't helpfully regenerate the looping entry on top of mine. in /etc/default/grub:
GRUB_DISABLE_OS_PROBER=true
then sudo update-grub. the resulting /boot/grub/grub.cfg has exactly two top-level menuentries, Ubuntu and Windows Boot Manager, and the only chainloader line in it points at bootmgfw_real.efi. the full chain now: firmware → \EFI\Microsoft\Boot\bootmgfw.efi (which is GRUB) → menu → pick Windows → chainload bootmgfw_real.efi (the real manager) → Windows. both OSes boot, across real reboots, in both directions. done.
the whole saga exists because a Windows Update can silently reorder your bootloader. it can because UEFI requires firmware to expose GetVariable/SetVariable Runtime Services that stay callable after boot, so BootOrder, Boot0000, BootNext all live in NVRAM as runtime-writable UEFI variables. Windows surfaces them through Get/SetFirmwareEnvironmentVariable, and bcdedit /set {fwbootmgr} … is just a front-end over that, gated behind SeSystemEnvironmentPrivilege (admin) and only in UEFI mode. userspace tool, kernel-mediated, admin-only, and capable of dropping your GRUB entry on the next feature update. (Linux's equivalent is efibootmgr; same NVRAM, same power.) on most machines, putting Ubuntu first in that order is enough. this HP's firmware is the nastier case: it lets the OS write the order, reports success, then ignores it, so the only durable lever left is the one path the firmware can't route around.
the hijack lives at a path Windows considers its own, which means Windows can take it back:
\EFI\Microsoft\Boot\bootmgfw.efi back to the real manager, silently undoing the hijack → you boot straight to Windows again, no GRUB. if GRUB vanishes after a Windows update, this is why. recovery is one line: sudo cp /boot/efi/EFI/ubuntu/grubx64.efi /boot/efi/EFI/Microsoft/Boot/bootmgfw.efi.Microsoft\Boot\ can also clobber or remove bootmgfw_real.efi. if GRUB shows up but Windows stops booting, re-copy it from the .bak.shimx64.efi approach and place grub where shim expects to find it.BootNext is the single cleanest diagnostic: if a consumed one-time override still boots the wrong OS, the firmware is ignoring NVRAM entirely, stop editing it.bootmgfw.efi hijack is a legitimate move, but it creates a trap: the file's name is now a lie, and any chainloader (os-prober, an update, future you) that points at it loops forever. redirect Windows to a preserved copy and disable os-prober.md5sum/byte-compare is the fast oracle for "did something overwrite this?". two files that should differ sharing a hash is the tell.