So recently my framework 16 laptop (AMD Ryzen 7 7840HS w/ Radeon 780M Graphics + Ryzen GPU expansion bay) has been freezing (until I hard reboot by holding the power button down for a time) randomly, and I've reasonably ruled out kernel bugs by booting old NixOS generations.
After some digging, I found that almost always the bugs happen after very specific dmesg errors:
[ 834.609055] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 2309.169094] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 2309.170021] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 2309.170075] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 3906.621059] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 3906.622065] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 3968.060038] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 3968.061023] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 3968.062018] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 3968.062036] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 3968.062045] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 3968.062046] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 3968.062050] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 3968.062083] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 4090.930227] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 4090.931032] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 4152.381046] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 4152.381062] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 4152.382099] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 4213.830047] nix-fw16 kernel: tpm tpm0: Operation Timed out
[ 6487.102025] nix-fw16 kernel: tpm tpm0: Operation Timed out
After which usually (but not always) I would get some hangups with the integrated amdgpu (SMU: No response msg_reg). Very notably, these messages appear at very similar times across unrelated boots, and in fact, would always occur at some multiple of 61.44 relative to 3968.06.
It turns out that the kernel polls the TPM for RNG reasons every 61.44 seconds. This can be verified by running the bpftrace command shown below. But why does this occur around ~3968.06 seconds into the boot process? Well, apparently the TPM specification says this:
The technology used for non-volatile storage may make the update rate for NV Clock an endurance issue. To mitigate this, the interval between updates of NV Clock from Clock is allowed to be as long as once per 222 milliseconds.
So if this were indeed true, reading the TPM clock and waiting for it to near 222 milliseconds, I would expect the next read by the kernel to fail right?
So essentially, I waited around and ran
nix shell nixpkgs#tpm2-tools -c sh -c 'sudo "$(command -v tpm2_readclock)" -T device:/dev/tpmrm0'
to check the clock and
nix shell nixpkgs#bpftrace -c sh -c 'sudo "$(command -v bpftrace)" -B line -e "
kprobe:tpm_transmit_cmd { @s[tid] = nsecs; }
kretprobe:tpm_transmit_cmd /@s[tid]/ {
time(\"%H:%M:%S \");
printf(\"%s %d ms ret=%d\n\", comm, (nsecs - @s[tid]) / 1000000, retval);
delete(@s[tid]);
}"'
to poll when the kernel reads the TPM (thanks claude for this atrocious command).
I expected the TPM to write at around 2:30 AM, and and by watching the output of the bpftrace command, for the crash to happen at the next poll time of 2:30:53 (the previous one happened at 2:29:52), and indeed this is what happened.
So I suspect that the fault happens following a (perhaps failed) attempt for the TPM to sync the clock to the flash (or wherever). I've temporarily disabled the TPM in the BIOS (BIOS settings save properly), and this seemed to have mitigated the issue, but this is truly something very strange. This started to happen after I dropped my computer on the ground, so perhaps something hardware damage related, but regardless is very strange, though I have TPM timeout events dating to before the drop.
What are some next steps so I can get this problem fixed? I'm on the latest firmware according to fwupdmgr.
EDIT: some more digging: updating the TPM clock does nothing, it seems to care about the uptime (tpm2_readclock's time field) not the actual clock, but when I boot normally those start out pretty much synced.