SSH keys are passwords too

They're not passwords because you're not providing the thing you know directly. You each have half of a shared secret and use it to make the math add up. I can't put a password in a HSM or Secure Enclave and have it work the same as key authentication. I can use an SSH key that's fused to a device. I can't fuse a password to a device.