TIL AWS doesn't initialize any blocks on your EBS volume until you read from it

Basically if you never read a particular block from your volume after you initialize it, AWS never moves it out of S3 into EBS.

This can cause bizarrely slow performance for database or other random-read applications when you fire up a volume from a snapshot.

The official mitigation is to literally read every block once and then your volume will perform the way you expect:

  sudo fio --filename=/dev/xvdf --rw=randread --bs=128k --iodepth=32 --ioengine=libaio --direct=1 --name=volume-initialize