Persistent disks are implemented as EBS snapshots, so the process is something like:
1. Create EC2 instance for runner #1. Find out there is no existing snapshot, so an empty volume is created and attached.
2. Runner #1 runs exactly 1 job and shuts down. A snapshot is taken for the persistent volume. That's going to be used by later runners.
3. Create EC2 instance for runner #2. Create a new volume based on the last snapshot.
4. Assuming #2 is still running while a new job comes in. Create EC2 instance for runner #3. Create volume based on the same last snapshot.
5. Whenever a runner finishes, its persistent volume gets a snapshot taken. Outdated snapshots are automatically removed.
And yes we manage the AMI that the runner uses. We try out best to follow https://github.com/actions/runner-images and will automate this process very soon so it's always up-to-date.
Thanks for answering! Unless I'm misunderstanding, one issue with this method is since you're creating a new EBS volume from a snapshot every time the runner starts, the volume will be cold and there will be additional latency on the first reads from the volume. Seems like you could run into this penalty fairly often if you were constantly spinning up and down runners due to inactivity. Maybe something worth considering for v3 (spot instances would be nice to have too).
1. Create EC2 instance for runner #1. Find out there is no existing snapshot, so an empty volume is created and attached.
2. Runner #1 runs exactly 1 job and shuts down. A snapshot is taken for the persistent volume. That's going to be used by later runners.
3. Create EC2 instance for runner #2. Create a new volume based on the last snapshot.
4. Assuming #2 is still running while a new job comes in. Create EC2 instance for runner #3. Create volume based on the same last snapshot.
5. Whenever a runner finishes, its persistent volume gets a snapshot taken. Outdated snapshots are automatically removed.
And yes we manage the AMI that the runner uses. We try out best to follow https://github.com/actions/runner-images and will automate this process very soon so it's always up-to-date.
Edit: formatting