Author here. I flipped the fleet to 100% spot to cut the GPU bill ~60%. The ASG launched a second instance and almost 9s later began terminating the health box that had survived all night. The replacement lasted 20 mins. Then, nothing for 4h37m.
Users never saw an error and I slept through it. The post is how both of these can be true plus the roughly 200 line supervisor that has handled spot for me since.
Author here. I flipped the fleet to 100% spot to cut the GPU bill ~60%. The ASG launched a second instance and almost 9s later began terminating the health box that had survived all night. The replacement lasted 20 mins. Then, nothing for 4h37m.
Users never saw an error and I slept through it. The post is how both of these can be true plus the roughly 200 line supervisor that has handled spot for me since.
Happy to answer any questions!