Can't stop or delete VMs, getting resource not found errors

[fredsimard]

Hi! First, the context: we have a GitHub action that performs tests on a commit before merging. This action is, as far as I understand it, spinning up a VM in a project on Google cloud, performs the tests and then is supposed to spin down the virtual machine when done.

Now I have an issue: there are eight virtual machines in that project that are seemingly idle, cannot be stopped, cannot be deleted and gives me an error message in the likes of "resource not found" when I try to get more details on it. Oddly enough, they all have public IPs and I can ping them, but cannot SSH into them using Google Cloud.

Now, if these VMs were not costing us a cent, I wouldn't mind, instead they are charging us every month for each of those VM's that are doing absolutely nothing of value to us. I tried contacting support, but without having a plan that includes paid support I can't seem to be able to talk to a human being, therefore I'm coming here to try and get an idea or some path to a resolution to this problem. Before you ask, I do have Owner privileges on both the organization and the project.

Thanks in advance!

[ErnestoC]

A full error log of attempting to delete the instance would be good for this question. You can use the Cloud CLI to attempt to delete the idle instances, and see how it fails (using in conjunction with verbosity flags.)

Adding a reproducible snippet of your Github action would also be very useful in troubleshooting why they seem unable to be removed.

[fredsimard]

Alright, so I created a temporary VM just for this, installed and configured gcloud CLI, then selected the right project, ran gcloud compute instances list and it says there are 0 instances. I'm super confused, I can SEE them when I'm in the project in GCloud. Which one is correct?

[ErnestoC · Accepted solution]

Have you confirmed that these GCE instances are not part of a Managed Instance Group? In such a case, the MIG would need to be deleted before the instances can be removed from your project. Otherwise, the appropriate support channel for this issue would be acquiring a GCP support package, and creating a support case.

This is so that your project can be internally reviewed for the origin of these instances.

[fredsimard]

There are no instance groups in that project, or any other project. I'll see about getting support, but it's likely going to cost us more than what the phantom instances do.

Thanks anyways.