Delete already-assigned subnets from VPC Private Service Connect producer project?

[roaldvanloon]

Hi there,

I have a weird use case. This is what I did;

1. I've enabled Private Service Connect in order to try AlloyDB.
2. I created an IP reservation for 10.1.0.0/20
3. I created the PSC connection 'servicenetworking-googleapis-com' and assigned the 10.1.0.0/20 range
4. I created an AlloyDB instance in europe-west4, it got assigned IP 10.1.0.9.
5. The PSC connection now exported the route 10.1.0.0/24 (for europe-west4), as expected this /24 block was selected for this region from my /20 assignment.

So far, all as expected. However at this point I discovered that the 10.1.0.0/24 range conflicts with a range on-premises (that was not yet advertised to GCP), and I would like to connect to this AlloyDB instance from an instance on-premise that is in the conflicting IP range. So I deleted all of the above;

1. Deleted the AlloyDB instance
2. Deleted the PSC connection (the AlloyDB was the only thing using PSC in this project)

Next I modified the IP reservation, from 10.1.0.0/20 to 10.8.0.0/20, and assigned it to PSC.

Then the weirdness began;

1. I created an AlloyDB instance in europe-west4, it got assigned IP 10.1.0.14 (in the "old" range!).
2. I created an AlloyDB instance in europe-west3, it got assigned IP 10.8.1.12 (in a new /24 block from the "new" range).

Apparently, this 10.1.0.0/24 block is fixed. I tried everything; deleting everything, even disabling Service Management API. Nothing works. Also disabling PSC, exporting the route from on-premise to GCP, and then re-enabling PSC ("An IP range in the peer network (10.1.0.0/24) overlaps with an IP range in the local network (10.1.0.0/24) allocated by resource ....").

From the looks of it, this project seems doomed because I can't ever use Private Service Connect products (AlloyDB, but also Cloud SQL) anymore because of the conflicting IP range in the producer project. I can't imagine there isn't a way to manage these subnets in the producer project, but I can't seem to find out why.

Who can help?

[alexmoore]

One thought, when you removed all the configuration, did you also remove the allocated IP range?

You can check with:

gcloud compute addresses list --filter=purpose:VPC_PEERING

[roaldvanloon]

Hi,

Yes I removed the allocated IP range from the project. Even removed the VPC peering with the producer project itself. 

I think the issue is with the producer project (which is under Google's control); once a /24 block is claimed for a region from the reservation, it probably creates a subnet for that region using that block, which is never cleaned up. Even though it's not used anymore. 

[alexmoore]

Were you establishing the PSC via the AlloyDB console?  or some other mechanism?

Can you try deleting any remaining VPC peering, PSC references to "servicenetworking-googleapis-com", then creating the IP address range and the PSC connection for "Google Cloud Platform" in the VPC UI first, then head over to AlloyDB and deploy a cluster - when you pick the VPC it should just give a green tick that it is usable.

Let me know if that works?

[roaldvanloon]

Hi,

Thanks for taking the time to respond.

I've tried each three; created the PSC via the VPC UI, created it via AlloyDB console, created it via gcloud. Each time in between, I completely removed any VPC peerings and any PSC references (including the IP reservation and even disabling servicenetworking API). It still always keeps using the previous "old" IP block for the specific regions that used it in the past (first time I deploy something in a new region, it does take a /24 from the new reservation).

Additional issue I have now is that this "old" IP block is advertised from on-premises VPN. So I can't even enable PSC anymore because the servicenetworking-googleapis-com VPC peering can't be created (conflicting IP block), which basically means I can't ever use Memorystore, AlloyDB etc anymore in this project as long as the PSC producer project tries to export this "old" IP block that is conflicting.

[alexmoore]

Strange indeed.

When you create the service networking peering via the VPC interface, does it report back the new IP range?

Also, are there other services running in the project that are leveraging the peering? you mention memorystore for example.

[roaldvanloon]

If I now try to create the new service networking peering via the VPC interface, it reports back an error that the 10.1.0.0/24 range in the producer networking conflicts with the range 10.1.0.0/24 that's exported via VPN from on-premises;

"An IP range in the peer network (10.1.0.0/24) overlaps with an IP range in the local network (10.1.0.0/24) allocated by resource ...."

There is currently nothing running that is using the peering (no Memorystore).

[maitas]

I had a similar issue. After deleting the PSC, I run:
gcloud services vpc-peerings update \ --network=VPC_NETWORK_NAME \ --ranges=ALLOCATED_RANGES \ --service=servicenetworking.googleapis.com \ --force

To force GCP to clean the range and it work. This is an old thread, I know, but it might help other users.
From CloudSQL
https://cloud.google.com/sql/docs/postgres/configure-private-services-access

[netog]

I found this today after Googling for this issue.
I have tried this gcloud command with --force but that didn't do anything for me. The range was already released and the peering gone away.

The only thing I can see is there we have a few VMs which are in theory using those routes but they are in completely different subnets and were never in my psc ranges to start with so I'm confused.

I have opened a case with Google to see where this is going wrong.