This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Log in to ask a question

Mutating deployed model using mutateDeployedModel

Posted 4 weeks ago
anillspawar
Bronze 3
Bronze 3
Reply posted on --/--/---- --:-- AM

I am using vertex AI online prediction with a custom container. To save on autoscaling time, I am using mutateDeployedModel API(https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.endpoints/mutateDeploye...)

 The strange issue that I am facing is, I am able to change minimum replicas a few times and then it stops working. 

The API call was successful with replica_target got increased but actual replica count coun't increase as shown in below screenshot:

scaling_failed_0305.png

 

 

anillspawar_0-1714736141344.png

 

  • Is there any limit on the number of times, I can mutate a deployed model?
  • Any other way of changing the nodes of the deployed model

Thanks in advance!

Regards,

Anil

 

 

 

6 1 96
1 REPLY 1

Posted on --/--/---- --:-- AM
nceniza
Staff
Reply posted on --/--/---- --:-- AM

It is possible that it took so many request that it is timing out for changing replica count. Also I believe that replica count varies to the actual resource available at that moment, it is not reaching the target due to availability of the resource. Even then it will still try to reach its target. 

 

I would recommend to file a support case here as they have a better view for your logs and  resource usage.