LLM Foundation Model Tuning Failed at Prompt Validation

[bl21812]

Hello, 

I have been trying to finetune text-bison (002) with a custom dataset using the cloud console, but keep running into the issue shown below at the vertex-pipelines-prompt-validation step. I've tried giving the vertex service agent (service-<PROJECT_ID>@gcp-sa-aiplatform.iam.gserviceaccount.com) storage admin and object admin roles as well, but this issue still persists. Does anyone know what might be causing this?

Error:

service-<PROJECT_ID>@gcp-sa-aiplatform.iam.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).; Failed to read the gcs file.; Failed to create external task or refresh its state. Task:Project number: <PROJECT_ID>, Job id: <JOB_ID>, Task id: <TASK_ID>, Task name: vertex-pipelines-prompt-validation, Task state: DRIVER_SUCCEEDED, Execution name: projects/<PROJECT_ID>/locations/us-central1/metadataStores/default/executions/<JOB_ID>; Failed to handle the pipeline task. Task: Project number: <PROJECT_ID>, Job id: <JOB_ID>, Task id: <TASK_ID>, Task name: vertex-pipelines-prompt-validation, Task state: DRIVER_SUCCEEDED, Execution name: projects/<PROJECT_ID>/locations/us-central1/metadataStores/default/executions/<JOB_ID>

Thanks,

Brian

[danielazamora · Accepted solution]

Hi @bl21812! I believe the GCP engineering team fixed the bug. But you can confirm by rerunning the job from the console or submitting a new one.

[bl21812]

Thanks @danielazamora, can confirm that it seems to be working fine now!