Re: "Internal error encountered" "code: 13"

rodrigorobert · 12-26-2022 12:42 PM

Hello friends, I would like to ask you a question.
I am working with DocumentIA, and after uploading the documents and tagging them, when training the model, I got the following error:

error
code: 13
message: "Internal error encountered,
details: []

As you can see there are no details. I didn't find anything in the documentation either.

Can anyone tell me what that means and how to fix it.

Thank you very much!

anjelab

It will be hard on my end to reproduce since I don't have access on your project and there are no additional details in your error. I suggest to have a 1:1 Support. An engineer that has access to your project and will be dedicated on your case can investigate more thoroughly.

glarrain-cdd

@anjelab but for that a paid support plan is required. Is it not possible to get free support from GCP to work around a bug for which GCP itself is responsible?

glarrain-cdd

I have the same error. It is very unfortunate to receive such useless error messages.

{
"name": "projects/570***756/locations/us/operations/3597173263047540318",
"done": true,
"result": "error",
"response": {},
"metadata": {
"@type": "type.googleapis.com/google.cloud.documentai.uiv1beta3.TrainProcessorVersionMetadata",
"commonMetadata": {
"state": "FAILED",
"createTime": "2023-01-19T13:40:54.538320Z",
"updateTime": "2023-01-19T13:43:25.142621Z",
"resource": "projects/570***756/locations/us/processors/4c3efa0d70e12950/processorVersions/31ebba062b7b3bd6"
},
"trainingDatasetValidation": {},
"testDatasetValidation": {}
},
"error": {
"code": 13,
"message": "Internal error encountered.",
"details": []
}
}

kolban

Howdy ... I'm wondering if there might be additional diagnostics written to Cloud Logging? Maybe have a look in your Cloud Logging through the Google Cloud console and see if any errors are written/logged at the time of the API request error.

glarrain-cdd

Well, a retry a few hours later worked like a charm. There were no changes between the failed and successful executions. 😭

rodrigorobert

In our case, we discovered by chance, and not thanks to the information provided by GCP, that the problem was in the 'date' field. When we classified the date field as text, error 13 disappeared.

Weinstein86

We are having the same issue, worked fine for training as of a week ago and now failing every training attempt with error 13. No explanation or assistance on the error, what can be done?

Weinstein86

In the logs, I am seeing this: Task [4563861179629305857] was already retried over max attempts: 0"

rodrigorobert

Remove fields that are 'date'. We did this and it gave no more errors.

[image: Zentricx Logo]
[image: Instagram] <>
[image: Facebook] <>
[image: Linkedin] <>
[image: Web] <>
[image: Youtube] <>

[image: Grate][image: Cloud]
*Rodrigo Robert*
| Data Scientist, Zentricx
+ (PII Removed by Staff)
+ (PII Removed by Staff)
(PII Removed by Staff)
*Linking People to** CX*

*CONFIDENCIALIDAD: Este mensaje es privado y los archivos adjuntos al mismo
son confidenciales y dirigidos exclusivamente a los destinatarios de los
mismos.Por favor, si Ud. no es uno de dichos destinatarios, sírvase
notificarnos este hecho y no copie o revele su contenido a
terceros.CONFIDENTIALITY: This message and any attached files are
confidential and intended only for the named recipients.If you are not one
of them, please, notify us this fact and do not disclose its contents to
another person or take any copies.*

caiolagana

Dear colleagues

I am getting the same useless error "13" when trying to train a new model. Does anyone know how to get more information on the cause of the error?

"trainingDatasetValidation": {},

"testDatasetValidation": {}

},

"error": {

"code": 13,

"message": "Internal error encountered.",

"details": []

}

rodrigorobert

In my case, the problem was with the extraction of the date field, GCP
supports only some date formats.

[image: Zentricx Logo]
[image: Instagram] <>
[image: Facebook] <>
[image: Linkedin] <>
[image: Web] <>
[image: Youtube] <>

[image: Grate][image: Cloud]
*Rodrigo Robert*
| Data Scientist, Zentricx
+(PII Removed by Staff)
+(PII Removed by Staff)
(PII removed by staff)
*Linking People to** CX*

*CONFIDENCIALIDAD: Este mensaje es privado y los archivos adjuntos al mismo
son confidenciales y dirigidos exclusivamente a los destinatarios de los
mismos.Por favor, si Ud. no es uno de dichos destinatarios, sírvase
notificarnos este hecho y no copie o revele su contenido a
terceros.CONFIDENTIALITY: This message and any attached files are
confidential and intended only for the named recipients.If you are not one
of them, please, notify us this fact and do not disclose its contents to
another person or take any copies.*

caiolagana

Dear rodrigorobert

Thanks for your reply. Following your suggestion, I have already replaced all "Data" fields by "Text" field, without success. Most likely, the problem is in the labeling of some specific document. I removed all documents from Training and Test sets, and will re-insert in sets of 10, until the problematic document shows up. This is a stupid way to debug but unfortunately the only available so far. Thats why I would like to know some way of retrieving log errors. Thanks again for your help!

rodrigorobert

Dear, thank you for your reply as well. Error '13' says nothing, it leaves
us no other less stupid ways. I hope your problem is solved, good luck with
that!

[image: Zentricx Logo]
[image: Instagram] <>
[image: Facebook] <>
[image: Linkedin] <>
[image: Web] <>
[image: Youtube] <>

[image: Grate][image: Cloud]
*Rodrigo Robert*
| Data Scientist, Zentricx
(PII Removed by Staff)
(PII Removed by Staff)
(PII Removed by Staff)
*Linking People to** CX*

*CONFIDENCIALIDAD: Este mensaje es privado y los archivos adjuntos al mismo
son confidenciales y dirigidos exclusivamente a los destinatarios de los
mismos.Por favor, si Ud. no es uno de dichos destinatarios, sírvase
notificarnos este hecho y no copie o revele su contenido a
terceros.CONFIDENTIALITY: This message and any attached files are
confidential and intended only for the named recipients.If you are not one
of them, please, notify us this fact and do not disclose its contents to
another person or take any copies.*

aacardona

I had the same error (see below from 'Vertex Pipelines Job logs') and in my case, after trying a multitude of things, I discovered that it was related to a lack of user rights to read/write to the GCS of the bucket. I did grant 'Storage Admin' to the user running the project (at the project level but I guess this should also work at the bucket level) and then the tuning pipeline job ran as expected. It was a trial and error to find this so I encourage you to have a look at user roles at the project and bucket level to see if that works.

error: {

code: 13

message: "Internal error encountered. Please try again"

}

rodrigorobert

Thank you very much dear for joining the thread and giving your contribution, without a doubt I will consider it in future opportunities. As you say, most of the time it's trial and error. Kind regards!

carohe

Dear all, I am still getting this error with code 13. I did all the things suggested here. It did not manage to start even one pipeline in VertexAI "fine tune" model. It is frustrating

carohe

Like glarrain-cdd suddenly it started working without changing anything. But then I ran into new problems. My quotas were set at zero for NVIDIA A100 80GB GPUs...

kishantripathi

I am getting an internal error encountered while starting the fine-tuning job on the cloud console. I have tried all the ways suggested but still facing the same error. Can you please help me to get rid of this error?