How to debug AWS internal errors when updating the cluster?

[ivan-aracki]

During many different update commands for an Anthos cluster I'm getting following output:

Updating cluster [anthos-ireland]...⠛ERROR: Internal error encountered.
Updating cluster [anthos-ireland]... Cluster is being health-checked...failed.
ERROR: (gcloud.container.aws.clusters.update) Internal error encountered.

When I describe the mentioned operation with "gcloud container aws operations describe ID" I'm getting this:

done: true
error:
code: 13
message: Internal error encountered.
metadata:
'@type': type.googleapis.com/google.cloud.gkemulticloud.v1.OperationMetadata
createTime: '2022-05-23T11:02:02.366522Z'
endTime: '2022-05-23T11:37:11.360113Z'
errorDetail: Internal error encountered.
statusDetail: Cluster is being health-checked

How can we debug those internal errors and what that code cipher mean?

[bkauf · Accepted solution]

Hi Ivan, can you check the anthos. cluster logs in GCP cloud operations logging for clues? Also the following AWS command might provide some insight into whats happening with the EC2 instances

aws ec2 get-console-output --instance-id=[INSTANCE_ID]

[ivan-aracki]

Thanks @bkauf , actually

aws ec2 get-console-output --instance-id=[ID] --latest --output text

will be really useful to use. Regarding the cluster logs, there was nothing extraordinary during those issues.

Thanks. 🙂