Status = StatusCode.INTERNAL error

Help & Support
1

Hello,

When using the API to run predictions, I recently started receiving the following error:

<_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = “Internal error encountered.”
debug_error_string = “UNKNOWN:Error received from peer ipv4:216.239.32.223:443 {grpc_message:“Internal error encountered.”, grpc_status:13}”

Is the cause for this known? I never received this error in the past.

Any help is appreciated!

2

Hi @adevarajan,

Thanks so much for the report! It looks like your requests were hitting a bad machine, which we’ve now taken out of circulation. Hopefully you won’t get this error anymore, though please do let us know if this happens again!

Thanks

Tom

3

Hi Tom!

Thanks so much!

Since the fix, I haven’t run into the error. Will update if I face any issues.

Best,

Archit

4

Hi @tward ,

Ran into the same issue again:

<_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = “Internal error encountered.”
debug_error_string = “UNKNOWN:Error received from peer ipv4:216.239.38.223:443 {grpc_status:13, grpc_message:“Internal error encountered.”}”>

5

Thanks for reporting again, yes we also noticed this rear up and are investigating more thoroughly now.

6

Hi,

I have also been hitting this occasionally on the ISM API endpoint. I tried to see if there is a pattern associated with when this occurs but it appears to be relatively random. Since this thread dates to about a week ago, I wanted to check in to see if the issue is now known or if it’s still expected to hit these. Thanks!

Best,

Valeh

7

Hi all,

We rolled out a fix earlier this week, which we hope has resolved the issue. We believe the problem was a rare race condition that caused an unrecoverable error, which we initially attributed to a hardware failure. We’ve had ~48hrs without any further errors, so :crossed_fingers: this is now more permanently fixed.

Thanks!