Sporadic Service Unavailable problem

We are using xml-rpc with OAuth2 bear token. Starting from UTC 8/22/24 8:55 PM, there have been some exceptions “Service Unavailable” when sending request to endpoint https://api.infusionsoft.com/crm/xmlrpc. It happens sporadically. Does anyone see this error as well?

1 Like

Same here. Started yesterday afternoon, its not consistent, but is DEF noticable.

This is almost always the tenant (app) you are accessing is down or struggling. You can also get this sometimes if you happen to make a request right when we are deploying new versions.

1 Like

Its been happening for a full day at this point, i can get to our app just fine, is there something that I can check on our app to see if that is the cause?

@bradb, I can also confirm that this occurred on querying the “Contact” and “ContactGroup” table on a particular tenant at different intervals.

ERROR: 5 - Didn’t receive 200 OK from remote server (HTTP/1.1 503 Service Unavailable)

So something has been occurring today.

1 Like

Can someone from Keap or Infusionsoft please jump in on this thread. I’m on with chat support and I’m unbelievably frustrated at this point.

I am already here :slight_smile:

1 Like

Is this something that you can assist with? I’m being requested to provide things that I’ll need to make production changes for via the support chat.

Thank you btw, just frustrated…

Can you be more specific on what you are needing? If support is asking for additional data, i will likely be asking for it as well.

Just want to know what is wrong and how to resolve the issue with the 503 response. Support said that they need full request information. You suggested that its an app issue, just looking to get this resolved.

Heres the case number if that helps : Here’s our case number: 03360362.

503 are expected from time to time. Our tenants are deployed to shared VMs, so a noisy neighbor can cause 503s for all tenants on that VM or your tenant might be the noisy one. Also if you have request in flight when a VM is shutdown you will get a 503. It can be difficult to track down the exact reason a 503 occurs, but you should be retrying requests when they error with an exponential backoff. Can you give more details how often you are getting these 503s, rough time frame, and which tenant?

We’re on tenant ef161.
We do have exponential back off setup ( up to 5 retries not including the initial request ).
Here’s the timestamps of them ( there are a TON of them compared to yesterday morning ).
( exporting the timestamps for the last few hours of the errors now, will update once it completes )

Here is a list of the timestamps:
receiveTimestamp
2024-08-23T18:37:33.527378240Z
2024-08-23T18:37:32.199113618Z
2024-08-23T18:37:32.421644082Z
2024-08-23T17:59:23.748982244Z
2024-08-23T17:59:22.746053121Z
2024-08-23T17:43:10.955451589Z
2024-08-23T17:43:09.614104924Z
2024-08-23T17:43:09.667202205Z
2024-08-23T16:49:32.681631819Z
2024-08-23T16:49:31.682649079Z
2024-08-23T16:42:42.622808519Z
2024-08-23T16:42:41.622312340Z
2024-08-23T16:32:00.671298044Z
2024-08-23T16:31:59.320833935Z
2024-08-23T16:31:59.613810587Z
2024-08-23T16:30:07.734180913Z
2024-08-23T16:30:06.408495829Z
2024-08-23T16:30:06.683899707Z
2024-08-23T16:00:44.026964732Z
2024-08-23T16:00:42.696170856Z
2024-08-23T16:00:42.959637935Z
2024-08-23T15:57:36.511417817Z
2024-08-23T15:57:35.511696013Z
2024-08-23T15:47:37.238177852Z
2024-08-23T15:47:36.238523155Z
2024-08-23T15:47:36.260975763Z

I can see an increase starting yesterday around 19:00 UTC.

yeah, that sounds about right for sure, def started yesterday afternoon.

I am taking a look to see what was going with that VM.

1 Like

Awesome, Thank you Brad!!!

There were a few very large spikes of traffic to that VM from a few different tenants during that timeframe.

OK, would that still be causing issues today?

I am still seeing occasional spikes.

1 Like