1A user sends a requestYour app calls its normal AI model.
2The model cannot answer nowThe provider returns an error such as 429 or 503.
3Your app chooses one safe actionRetry, use an approved backup, or stop.
The goal is not “answer at any cost.” The goal is to stay available
without changing safety, data, output, or tool rules.
Lesson 1 · the three choices
Every failed request ends with one action
1 · Wait and retry
Try the same model again after a short, increasing delay.
Use when the problem looks temporary and the request still has time left.
2 · Use an approved backup
Send the request to a backup model or location that was approved before the outage.
Use when retrying is no longer sensible and every safety check still passes.
3 · Stop safely
Return a controlled error instead of guessing or weakening a rule.
Use when the request is invalid, access is denied, or the backup is not proven safe.
Simple exampleA model is briefly busy. Wait 1 second and retry. If the retry budget is used up, switch only to a pre-approved backup. If no safe backup exists, stop.
Lesson 2 · the decision
Use the same three questions for every provider
01
What failed?
Temporary capacity, an invalid request, missing access, or a safety block?
→
02
Can we retry?
Check how many attempts and how much user waiting time remain.
→
03
Is the backup safe?
If retry is over, verify the backup. Otherwise stop.
The model or cloud may change. The safety policy, expected output, approved data
locations, and allowed tools must stay the same.
429Too many requests. Slow down and respect the provider’s wait signal.
503The service is temporarily unavailable or has insufficient capacity.
Retry budgetA hard limit on attempts and total waiting time.
FallbackA backup chosen and tested before the incident.
Circuit breakerA temporary pause that prevents repeated calls to a failing service.
Fail closedStop when safety or authorization is uncertain.
Lesson 3 · provider versus application
Cloud routing helps, but your application still owns the final decision
Providers offer different capacity, endpoint, and routing features. They can reduce failures, but they do not automatically preserve your application’s safety contract or recover its databases, tools, and other dependencies.
ProviderManaged helpYour application still owns
AWS BedrockGeographic or global cross-Region inference; provisioned optionsRetry limits, approved geographies, backup policy, and dependent services
Google Vertex AIGlobal endpoints and SDK retries for transient 408, 429, and 5xx errorsLatency budget, regional fallback, idempotency, and safety parity
Microsoft FoundryDeployment types, model router, and multi-Region architecture guidanceRecovery orchestration and every customer-managed stateful dependency
OCI Generative AIOn-demand or dedicated serving; dynamic throttling guidanceBackoff, alternate endpoint planning, and compatibility checks
IBM watsonx.aiConfigurable SDK retries for 429, 503, 504, and 520End-to-end budget, backup approval, and workload-specific limits
Important: error meanings and SDK defaults differ by provider and can change. Treat provider documentation as input to your policy, not as the policy itself.
Practice 1 · recognize the problem
Decide whether to retry, fix the request, or stop safely
Read one case at a time. The explanation appears immediately after your choice.
HTTP 429
Case 1 of 8 · choose the response
Quota or rate-limit pressure
Progress saves locally.
Lesson 4 · prevent retry storms
A circuit breaker works like a fuse for a failing model
It begins closed, so calls may pass. Repeated failures open it and pause calls. Later, one half-open test checks whether the provider recovered.
Failure503 retryable
Attempts2 / 3 used
Elapsed3.7 / 5.0 s
Next delay1.24 s fits?
1
Closed
503 #1
Retry with jitter
2
Closed
503 #2
Spend final retry
3
Open
Threshold hit
Stop primary calls
4
Open
Timer
Protect dependencies
5
Half-open
Probe only
Test recovery safely
6
Closed
Probe succeeds
Start a new budget
Closed
Retry with jitter
Lesson 5 · approve backups before an outage
A backup must pass every check, or the app stops
BackupModelLocationTools
A · summarizer-b✓✓✓
B · model-x×✓✓
C · summarizer-b✓×✓
D · summarizer-b✓✓×
Same jobThe backup accepts the same input and returns the expected output shape.
Same safety and data rulesThe backup keeps required controls and processes data only in approved locations.
Explicit approvalThe model, location, and tools must appear on the allowlist.
No duplicate side effectsIf a tool may already have acted, an idempotency key or execution record must prevent a duplicate.
Candidate A may receive traffic. B, C, and D stop with a reason.
Lesson 6 · make decisions explainable
Record what happened and why the router chose its action
stopfail-fast, fail-closed, unsafe target, or unsupported state
A reason is mandatory on retry, fallback, and stop.
Guided example
Choose the final action for a realistic outage
The app tried the primary model twice. A later 429 asks it to wait 45 seconds, but the user-facing request may wait only 5 seconds. A fully approved backup is available.