Generative AI applications fail in ways that do not resemble ordinary deterministic software. A request may succeed at HTTP level while returning the wrong source, an invented fact, an unsafe tool action or an answer that changed after a prompt or index release. AWS AIP-C01 testing and troubleshooting expects developers to identify the failing stage and choose evidence-led repairs. This article provides an incident method that separates model, data, application, permission and operational failures.
Capture the defect and localize the failing stage
Capture the user’s expected task, the actual output, the time, tenant scope, request ID, relevant model/prompt/index versions and whether the error is reproducible. Distinguish wrong content, missing evidence, disclosure, tool side effect, timeout, cost spike and formatting. Each needs a different first diagnostic. A chatbot saying ‘done’ is not confirmation a ticket was created. Collect transaction state from the downstream system and review whether the user had authorization to request it. Avoid publishing sensitive raw prompts into an unrestricted incident channel.
Start with the original query and the passages the retriever selected. Were those passages authorized, current and sufficient to answer? If the correct source was absent, inspect document ingestion, chunking, metadata filters, lexical/vector matching and reranking before changing the prompt. If the correct evidence was retrieved but the model ignored it, investigate prompt context ordering, truncation, model behavior and structured output rules. The query evaluation practice gives independent retrieval metrics.
Test releases using versioned datasets and configuration evidence
Keep versioned tests for ordinary requests, exact identifiers, ambiguous wording, conflicting policies, wrong-tenant access, malicious source text and unavailable tools. Score task outcome rather than only response fluency. Mark critical authorization or safety failures as release blockers even when average relevance improves. A golden request should include its expected evidence and disallowed outcomes, not a single exact generated sentence that penalizes harmless wording variation. AWS domain-five objectives cover evaluation and troubleshooting across the whole application.
Compare the exact prompt asset, model identifier, inference parameters, guardrail settings, tool definitions and prior release. A subtle system-message modification may alter output shape or tool choice while the application code remains unchanged. Replay the same safe test set against old and new configurations and look at errors by task class. Do not assume that a larger model will resolve a missing permission check or stale document. When rolling back, restore a compatible set of behavior-shaping assets, not just one prompt string.
Distinguish rate limits, IAM failures and unsafe tool behavior
Trace authentication, API Gateway, queueing, knowledge retrieval, model inference, guardrail processing and tool invocation timestamps. Bedrock throttling may require backoff or capacity changes; an invalid IAM policy needs access repair; a tool timeout after an external write needs state reconciliation. Check model usage and input/output tokens because long outputs can dominate latency. A request may have completed its model response but failed in a downstream workflow. Use bounded retries and circuit breakers so diagnosis does not worsen cost or availability.
Review the sequence of tool proposals, validation decisions, approvals, executed calls and committed results. The model may request repeated search because the first response was malformed, or repeatedly attempt a tool it cannot access. Enforce a maximum step budget and inspect whether a loop came from insufficient tool descriptions, wrong state handling or a blocked action. Correct the authorization or workflow logic, not simply instruct the model to be more careful. Test that untrusted tool output cannot inject higher-priority instructions.
Detect stale retrieval data and evaluate safety interventions
A knowledge base may return fluent answers long after ingestion stopped. Check source modification time, ingestion-job outcome, index build/version, metadata filters, document deletion and cache invalidation. Compare a known-current source to the passages actually supplied to generation. If a confidential file was revoked but remains in vector search, treat it as a privacy incident until propagation is verified. A model swap cannot fix this data-plane defect. Keep corpus ownership and deletion evidence with the query trace so the team can reproduce the failure.
Bedrock Guardrails and application filters may block harmful content, personal data or prompt attacks, but they can also deny legitimate tasks. Track the configured guardrail version, intervention type, relevant inputs and the approved policy. Use restricted evidence where unsafe text contains personal material. Test near-boundary requests to determine whether a policy is too broad or a model response triggered a valid rule. Do not disable a control in production merely to eliminate support complaints; propose a scoped reviewed change with a regression suite.
Use canaries, rollback and stage-specific incident repair
Promote a candidate only after deterministic software tests, schema checks, retrieval evaluation, safety denial tests and task-level quality review. Observe a small safe traffic cohort for errors, token cost, latency, tool success and source freshness. Keep a known-compatible prior model route, prompt version, index and tool API contract. A release can fail because a newer prompt references a tool schema that the previous application version cannot parse; rolling back one component may deepen the outage. Design reversible dependencies explicitly.
A customer asks whether an order is refundable. The assistant cites an old policy and offers a refund amount that the billing system would reject. The incident responder first checks tenant identity, retrieved source IDs and effective dates, then index version and metadata filtering. If stale content entered context, correct the corpus and sync; if current content was present but ignored, review prompt and model behavior. Confirm that no unauthorized refund action executed before declaring recovery. Run the same case through the candidate and prior release to verify the correction.
A useful incident record includes what failed, user impact, source and code/config versions, root-cause evidence, containment, rollback or repair, owner and tests added to prevent repetition. Measure time to explain the defect, not just time to restart the service. Link alert thresholds to real task quality and resource limits; an HTTP 200 rate alone cannot reveal wrong-customer answers. Feed learnings into prompt review, index maintenance, tool contracts and capacity planning. Distinguish hypothetical examples from tests actually executed in an AWS account.
Reconcile committed tool state and prevent recurrence
Introduce a second failure where the model-generated ticket reference is correct, but the downstream service already processed the request and its reply was lost. Look for the idempotency key and committed transaction status. Repeating the action because the model says ‘try again’ can create duplicate records. If the tool used an overly broad role, treat that as a control incident rather than a formatting bug. Separate user-facing recovery from the forensic trace so private case notes are not copied into ordinary dashboards.
Close the incident by adding a regression case that fails on the old configuration and passes on the corrected candidate. Run it alongside unrelated safety and privacy tests to guard against accidental weakening of the system. Preserve model and prompt versions, source ingestion state, route policy and deployment revision as rollback evidence. A postmortem should state who owned the failure, the measurable impact and the specific future alert that would find recurrence earlier. It should never confuse a simulated run with a verified production outcome.
GenAI troubleshooting improves when operators preserve the chain from user intent to source evidence, model route, tool execution and persisted result. A reliable fix targets the failing stage and proves the repaired application still meets authorization, quality, cost and recovery requirements.