The answer is only as good as the evidence retrieved
A support assistant can use a powerful language model and still give the wrong warranty policy if retrieval returns the wrong revision. Retrieval-augmented generation, or RAG, helps a model ground its answer in external content, but it creates an additional engineering system: source ingestion, document parsing, chunking, embeddings, index schemas, retrieval, authorization, reranking, evidence packaging, and answer verification. Microsoft AI-103 expects developers to make these choices, not simply attach a search service to an agent.
Use a controlled dataset: a product handbook, two versions of a return policy, a confidential supervisor exception, a scanned invoice, and a troubleshooting article with the same product code but a different model year. Give each source a stable identity, effective date, owner, sensitivity label, and ingestion timestamp. Write questions that require exact code lookup, paraphrased concept matching, and a cross-document comparison. This makes it possible to identify which stage lost essential evidence.
Success means more than receiving an answer. The assistant must cite an accessible, current source; reject unsupported claims; distinguish a policy explanation from an approved return; and make a safe decision when evidence conflicts.
An index designed only to make a search demonstration look relevant is insufficient when the same content will support user-specific retrieval and downstream agent decisions. Preserve access labels and source revision metadata throughout ingestion, ranking and answer generation so that the application can prove why the selected evidence was eligible.
Prepare sources for searchable, versioned evidence
An indexer cannot repair a document collection that has poor ownership metadata or no version control. Preserve original file IDs, document titles, modification dates, ACL-related fields, and source pointers. Normalize character encodings and recognize headings, tables, captions, and attached notes. If a contract’s exception appears in a footnote, a chunking strategy that discards footnotes may produce an incorrect answer even though every API call succeeds.
Chunk text at boundaries that preserve meaning. A naive fixed-size window can separate a price limit from its regional qualifier. Overlapping chunks may protect local context but create duplication and extra embedding cost. For regulatory documents, use section-aware splitting and retain parent-child links; for short policies, a whole-section chunk may be sufficient. Test the chunk strategy with questions that refer to information at boundaries and in tables.
Decide whether embeddings are created outside search or by integrated vectorization. Ensure the embedding dimensions, model, and vector-field configuration are compatible. Keep the text used for keyword search alongside embedding fields where hybrid retrieval is needed, and retain filterable metadata such as department and effective date. Avoid re-embedding documents on every query; changes in embeddings should have their own index-version and reindexing plan.
Document extraction is relevant at ingestion. OCR, layout analysis, and Content Understanding can turn scanned pages and tables into structured representations, but source provenance must survive that conversion. It should not allow an extractor’s summary to be mistaken for the original contract clause.
Compare keyword, vector, and hybrid retrieval
Keyword search shines when the question contains an exact warranty SKU, a legal clause number, or a product identifier. Vector search helps when the user says “my laptop stopped charging” but the document discusses “power delivery faults.” Neither method is universally superior. Azure AI Search supports both searchable text and vector fields, so the same corpus can be used to compare approaches under identical access and freshness constraints.
A hybrid request combines full-text and vector queries. Microsoft describes parallel execution and fusion of ranked lists, commonly through Reciprocal Rank Fusion. Optional semantic ranker further evaluates the candidate results. The relevant hybrid search documentation distinguishes retrieval from the later generative answer. Semantic ranker can improve ordering among retrieved documents; it does not automatically fetch a missing document that never made it into the candidate set.
Measure retrieval separately from generation. For 30 representative questions, label which document sections constitute the correct evidence. Track whether they appear in the top candidates, where they rank, how many tokens are required to include the key passage, and whether irrelevant near-matches displace the source. Include exact identifiers, synonyms, acronyms, misspellings, and two languages where the application supports them. A great final answer on one sample cannot replace these tests.
Do not treat vector similarity as a permission or truth signal. A close embedding match can lead to a stale bulletin or an unrelated department’s exception. Apply source filtering and effective-date rules, inspect the document identity, and use application logic for authoritative rules. If two active policies disagree, preserve the disagreement and escalate rather than selecting one because it scores a few points higher.
Enforce permissions before content reaches the model
The most dangerous retrieval failure is serving an employee data they should not see. Restrict search results by user identity and content permissions at query time. Microsoft describes document-level trimming and permission-aware knowledge sources, but exact support differs across API versions and source types. An application must validate identity propagation and source entitlements instead of assuming a private network or the presence of an ACL field guarantees enforcement.
Use three test users: an employee, a departmental supervisor, and a support administrator. Ask each the same question about a confidential exception. Inspect the raw retrieval results, not only the final user-facing answer. If unauthorized data appears in the retrieved context and the model merely avoids quoting it, the boundary has already failed. The correction belongs in ingestion metadata, query filters, or identity integration.
Separate the application’s service credential from the end-user authorization context. Permission-sensitive search may require both. Do not use one administrator credential to retrieve all documents and then tell the model to redact invisible information. Keep returned source IDs and access-filter decisions in an audit trace without unnecessarily logging full private documents. Use least-privileged search index access for service identities.
Test change propagation: revoke a user’s access and query again; add a confidential document; and update a public policy to a new version. The expected visibility rules should remain effective during indexing and cache refresh. When permission changes are eventually consistent, define a safe policy for delayed propagation and avoid claiming immediate revocation unless the underlying service supports it.
Choose classic RAG or agentic retrieval deliberately
Classic RAG is straightforward: the application sends a query to search, receives relevant passages, and passes bounded evidence to a model for answer generation. This is often appropriate for predictable questions and deployments requiring generally available features. You control query construction, filters, retries, citation handling, and the evidence-to-prompt boundary, which simplifies debugging.
Microsoft’s newer agentic retrieval architecture introduces knowledge sources and knowledge bases. Depending on API version, features can include multi-query planning, activity logs, answer synthesis, and structured responses. In September 2026 documentation, the generally available retrieval API and preview LLM-planning features have different status. Do not assume the portal’s preview configuration is covered by the same production support promises as a stable classic-search API.
An example of useful decomposition is “Can a remote employee replace a 2024 laptop, and which manager must approve?” The question combines employment policy, hardware age, and delegation rules. A planned series of narrower searches may recover more evidence than one broad query. But if the normal workload is exact warranty lookup, the added planning call can increase cost and latency without improving correctness.
Compare the two patterns on the same complex questions. Include model-call overhead, number of searches, evidence coverage, citation correctness, permission filtering, and the fraction of answers that still require review. Choose a retrieval path based on measurable utility and risk. Feature novelty is not itself an architecture reason.
Connect retrieval to answer generation and agents
Keep retrieved content explicitly separate from instructions. A policy document may contain a sentence that appears to direct the assistant; it has authority as policy evidence only within its subject, not authority to change model goals or call tools. Limit source size, label document provenance, and require that generated factual claims have an attributable passage. A citation to a generally relevant page is insufficient when the answer depends on a particular exception.
When an agent uses search as a tool, the tool result should carry structured source IDs, freshness metadata, access outcome, and a clear error shape. A timeout is not “no policy exists”; a forbidden result is not the same as an empty index; a partial ingestion is not a successful complete search. The Foundry agent exercise connects those distinctions to safe ticket proposals and human approval.
Evaluate the generated answer on supported facts, correct abstention, and citation fidelity. Create an example where a retired policy says 60 days but the active policy says 30; the answer must choose the correct effective rule and mention why the older material is excluded if relevant. For a supervisor-only exception, an ordinary employee should receive the public rule without exposure to confidential criteria.
Remember that indexing and retrieval are continuously changing systems. Add monitoring for failed ingestion, stuck jobs, search latency, index size, relevance drift, embedding changes, and permission mismatches. A stable model deployment can degrade as the document collection changes. That makes operational ownership of the knowledge pipeline an ongoing responsibility rather than a one-time setup step.
A reproducible RAG evaluation lab
Build a 30-question benchmark with the expected source revision and access level for each question. Divide it into exact lookup, concept matching, combined-criteria questions, denied access, stale documents, and absent evidence. Run keyword-only, vector-only, hybrid, and optionally hybrid-plus-semantic configurations against the same index snapshot. Record recall, position of correct evidence, query latency, cost, and failure categories. Change only one retrieval variable at a time.
For a complete end-to-end test, ask the assistant to decide whether a specific broken laptop can be replaced. The correct path authenticates the employee, retrieves current policy and product information, explains the decision with citations, and offers a ticket proposal only if authorized. The answer should not assert that a replacement has been ordered. Interrupt search or place malicious text inside a retrieved bulletin to prove the system fails safely.
Finally, write a source-and-index operations note: what gets reindexed when a policy changes, how you identify which answers used an old revision, how you revoke access, what telemetry reveals low retrieval quality, and which API features are GA or preview. Those questions turn a plausible RAG demo into an AI-103-quality engineering exercise and a maintainable knowledge application.