CAREERS & STRATEGY / TECHNICAL EXPLAINER

How To Build AI Solutions On Azure: Certification Exam Preparation

Building an AI application on Azure in October 2026 requires a current technical plan, not a recycled study path for the retired AI-102 exam. Microsoft […]

In this article

Building an AI application on Azure in October 2026 requires a current technical plan, not a recycled study path for the retired AI-102 exam. Microsoft retired Azure AI Engineer Associate (AI-102) on June 30, 2026 and introduced the Azure AI Apps and Agents Developer Associate credential, assessed by AI-103. The AI-103 assessment emphasizes Python, Microsoft Foundry, generative and agentic solutions, multimodal services, text processing, information extraction, security and operations. Those are useful engineering categories whether or not the reader takes the exam.

The project in this article is a controlled document assistant. It should answer from approved knowledge, explain its evidence, decline unsupported requests and create a draft action only when a user authorizes it. The goal is an end-to-end implementation plan that can be tested against real failure cases.

Define the minimum valuable product

Choose a narrow source collection, such as a team's approved maintenance procedures. Define who may use the application, what questions it can answer, which document versions are authoritative and what it must never do. Write acceptance examples before selecting services: a user asks a supported question and receives a grounded answer; asks about a withdrawn procedure and gets the current replacement; requests restricted information and receives no protected text; or asks something not covered and receives a clear uncertainty response.

Make the product contract measurable. Specify expected latency, maximum operating cost, citation precision and how long the search index may lag source changes. Decide which requests require a human review. A proof of concept that gives an eloquent answer without a source reference is not yet a dependable document assistant.

Separate the initial informational product from actions such as creating tickets. First make retrieval and generation reliable. Only then introduce a tool that can affect another service. This keeps failures easier to isolate and avoids using unnecessary orchestration to solve a simple search problem.

Set up secure Foundry and application access

Treat the Foundry project, model deployment, retrieval service and application as separate permission boundaries. Use managed identities or appropriately scoped service credentials, enforce private or restricted access where required and keep secrets out of source code. Avoid giving an application identity administrative access simply to work around an integration error. Record who can deploy a model, change an index and invoke an agent tool.

Configure quotas, request limits and budget alerts before load testing. If a user can submit an unbounded amount of text or trigger repeated tool calls, the system can become expensive or unstable even without malicious intent. Define input constraints and safe failure behavior for rate limiting. Document the model and deployment versions so testers know which system produced a particular answer.

A typical build uses a Python service that validates requests, invokes the appropriate model or agent, and returns structured responses. Keep the data-access layer independent from the user interface. This makes it easier to test access decisions without requiring a browser or a model to execute every time.

Ingest and retrieve documents without losing meaning

A document pipeline should maintain source identity, version and access policy. Extract text and relevant layout, clean it conservatively and split it so a rule and its exceptions remain interpretable together. Include titles, section identifiers and update dates as metadata. Do not treat retrieval chunks as anonymous strings when users need to verify the evidence.

Evaluate both lexical and semantic retrieval in Azure AI Search. A user may know the exact procedural code, while another describes the task in ordinary language. Hybrid retrieval can address both, but relevance must be measured on real examples. Ask whether the correct document is in the candidate set, whether the relevant section ranks highly and whether the index exposes the right version.

Security filters must apply during retrieval. If an employee cannot read a restricted incident report, the search results should not return its contents to a generative model as a supposedly internal passage. Preserve the connection between each result and the permissions used to retrieve it. Treat access-control changes as triggers for reindexing or permission re-evaluation.

Produce answers that can be checked

Provide the model with retrieved evidence and concise behavioral constraints. Make it distinguish source facts from explanation, identify unsupported requests and avoid inventing missing details. A citation should refer to a passage that actually supports the claim, not simply any document retrieved in the same turn. Test what happens when sources contradict each other or an obsolete document ranks above a current one.

Develop evaluation questions with an expected source and response category. Include a question whose answer is absent, a policy exception buried in a longer section, a renamed service and an adversarial prompt inside a retrieved document telling the agent to ignore its rules. The system should follow its own trusted instructions and treat retrieved material as data. Record unsupported or unsafe outputs, then determine whether the failure came from indexing, retrieval, prompting or the model.

Do not measure only user satisfaction or lexical similarity to one preferred answer. Review factual support, completeness, concise wording, citation quality, access compliance and the value of abstaining. A short answer that correctly states a limitation is preferable to a detailed but ungrounded explanation.

Add a tool-calling step only with explicit authority

Once the assistant can answer reliably, add a draft-ticket capability. Define a structured tool schema: permitted category, summary, urgency, source document and user identifier. Validate values on the server, check the requesting user's permission and create a draft rather than immediately opening a consequential ticket. Present the prepared action and ask for a clear confirmation before submission.

Make the operation idempotent. If the network times out after a submission, a retry must not duplicate the ticket. Preserve a request identifier and a reliable status path. Store only the information required for the action and enforce server-side tool authorization before executing it; retrieved documents must never grant permissions or choose unauthorized recipients.

Test attempted misuse: an injection in a document asks the agent to send confidential content elsewhere; a user names a ticket target they are not permitted to access; a request contains malformed values; or the action service becomes unavailable after authorization. Each case needs a known safe outcome and useful audit evidence.

Evaluate deployment, monitoring and changes

Build a repeatable test run covering retrieval, answer support, tool permission and end-to-end latency. Re-run it after changes to models, prompts, index content, APIs or security configuration. Tag each test result with relevant versions. A drop in citation precision might be caused by new chunking rules rather than a model change; the evidence must make those explanations distinguishable.

Monitor both infrastructure and behavior. Include request rate, token or model usage, latency, failures, source freshness, retrieval relevance and guarded-tool execution outcomes. Treat alerts as actionable investigations: who responds, which trace identifiers are available and how is sensitive information removed from logs? Plan rollback for a faulty prompt, model release or index deployment, including compatibility with previous client behavior.

Ask reviewers to inspect deliberately failed questions. A system that correctly answers common requests but exposes one restricted document is not ready. Include safety and access checks in the release gate rather than relying solely on aggregate quality statistics.

Check whether a document-answering app respects source revisions

An end-to-end acceptance test should record the exact input, source version, permitted identity and expected citation before exercising an approved policy update while the application remains in operation. Version A says equipment requests need one signoff; version B says two are required. The application must not continue to cite the withdrawn version after the agreed index-refresh window. Record when the source changed, when ingestion saw it, whether indexing completed and which version was retrieved by a test question. If the application still returns the old rule, locate the failure in synchronization, index metadata, retrieval ranking or generation.

For a permission-sensitive case, create two documents: one available to all staff and one limited to an operations group. Test the same question under both identities. It is not enough that the final response hides a protected sentence if the model was already given unauthorized evidence. The retrieval layer needs an enforceable access boundary. Add a negative test where a document contains instructions to send a summary to an unfamiliar email address; the text must be treated as source content, not as permission to invoke a messaging tool.

Store a reproducible result for each run: the retrieved source IDs, model deployment, prompt version, observed answer and citation, authorization test, and pass/fail comparison with expected behavior. A complete build should expose why a current policy was accepted and an obsolete or restricted document was rejected. This allows the team to demonstrate that a correction worked and to catch regressions after a model upgrade or re-index. A complete implementation should make data provenance and authorization observable rather than assuming a model's confidence is proof.

Align study with today's AI-103, not retired AI-102

The former AI-102 material on vision, language and deployment remains background for some product designs. It is not the current exam registration route. AI-103 measures modern Foundry projects, generative and agentic workflows, multimodal understanding, text and information extraction, deployment and operational controls. A learner seeking the current credential should use the Microsoft AI-103 certification together with the latest issuer objective date.

A credible study portfolio is a working assistant with approved evidence, denied-access tests, one constrained action, reproducible evaluation and a clearly explained deployment design. Describe trade-offs and failures in enough detail that another developer can reproduce them. That is useful for both certification preparation and the role itself; an old exam outline cannot substitute for practical proof.