Separate media generation from visual understanding
AI-103 covers two tasks that use similar inputs but have different correctness standards: creating or editing imagery and interpreting visual evidence. A generation system is judged by whether it produces an acceptable asset under instructions, safety constraints, and rights requirements. An understanding system is judged by whether it accurately describes what actually appears in a source image or video. Treating a generated picture as visual evidence for an unrelated real-world claim is a category error.
Build a small product-support experience. The business needs approved promotional graphics and also receives customer photographs of damaged devices. The first workflow may create or edit images from prompts and references; the second must identify visible damage, extract legible labels, and explain uncertainty to a technician. Keep their data handling and user promises separate even if they share a Foundry project.
The published AI-103 objectives explicitly include text-to-image and video generation, image and video editing, captions, visual question answering, alt text, Content Understanding, segmentation, and responsible AI for multimodal content. It does not mean every model in every region supports every capability. Verify the model’s supported input and output modalities, deployment type, preview status, and safety conditions before designing the user interface.
Choose the generation workflow and controls
A controlled asset-generation workflow starts with a creative brief: subject, intended audience, brand style, permissible reference media, prohibited content, and approval criteria. The model then receives a prompt and, when supported, reference material or edit inputs. Record model deployment, prompt revision, asset IDs, region, usage rights, moderation result, and the review decision. This is necessary for repeatability and brand governance.
Image-to-image editing and masked inpainting are different from generating a new composition. In an inpainting task, identify which area may change and which pixels or composition features should remain stable. Test whether the model inadvertently alters protected logos, faces, text, or background details. A prompt claiming “keep everything else identical” does not prove that the output is unchanged; compare the actual images and run a human acceptance review.
Video generation and editing add temporal consistency. A prompt might produce individually plausible frames that contradict one another about an object’s position or a person’s clothing. Define sequence, duration, reference treatment, audio policy, and any editing restrictions. Test whether a change to one segment disrupts adjacent shots. Model offerings and edit endpoints evolve quickly; use versioned product documentation instead of inventing a universal API call.
Microsoft documents MAI image models among its Foundry options, with preview labels on specific releases. Preview status affects production suitability and support commitments. For a Microsoft AI-103 lab, record what you tested and treat newer preview editing capabilities as demonstrations until the deployment’s actual readiness and terms support the intended business release.
Ground captions and visual questions in evidence
An image-understanding application needs to distinguish observable facts from speculation. A photograph may show a cracked device screen but not prove the cause, purchase date, warranty coverage, or whether the damage is accidental. Ask the model for a concise caption, a detailed technician note, and answers to specific visual questions, then compare each against a human-annotated reference.
Include counterexamples. Use blurry labels, partially hidden objects, reflections resembling characters, tiny warning symbols, and two similar device models. Test whether the model says the label is unreadable instead of inventing a serial number. Accurate uncertainty is a positive outcome. A caption that confidently adds a nonexistent charger or battery defect is a factual failure even if it sounds professional.
Accessible alt text has a narrower purpose than a marketing description. It should communicate meaningful visual content in the context where an image appears, while avoiding irrelevant speculation and repeating nearby visible text unnecessarily. A useful test asks whether a screen-reader user would understand the important device defect or chart relationship. Extended image descriptions can support complex visuals but should preserve an appropriate reading order and identify uncertainty.
For visual question answering, retain links between the question, specific image regions or frames, and the answer. A final response that includes a helpful-looking paragraph but no supporting visual basis should not be accepted as grounded. If an image contains document text, determine whether a dedicated extraction pipeline is more reliable than asking a general model to infer exact numeric fields.
Analyze video as a sequence, not one picture
A security camera clip, assembly demonstration, or product troubleshooting video contains changing visual evidence. Sample key frames and timeline segments in a way that preserves event order, and align descriptions with timestamps or segment identifiers. A one-sentence summary of a whole clip may omit the step that caused a device fault or misstate whether an action happened before or after an alert.
Azure Content Understanding can produce structured video-oriented outputs, including scene and segment information. Some features and schema choices depend on API version and analyzer configuration. Validate what fields are present for the selected version rather than assuming all video models yield identical timecodes, key frames, and descriptions.
Create three evaluation clips: a normal device replacement procedure, the same procedure with a missing safety step, and a clip where the relevant component is occluded. Require the system to cite frames or segments and to report when the evidence is insufficient. A correct summary of the normal clip is not enough; detection of a subtle omission reveals the quality needed for operational use.
Audio can change the meaning of a video. A spoken instruction may refer to an object outside the frame, while an image may contradict the transcript. Decide whether the product needs independent speech transcription, audiovisual reconciliation, or a human review step. Preserve provenance for both channels. A model should not silently override clear visual evidence because the audio is more linguistically fluent.
Use Content Understanding for structured visual facts
Microsoft’s Content Understanding framework provides analyzers for documents, images, audio, and video. An analyzer determines processing options, extracted elements, and structured fields. General multimodal understanding can help interpret a visual scene, but domain-specific extraction may require explicit schemas and confidence or source metadata.
Build a small image intake schema containing device type, visible damage type, affected region, readability of the serial label, and evidence location. A useful output distinguishes “screen crack visible” from “warranty invalid.” The first is potentially a visual observation; the second requires business policy and authorized customer records. Send each fact to the system responsible for validating it.
Content Understanding API version 2025-11-01 is documented as GA for production-oriented multimodal extraction, while version 2026-06-01-preview adds capabilities with different readiness. Label both when discussing features such as agentic document reasoning or richer segmentation. An implementation should state which version and analyzer were used, what region supports it, and which fields require downstream validation.
When image evidence becomes searchable, keep original media identifiers, derived descriptions, timestamps, and access permissions. Do not discard the source image after creating an embedding or caption if the workflow requires later verification. The extraction pipeline explains how to connect multimodal source provenance to retrieval and agents without turning an inferred description into unquestioned truth.
Defend against visual prompt injection and unsafe output
Images can contain instructions aimed at the assistant: a label saying “ignore previous rules” or a QR code pointing to unapproved content. The application may need to read that text as part of a visual analysis, but it must not treat it as an instruction to change tools, retrieve restricted information, or transfer data. A prompt shield or classifier helps detect threats; the action service still needs independent permission checks.
Evaluate content moderation separately for inputs and generated outputs. An image request may be disallowed by a business brand policy even when it is not universally unsafe. A genuine photo can contain sensitive identifiers that should be masked in an output. Decide which categories are blocked, reviewed, or annotated and how those decisions appear to the user. Record moderation outcomes without retaining unnecessary media.
Watermarks and provenance controls may be appropriate for generated assets. Do not make an absolute claim that every deployment automatically adds them or that they prove authenticity. The business needs a defined labeling and retention policy supported by the chosen service and downstream publishing workflow. For reference assets, verify consent, licensing, and permitted transformation conditions before sending them to a model.
Evaluate quality with different rubrics for each task
For generated images, score adherence to the brief, unwanted changes, output safety, license constraints, composition quality, and reviewer acceptance. For video, add temporal consistency and segment-level edits. For understanding, score object and region recognition, grounded captions, accurate abstention, timestamp fidelity, and source traceability. A single aesthetic score is not an acceptable replacement for factual accuracy.
Prepare positive and negative cases for each modality. A clean product photograph and a blurred one; a valid edit mask and one that covers a protected logo; a normal assembly clip and one missing an essential safety step; a caption where the right answer is “cannot determine.” Apply the same dataset when the model or API version changes. A model upgrade that improves creative quality can worsen exact label recognition.
Link visual intake to the agent only through a validated structured result. If the model says a customer’s device is model X but confidence is weak, the system should request a manual check before retrieving policy or drafting a return. Do not let visually inferred facts initiate a purchase or warranty approval without corroborating business information and authorization.
A practical multimodal lab for AI-103
Start with three still images of devices and define the questions a support technician needs answered. Generate captions, detailed descriptions, and field extraction, preserving evidence and refusal reasons. Next, use a controlled product photo for an image edit and inspect whether the specified mask is respected. Record model, deployment, region, and version so the test can be repeated.
Add a short assembly video and label critical sequence steps and timestamps. Ask the analysis workflow to identify a skipped step, distinguish visible evidence from guesses, and produce a safe technician summary. Insert an adversarial phrase on a box or screen and confirm that the agent ignores it as a command. Do not use private customer media in public examples.
Evaluate the media workflow separately for image generation, editing, video understanding and accessible descriptions. A model may correctly identify a product in a still photograph yet hallucinate a step in a video that was never visible. Preserve original media and timecodes, distinguish observations from inferences, and require human confirmation before a generated inspection result changes a maintenance record. The output is ready for operational use only when those differences are measurable and the privacy and safety boundaries survive adversarial examples.