Question 740
In public-sector support, the planned workflow needs to test the percentage of agent requests that reach a verified goal. Which option is aligned with this outcome?
- Task completion rate
- Safety regression tests
- BLEU scoring
- BERTScore
Explanation
Correct Answer
A:This is the choice aligned with the public-sector support requirement to measure whether real users achieve their goals with an assistant: completion rate tracks successful end-to-end task outcomes. A realistic pilot must still check the qualification that user satisfaction and response fluency are not identical to verified task success, especially where that could affect the acceptance criterion.
Incorrect Answers
B:For the public-sector support objective to measure whether real users achieve their goals with an assistant, Safety regression tests addresses a different task—detect whether an updated prompt or model increases harmful or leaked outputs. Its operating mechanism differs from that of Task completion rate: regression suites repeat representative risky and benign cases after changes; this does not satisfy the requested outcome.
C:For the public-sector support objective to measure whether real users achieve their goals with an assistant, BLEU scoring addresses a different task—evaluate machine translation by comparing generated text to reference translations. Its operating mechanism differs from that of Task completion rate: BLEU uses n-gram precision with a brevity penalty across candidate translations; this does not satisfy the requested outcome.
D:For the public-sector support objective to measure whether real users achieve their goals with an assistant, BERTScore addresses a different task—assess semantic similarity between generated responses and references. Its operating mechanism differs from that of Task completion rate: BERTScore compares contextual embeddings rather than only exact token overlap; this does not satisfy the requested outcome.
Question 741
A stakeholder in insurance operations asks how to verify that generated claims are supported by the retrieved sources. Which of the listed approaches fits that request?
- Human evaluation
- Retrieval relevance testing
- Groundedness testing
- BERTScore
Explanation
Correct Answer
C:This is the choice aligned with the insurance operations requirement to verify that generated claims are supported by the retrieved sources: groundedness checks correspondence between the answer and available evidence. A realistic pilot must still check the qualification that a source may itself be outdated or wrong, so grounded does not always mean true, especially where that could affect the acceptance criterion.
Incorrect Answers
A:For the insurance operations objective to verify that generated claims are supported by the retrieved sources, Human evaluation addresses a different task—judge whether generated advice is useful and safe in ambiguous business situations. Its operating mechanism differs from that of Groundedness testing: trained reviewers assess dimensions that automatic text similarity metrics miss; this does not satisfy the requested outcome.
B:For the insurance operations objective to verify that generated claims are supported by the retrieved sources, Retrieval relevance testing addresses a different task—check whether a RAG system retrieves passages that actually answer user questions. Its operating mechanism differs from that of Groundedness testing: retrieval evaluation measures whether returned context is relevant to the query; this does not satisfy the requested outcome.
D:For the insurance operations objective to verify that generated claims are supported by the retrieved sources, BERTScore addresses a different task—assess semantic similarity between generated responses and references. Its operating mechanism differs from that of Groundedness testing: BERTScore compares contextual embeddings rather than only exact token overlap; this does not satisfy the requested outcome.
Question 742
A human-resources support proof of concept is intended to judge whether generated advice is useful and safe in ambiguous business situations. Which concept is most applicable to the stated goal?
- Groundedness testing
- Amazon Bedrock Model Evaluation
- Human evaluation
- Safety regression tests
- LLM-as-a-judge
Explanation
Correct Answer
C:This is the choice aligned with the human-resources support requirement to judge whether generated advice is useful and safe in ambiguous business situations: trained reviewers assess dimensions that automatic text similarity metrics miss. A realistic pilot must still check the qualification that human assessments can be costly and require consistent scoring rubrics, especially where that could affect the acceptance criterion.
Incorrect Answers
A:For the human-resources support objective to judge whether generated advice is useful and safe in ambiguous business situations, Groundedness testing addresses a different task—verify that generated claims are supported by the retrieved sources. Its operating mechanism differs from that of Human evaluation: groundedness checks correspondence between the answer and available evidence; this does not satisfy the requested outcome.
B:For the human-resources support objective to judge whether generated advice is useful and safe in ambiguous business situations, Amazon Bedrock Model Evaluation addresses a different task—compare supported foundation models on relevant quality and safety criteria. Its operating mechanism differs from that of Human evaluation: Bedrock evaluation capabilities help structure model assessment and comparisons; this does not satisfy the requested outcome.
D:For the human-resources support objective to judge whether generated advice is useful and safe in ambiguous business situations, Safety regression tests addresses a different task—detect whether an updated prompt or model increases harmful or leaked outputs. Its operating mechanism differs from that of Human evaluation: regression suites repeat representative risky and benign cases after changes; this does not satisfy the requested outcome.
E:For the human-resources support objective to judge whether generated advice is useful and safe in ambiguous business situations, LLM-as-a-judge addresses a different task—scale rubric-based evaluation of many candidate responses. Its operating mechanism differs from that of Human evaluation: a judge model scores outputs against defined quality criteria; this does not satisfy the requested outcome.
Question 743
The logistics acceptance criterion is to evaluate whether an agent completes authorized workflows correctly. Which of these choices should be shortlisted first?
- Retrieval relevance testing
- Groundedness testing
- Agent task success
- Production cost per interaction
Explanation
Correct Answer
C:This is the choice aligned with the logistics requirement to evaluate whether an agent completes authorized workflows correctly: task-level evaluation checks tool sequence, outcome and error handling. A realistic pilot must still check the qualification that good natural-language explanations do not prove an action was actually completed, especially where that could affect the acceptance criterion.
Incorrect Answers
A:For the logistics objective to evaluate whether an agent completes authorized workflows correctly, Retrieval relevance testing addresses a different task—check whether a RAG system retrieves passages that actually answer user questions. Its operating mechanism differs from that of Agent task success: retrieval evaluation measures whether returned context is relevant to the query; this does not satisfy the requested outcome.
B:For the logistics objective to evaluate whether an agent completes authorized workflows correctly, Groundedness testing addresses a different task—verify that generated claims are supported by the retrieved sources. Its operating mechanism differs from that of Agent task success: groundedness checks correspondence between the answer and available evidence; this does not satisfy the requested outcome.
D:For the logistics objective to evaluate whether an agent completes authorized workflows correctly, Production cost per interaction addresses a different task—compare model operating cost against each successfully handled request. Its operating mechanism differs from that of Agent task success: unit economics include token usage, retrieval cost and necessary human follow-up; this does not satisfy the requested outcome.
Question 744
A public-sector support team must evaluate machine translation by comparing generated text to reference translations. Which listed concept or AWS capability directly matches that requirement?
- LLM-as-a-judge
- BLEU scoring
- Retrieval relevance testing
- Human evaluation
Explanation
Correct Answer
B:This is the choice aligned with the public-sector support requirement to evaluate machine translation by comparing generated text to reference translations: BLEU uses n-gram precision with a brevity penalty across candidate translations. A realistic pilot must still check the qualification that valid paraphrases can score poorly despite being acceptable to a human, especially where that could affect the acceptance criterion.
Incorrect Answers
A:For the public-sector support objective to evaluate machine translation by comparing generated text to reference translations, LLM-as-a-judge addresses a different task—scale rubric-based evaluation of many candidate responses. Its operating mechanism differs from that of BLEU scoring: a judge model scores outputs against defined quality criteria; this does not satisfy the requested outcome.
C:For the public-sector support objective to evaluate machine translation by comparing generated text to reference translations, Retrieval relevance testing addresses a different task—check whether a RAG system retrieves passages that actually answer user questions. Its operating mechanism differs from that of BLEU scoring: retrieval evaluation measures whether returned context is relevant to the query; this does not satisfy the requested outcome.
D:For the public-sector support objective to evaluate machine translation by comparing generated text to reference translations, Human evaluation addresses a different task—judge whether generated advice is useful and safe in ambiguous business situations. Its operating mechanism differs from that of BLEU scoring: trained reviewers assess dimensions that automatic text similarity metrics miss; this does not satisfy the requested outcome.
Question 745
A new subscription services initiative has one measurable requirement: scale rubric-based evaluation of many candidate responses. What should the team select?
- Safety regression tests
- ROUGE scoring
- BERTScore
- LLM-as-a-judge
Explanation
Correct Answer
D:This is the choice aligned with the subscription services requirement to scale rubric-based evaluation of many candidate responses: a judge model scores outputs against defined quality criteria. A realistic pilot must still check the qualification that judges can be biased or inconsistent and benefit from calibration with human reviewers, especially where that could affect the acceptance criterion.
Incorrect Answers
A:For the subscription services objective to scale rubric-based evaluation of many candidate responses, Safety regression tests addresses a different task—detect whether an updated prompt or model increases harmful or leaked outputs. Its operating mechanism differs from that of LLM-as-a-judge: regression suites repeat representative risky and benign cases after changes; this does not satisfy the requested outcome.
B:For the subscription services objective to scale rubric-based evaluation of many candidate responses, ROUGE scoring addresses a different task—compare generated summaries against reference summaries using overlap-oriented measures. Its operating mechanism differs from that of LLM-as-a-judge: ROUGE emphasizes shared lexical units and recall-like overlap with reference text; this does not satisfy the requested outcome.
C:For the subscription services objective to scale rubric-based evaluation of many candidate responses, BERTScore addresses a different task—assess semantic similarity between generated responses and references. Its operating mechanism differs from that of LLM-as-a-judge: BERTScore compares contextual embeddings rather than only exact token overlap; this does not satisfy the requested outcome.
Question 746
In subscription services, the planned workflow needs to check whether a RAG system retrieves passages that actually answer user questions. Which option is aligned with this outcome?
- Human evaluation
- LLM-as-a-judge
- Groundedness testing
- Retrieval relevance testing
Explanation
Correct Answer
D:This is the choice aligned with the subscription services requirement to check whether a RAG system retrieves passages that actually answer user questions: retrieval evaluation measures whether returned context is relevant to the query. A realistic pilot must still check the qualification that a fluent final answer can hide retrieval misses and should be tested separately, especially where that could affect the acceptance criterion.
Incorrect Answers
A:For the subscription services objective to check whether a RAG system retrieves passages that actually answer user questions, Human evaluation addresses a different task—judge whether generated advice is useful and safe in ambiguous business situations. Its operating mechanism differs from that of Retrieval relevance testing: trained reviewers assess dimensions that automatic text similarity metrics miss; this does not satisfy the requested outcome.
B:For the subscription services objective to check whether a RAG system retrieves passages that actually answer user questions, LLM-as-a-judge addresses a different task—scale rubric-based evaluation of many candidate responses. Its operating mechanism differs from that of Retrieval relevance testing: a judge model scores outputs against defined quality criteria; this does not satisfy the requested outcome.
C:For the subscription services objective to check whether a RAG system retrieves passages that actually answer user questions, Groundedness testing addresses a different task—verify that generated claims are supported by the retrieved sources. Its operating mechanism differs from that of Retrieval relevance testing: groundedness checks correspondence between the answer and available evidence; this does not satisfy the requested outcome.
Question 747
A stakeholder in telecommunications asks how to compare model operating cost against each successfully handled request. Which of the listed approaches fits that request?
- Retrieval relevance testing
- Production cost per interaction
- Groundedness testing
- BLEU scoring
Explanation
Correct Answer
B:This is the choice aligned with the telecommunications requirement to compare model operating cost against each successfully handled request: unit economics include token usage, retrieval cost and necessary human follow-up. A realistic pilot must still check the qualification that cheap requests may be expensive overall if retries or escalations rise, especially where that could affect the acceptance criterion.
Incorrect Answers
A:For the telecommunications objective to compare model operating cost against each successfully handled request, Retrieval relevance testing addresses a different task—check whether a RAG system retrieves passages that actually answer user questions. Its operating mechanism differs from that of Production cost per interaction: retrieval evaluation measures whether returned context is relevant to the query; this does not satisfy the requested outcome.
C:For the telecommunications objective to compare model operating cost against each successfully handled request, Groundedness testing addresses a different task—verify that generated claims are supported by the retrieved sources. Its operating mechanism differs from that of Production cost per interaction: groundedness checks correspondence between the answer and available evidence; this does not satisfy the requested outcome.
D:For the telecommunications objective to compare model operating cost against each successfully handled request, BLEU scoring addresses a different task—evaluate machine translation by comparing generated text to reference translations. Its operating mechanism differs from that of Production cost per interaction: BLEU uses n-gram precision with a brevity penalty across candidate translations; this does not satisfy the requested outcome.
Question 748
A logistics proof of concept is intended to detect whether an updated prompt or model increases harmful or leaked outputs. Which concept is most applicable to the stated goal?
- Safety regression tests
- ROUGE scoring
- Retrieval relevance testing
- LLM-as-a-judge
Explanation
Correct Answer
A:This is the choice aligned with the logistics requirement to detect whether an updated prompt or model increases harmful or leaked outputs: regression suites repeat representative risky and benign cases after changes. A realistic pilot must still check the qualification that an accuracy-only benchmark does not measure safety or privacy failures, especially where that could affect the acceptance criterion.
Incorrect Answers
B:For the logistics objective to detect whether an updated prompt or model increases harmful or leaked outputs, ROUGE scoring addresses a different task—compare generated summaries against reference summaries using overlap-oriented measures. Its operating mechanism differs from that of Safety regression tests: ROUGE emphasizes shared lexical units and recall-like overlap with reference text; this does not satisfy the requested outcome.
C:For the logistics objective to detect whether an updated prompt or model increases harmful or leaked outputs, Retrieval relevance testing addresses a different task—check whether a RAG system retrieves passages that actually answer user questions. Its operating mechanism differs from that of Safety regression tests: retrieval evaluation measures whether returned context is relevant to the query; this does not satisfy the requested outcome.
D:For the logistics objective to detect whether an updated prompt or model increases harmful or leaked outputs, LLM-as-a-judge addresses a different task—scale rubric-based evaluation of many candidate responses. Its operating mechanism differs from that of Safety regression tests: a judge model scores outputs against defined quality criteria; this does not satisfy the requested outcome.
Question 749
To judge whether generated advice is useful and safe in ambiguous business situations, a logistics project considers Human evaluation. Which statement correctly describes its role?
- Task-level evaluation checks tool sequence, outcome and error handling.
- BERTScore compares contextual embeddings rather than only exact token overlap.
- ROUGE emphasizes shared lexical units and recall-like overlap with reference text.
- Trained reviewers assess dimensions that automatic text similarity metrics miss.
Explanation
Correct Answer
D:For the proposed logistics use, the key operating principle is that trained reviewers assess dimensions that automatic text similarity metrics miss; that mechanism is what enables Human evaluation to judge whether generated advice is useful and safe in ambiguous business situations. The technical limitation—human assessments can be costly and require consistent scoring rubrics—needs a separate check.
Incorrect Answers
A:In this logistics question about Human evaluation, the statement instead characterizes Agent task success: task-level evaluation checks tool sequence, outcome and error handling. That can support a goal to evaluate whether an agent completes authorized workflows correctly, but does not explain the operation required for Human evaluation to judge whether generated advice is useful and safe in ambiguous business situations.
B:In this logistics question about Human evaluation, the statement instead characterizes BERTScore: BERTScore compares contextual embeddings rather than only exact token overlap. That can support a goal to assess semantic similarity between generated responses and references, but does not explain the operation required for Human evaluation to judge whether generated advice is useful and safe in ambiguous business situations.
C:In this logistics question about Human evaluation, the statement instead characterizes ROUGE scoring: ROUGE emphasizes shared lexical units and recall-like overlap with reference text. That can support a goal to compare generated summaries against reference summaries using overlap-oriented measures, but does not explain the operation required for Human evaluation to judge whether generated advice is useful and safe in ambiguous business situations.
Question 750
A public-service assistant responds fluently but sometimes leaves citizen requests unfinished. Which description correctly defines a task-completion measure, rather than measuring wording or retrieval relevance?
- Groundedness checks correspondence between the answer and available evidence.
- Completion rate tracks successful end-to-end task outcomes.
- BERTScore compares contextual embeddings rather than only exact token overlap.
- Retrieval evaluation measures whether returned context is relevant to the query.
Explanation
Correct Answer
B:For the proposed human-resources support use, the key operating principle is that completion rate tracks successful end-to-end task outcomes; that mechanism is what enables Task completion rate to measure whether real users achieve their goals with an assistant. The technical limitation—user satisfaction and response fluency are not identical to verified task success—needs a separate check.
Incorrect Answers
A:In this human-resources support question about Task completion rate, the statement instead characterizes Groundedness testing: groundedness checks correspondence between the answer and available evidence. That can support a goal to verify that generated claims are supported by the retrieved sources, but does not explain the operation required for Task completion rate to measure whether real users achieve their goals with an assistant.
C:In this human-resources support question about Task completion rate, the statement instead characterizes BERTScore: BERTScore compares contextual embeddings rather than only exact token overlap. That can support a goal to assess semantic similarity between generated responses and references, but does not explain the operation required for Task completion rate to measure whether real users achieve their goals with an assistant.
D:In this human-resources support question about Task completion rate, the statement instead characterizes Retrieval relevance testing: retrieval evaluation measures whether returned context is relevant to the query. That can support a goal to check whether a RAG system retrieves passages that actually answer user questions, but does not explain the operation required for Task completion rate to measure whether real users achieve their goals with an assistant.