A mobile app start-up company is implementing an AI-based chat assistant for e-commerce customers. In the process of planning the testing, the team realizes that the specifications are insufficient. Which testing approach should be used to test this system?
Correct Answer: A
The syllabus describes exploratory testing as especially useful when there are poor specifications or test oracle problems: "Exploratory testing is especially useful when there are poor specifications or test oracle problems, which is often the case for AI-based systems." (Reference: ISTQB CT-AI Syllabus v1.0, Section 9.6, page 70 of 99)
Question 72
Which ONE of the following tests is LEAST likely to be performed during the ML model testing phase? SELECT ONE OPTION
Correct Answer: C
The question asks which test is least likely to be performed during the ML model testing phase. Let's consider each option: Testing the accuracy of the classification model (A): Accuracy testing is a fundamental part of the ML model testing phase. It ensures that the model correctly classifies the data as intended and meets the required performance metrics. Testing the API of the service powered by the ML model (B): Testing the API is crucial, especially if the ML model is deployed as part of a service. This ensures that the service integrates well with other systems and that the API performs as expected. Testing the speed of the training of the model (C): This is least likely to be part of the ML model testing phase. The speed of training is more relevant during the development phase when optimizing and tuning the model. During testing, the focus is more on the model's performance and behavior rather than how quickly it was trained. Testing the speed of the prediction by the model (D): Testing the speed of prediction is important to ensure that the model meets performance requirements in a production environment, especially for real-time applications. Reference: ISTQB CT-AI Syllabus Section 3.2 on ML Workflow and Section 5 on ML Functional Performance Metrics discuss the focus of testing during the model testing phase, which includes accuracy and prediction speed but not the training speed.
Question 73
Upon testing a model used to detect rotten tomatoes, the following data was observed by the test engineer, based on certain number of tomato images. For this confusion matrix which combinations of values of accuracy, recall, and specificity respectively is CORRECT? SELECT ONE OPTION
Correct Answer: A
To calculate the accuracy, recall, and specificity from the confusion matrix provided, we use the following formulas: * Confusion Matrix: * Actually Rotten: 45 (True Positive), 8 (False Positive) * Actually Fresh: 5 (False Negative), 42 (True Negative) * Accuracy: * Accuracy is the proportion of true results (both true positives and true negatives) in the total population. * Formula: Accuracy=TP+TNTP+TN+FP+FN\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}Accuracy=TP+TN+FP+FNTP+TN * Calculation: Accuracy=45+4245+42+8+5=87100=0.87\text{Accuracy} = \frac{45 + 42}{45 + 42 + 8 + 5} = \frac{87}{100} = 0.87Accuracy=45+42+8+545+42=10087=0.87 * Recall (Sensitivity): * Recall is the proportion of true positive results in the total actual positives. * Formula: Recall=TPTP+FN\text{Recall} = \frac{TP}{TP + FN}Recall=TP+FNTP * Calculation: Recall=4545+5=4550=0.9\text{Recall} = \frac{45}{45 + 5} = \frac{45}{50} = 0.9 Recall=45+545=5045=0.9 * Specificity: * Specificity is the proportion of true negative results in the total actual negatives. * Formula: Specificity=TNTN+FP\text{Specificity} = \frac{TN}{TN + FP}Specificity=TN+FPTN * Calculation: Specificity=4242+8=4250=0.84\text{Specificity} = \frac{42}{42 + 8} = \frac{42} {50} = 0.84Specificity=42+842=5042=0.84 Therefore, the correct combinations of accuracy, recall, and specificity are 0.87, 0.9, and 0.84 respectively. References: ISTQB CT-AI Syllabus, Section 5.1, Confusion Matrix, provides detailed formulas and explanations for calculating various metrics including accuracy, recall, and specificity. "ML Functional Performance Metrics" (ISTQB CT-AI Syllabus, Section 5).
Question 74
Which of the following aspects is a challenge when handling test data for an AI-based system?
Correct Answer: A
The syllabus explicitly mentions challenges of handling personal data and ensuring privacy when testing AI-based systems: "The management of personal data and sensitive data is often a concern during testing, as testing typically requires realistic data and it is difficult to fully anonymize data."
Question 75
Which statement regarding the use of training, validation, and test data sets is correct? Choose ONE option (1 out of 4)
Correct Answer: D
The ISTQB CT-AI syllabus (Section3.2 - Model Evaluation) specifies the correct usage oftraining, validation, andtestdatasets. It emphasizes that thetest dataset must be representative of the real operational dataand must beequivalent in distribution to the training and validation sets, ensuring a fair and unbiased evaluation. Option D precisely matches this requirement. Option A contradicts the syllabus because validation and test sets servedifferent purposes: validation is for tuning, test is for final evaluation. Combining them undermines the reliability of results. Option B is incorrect because even with limited data, the syllabus recommends maintaining a test set or using techniques such as cross-validationrather than eliminating testing. Option C is wrong because equal distribution (33/33/33) isnot recommended; typically, the training set is much larger (e.g., 70-80%). Thus, OptionDis the only statement aligned with the syllabus' guidance.