Per AAISM's ML lifecycle controls, hyperparameter tuning is performed on the validation set, reserving the test set strictly for final, unbiased performance estimation. The training set is used to fit parameters; the validation set guides model selection and hyperparameter optimization; the test set is untouched until the end to prevent leakage and optimistic bias. "Configuration" is not a dataset type in the lifecycle split.
References:* AI Security Management (AAISM) Body of Knowledge: Model Development Controls- Data Splitting and Evaluation Integrity* AAISM Study Guide: Overfitting Avoidance; Validation vs. Test Separation; Leakage Prevention* AAISM Mapping to Standards: Evaluation Integrity-Hold-out Protocols and Tuning Practices

AAISM identifies percentage of AI projects in compliance as the most relevant KRI for evaluating AI risk management effectiveness. This metric directly reflects adherence to governance, regulatory, and security requirements. The number of models deployed (A) or systems with AI components (B) indicate scale, not risk management quality. Training requests (D) show awareness levels but do not measure effectiveness of risk management. Compliance percentage provides a direct, measurable indication of how well risks are being governed and mitigated.
References:
AAISM Exam Content Outline - AI Risk Management (Risk Metrics and Compliance) AI Security Management Study Guide - Key Risk Indicators in AI Programs

AAISM classifies learning paradigms by the presence of labeled targets. Creating categories (labels) and training on them is supervised learning, where input features are mapped to known outputs and optimization minimizes prediction error against ground truth. Unsupervised (B) discovers structure without labels; reinforcement (A) optimizes behavior via rewards; "machine learning" (C) is the broad field, not the specific technique.
References: AI Security Management (AAISM) Body of Knowledge - AI/ML Foundations; Learning Paradigms and Data Requirements. AAISM Study Guide - Supervised vs. Unsupervised vs. Reinforcement Learning; Label Quality and Model Performance Dependencies.

AAISM highlights that while accuracy and performance metrics are important, the rate of drift is the most critical KRI for AI systems. Model drift occurs when input data or environmental conditions shift, causing the system to degrade and produce unreliable outputs. This risk indicator directly reflects whether the AI continues to function as intended over time. Accuracy rates and response times are performance metrics, not primary risk signals. The amount of data in the model does not reliably indicate exposure to risk. Therefore, the greatest KRI for ongoing assurance and governance is the rate of drift.
References:
AAISM Study Guide - AI Risk Management (Monitoring and Drift Detection) ISACA AI Security Management - Key Risk Indicators for AI Systems