Unit 3: Evaluating Models
Class 10 Artificial Intelligence notes — Model Evaluation, Train-Test Split, Accuracy & Error, Confusion Matrix, Precision, Recall and F1 Score explained clearly, with solved MCQs, Assertion-Reasoning, case studies with explanations, and a quick-revision concept map.
📘 Concept Notes
3.1 Importance of Model Evaluation
Model Evaluation is the process of using different evaluation metrics to understand a machine learning model's performance — like a report card for your AI model.
- An AI model gets better with constructive feedback.
- The cycle: build a model → get feedback from metrics → make improvements → repeat until you achieve a desirable accuracy.
- Need of Model Evaluation: It reveals the model's strengths, weaknesses, and suitability for the task — essential for building trustworthy and reliable AI systems.
3.2 Splitting the Training Set Data for Evaluation
Train-Test Split
- A technique for evaluating the performance of a machine learning algorithm; usable for any supervised learning algorithm.
- The dataset is divided into two subsets: the training dataset and the testing dataset.
- Appropriate when a sufficiently large dataset is available.
Need of Train-Test Split
- The training dataset is used to make the model learn.
- The testing dataset's inputs are fed to the trained model; its predictions are compared to the expected (actual) values.
- Objective: estimate how the model performs on new, unseen data.
- Important rule: never evaluate a model on the same data used to build it — it will simply memorize the training set. This is called overfitting.
3.3 Accuracy and Error
- Accuracy: An evaluation metric that measures the total number of correct predictions a model makes. It is directly proportional to model performance.
- Error: The difference between a model's prediction and the actual outcome; quantifies how often the model makes mistakes.
Key Points
- The goal is to minimize error and maximize accuracy.
- Real-world data can be messy — even the best models make mistakes.
- Focusing only on accuracy isn't always ideal — e.g., in medical diagnosis, avoiding false healthy/sick labels may matter more than raw accuracy.
- The right metric depends on the specific task and its requirements.
Formulas
- Absolute Error = |Actual − Predicted|
- Error Rate = Error / Actual
- Accuracy = 1 − Error Rate
- Accuracy % = Accuracy × 100
3.4 Evaluation Metrics for Classification
Classification refers to a problem where a specific class label is predicted from the given input data (e.g., sorting items into "fruits & vegetables" vs. "grocery").
Popular Classification Metrics: Confusion Matrix, Classification Accuracy, Precision, Recall, F1 Score.
Confusion Matrix
- A handy tabular presentation of a model's accuracy across two or more classes.
- Actual values on the y-axis, predicted values on the x-axis.
- Each cell shows the number of predictions falling into that category.
| Term | Definition |
|---|---|
| True Positive (TP) | Model correctly predicts the positive class |
| True Negative (TN) | Model correctly predicts the negative class |
| False Positive (FP) | Model wrongly predicts positive for an actual negative |
| False Negative (FN) | Model wrongly predicts negative for an actual positive |
Classification Accuracy
The number of correct predictions as a ratio of all predictions made.
Formula: Accuracy = (TP + TN) / (TP + TN + FP + FN)
Limitation: Reliable only with a balanced dataset and when all errors are equally important — rarely true in real life. Example: in an unbalanced dataset (900 "Yes," 100 "No"), a model that always predicts "Yes" still shows 90% accuracy despite being useless — hence the need for Precision, Recall, or F1 Score.
Precision
The ratio of correctly classified positive examples to the total predicted positive examples.
Formula: Precision = TP / (TP + FP)
Used when: False Positives (FP) are costly and must be minimized — e.g., predicting good weather for a satellite launch, where a false "good weather" prediction can be disastrous.
Recall (also called Sensitivity or True Positive Rate)
Measures how many actual positives the model correctly identifies.
Formula: Recall = TP / (TP + FN)
Used when: False Negatives (FN) are costly and must be minimized — e.g., in COVID-19 detection, missing an actual positive case could delay treatment and spread infection.
F1 Score
Combines Precision and Recall into a single measure.
Formula: F1 Score = (2 × Precision × Recall) / (Precision + Recall)
Used when: the dataset is unbalanced and it's unclear whether FP or FN matters more.
Ethical Concerns Around Model Evaluation
Model evaluation must also consider fairness, bias, and ethical impact — not just numerical accuracy — to ensure the model doesn't disadvantage any group unfairly.
✅ Practice Zone — MCQs, Assertion-Reasoning, Reflection & Case Studies
All options are shown upfront. Tap + on any question to reveal the correct answer and explanation.
Test Yourself (MCQ)
1. A student solved 90 out of 100 questions correctly in a multiple-choice exam. What is the error rate of the student's answers?
Explanation: Error Rate = Incorrect answers / Total answers = 10/100 = 10%.
2. In a spam email detection system, out of 1000 emails received, 300 are spam. The system correctly identifies 240 spam emails as spam, but it also marks 60 legitimate emails as spam. What is the precision of the system?
Explanation: Precision = TP/(TP+FP) = 240/(240+60) = 240/300 = 80%.
3. A teacher's marks prediction system predicts the marks of a student as 75, but the actual marks obtained by the student are 80. What is the absolute error in the prediction?
Explanation: Absolute Error = |Actual − Predicted| = |80 − 75| = 5.
4. The goal when evaluating an AI model is to:
Explanation: A well-evaluated model should make as few mistakes (errors) as possible while correctly predicting outcomes (accuracy) as often as possible.
5. A high F1 score generally suggests:
Explanation: Since F1 Score is the harmonic mean of Precision and Recall, a high F1 score only results when both metrics are reasonably high and balanced.
6. How is the relationship between model performance and accuracy described?
Explanation: Better model performance results in more correct predictions, so accuracy increases as performance increases — a directly proportional relationship.
Assertion & Reasoning
A&R-1. Assertion (A): Accuracy is an evaluation metric that allows you to measure the total number of predictions a model gets right.
Reasoning (R): The accuracy of the model and performance of the model is directly proportional, and hence the better the performance of the model, the more accurate the predictions.
Explanation: Both statements are correct definitions from the chapter, and R correctly explains why accuracy is used to measure performance.
A&R-2. Assertion (A): The sum of the values in a confusion matrix's row represents the total number of instances for a given actual class.
Reasoning (R): This enables the calculation of class-specific metrics such as precision and recall, essential for evaluating a model's performance across classes.
Explanation: Row sums in a confusion matrix do represent actual class totals, and this is precisely what allows precision and recall to be calculated per class.
Reflection Time — Exam-Style Q&A
Q1. What will happen if you deploy an AI model without evaluating it with known test set data?
Without evaluation, there is no way to know how the model will perform on new, unseen data. The model might be overfitted — performing well only on training data but poorly in the real world — leading to unreliable or incorrect predictions once deployed.
Q2. Do you think evaluating an AI model is that essential in an AI project cycle?
Yes. Model Evaluation is an essential stage of the AI project cycle because it determines whether the model is fit for purpose. It identifies the model's strengths and weaknesses, allows comparison between models, and ensures reliable performance on real-world data before deployment.
Q3. Explain train-test split with an example.
Train-Test Split is a technique for evaluating a machine learning algorithm by dividing the dataset into a training dataset and a testing dataset. The training dataset is used to make the model learn, and the testing dataset is used to generate predictions, which are compared with actual values to estimate performance.
Example: Of 1000 house-price records, 800 (training set) are used to train the model, and the remaining 200 (testing set) check how accurately it predicts prices it has never seen.
Q4. "Understanding both error and accuracy is crucial for effectively evaluating and improving AI models." Justify this statement.
Error shows how far off predictions are from actual outcomes, while Accuracy shows how often the model is correct overall. A model can have high accuracy but still make critical errors on an unbalanced dataset. Understanding both helps identify exactly where the model is going wrong, guiding targeted improvements — for example, in medical diagnosis, both metrics must be considered to avoid dangerous misclassifications.
Q5. What is classification accuracy? Can it be used all the time for evaluating AI models?
Classification Accuracy is the number of correct predictions made as a ratio of all predictions made: Accuracy = (TP + TN) / (TP + TN + FP + FN).
No, it cannot be used all the time. It is reliable only with a balanced dataset and when all errors are equally important. In a highly unbalanced dataset (900 "Yes," 100 "No"), a model that always predicts "Yes" shows 90% accuracy despite being useless — so Precision, Recall, or F1 Score should be used instead.
Case Study–Based Questions
Case Study Q1. Identify which metric (Precision or Recall) should be used in the following cases, and why.
a) Email Spam Detection
Answer Precision. Explanation: A false positive (legitimate email marked spam) risks the user missing important emails, so false positives must be minimized.
b) Cancer Diagnosis
Answer Recall. Explanation: A false negative (missing an actual cancer patient) delays critical treatment, so false negatives must be minimized.
c) Legal Cases (Innocent until proven guilty)
Answer Precision. Explanation: A false positive (wrongly convicting an innocent person) is a serious miscarriage of justice, so false positives must be minimized.
d) Fraud Detection
Answer Recall. Explanation: A false negative (missing actual fraud) causes financial loss, so false negatives must be minimized.
e) Safe Content Filtering (like Kids YouTube)
Answer Recall. Explanation: A false negative (unsafe content wrongly marked safe) risks harm to a child, so false negatives must be minimized.
Case Study Q2a. Spam Email Detection — TP=150, FP=50, TN=750, FN=50 (Total=1000). Find Accuracy, Precision, Recall, F1-Score.
| Predicted Spam (1) | Predicted Not Spam (0) | |
|---|---|---|
| Actual Spam (1) | TP = 150 | FN = 50 |
| Actual Not Spam (0) | FP = 50 | TN = 750 |
Accuracy = (150+750)/1000 = 90% Precision = 150/200 = 75% Recall = 150/200 = 75% F1 Score = 75%
Explanation: Precision and Recall are equal and reasonably high, showing the model correctly identifies spam without excessive false alarms or missed spam.
Case Study Q2b. Credit Scoring (Loan Default) — TP=90, FP=40, TN=820, FN=50 (Total=1000). Find Accuracy, Precision, Recall, F1-Score.
| Predicted Default (1) | Predicted No Default (0) | |
|---|---|---|
| Actual Default (1) | TP = 90 | FN = 50 |
| Actual No Default (0) | FP = 40 | TN = 820 |
Accuracy = (90+820)/1000 = 91% Precision = 90/130 ≈ 69.23% Recall = 90/140 ≈ 64.29% F1 Score ≈ 66.67%
Explanation: The model misses several actual defaulters (FN=50) and wrongly flags some good applicants (FP=40). Since missing a defaulter (FN) is costlier for a bank, Recall should ideally be improved further.
Case Study Q2c. Fraud Detection — TP=80, FP=30, TN=850, FN=40 (Total=1000). Find Accuracy, Precision, Recall, F1-Score.
| Predicted Fraud (1) | Predicted Legit (0) | |
|---|---|---|
| Actual Fraud (1) | TP = 80 | FN = 40 |
| Actual Legit (0) | FP = 30 | TN = 850 |
Accuracy = (80+850)/1000 = 93% Precision = 80/110 ≈ 72.73% Recall = 80/120 ≈ 66.67% F1 Score ≈ 69.57%
Explanation: Since fraud detection prioritizes Recall (missing fraud is costly), a Recall of ~67% suggests the model still misses a notable share of fraud cases and should be improved further.
Case Study Q2d. Medical Diagnosis — TP=120, FP=20, TN=800, FN=60 (Total=1000). Find Accuracy, Precision, Recall, F1-Score.
| Predicted Disease (1) | Predicted No Disease (0) | |
|---|---|---|
| Actual Disease (1) | TP = 120 | FN = 60 |
| Actual No Disease (0) | FP = 20 | TN = 800 |
Accuracy = (120+800)/1000 = 92% Precision = 120/140 ≈ 85.71% Recall = 120/180 ≈ 66.67% F1 Score = 75%
Explanation: For medical diagnosis, Recall should ideally be maximized to avoid missing true patients. A Recall of ~67% means 60 actual patients (FN) were missed — the model should be tuned to reduce false negatives.
Case Study Q2e. Inventory Management (Stock-Out Prediction) — TP=100, FP=50, TN=800, FN=50 (Total=1000). Find Accuracy, Precision, Recall, F1-Score.
| Predicted Out of Stock (1) | Predicted In Stock (0) | |
|---|---|---|
| Actual Out of Stock (1) | TP = 100 | FN = 50 |
| Actual In Stock (0) | FP = 50 | TN = 800 |
Accuracy = (100+800)/1000 = 90% Precision = 100/150 ≈ 66.67% Recall = 100/150 ≈ 66.67% F1 Score ≈ 66.67%
Explanation: Since Precision equals Recall here, the model is equally likely to falsely predict a stock-out as it is to miss one — an F1 score of ~67% shows room for improvement in forecasting inventory needs.