1. Training dataset

The training dataset contains examples used by the algorithm to learn patterns.

For policy-renewal prediction:

CustomerAge + Premium + Claims + Complaints → Renewed
During training, the algorithm examines both:
  • Features: input columns
  • Label: correct historical outcome

It adjusts its internal parameters to reduce prediction errors.

Think of it as:

Training dataset = lessons + solved examples
A model’s performance on its own training data is not sufficient evidence of production quality.

2. Validation dataset

The validation dataset is used during model development to compare alternatives and make design choices.

It can help select:

  • Algorithm
  • Feature combination
  • Hyperparameters
  • Classification threshold
  • Data transformations
  • Model version

A hyperparameter is a setting selected before or during the training process rather than learned directly from individual rows.

Examples include:

  • Number of trees
  • Maximum tree depth
  • Learning rate
  • Number of training iterations

Think of validation data as:

Validation dataset = practice examination

You use its result to improve your approach, but it is not the final independent examination.

3. Test dataset

The test dataset provides the final evaluation of the selected model.

It must remain untouched while:

  • Creating features
  • Selecting algorithms
  • Adjusting hyperparameters
  • Choosing thresholds
  • Comparing experiments

Think of it as:

Test dataset = sealed final examination
 
You should evaluate the final selected solution on it only after development decisions are complete.

4. A common split

A reasonable starting split for a sufficiently large dataset is:

Dataset Percentage Purpose
Training 70% Learn model parameters
Validation 15% Compare and tune models
Test 15% Final independent evaluation

This is not a universal rule.

Other possible splits include:

80% training + 20% test 
80% training + 10% validation + 10% test 
60% training + 20% validation + 20% test
The correct choice depends on:
  • Total records
  • Number of rare cases
  • Time dependence
  • Number of model experiments
  • Business risk

5. Generalization

Generalization means the model performs well on new examples drawn from the real-world population it is intended to serve.

A useful model should learn:

Reusable relationship between features and outcome

It should not simply memorize:

Customer 104 renewed 
Customer 105 did not renew 
Customer 106 renewed
 

6. Overfitting

Overfitting happens when a model learns training examples—including their noise and accidental details—too closely.

Typical sign:

 
Training accuracy:   99%
Validation accuracy: 72%

The large difference suggests that the model performs well on familiar records but poorly on unseen records.

Software analogy:

A method passes tests written around its exact implementation, 
but fails when it receives realistic production inputs.
 

Common causes include:

  • Model is unnecessarily complex.
  • Dataset is small.
  • Features contain identifiers.
  • Duplicate records exist across splits.
  • Training continues too aggressively.
  • Too many alternatives are tuned against the same validation set.

7. Underfitting

Underfitting means the model is too simple—or insufficiently trained—to learn the important pattern.

Typical result:

Training accuracy: 61% 
Validation accuracy: 59%
Both scores are poor.

Comparison:

Situation Training performance Unseen-data performance
Underfitting Poor Poor
Appropriate fit Good Similar and good
Overfitting Excellent Much worse

8. Random splitting

A random split assigns rows randomly to training, validation and test datasets.

This may work when:

  • Rows are reasonably independent.
  • Data does not represent a time sequence.
  • Multiple rows from one entity will not leak information.
  • Production conditions resemble the complete dataset.

Example:

10,000 independent support tickets 
           ↓ 
Random assignment 
          ↓ 
7,000 training 
1,500 validation 
1,500 test
 

9. Grouped splitting

Suppose one customer owns five policies.

A row-level random split could place:

Customer 501 policies 1–4 → Training
Customer 501 policy 5     → Test

The model may indirectly recognize the same customer through related features. The test result can become unrealistically optimistic.

A grouped split keeps every row belonging to the same entity in one subset:

Customer 501 → Training only 
Customer 782 → Validation only 
Customer 910 → Test only

Possible grouping keys include:

  • Customer ID
  • Candidate ID
  • Patient ID
  • Device ID
  • Exam session ID
  • Document family

The grouping key controls the split; it does not necessarily become a model feature.

10. Time-based splitting

Random splitting is dangerous when predicting future events.

For example:

Training: January 2024–December 2024
Validation: January 2025–March 2025 
Test: April 2025–June 2025

This better simulates the production question:

Can a model trained on the past predict a later period?

Use time-based splitting for:

  • Sales forecasting
  • Server-failure prediction
  • Fraud detection
  • Policy-renewal prediction
  • Demand prediction
  • Sensor monitoring

11. Preprocessing leakage

Suppose you calculate the average premium using the complete dataset before splitting it:

 
Training + validation + test
              ↓
Calculate global average
              ↓
Fill missing values

Information from validation and test records has influenced training preparation.

The safer order is:

Split first
     ↓
Learn transformation values from training data only
     ↓ 
Apply those values to validation and test data

This rule applies to:

  • Average or median calculation
  • Normalization
  • Category dictionaries
  • Feature selection
  • Outlier thresholds
  • Text vocabulary construction
PreviousNext