Scenario

We have policy-renewal history and want to create:

70% training
15% validation 
15% test

Step 1: Install ML.NET

Create a console application:

dotnet new console -n DataSplitDemo 
cd DataSplitDemo 
dotnet add package Microsoft.ML

Step 2: Define the input class

Replace Program.cs with:

using Microsoft.ML;

public sealed class PolicyRenewalRow
{
    public float CustomerAge { get; set; }
    public float PremiumAmount { get; set; }
    public float ClaimCount { get; set; }
    public float ComplaintCount { get; set; }
    public bool Renewed { get; set; }
}

var rows = Enumerable.Range(1, 100)
    .Select(index => new PolicyRenewalRow
    {
        CustomerAge = 20 + index % 55,
        PremiumAmount = 5000 + index * 250,
        ClaimCount = index % 4,
        ComplaintCount = index % 5,
        Renewed = index % 3 != 0
    })
    .ToList();

var mlContext = new MLContext(seed: 42);
IDataView completeData = mlContext.Data.LoadFromEnumerable(rows);
The seed makes random operations reproducible.
This is important for:
  • Debugging
  • Comparing experiments
  • Reproducing reported results

Step 3: Reserve 15% for final testing

var developmentAndTest = mlContext.Data.TrainTestSplit(
    completeData,
    testFraction: 0.15,
    seed: 42);

IDataView developmentData = developmentAndTest.TrainSet;
IDataView testData = developmentAndTest.TestSet;

The test data should now be treated as sealed.

Step 4: Create validation data

The development set contains approximately 85% of the original records.

We need approximately 15% of the complete dataset for validation:

15 ÷ 85 ≈ 0.1765

Split about 17.65% from the development set:

var trainingAndValidation = mlContext.Data.TrainTestSplit(
    developmentData,
    testFraction: 0.1765,
    seed: 42);

IDataView trainingData = trainingAndValidation.TrainSet;
IDataView validationData = trainingAndValidation.TestSet;

This produces approximately:

Training   : 70%
Validation : 15%
Test       : 15%

Step 5: Verify row counts

long trainingCount = trainingData.GetRowCount() ?? 0;
long validationCount = validationData.GetRowCount() ?? 0;
long testCount = testData.GetRowCount() ?? 0;

Console.WriteLine($"Training rows:   {trainingCount}");
Console.WriteLine($"Validation rows: {validationCount}");
Console.WriteLine($"Test rows:       {testCount}");
Console.WriteLine($"Total rows:      " +
                  $"{trainingCount + validationCount + testCount}");
Run it:
dotnet run

The exact counts may vary slightly because splitting is probabilistic.

Step 6: Understand how each subset will be used

Later, the flow will become:

// Conceptual preview—we have not created the pipeline yet.

var model = pipeline.Fit(trainingData);

var validationPredictions = model.Transform(validationData);
// Compare model choices and tune settings here.

var finalPredictions = model.Transform(testData);
// Use only after selecting the final model.

Do not repeatedly inspect test results and then modify the model. Doing that turns the test set into another validation set.

Step 7: Time-based splitting with SQL

For renewal prediction, chronological splitting may be more realistic:

SELECT *,
    CASE
        WHEN PredictionDate < '2025-01-01'
            THEN 'Training'

        WHEN PredictionDate >= '2025-01-01'
         AND PredictionDate <  '2025-04-01'
            THEN 'Validation'

        WHEN PredictionDate >= '2025-04-01'
         AND PredictionDate <  '2025-07-01'
            THEN 'Test'
    END AS DatasetPurpose
FROM PolicyRenewalHistory
WHERE PredictionDate < '2025-07-01';

Verify the distribution:

WITH SplitData AS
(
    SELECT
        Renewed,
        CASE
            WHEN PredictionDate < '2025-01-01'
                THEN 'Training'
            WHEN PredictionDate < '2025-04-01'
                THEN 'Validation'
            WHEN PredictionDate < '2025-07-01'
                THEN 'Test'
        END AS DatasetPurpose
    FROM PolicyRenewalHistory
    WHERE PredictionDate < '2025-07-01'
)
SELECT
    DatasetPurpose,
    COUNT(*) AS TotalRows,
    SUM(CASE WHEN Renewed = 1 THEN 1 ELSE 0 END) AS RenewedRows,
    SUM(CASE WHEN Renewed = 0 THEN 1 ELSE 0 END) AS NonRenewedRows
FROM SplitData
GROUP BY DatasetPurpose;

Each subset should be inspected for:

  • Adequate row count
  • Both label classes
  • Unexpected period changes
  • Missing values
  • Duplicate entities
  • Representative business categories

Step 8: Prevent customer overlap

Check whether a customer appears in more than one subset:

WITH SplitData AS
(
    SELECT
        CustomerId,
        CASE
            WHEN PredictionDate < '2025-01-01'
                THEN 'Training'
            WHEN PredictionDate < '2025-04-01'
                THEN 'Validation'
            ELSE 'Test'
        END AS DatasetPurpose
    FROM PolicyRenewalHistory
    WHERE PredictionDate < '2025-07-01'
)
SELECT CustomerId
FROM SplitData
GROUP BY CustomerId
HAVING COUNT(DISTINCT DatasetPurpose) > 1;

A returned customer is not automatically an error. You must decide based on the intended production scenario.

For example:

  • Predicting future policies for existing customers may permit overlap.
  • Evaluating performance on entirely new customers requires customer-grouped splitting.

The evaluation design must match the production question.

PreviousNext