1. What is a data pipeline?

A data pipeline is a defined sequence that moves and transforms data from source systems into a usable destination.

Traditional application pipeline:

HTTP request → Validation → Business service → Database

ML training pipeline:

Source data
    ↓
Extract
    ↓
Validate
    ↓
Create point-in-time features
    ↓
Split data
    ↓
Fit transformations
    ↓
Train model
    ↓
Evaluate
    ↓
Save approved model

Production scoring pipeline:

Current business record
    ↓
Create the same features
    ↓
Apply saved transformations
    ↓
Run saved model
    ↓
Store prediction
    ↓
Trigger approved business action

2. Training pipeline

The training pipeline creates a model from historical labeled data.

Its responsibilities include:

• Choosing an approved historical period
• Extracting labeled examples
• Validating schema and quality
• Creating features
• Splitting data
• Fitting transformations
• Training the algorithm
• Measuring performance
• Saving the model and metadata

Training may happen:

• Manually during experimentation
• Weekly or monthly
• When enough new labels become available
• When monitoring detects deterioration
• After an approved feature change

Training does not normally need to run for every prediction.

3. Scoring pipeline

Scoring, also called inference, means using a trained model to make predictions.

Example:

Policy features
        ↓
Saved model
        ↓
Non-renewal probability = 0.78

The scoring pipeline does not normally relearn the model. It loads an approved trained model and applies it.

Scoring can be:

Online scoring

One or a few predictions are requested immediately.

ASP.NET Core request
        ↓
Build features
        ↓
Model prediction
        ↓
Return result

Use it when a user or system needs a low-latency result.

Batch scoring

Many records are scored together on a schedule.

Every morning
      ↓
Find policies reaching 30-day expiry point
      ↓
Score all eligible policies
      ↓
Create ranked retention worklist

For policy renewal, daily batch scoring may be simpler and cheaper than a real-time API.

4. Reproducibility

Reproducibility means another approved run can recreate the same result using the same:

• Data
• Code
• Query
• Feature definitions
• Package versions
• Random seed
• Parameters
• Environment

A weak experiment record says:

“We trained the model using recent policy data.”

A reproducible record says:

Dataset snapshot: PolicyRenewalTraining_v3
As-of date: 2026-06-30
SQL feature query version: 4
Code commit: a21f9c7
ML.NET package: recorded version
Random seed: 42
Training period: 2023-01-01 to 2025-12-31
Validation period: 2026-01-01 to 2026-03-31
Test period: 2026-04-01 to 2026-06-30
Model artifact: policy-renewal-v7.zip

5. Dataset snapshot

A dataset snapshot is a fixed representation of the data used for one training run.

Why not simply rerun the current SQL query later?

Because operational data changes:

• Incorrect rows are corrected.
• Claims are added retrospectively.
• Customers are merged.
• Categories are renamed.
• Records are deleted.
• Late outcomes arrive.
• Business rules change.

If the source changes, the same query may produce a different dataset.

A snapshot can be:

• Immutable CSV or Parquet file
• Versioned SQL extract table
• Versioned Azure Machine Learning data asset
• Immutable object-storage artifact

6. Data lineage

Data lineage records where data came from and how it changed.

Example:

Policies + Claims + Complaints
             ↓
SQL feature query v4
             ↓
Dataset snapshot v3
             ↓
ML.NET feature pipeline v2
             ↓
Model v7
             ↓
Production predictions

Lineage helps answer:

• Which data trained this model?
• Which query produced the features?
• Which model generated a prediction?
• Can an incorrect source record affect the result?
• Which predictions require review after a defect?

7. Schema contract

A schema contract defines the structure expected by the pipeline.

Example:

A schema change should not silently enter production.

Examples of breaking changes:

PremiumAmount: decimal → formatted string
PolicyType renamed to ProductCategory
NULL begins appearing in PreviousRenewalCount
DidNotRenew meaning changes

8. Training-serving skew

Training-serving skew means features are calculated differently during training and production scoring.

Example:

Training:
ComplaintsLast90Days includes complaints before prediction date.

Production:
ComplaintsLast90Days accidentally includes today’s post-prediction events.

Another example:

Training premium is stored in rupees.
Production API supplies premium in paise.

The application may compile successfully while predictions become unreliable.

The strongest protection is to reuse:

• The same feature definitions
• The same transformations
• The same category encoding
• The same saved ML.NET pipeline

9. Idempotency

An idempotent pipeline produces the same logical result when rerun with the same inputs.

Suppose daily scoring is accidentally run twice.

Incorrect behaviour:

First run  → creates retention task
Second run → creates another identical task

Correct behaviour:

Unique key:
PolicyId + PredictionDate + ModelVersion

Second run detects existing prediction
and safely updates or skips it.

Idempotency is familiar from reliable payment, migration and background-job design.

10. Version the complete prediction

A production prediction should record more than a probability.

PolicyId
PredictionTimestamp
PredictionMoment
ModelVersion
FeatureDefinitionVersion
RiskProbability
ThresholdVersion
PredictedClass
ActionTaken
ReviewerOutcome

This provides traceability and future labeled data.