1. What is a dataset?

A dataset is an organized collection of examples used for analysis, training or evaluation.

In SQL terms, a simple dataset resembles the output of a query:

SELECT
    CustomerAge,
    PremiumAmount,
    ClaimCount,
    PreviousRenewalCount,
    Renewed
FROM PolicyHistory;

Each returned row represents one example.

CustomerAge PremiumAmount ClaimCount PreviousRenewalCount Renewed
34 12500 0 2 1
52 18400 3 0 0
41 15000 1 4 1

For this dataset:

  • One row = one historical policy
  • Input columns = features
  • Expected result = label
  • Entire collection = dataset

2. What is a feature?

A feature is an input supplied to a model so it can make a prediction.

For policy-renewal prediction, possible features include:

  • Customer age
  • Premium amount
  • Number of claims
  • Previous renewal count
  • Number of support complaints
  • Policy type
  • Payment delays

In C#, features can be represented as properties:

public sealed class PolicyTrainingRow
{
    public float CustomerAge { get; set; }
    public float PremiumAmount { get; set; }
    public float ClaimCount { get; set; }
    public float PreviousRenewalCount { get; set; }
    public bool Renewed { get; set; }
}

A database column does not automatically become a useful feature.

For example:

PolicyId = 984729

This value identifies a record, but normally does not explain whether the customer will renew.

3. What is a label?

A label is the known result that we want a supervised ML model to learn.

For renewal prediction:

Renewed = true or false

For support-ticket routing:

Category = Billing, Technical, Retention or General

For premium prediction:

ExpectedPremium = 18,500

Different problems have different label types:

Problem Label ML task
Will a policy renew? Yes/No Binary classification
Which department handles this ticket? Department name Multiclass classification
What will the house sell for? Price Regression
How many orders next month? Future order count Forecasting

Not every ML problem has a label. Clustering, for example, can group records without predefined outcomes. We will study supervised and unsupervised learning later.

4. Features versus labels versus identifiers

Consider this table:

Column Role
PolicyId Identifier
CustomerAge Feature
PremiumAmount Feature
ClaimCount Feature
Renewed Label
RenewalProcessedBy Possibly irrelevant operational data

A useful question for every column is:

“Would this information genuinely be available at the moment the prediction is requested?”

If the answer is no, it must not be used as an input.

5. Data leakage

Data leakage occurs when training data contains information that would not actually be available when making a real prediction.

Suppose we want to predict whether a policy will renew. This is dangerous:

Feature Value
RenewalPaymentDate 15 August
RenewalReceiptNumber RCP-10852
Label: Renewed Yes

The payment date and receipt number reveal that renewal already happened.

A model trained using them may show almost perfect accuracy during testing, but it cannot make a useful advance prediction.

This is similar to writing an exam while already having the answer sheet.

6. Types of data

Structured data

Data organized into a fixed schema:

  • SQL tables
  • CSV files
  • Excel sheets
  • Customer records
  • Transactions
  • Sensor measurements

Example:

PolicyId | Premium | ClaimCount | Renewed

This is naturally familiar to .NET and SQL developers.

Semi-structured data

Data has recognizable fields but not necessarily a relational-table structure:

  • JSON
  • XML
  • Application logs
  • API responses
  • Event messages

Example:

{
  "policyId": 101,
  "premium": 15000,
  "events": ["login", "quote-viewed", "payment-started"]
}

Unstructured data

Data does not arrive as fixed rows and columns:

  • PDF documents
  • Emails
  • Images
  • Audio
  • Video
  • Support conversations
  • Free-form text

Unstructured data normally needs processing before a traditional ML model can use it.

For example:

Support-ticket text
        ↓
Text transformation
        ↓
Numeric feature vector
        ↓
Classification model

7. Common data-quality problems

Missing values

CustomerAge = NULL

Options include:

  • Reject the record
  • Correct it from the source
  • Replace it with a suitable calculated value
  • Create an “Unknown” category
  • Use an algorithm that can handle missing values

Never replace missing values blindly.

Duplicate records

The same policy may appear multiple times and distort the model.

Invalid values

Examples:

CustomerAge = -5 
PremiumAmount = -12000 
ClaimCount = 99999 
Renewed = NULL

Inconsistent categories

Active 
ACTIVE 
active 
A
These could incorrectly appear as four different categories.

Outliers

An outlier is a value far outside the usual pattern.

Example:

Normal premium range: ₹5,000–₹100,000 
Recorded premium: ₹90,000,000

An outlier may be:

  • A data-entry error
  • A valid exceptional case
  • Fraud
  • An important rare event

Do not delete every outlier automatically.

Class imbalance

Suppose fraud data contains:

99,500 normal transactions 
500 fraudulent transactions

A useless model that predicts “normal” every time would achieve:

99,500 ÷ 100,000 = 99.5% accuracy
Therefore, accuracy alone can be misleading. We will study better metrics later.

 

PreviousNext