Part A: Training or scoring?

Mark each activity:

  1. Fit category encoding using historical policy types.
  2. Load an approved model.zip.
  3. Learn normalization minimum and maximum values.
  4. Predict risk for today’s eligible policies.
  5. Compare algorithms using validation data.
  6. Create today’s retention worklist.
  7. Evaluate the final model using the sealed test set.
  8. Store the model version with each prediction.

Choose:

Training pipeline 
Scoring pipeline 
Both or shared concern

Part B: Find reproducibility gaps

Audit this experiment note:

Model trained on recent data.
Used the normal SQL query.
Accuracy was good.
Saved model-final.zip.

Identify at least eight missing items, such as:

  • Exact data period
  • Dataset version
  • Query version
  • Row count
  • Feature version
  • Code version
  • Random seed
  • Package versions
  • Split definition
  • Evaluation metrics
  • Model version

Part C: Detect training-serving skew

Identify the mismatch:

Training Production
Premium in rupees Premium in paise
Policy type trimmed and uppercase Raw policy type
Complaints in previous 90 days Lifetime complaints
Missing claims represented separately Missing claims converted to zero
Customer tenure in months Customer tenure in days

Write how each mismatch could be prevented.

Part D: Design the Assessment OS pipeline

Complete:

Prediction unit:
Prediction moment:
Source tables:
Feature query version:
Dataset snapshot:
Grouping key:
Training split:
Validation split:
Test split:
Feature pipeline version:
Model version:
Scoring mode:
Prediction audit fields:
Human review outcome:
Unique key preventing duplicate predictions:

Part E: Create a manifest

Using the C# TrainingRunManifest, create one manifest for:

Assessment OS exam-session risk model

Use reasonable sample values and serialize it to JSON.

Do not place passwords, connection strings or API keys inside a run manifest.

Previous