BUSINESS SYSTEMS
│
├── SQL tables
├── CSV/Excel
├── APIs and JSON
├── Logs
├── Documents
└── Images, audio and video
          │
          ▼
    DATA COLLECTION
          │
          ▼
    DATA VALIDATION
    ├── Missing values?
    ├── Duplicates?
    ├── Invalid ranges?
    ├── Inconsistent categories?
    ├── Sensitive data?
    └── Future information leaked?
          │
          ▼
    DATASET DESIGN
    ├── Identifier: identifies the row
    ├── Features: inputs available at prediction time
    └── Label: correct historical outcome
          │
          ▼
    MODEL DEVELOPMENT
    ├── Training data
    ├── Validation data
    └── Test data
          │
          ▼
       MODEL
          │
          ▼
    NEW FEATURES → PREDICTION

Remember

Database table ≠ automatically valid training dataset

Available column ≠ useful feature

High data volume ≠ high data quality

Historical outcome ≠ future feature
PreviousNext