ML Pipeline Workflow

Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment.

Do not use this skill when

• The task is unrelated to ml pipeline workflow
• You need a different domain or tool outside this scope

Instructions

• Clarify goals, constraints, and required inputs.
• Apply relevant best practices and validate outcomes.
• Provide actionable steps and verification.
• If detailed examples are required, open resources/implementation-playbook.md.

Overview

This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring.

Use this skill when

• Building new ML pipelines from scratch
• Designing workflow orchestration for ML systems
• Implementing data → model → deployment automation
• Setting up reproducible training workflows
• Creating DAG-based ML orchestration
• Integrating ML components into production systems

What This Skill Provides

Core Capabilities

1. Pipeline Architecture
2. End-to-end workflow design
3. DAG orchestration patterns (Airflow, Dagster, Kubeflow)
4. Component dependencies and data flow

5. Error handling and retry strategies

6. Data Preparation

7. Data validation and quality checks
8. Feature engineering pipelines
9. Data versioning and lineage

10. Train/validation/test splitting strategies

11. Model Training

12. Training job orchestration
13. Hyperparameter management
14. Experiment tracking integration

15. Distributed training patterns

16. Model Validation

17. Validation frameworks and metrics
18. A/B testing infrastructure
19. Performance regression detection

20. Model comparison workflows

21. Deployment Automation

22. Model serving patterns
23. Canary deployments
24. Blue-green deployment strategies
25. Rollback mechanisms

Reference Documentation

See the references/ directory for detailed guides: - data-preparation.md - Data cleaning, validation, and feature engineering - model-training.md - Training workflows and best practices - model-validation.md - Validation strategies and metrics - model-deployment.md - Deployment patterns and serving architectures

Assets and Templates

The assets/ directory contains: - pipeline-dag.yaml.template - DAG template for workflow orchestration - training-config.yaml - Training configuration template - validation-checklist.md - Pre-deployment validation checklist

Usage Patterns

Basic Pipeline Setup

# 1. Define pipeline stages
stages = [
    "data_ingestion",
    "data_validation",
    "feature_engineering",
    "model_training",
    "model_validation",
    "model_deployment"
]

# 2. Configure dependencies
# See assets/pipeline-dag.yaml.template for full example

Production Workflow

1. Data Preparation Phase
2. Ingest raw data from sources
3. Run data quality checks
4. Apply feature transformations

5. Version processed datasets

6. Training Phase

7. Load versioned training data
8. Execute training jobs
9. Track experiments and metrics

10. Save trained models

11. Validation Phase

12. Run validation test suite
13. Compare against baseline
14. Generate performance reports

15. Approve for deployment

16. Deployment Phase

17. Package model artifacts
18. Deploy to serving infrastructure
19. Configure monitoring
20. Validate production traffic

Best Practices

Pipeline Design

• Modularity: Each stage should be independently testable
• Idempotency: Re-running stages should be safe
• Observability: Log metrics at every stage
• Versioning: Track data, code, and model versions
• Failure Handling: Implement retry logic and alerting

Data Management

• Use data validation libraries (Great Expectations, TFX)
• Version datasets with DVC or similar tools
• Document feature engineering transformations
• Maintain data lineage tracking

Model Operations

• Separate training and serving infrastructure
• Use model registries (MLflow, Weights & Biases)
• Implement gradual rollouts for new models
• Monitor model performance drift
• Maintain rollback capabilities

Deployment Strategies

• Start with shadow deployments
• Use canary releases for validation
• Implement A/B testing infrastructure
• Set up automated rollback triggers
• Monitor latency and throughput

Integration Points

Orchestration Tools

• Apache Airflow: DAG-based workflow orchestration
• Dagster: Asset-based pipeline orchestration
• Kubeflow Pipelines: Kubernetes-native ML workflows
• Prefect: Modern dataflow automation

Experiment Tracking

• MLflow for experiment tracking and model registry
• Weights & Biases for visualization and collaboration
• TensorBoard for training metrics

Deployment Platforms

• AWS SageMaker for managed ML infrastructure
• Google Vertex AI for GCP deployme