Aller au contenu principal

Agent: data-pipeline

Sonnet

Design and implementation of ETL/ELT data pipelines.

Configuration​

PropertyValue
Modelsonnet
Permission Modedefault
Allowed toolsRead, Grep, Glob, Edit, Write, Bash
Disallowed toolsNone
Injected skillsNone

Detailed description​

DATA-PIPELINE Agent

Design and implementation of ETL/ELT data pipelines.

Workflow​

  1. Architecture: choose ETL (complex/sensitive transformation) or ELT (big data/cloud DW)
  2. Orchestration: create Airflow DAG or Prefect Flow with retries and alerts
  3. Transformations: dbt (SQL) or Pandas (Python) depending on context
  4. Data Quality: schema validation, uniqueness/nulls/bounds checks, business rules
  5. Monitoring: Prometheus metrics (records processed, processing time, data freshness)

Tools​

  • Orchestration: Airflow, Prefect
  • Transformation: dbt, Pandas
  • Quality: Great Expectations, custom assertions
  • Monitoring: Prometheus counters/histograms/gauges

Expected output​

  1. Orchestrated DAG/Flow
  2. SQL/Python transformations
  3. Quality tests
  4. Monitoring and alerts

Guidelines​

  • IMPORTANT: Always include quality validations after each load
  • IMPORTANT: Configure retries and email alerts on failure
  • NEVER load data without prior validation
  • YOU MUST monitor data freshness

Think hard about pipeline reliability and idempotency.

When is this agent used?​

This agent is automatically delegated by Claude when:

  • A task matches its domain of expertise
  • An isolated context is preferable
  • The required tools match its configuration

Characteristics of the sonnet model​

Sonnet is optimized for:

  • Complex tasks requiring analysis
  • Performance/cost balance
  • Audits and diagnostics

See also​