Skip to main content

Running Your Pipelines in Dagster

Quick Start: Run Your First Pipeline​

  1. Navigate to the Assets page
  2. Click Materialize all in the top right
  3. Confirm the run
  4. Monitor progress in the Runs page

Method 2: Run Specific Assets​

  1. Go to Assets page
  2. Select assets by checking boxes
  3. Click Materialize selected
  4. Useful for testing individual components

Method 3: Launch a Job​

  1. Navigate to Jobs page
  2. Find your job (e.g., meltano_dbt_pipeline)
  3. Click Launch Run
  4. Configure if needed (usually not required)

Understanding Your Pipeline Structure​

Typical Pipeline Flow​

Meltano Extractors → Raw Data → dbt Models → Analytics Tables
    (Assets)         (Assets)      (Assets)        (Assets)

Asset Dependencies​

  • Meltano assets must complete before dbt assets
  • Dependencies are automatically managed
  • Failed upstream assets prevent downstream execution

Monitoring Your Runs​

Run Status Indicators​

  • 🟢 Success: Asset materialized successfully
  • 🔵 In Progress: Currently executing
  • 🔴 Failed: Error occurred (check logs)
  • ⚪ Queued: Waiting to start

Viewing Run Details​

  1. Click on any run in the Runs page
  2. See:
    • Timeline of execution
    • Logs for each step
    • Asset materializations
    • Error messages (if any)

Understanding Logs​

Meltano Logs​

2024-01-15 10:30:45 [info] Running extract/load for tap-salesforce
2024-01-15 10:30:46 [info] Updated state: 2024-01-15T10:30:46
2024-01-15 10:31:15 [info] Extracted 1,543 records

dbt Logs​

10:31:20  Running dbt...
10:31:25  Found 23 models, 15 tests
10:31:30  Completed successfully

Validating Your Migration​

Checklist for First Run​

  • All Meltano extractors complete successfully
  • Data lands in expected raw tables
  • dbt models build without errors
  • Row counts match expectations
  • Schedules show correct next run time

Comparing with Arch​

Check Data Freshness​

  1. Go to Assets page
  2. Look at "Last Materialized" timestamps
  3. Compare with last Arch run times

Verify Row Counts​

-- Run in your data warehouse
SELECT COUNT(*) FROM raw.salesforce_accounts;
SELECT COUNT(*) FROM analytics.customer_summary;

Handling Common Scenarios​

Rerunning Failed Assets​

  1. Click on the failed asset
  2. Review error in logs
  3. Click Re-materialize after fixing issue

Running Historical Backfills​

  1. Select assets needing backfill
  2. Click Materialize selected
  3. Choose "Custom Run Config"
  4. Specify date range for Meltano

Partial Pipeline Runs​

Run only dbt models (skip extraction):

  1. Select only dbt assets
  2. Materialize selected
  3. Useful for model development

State Management​

Meltano State​

  • Automatically managed by platform
  • Tracks last successful extraction
  • Prevents duplicate data
  • No manual intervention needed

Viewing State​

  1. Click on Meltano asset
  2. Check "Metadata" tab
  3. See last bookmark value

Scheduling Runs​

View Existing Schedules​

  1. Go to Schedules page
  2. See all migrated schedules
  3. Check next tick time

Schedule Status​

  • ▶️ Running: Schedule is active
  • ⏸️ Stopped: Manually paused
  • 🔄 Refreshing: Updating schedule

Manual Schedule Control​

  • Start/Stop: Toggle schedule on/off
  • Test: Run immediately regardless of schedule
  • Edit: Modify cron expression (requires code change)

Troubleshooting​

Common Issues​

"Asset not found" Error​

  • Ensure all dependencies materialized
  • Check if asset definition exists
  • Reload definitions in UI

Meltano State Issues​

  • DO NOT manually modify state
  • Contact support for state resets
  • Check logs for specific state errors

dbt Compilation Errors​

  • Review dbt logs for model name
  • Check for missing dependencies
  • Validate warehouse permissions

Getting Run URLs​

  1. Click on any run
  2. Copy URL from browser
  3. Share with support for help

Best Practices​

Daily Operations​

  1. Check Assets page for freshness
  2. Review any failed runs
  3. Monitor schedule execution
  4. Validate critical metrics

Development Workflow​

  1. Test changes with single asset runs
  2. Validate in development branch first
  3. Monitor first production run closely
  4. Document any configuration changes

Next Steps​