Overview
Pipeline management in Entegrata provides a centralized interface for organizing and controlling your data integration workflows. From this dashboard, you can create new pipelines, monitor their execution status, deploy changes to production, and manage the entire lifecycle of your data processing jobs.Accessing Pipeline Management
To access the pipeline management interface:- Log in to the Entegrata Admin Portal
- Navigate to the Pipelines tab in the main navigation
- You’ll see the pipeline list view with all pipelines in your instance
Pipeline List View
The pipeline list view displays all pipelines with the following information:Pipeline Columns
The descriptive name of the pipeline (e.g., “Client Data Processing Pipeline”)
A brief description of what the pipeline does and what data it processes
When the pipeline configuration was last updated
When the pipeline was last executed (manually or automatically)
Current pipeline status combining deployment status and run status:
- Draft - Pipeline has not been deployed yet
- Deployed - Pipeline is deployed and ready to run
- Active - Pipeline is deployed and scheduled to run automatically
- Paused - Pipeline is deployed but paused from automatic execution
- Running - Pipeline is currently executing
- Failed - Pipeline last run encountered errors
- Success - Pipeline last run completed successfully
Quick action menu for pipeline operations (see below)
Pipeline Actions Menu
Each pipeline has an actions menu (⋮) that provides quick access to common operations:- Edit - Modify pipeline name and description
- Duplicate - Create a copy of the pipeline with all its mappings
- Run - Execute the pipeline immediately (manual trigger)
- Dry Run - Test the pipeline without writing data to production
- Deploy - Publish all mapped data types to production
- Pause/Activate - Toggle automatic execution on or off
- View Mappings - Navigate to the data mapping editor
- Delete - Remove the pipeline (requires confirmation)
Pipeline Lifecycle
Understanding the pipeline lifecycle helps you manage your data integration effectively:Draft
When first created, pipelines are in draft status. You can configure data type mappings, but the pipeline is not yet available for execution in production.
Deployed
After deploying, the pipeline’s DLT scripts are generated and published to your data warehouse. The pipeline can now be executed but won’t run automatically unless activated.
Active
Active pipelines are deployed and configured to run automatically based on their trigger settings (e.g., scheduled execution).
Pipeline Trigger Types
When creating or editing a pipeline, you can configure how it should be triggered:Manual Trigger
Pipelines run only when explicitly triggered by a user through the “Run” action. Best for:- Ad-hoc data migrations
- Testing and development
- One-time data loads
Event-Driven Trigger
Pipelines run automatically when specific events occur (e.g., new data detected in source). Best for:- Real-time or near-real-time data processing
- Event-based workflows
- Triggered by external systems
Event-driven triggers require additional configuration with your Collector and may not be available for all data sources.
No Trigger (Manual Only)
Pipeline has no automatic trigger configured and must be run manually.Searching and Filtering Pipelines
Use the search bar at the top of the pipeline list to quickly find pipelines by name. The search supports:- Partial name matching
- Case-insensitive search
- Real-time filtering as you type
Pipeline Execution Monitoring
Monitor pipeline execution through several indicators:Run Status Indicators
- Success (Green) - Pipeline completed without errors
- Running (Blue) - Pipeline is currently executing
- Failed (Red) - Pipeline encountered errors during execution
- Pending (Gray) - Pipeline is queued for execution
Best Practices
Common Operations
Creating a New Pipeline
Modifying Existing Pipelines
Edit Pipeline
Update pipeline name, description, and settings
Duplicate Pipeline
Create a copy of an existing pipeline
Execution and Deployment
Run Pipeline
Execute pipelines manually or on schedule
Deploy Pipeline
Publish pipelines to production
Manage Status
Pause, activate, or monitor pipeline status
Related Topics
Data Mapping
Learn how to configure data type mappings within pipelines
Managing Data Types
Create and manage the data types your pipelines process
Troubleshooting
Pipeline Won’t Run
- Check that the pipeline is deployed (not in draft status)
- Verify the pipeline is activated (not paused)
- Ensure your Collector data sources are connected and accessible
Pipeline Failing Consistently
- Check source data quality and schema changes
- Verify required fields have valid mappings or default values
- Review data type field requirements
Pipeline Running Too Long
- Check for inefficient field mappings or complex transformations
- Verify source queries are optimized with appropriate filters
- Review data volumes and adjust processing strategies
For additional support, contact your Entegrata administrator or submit a ticket through the Support Portal.