> ## Documentation Index
> Fetch the complete documentation index at: https://support.entegrata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipelines Overview

> Comprehensive guide to Entegrata pipeline management and data mapping capabilities

## Introduction

The Entegrata Pipeline system enables you to create, configure, and manage data pipelines that transform and load data from your Collector sources into the Entegrata platform. Pipelines provide a powerful framework for defining how your source data maps to standardized data types, allowing you to test configurations before deploying them to production.

<Info>
  Pipelines are the foundation of your data integration strategy. They define the transformation logic that converts raw data from your sources into structured, queryable data types.
</Info>

## What are Pipelines?

Pipelines in Entegrata serve two primary functions:

1. **Data Orchestration** - Define when and how your data processing jobs execute
2. **Data Mapping** - Configure the transformation logic that maps source data to canonical data types

Each pipeline contains one or more data type mappings that specify:

* Which data sources to pull from
* How fields map between source and target schemas
* Transformation logic for data cleansing and enrichment
* Default values and data quality rules

## Key Features

<CardGroup cols={2}>
  <Card title="Pipeline Management" icon="gear">
    Create, edit, duplicate, and delete pipelines with full version control and audit history
  </Card>

  <Card title="Visual Mapping Editor" icon="diagram-project">
    Use the intuitive drag-and-drop interface to map source fields to data type fields
  </Card>

  <Card title="Test Before Deploy" icon="flask">
    Run pipelines in test mode to validate mappings without affecting production data
  </Card>

  <Card title="Scheduled Execution" icon="clock">
    Configure pipelines to run automatically on schedules or trigger them manually
  </Card>

  <Card title="Multi-Source Support" icon="database">
    Map data from multiple sources into a single data type with primary and related sources
  </Card>

  <Card title="Advanced Transformations" icon="wand-magic-sparkles">
    Use COALESCE, CONCAT, CASE statements, and custom SQL for complex field mappings
  </Card>
</CardGroup>

## Pipeline Workflow

The typical pipeline workflow follows these stages:

<Steps>
  <Step title="Create Pipeline">
    Set up a new pipeline with a name, description, and trigger configuration
  </Step>

  <Step title="Add Data Types">
    Define which data types (entities) this pipeline will process
  </Step>

  <Step title="Configure Sources">
    Connect to your Collector data sources and select the tables or resources to use
  </Step>

  <Step title="Map Fields">
    Use the visual editor to map source fields to data type fields, applying transformations as needed
  </Step>

  <Step title="Test Execution">
    Run the pipeline in dry-run mode to validate your mappings and catch errors
  </Step>

  <Step title="Deploy">
    Publish your pipeline to production, generating optimized DLT scripts
  </Step>

  <Step title="Monitor">
    Track pipeline execution status, view logs, and troubleshoot issues
  </Step>
</Steps>

## Pipeline vs. Data Mapping

It's important to understand the relationship between pipelines and data mappings:

* **Pipeline** - The container that organizes and schedules data processing jobs. A pipeline can include multiple data type mappings.
* **Data Mapping** - The specific configuration that maps one or more source tables to a single data type (entity).

Think of a pipeline as a project that groups related data mappings together. For example, you might have a "Client Data Processing Pipeline" that includes mappings for Client, Contact, and Address data types.

## Getting Started

<CardGroup cols={2}>
  <Card title="Pipeline Management" icon="list-check" href="/data/pipeline/overview">
    Learn how to create and manage pipelines
  </Card>

  <Card title="Data Mapping" icon="diagram-project" href="/data/pipeline/data-mapping-overview">
    Explore the visual mapping editor and field configuration
  </Card>

  <Card title="Creating Pipelines" icon="plus" href="/data/pipeline/creating-pipelines">
    Step-by-step guide to creating your first pipeline
  </Card>

  <Card title="Running Pipelines" icon="play" href="/data/pipeline/running-pipelines">
    Execute and monitor pipeline jobs
  </Card>
</CardGroup>

## Common Use Cases

### Daily Client Data Refresh

Create a scheduled pipeline that runs nightly to pull updated client information from your CRM, mapping it to standardized Client and Contact data types.

### Historical Data Migration

Build a one-time pipeline to migrate legacy data, using complex field mappings to handle data quality issues and schema differences.

### Multi-System Integration

Configure pipelines that combine data from multiple sources (e.g., CRM, ERP, billing systems) into unified data types for comprehensive reporting.

### Incremental Updates

Set up event-driven pipelines that process only new or changed records, ensuring your Entegrata data stays current with minimal processing overhead.

## Next Steps

Ready to start building pipelines? Choose your path:

<CardGroup cols={3}>
  <Card title="Create Your First Pipeline" icon="rocket" href="/data/pipeline/creating-pipelines">
    Follow our quickstart guide
  </Card>

  <Card title="Understanding Data Types" icon="shapes" href="/data/pipeline/managing-entities">
    Learn about data type management
  </Card>

  <Card title="Advanced Mapping" icon="code" href="/data/pipeline/multi-field-mapping">
    Explore transformation options
  </Card>
</CardGroup>