> ## Documentation Index
> Fetch the complete documentation index at: https://support.entegrata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploying Pipelines

> How to publish pipelines to production and generate DLT scripts

## Overview

Deploying a pipeline publishes all configured data type mappings to your production data warehouse, generating optimized Delta Live Tables (DLT) scripts that transform source data according to your mapping configurations. Deployment is required before a pipeline can execute with real data.

<Info>
  Deployment is a one-way process that generates production-ready code. Always test your mappings thoroughly with dry runs before deploying.
</Info>

## What Deployment Does

When you deploy a pipeline, the system:

1. **Validates Configuration** - Checks that all mappings are complete and valid
2. **Generates DLT Scripts** - Creates optimized SQL/Python code for data transformations
3. **Publishes to Warehouse** - Deploys scripts to your Databricks environment
4. **Updates Status** - Changes pipeline status from Draft to Deployed
5. **Enables Execution** - Makes the pipeline available for running (manual or scheduled)

## Deployment Status

Pipelines have one of these deployment statuses:

<ResponseField name="Draft" type="status">
  Pipeline has been created but not yet deployed. Cannot execute with real data.
</ResponseField>

<ResponseField name="Deployed" type="status">
  Pipeline has been published to production and is ready to execute.
</ResponseField>

<Info>
  After deployment, any changes to data type mappings require redeployment to take effect in production.
</Info>

## Prerequisites

Before deploying a pipeline, ensure:

* **Mappings Are Complete** - All data types have configured field mappings
* **Sources Are Configured** - Primary and related sources are properly connected
* **Required Fields Are Mapped** - All mandatory fields have valid mappings or defaults
* **Testing Is Done** - Dry runs have been performed and validated
* **Permissions Are Granted** - You have deployment permissions

<Warning>
  Deploying overwrites any previous deployment of the same pipeline. Ensure you're ready to replace the existing configuration.
</Warning>

## Deploying a Pipeline

<Steps>
  <Step title="Navigate to Pipeline List">
    Log in to the [Entegrata Admin Portal](https://admin.entegrata.com) and go to the **Pipelines** tab.
  </Step>

  <Step title="Locate the Pipeline">
    Find the pipeline you want to deploy.
  </Step>

  <Step title="Verify Mappings Are Ready">
    Before deploying, click the pipeline name to review:

    * All intended data types are configured
    * Field mappings are complete
    * Sources are properly connected
    * Transformations are tested

    <Tip>
      Run a dry run to validate mappings before deploying. This catches errors without impacting production.
    </Tip>
  </Step>

  <Step title="Return to Pipeline List">
    Navigate back to the pipeline list after verifying mappings.
  </Step>

  <Step title="Open Actions Menu">
    Click the three-dot menu (⋮) in the Actions column for the pipeline you want to deploy.

    <Frame>
      <img src="https://mintcdn.com/entegrata/sWO-t3fu8JP8xl73/images/pipelines/pipeline_list_row_options.png?fit=max&auto=format&n=sWO-t3fu8JP8xl73&q=85&s=dafdee9df07abc831baec79f1f4a8484" alt="Pipeline actions menu" width="1200" height="748" data-path="images/pipelines/pipeline_list_row_options.png" />
    </Frame>
  </Step>

  <Step title="Select Deploy">
    From the actions menu, click **Deploy** to open the deployment confirmation dialog.

    <Frame>
      <img src="https://mintcdn.com/entegrata/sWO-t3fu8JP8xl73/images/pipelines/pipeline_list_deploy_option.png?fit=max&auto=format&n=sWO-t3fu8JP8xl73&q=85&s=bda3b020130e645fbc91c949f232f149" alt="Deploy option in actions menu" width="2724" height="656" data-path="images/pipelines/pipeline_list_deploy_option.png" />
    </Frame>
  </Step>

  <Step title="Review Deployment Information">
    The deployment confirmation dialog shows:

    * Pipeline name
    * Number of data types that will be deployed
    * Warning about replacing existing deployment (if applicable)

    <Warning>
      Deployment will deploy all mapped data types in this pipeline. Ensure you've reviewed all mappings, not just the ones you recently changed.
    </Warning>
  </Step>

  <Step title="Confirm Deployment">
    Click **Deploy Pipeline** to proceed with deployment.

    The deployment process begins and may take several minutes depending on:

    * Number of data types being deployed
    * Complexity of field mappings
    * Size of transformation logic
  </Step>

  <Step title="Wait for Completion">
    Monitor the deployment progress. During deployment:

    * A loading indicator shows deployment is in progress
    * You cannot modify the pipeline
    * Other operations on the pipeline are blocked

    <Note>
      Do not close the browser or navigate away during deployment. Wait for the confirmation message.
    </Note>
  </Step>

  <Step title="Verify Deployment Success">
    Upon successful deployment, you'll see:

    * Success confirmation message
    * Pipeline status changes to **Deployed**
    * Last Modified timestamp is updated
    * Pipeline is now ready to run
  </Step>
</Steps>

## After Deployment

Once a pipeline is deployed, you can:

### Run the Pipeline

Execute the pipeline to process data using your deployed mappings.

<CardGroup cols={1}>
  <Card title="Running Pipelines" icon="play" href="/data/pipeline/running-pipelines">
    Learn how to execute deployed pipelines
  </Card>
</CardGroup>

### Activate for Scheduled Runs

If the pipeline has a scheduled trigger, activate it to enable automatic execution.

<CardGroup cols={1}>
  <Card title="Managing Pipeline Status" icon="toggle-on" href="/data/pipeline/managing-pipeline-status">
    Learn how to activate pipelines for scheduled execution
  </Card>
</CardGroup>

### Monitor Execution

Track pipeline runs and review execution logs to ensure everything works as expected.

### Make Updates

If you need to change mappings after deployment:

1. Modify the mappings in the pipeline editor
2. Test with dry run
3. Deploy again to publish changes

<Info>
  Changes to mappings are not reflected in production until you redeploy. Draft changes don't affect running pipelines.
</Info>

## Redeployment

Redeploying an already-deployed pipeline follows the same process as initial deployment. Use redeployment when:

### Mapping Changes

After modifying field mappings, transformations, or data sources.

### Adding Data Types

After adding new data type mappings to an existing pipeline.

### Fixing Errors

After correcting mapping errors discovered during execution.

### Schema Updates

After changes to source or target schemas that require mapping updates.

<Warning>
  Redeployment replaces the entire pipeline deployment. Test all mappings, not just changed ones, to avoid breaking working configurations.
</Warning>

## Deployment Validation

The deployment process validates:

### Required Fields

* All mandatory data type fields have mappings or default values
* No orphaned field references

### Source Connections

* All referenced sources are accessible
* Source tables/resources exist

### Transformation Logic

* SQL expressions are syntactically correct
* Functions used are supported
* Type conversions are valid

### Data Type Configuration

* Data types are properly configured
* No circular dependencies
* Primary keys are defined

<Note>
  If validation fails, deployment is aborted and you'll see specific error messages. Fix the errors and try deploying again.
</Note>

## Deployment Best Practices

<Tip>
  **Test Thoroughly Before Deploying**

  Always perform these checks before deployment:

  1. Run dry run to validate query logic
  2. Review all field mappings visually
  3. Check for any validation warnings
  4. Verify source connections are active
  5. Confirm required fields are mapped

  This prevents deploying broken configurations.
</Tip>

<Tip>
  **Deploy During Maintenance Windows**

  For production pipelines:

  * Deploy during scheduled maintenance windows
  * Avoid deployment during active pipeline runs
  * Notify team members of deployment timing
  * Have rollback plan ready if issues arise
</Tip>

<Tip>
  **Document Deployment Changes**

  After deploying, update the pipeline description to note:

  * What was changed in this deployment
  * Date of deployment
  * Reason for changes
  * Any known issues or considerations

  Example: "Deployed 2024-10-02: Added Account data type mapping, updated Client email field transformation"
</Tip>

<Tip>
  **Monitor First Run After Deployment**

  After deploying:

  1. Run the pipeline manually first (don't rely on schedule)
  2. Watch execution logs closely
  3. Verify data quality in warehouse
  4. Check row counts match expectations
  5. Validate transformations worked correctly

  Only activate for scheduled runs after confirming success.
</Tip>

<Tip>
  **Version Control Significant Changes**

  Before major redeployments:

  * Duplicate the pipeline to create a backup
  * Name it with version number or date
  * Keep the backup until new version is validated
  * Allows quick rollback if needed
</Tip>

## Deployment Strategies

### Blue-Green Deployment

For critical pipelines, use a blue-green strategy:

1. **Blue (Current)**: Keep existing deployed pipeline running
2. **Green (New)**: Create duplicate pipeline with changes
3. **Test Green**: Thoroughly test the new version
4. **Switch**: Deactivate blue, activate green
5. **Monitor**: Watch green for issues
6. **Cleanup**: Delete blue after green is validated

### Phased Deployment

For complex changes, deploy in phases:

1. **Phase 1**: Deploy subset of data types
2. **Validate**: Test phase 1 thoroughly
3. **Phase 2**: Deploy additional data types
4. **Validate**: Test phase 2 thoroughly
5. **Continue**: Add remaining data types incrementally

### Rollback Strategy

Always have a rollback plan:

1. **Before Deployment**: Duplicate pipeline as backup
2. **After Deployment**: Keep backup active initially
3. **If Issues Arise**: Reactivate backup pipeline
4. **Fix Issues**: Correct problems in new version
5. **Redeploy**: Try again after fixes

## Troubleshooting Deployment

### Deployment Fails with Validation Errors

**Problem:** Deployment is blocked due to validation errors.

**Solutions:**

* Review error messages for specific issues
* Check that all required fields are mapped
* Verify source connections are accessible
* Fix transformation syntax errors
* Ensure data types are properly configured

### Deployment Hangs or Takes Too Long

**Problem:** Deployment process doesn't complete or runs for excessive time.

**Solutions:**

* Wait at least 10-15 minutes for complex pipelines
* Check network connection stability
* Refresh page after 15+ minutes to see if it completed
* Contact support if consistently hanging

### Deployment Succeeds But Pipeline Fails

**Problem:** Deployment completes successfully but pipeline execution fails.

**Solutions:**

* Deployment validates configuration, not runtime execution
* Run dry run to identify execution issues
* Check source data availability and quality
* Review transformation logic with actual data
* Verify warehouse permissions for writing data

### Can't Deploy (No Deploy Option)

**Problem:** Deploy option is not available in actions menu.

**Solutions:**

* Verify you have deployment permissions
* Check if pipeline is already deployed (may need to redeploy instead)
* Ensure pipeline has at least one data type mapping configured
* Verify you're not in the middle of editing the pipeline

### Deployed But Changes Not Reflected

**Problem:** After deployment, changes aren't visible in pipeline execution.

**Solutions:**

* Ensure you redeployed after making changes
* Wait a few minutes for deployment to propagate
* Run the pipeline fresh (not continuing previous run)
* Clear any cached query plans in warehouse
* Verify changes were saved before deploying

## Related Topics

<CardGroup cols={2}>
  <Card title="Running Pipelines" icon="play" href="/data/pipeline/running-pipelines">
    Execute deployed pipelines
  </Card>

  <Card title="Managing Status" icon="toggle-on" href="/data/pipeline/managing-pipeline-status">
    Activate deployed pipelines for scheduled runs
  </Card>

  <Card title="Data Mapping" icon="diagram-project" href="/data/pipeline/data-mapping-overview">
    Configure mappings before deployment
  </Card>

  <Card title="Duplicating Pipelines" icon="copy" href="/data/pipeline/duplicating-pipelines">
    Create backups before major deployments
  </Card>
</CardGroup>