# Data Encryption (CMK) Source: https://support.entegrata.com/administration/instance/data-encryption Manage Customer-Managed Keys for enhanced data encryption and security compliance ## Overview Customer-Managed Keys (CMK) provide enhanced security for your Entegrata instance by allowing you to control the encryption keys used to protect your data in Azure. With CMK, you maintain ownership and control over the cryptographic keys, meeting compliance requirements and providing additional security layers beyond Azure's default encryption. CMK implementation requires uploading PEM-formatted encryption keys to your Entegrata instance. The system monitors key expiration and provides visual alerts to ensure continuous data protection. ## Why Use Customer-Managed Keys? ### Enhanced Security Control **Your Keys, Your Control:** * You generate and own the encryption keys * Keys never leave your control during generation * You decide when to rotate or revoke keys or let Entegrata rotate them yearly by default * Independent from cloud provider's key management ### Flexibility in Key Management Rotate encryption keys on demand to meet security policies and compliance requirements or yearly by default. Immediately revoke access by removing keys, providing an additional security control in breach scenarios. ## Understanding CMK Status ### Status Badge Colors The CMK card displays color-coded badges indicating key expiration status for quick assessment:
Red Badge - Key expires in less than 1 day (immediate action required) Yellow Badge - Key expires in less than 7 days (plan key rotation soon) Blue Badge - Key expires in less than 30 days (schedule key rotation) Green Badge - Key valid for 30+ days (no immediate action needed)
If your CMK shows a **red badge** (expires in less than 1 day), upload a new key immediately. Expired keys may prevent access to encrypted data and cause service disruptions. ## Generating a PEM Key File ### Prerequisites Before generating a CMK, ensure you have: * **[OpenSSL](https://openssl-library.org/source/) installed** on your system (or access to key generation tools) * **Secure key storage** for backup copies * **Documentation process** for tracking key generation ### Using OpenSSL to Generate Keys Access a terminal with OpenSSL installed. Verify installation: ```bash theme={null} openssl version ``` Generate a 4096-bit RSA private key: ```bash theme={null} openssl genrsa -out cmk-private-key.pem 4096 ``` The 4096-bit key length provides strong encryption suitable for most compliance requirements. You can use 2048-bit for basic needs or 8192-bit for maximum security. Set appropriate file permissions to protect the key: ```bash theme={null} chmod 600 cmk-private-key.pem ``` Never share your private key file. Store it securely with restricted access and maintain encrypted backups. Confirm the key is in proper PEM format: ```bash theme={null} openssl rsa -in cmk-private-key.pem -check -noout ``` Expected output: `RSA key ok` ### Key Generation Best Practices **Security recommendations:** * Generate keys on secure, offline systems when possible * Use strong key lengths (4096-bit or higher) * Store backup copies in encrypted, secure locations * Document key generation date and purpose * Never transmit keys over unsecured channels ## Uploading a Customer-Managed Key ### File Format Requirements **PEM File Requirements:** * File extension: `.pem` * Format: Privacy Enhanced Mail (PEM) encoded * Key type: RSA private key * Minimum key length: 2048 bits (4096 bits recommended) * Character encoding: ASCII/UTF-8 * No password protection on the key file The PEM file must be an unencrypted private key. Password-protected or encrypted PEM files will be rejected by the upload process. ### Upload Process Navigate to Instance Settings and click **Manage Data Encryption** on the CMK card. Click **Choose File** or **Select PEM File** button to open the file browser. Only files with `.pem` extension are accepted. Other file types will be rejected. Click **Upload** The system validates the file format before accepting the upload. Invalid files will be rejected with an error message. ### During Upload Process While the infrastructure is in TRANSITIONING state: * All other instance configuration changes are disabled * The upload typically takes 5-15 minutes * You can monitor progress on the loading screen ### What Happens During Upload The CMK upload process involves several backend operations: 1. **File Validation** - System verifies PEM format and key strength 2. **Secure Transfer** - Key is sent via Service Bus 3. **Customer Environment Processing** - Key is stored in your Azure subscription 4. **Key Vault Update** - Azure Key Vault is configured with the new CMK 5. **Infrastructure Update** - Instance infrastructure is updated to use the new key 6. **Encryption Migration** - Data encryption keys are re-encrypted with new CMK 7. **Status Update** - System records new expiration date and status ## CMK Expiration and Renewal ### Why Key Expiration Matters Encryption keys should be rotated regularly to: * **Minimize exposure** from potential key compromise * **Meet compliance requirements** mandating periodic key rotation * **Reduce cryptographic risk** from long-term key usage * **Maintain security posture** with current best practices Most compliance frameworks (SOC 2, ISO 27001, PCI-DSS) require key rotation at least annually. Many organizations implement yearly rotation schedules for sensitive data. ### Expired Key Recovery If a CMK expires before rotation: * Data access may be impacted * Some encrypted services may fail * Immediate key upload is required * Contact Entegrata support if data access issues occur **Recovery steps for expired keys:** 1. **Immediate action** - Generate and upload a new key as quickly as possible 2. **Service verification** - Test critical services after new key is deployed 3. **Root cause analysis** - Document why expiration occurred 4. **Process improvement** - Adjust monitoring and rotation schedules to prevent recurrence ## Best Practices for Key Management ### Key Generation and Storage * **Generate offline** - Create keys on secure, non-networked systems when possible * **Encrypted backups** - Store backup copies in encrypted archives * **Access control** - Limit key access to authorized security personnel only * **Audit logging** - Document all key generation, access, and rotation activities * **Physical security** - Store backup keys in secure physical locations (safes, vaults) ### Key Lifecycle Management Create keys using approved methods and tools with documented procedures Securely transfer keys to authorized personnel using encrypted channels Upload keys to Entegrata during planned maintenance windows Monitor expiration status and system health throughout key lifetime Replace keys before expiration following planned rotation schedule Securely archive retired keys per data retention policies Permanently destroy keys after retention period using approved methods ### Multi-Person Approval For high-security environments, implement multi-person approval for CMK operations: * One person generates the key * Another person reviews and approves the key * A third person uploads the key to Entegrata * All actions are logged and auditable ## Troubleshooting ### Common Upload Errors **Error:** "The uploaded file is not a valid PEM file" **Solutions:** * Verify the file has `.pem` extension * Ensure the file contains `-----BEGIN RSA PRIVATE KEY-----` header * Check that the key is not password-protected * Regenerate the key using OpenSSL if format is uncertain * Verify no corruption occurred during file transfer **Error:** "Key length does not meet minimum security requirements" **Solutions:** * Generate a new key with at least 2048-bit length * Recommended: Use 4096-bit keys for enhanced security * Verify key properties with: `openssl rsa -in key.pem -text -noout` **Error:** "Upload operation timed out" **Solutions:** * Check your internet connection * Verify no firewall or proxy is blocking the upload * Try uploading again during off-peak hours * Contact Entegrata support if timeouts persist **Error:** "Cannot upload CMK while infrastructure is transitioning" **Solutions:** * Wait for current infrastructure operations to complete * Check if other administrators are making instance changes * Monitor the infrastructure status on the loading screen * Try upload again after status returns to IDLE **Error:** "File exceeds maximum allowed size" **Solutions:** * Ensure you're uploading only the private key, not a certificate chain * Verify the file contains only the key, no extra content * Standard PEM keys should be under 10KB ### Deployment Stuck in Transitioning If CMK upload remains in TRANSITIONING state for more than 30 minutes: 1. **Do not close browser** - Keep the page open to monitor progress 2. **Check infrastructure status** - Review any error messages on loading screen 3. **Wait for timeout** - System may still be processing 4. **Contact support** - If stuck beyond 60 minutes, contact Entegrata support 5. **Provide details** - Share upload timestamp, file details, and any error messages ### Data Access Issues After Upload If you experience data access problems after CMK upload: 1. **Verify upload success** - Confirm green status badge appears 2. **Check expiration date** - Ensure new expiration date is displayed 3. **Test specific services** - Identify which services are affected 4. **Review error logs** - Check application logs for encryption errors 5. **Contact support immediately** - CMK issues may require backend intervention ## Security Considerations ### Key Security Warnings **Critical Security Reminders:** * Never share your CMK private key files * Never commit PEM files to version control systems * Never send keys via email or unencrypted channels * Never store keys on shared network drives without encryption * Never reuse keys across different environments or systems ## Related Documentation Configure secure network connectivity Manage outbound access controls View and upgrade system versions Learn about instance administration # Firewall Rules Source: https://support.entegrata.com/administration/instance/firewall-rules Configure and manage firewall rules to control network access for your Entegrata instance ## Overview Firewall rules control which external services and resources your Entegrata instance can communicate with. Rules are organized by data source — making it easy to see exactly which policies apply to each integration — and by system-level rules for infrastructure-wide access. This is a critical security feature that: * Restricts outbound connections to only approved destinations * Prevents unauthorized data exfiltration * Ensures compliance with security policies * Enables safe communication with data sources and third-party services Firewall rules complement your network connection settings (S2S VPN or VNet Peering) by controlling what your Entegrata instance can access once connected to your infrastructure. ## Navigating Firewall Settings To access firewall settings, navigate to **Instance Settings** and click **Manage Firewall Rules** on the Firewall Settings card. The card displays a summary of your current application and network rule counts. Firewall Settings Card The firewall management interface is organized into two tabs: Firewall Overview — Data Sources Tab Lists all data sources that have firewall policies configured. Each data source appears as an expandable card showing: * **Data source icon and name** — identifies the integration * **Status badge** — **Active** (at least one policy configured) or **Inactive** (no policies yet) * **Expand/collapse arrow** — click to view individual policies Use the **Search data sources** field at the top to quickly filter the list. Contains system-level firewall rules that apply to your Entegrata infrastructure rather than a specific data source. These include rules for services like Microsoft Fabric and other platform-wide dependencies. ## Understanding Firewall Policies ### Policy Types Each firewall policy is one of two types: **What they are**: Domain-based policies that match Fully Qualified Domain Names (FQDNs) **Use cases**: * Cloud APIs (e.g., `api.example.com`) * SaaS platforms (e.g., `app.salesforce.com`) * Services with dynamic IPs **Supported formats**: * Exact domains: `api.example.com` * Wildcard subdomains: `*.example.com` Wildcard rules like `*.com` are not allowed. The wildcard can only replace one subdomain level, e.g., `*.example.com`. **What they are**: IP address-based policies for specific network destinations **Use cases**: * On-premises servers with static IPs * Internal network resources * Database servers without domain names **Supported formats**: * Individual IPs: `192.168.1.100` * CIDR ranges: `10.0.0.0/24` * Single IP as CIDR: `203.0.113.50/32` If you enter an IP without a CIDR suffix (e.g., `192.168.1.100`), the system automatically treats it as `/32` (single IP). ### Policy Display When you expand a data source or view the System tab, each policy row shows: Expanded Data Source with Policies A descriptive label for the policy (e.g., "ADP OAuth", "Database Server", "Fabric"). Shows the current state of the policy: * **Pending** — The policy has been saved but not yet deployed. It will become active after you click **Apply Changes** and the infrastructure transition completes. * **Active** — The policy is deployed and enforced on your infrastructure. Shows **application** (FQDN-based) or **network** (IP/CIDR-based). The FQDN or IP address/CIDR range (e.g., `accounts.adp.com`, `1.2.3.4/32`). A three-dot menu with **Edit** and **Delete** options. ## Adding a Data Source Use this workflow to add firewall policies for a new data source integration. On the **Data Sources** tab, click **Add Data Source** in the top-right corner. A dialog appears showing all available data sources with icons. Use the search field to filter the list. Add Data Source Dialog Click the data source you want to add (e.g., 3E, Aderant, Iridium). The data source is added to your list in an **Inactive** state and automatically expanded, showing the message: *"This data source requires at least 1 policy to activate."* Click **Add Policies** to open the guided policy configuration dialog. This dialog is tailored to the selected data source and shows: **Required** — Policies that must be configured for the data source to function (e.g., a database server IP address). **Suggested** — Optional policies that may be needed depending on your configuration (e.g., a DNS server). For each policy: 1. Enter the IP address/CIDR range or domain in the text field 2. Click **Add** to save the policy 3. Click **Done** when finished Guided Policy Configuration Dialog The policy type (network or application) is pre-selected based on what the data source requires. Placeholder text shows the expected format (e.g., `e.g. 10.0.0.1/32`). Once policies are added, they appear with a **Pending** status. The data source is now listed but the policies are not yet enforced on your infrastructure. Data Source and Policy in Pending State Click **Apply Changes** to deploy the new policies to your infrastructure. Once the infrastructure transition completes, the policy status changes from **Pending** to **Active** and the data source becomes fully operational. Policies remain in **Pending** state until you click **Apply Changes** and the transition completes. They are not enforced on your infrastructure until they reach **Active** status. ### Removing a Data Source If a newly added data source is not needed, click **Remove** in the expanded data source card to remove it before applying changes. ## Adding a System Policy Use this workflow to add custom system-level firewall policies. Click the **System** tab at the top of the firewall settings page. System Tab Click the **+ Add** button at the bottom of the System rules list. Fill in the dialog fields: * **Type**: Select **Network** (for IP/CIDR) or **Application** (for FQDN) * **Name**: Enter a descriptive policy name * **Value**: Enter the IP address/CIDR range or domain name Add Custom Policy Dialog The placeholder text updates based on your type selection — `e.g. 10.0.0.1/32` for Network or a domain format for Application. Click **Save** to add the policy. It appears in the System rules list immediately with a **Pending** status. Click **Apply Changes** to deploy the policy to your infrastructure. Once the transition completes, the policy status changes from **Pending** to **Active**. ## Adding a Policy to an Existing Data Source You can add additional policies to data sources that are already configured. On the **Data Sources** tab, click the data source card to expand it and view its current policies. Click the **+ Add** button at the bottom of the policy list within the expanded data source. The Add Custom Policy dialog opens. Select the policy type, enter a name and value, then click **Save**. The new policy appears with a **Pending** status. Click **Apply Changes** to deploy the updated configuration. The policy status changes to **Active** once the transition completes. ## Editing a Policy Expand the relevant data source on the **Data Sources** tab, or navigate to the **System** tab. Click the three-dot menu (⋮) on the right side of the policy row and select **Edit**. Three-dot Menu with Edit and Delete Modify the policy details as needed. The same validation rules apply as when creating a new policy. Edit Policy Dialog Click **Save**, then click **Apply Changes** to deploy the update to your infrastructure. ## Deleting a Policy Before deleting a policy, ensure it is not required for active data collection or critical integrations. Removing a policy may cause collectors or integrations to fail. Expand the relevant data source or navigate to the **System** tab. Click the three-dot menu (⋮) on the policy row and select **Delete**. Click **Apply Changes** to remove the policy from your infrastructure. The deletion is saved to your configuration immediately but the policy remains active on your infrastructure until you apply changes. ## Applying Firewall Changes ### Understanding the Save-Then-Apply Workflow Firewall management uses a **save-then-apply** workflow. When you add or edit a policy, it is saved with a **Pending** status. Pending policies are not enforced on your infrastructure until you click **Apply Changes** and the transition completes, at which point they become **Active**. Add, edit, or delete policies as needed. You can batch multiple changes together. Click the **Apply Changes** button to deploy all staged changes to Azure infrastructure. The system triggers an infrastructure update: * Status changes to **Transitioning** * A loading screen displays progress * The system polls for completion every 30 seconds * Changes are deployed via infrastructure as code Once the transition completes: * Instance status returns to **Idle** * All **Pending** policies transition to **Active** * New firewall policies are now enforced * Your instance can access newly allowed destinations * Removed destinations are now blocked ### Why Save-Then-Apply? **Safety**: Review all changes before deploying to production **Efficiency**: Batch multiple policy changes into a single infrastructure update **Flexibility**: Experiment with policies without immediately affecting infrastructure **Transparency**: Clear separation between configuration and deployment ### During Transition State While your instance is in **Transitioning** state: * Existing firewall policies remain active * Data collection continues normally * You cannot make additional firewall changes * You cannot modify other instance settings * You cannot apply new changes until transition completes Firewall infrastructure transitions typically take 5-15 minutes depending on the complexity of changes being deployed. ## Validation Rules **These will be accepted:** * `api.salesforce.com` * `*.azure.com` * `app.subdomain.example.com` * `service-123.cloud.provider.net` * `my-api.example.co.uk` **These will be rejected:** * `*.com` (wildcard too broad) * `example` (missing TLD) * `http://api.example.com` (includes protocol) * `api.example.com:443` (includes port) * `api..example.com` (double dots) **These will be accepted:** * `192.168.1.100` * `192.168.1.100/32` * `10.0.0.0/24` * `172.16.0.0/16` * `203.0.113.0/28` **These will be rejected:** * `256.1.1.1` (octet > 255) * `192.168.1.1/33` (CIDR > 32) * `192.168.1` (incomplete IP) * `2001:db8::1` (IPv6 not supported) * `192.168.1.1-192.168.1.10` (range notation not supported) ## Troubleshooting **Symptoms**: Collector reports connection failures despite policy being added **Solutions**: * Check if the policy status is still **Pending** — policies are not enforced until they reach **Active** status after applying changes * Wait for any Transitioning state to complete * Double-check the policy value matches the collector's actual endpoint * Ensure the policy uses the correct type (application for domains, network for IPs) * Check collector logs for the actual connection target **Symptoms**: Data source remains Inactive even after adding policies **Solutions**: * Ensure at least one **Required** policy has been added * Verify the policy value passed validation (no error messages in the form) * Try removing and re-adding the data source **Symptoms**: The data source you need is not listed **Solutions**: * Check the search field for typos * The data source may already be added — check the Data Sources tab * If the data source is not in the catalog, add rules manually via the **System** tab **Symptoms**: Connection fails despite a wildcard rule that should match **Solutions**: * `*.example.com` matches `api.example.com` but NOT `api.sub.example.com` * For nested subdomains, add multiple rules or use exact domains * `*.example.com` does not match `example.com` itself — add both if needed * Verify the policy has been applied (not just saved) **Symptoms**: Apply Changes triggered but state doesn't return to Idle **Solutions**: * Wait at least 15-20 minutes for infrastructure updates * Refresh the page to see current status * Check Azure portal for alerts in your subscription * If it persists beyond 30 minutes, contact Entegrata support ## Security Best Practices **Principle of Least Privilege** * Only allow access to services actively in use * Use specific FQDNs over wildcards when possible * Prefer /32 CIDR (single IP) over broad ranges * Use the guided policy setup when adding data sources — it recommends only what's needed **Regular Audits** * Review firewall policies quarterly * Remove data sources that are no longer in use * Verify policies match active collectors and integrations * Use the search feature to quickly audit specific data sources **Change Management** * Test policy changes in non-production environments first * Batch related changes and apply them together * Coordinate firewall changes with your security team * Plan changes during maintenance windows ## Related Documentation Configure S2S VPN or VNet Peering for network connectivity Learn about instance administration concepts Set up data collectors that use firewall rules Get help with firewall configuration issues # Network Connections Source: https://support.entegrata.com/administration/instance/network-connections Configure and manage network connections between your infrastructure and Entegrata ## Overview Network connections enable secure communication between your on-premises or Azure-based systems and your Entegrata instance. These connections allow Entegrata collectors to access your systems while maintaining network isolation and security. Entegrata supports three connection types: * **Site-to-Site (S2S) VPN** - Encrypted tunnel between your network and Entegrata * **VNet Peering** - Direct Azure Virtual Network connection for Azure-based infrastructure * **ExpressRoute** - Private, high-bandwidth connection via an Azure ExpressRoute circuit Network connections are configured during instance provisioning and can also be managed after go-live through the **Network Connections** page in Instance Settings. ## Connection Status The Network Connections card displays the current status of your connection with color-coded badges for quick identification. Connection Status Card ### Network Status Meanings
Connected - Network connection is active and functioning Transitioning - Connection is being established or modified Error - Connection failed or encountered an issue Not Connected - No active connection configured
If your connection status shows **Error** or **Not Connected** and you expect it to be active, contact Entegrata support immediately. Collectors will not be able to access your systems while the connection is unavailable. ## Network Configuration ### What Network Configuration Controls Network configuration defines which IP addresses and CIDR ranges are allowed to communicate with your Entegrata instance. This ensures that only authorized systems can establish network connections. ### Viewing Network Configuration To view your current configuration select \*\*Network Information: View Details \*\*on the Network Configuration card. The modal displays: * **Internal/API Whitelisting**: IPs Entegrata uses to call external APIs * **Firm Systems Access**: CIDR ranges allowed to access firm systems To view your current network configuration: ## IP Whitelisting ### Why IP Whitelisting Matters IP whitelisting ensures that only known and trusted systems can communicate with your Entegrata instance. This prevents unauthorized access attempts and provides an additional security layer beyond authentication. Entegrata collectors running in your environment must have their IP addresses whitelisted to successfully transmit data to your instance. Missing IPs will result in collection failures. Many regulatory frameworks require strict network access controls. IP whitelisting helps satisfy these requirements by providing traceable, auditable network access restrictions. ## CIDR Ranges for System Access ### Understanding CIDR Notation CIDR (Classless Inter-Domain Routing) notation allows you to specify ranges of IP addresses efficiently: | CIDR Range | Description | Number of IPs | | - | - | - | | `x.x.x.x/32` | Single IP address | 1 | | `x.x.x.x/24` | Class C subnet | 256 | | `x.x.x.x/16` | Class B subnet | 65,536 | | `x.x.x.x/8` | Class A subnet | 16,777,216 | ## Managing Network Connections ### How Connections Are Established Network connections are typically configured during instance provisioning: Entegrata's team assesses your network architecture to determine the optimal connection type (S2S VPN, VNet Peering, or ExpressRoute). Based on your infrastructure, the appropriate connection parameters are configured, including: * Gateway IPs (for S2S VPN) * Encryption settings (for S2S VPN) * Peering settings (for VNet Peering) * Circuit Resource ID and Authorization Key (for ExpressRoute) The connection is provisioned through Azure infrastructure, creating the secure network path between your systems and Entegrata. Status changes to **Connected** once the connection is successfully established and verified. ## Connection Types ### Site-to-Site (S2S) VPN An encrypted IPsec tunnel between your on-premises VPN device and Entegrata's Azure VPN Gateway. Best for on-premises or hybrid environments with existing VPN infrastructure. ### VNet Peering A direct Azure Virtual Network peering connection. Best for organizations with Azure-hosted data sources that want the lowest latency and simplest configuration. ### ExpressRoute A private, high-bandwidth connection using an Azure ExpressRoute circuit. Best for organizations that require dedicated connectivity with guaranteed bandwidth and lower latency than internet-based connections. ExpressRoute connections require: * **Circuit Resource ID** - The full ARM resource ID of your ExpressRoute circuit * **Authorization Key** - A single-use key from the circuit owner (required for new connections only) * **Address Prefixes** (optional) - IP ranges advertised over the circuit, used for CIDR overlap validation ExpressRoute uses Border Gateway Protocol (BGP) for dynamic routing. Address prefixes are informational and used for validation — the actual routing is handled by the circuit's BGP configuration. ## Troubleshooting Connection Issues If the status remains in **Transitioning** for more than 30 minutes: 1. Refresh your browser to ensure you're seeing the latest status 2. Check if other infrastructure changes are in progress 3. Contact Entegrata support if the issue persists If the connection displays **Error**: 1. Check the Network Connections card for error details 2. Verify your VPN gateway is running (for S2S VPN connections) 3. Confirm firewall rules aren't blocking the connection 4. Contact Entegrata support with error details If collectors fail to connect even with **Connected** status: 1. Verify collector IPs are in the network whitelist 2. Check collector logs for specific connection errors 3. Confirm collectors are using the correct endpoint URLs 4. Review your local firewall rules for outbound blocks ## Next Steps Configure firewall rules for additional security controls Manage Entegrata system versions and upgrades Configure customer-managed keys for data encryption Get help with network connection issues # Onboarding Complete Source: https://support.entegrata.com/administration/instance/onboarding-complete Your Entegrata instance is provisioned and ready for configuration. ## Overview Congratulations! Your Entegrata instance is now configured and ready for use. All infrastructure has been provisioned and network connectivity is established. Success confirmation screen ## What You've Accomplished Reviewed the onboarding process requirements and prepared necessary information for instance setup. Activated your Azure subscription and provisioned foundational infrastructure including Resource Group, Managed Identity, Service Bus, and Container App Environment. Established secure connectivity between Entegrata and your environment using Virtual Network Peering or Site-to-Site VPN. Uploaded your own Customer-Managed Key for encryption, or allowed Entegrata to generate one for you. ## Next Steps Now that your instance is configured, you can access the admin portal and begin configuring these features for production use: Add users and configure roles Set up data collectors Manage network settings Configure CMK settings Set up data governance Review configurations User email addresses must match their Azure Entra ID accounts for SSO to function correctly. ## Access Your Instance Access your Entegrata instance ## Need Help? Submit a support ticket Instance administration guide # Overview Source: https://support.entegrata.com/administration/instance/onboarding-overview Guided setup for provisioning your Entegrata instance and preparing it for data collection and governance. Welcome to Entegrata instance onboarding. This guided process walks you through provisioning your environment, configuring secure connectivity, and setting up customer-managed encryption. You can pause and resume onboarding at any time. Your progress is saved automatically. ## What You Will Do The onboarding process walks you through the essential steps to get your Entegrata instance up and running: * Activate your Azure subscription and provision infrastructure * Establish secure network connectivity between Entegrata and your environment * Configure optional Customer Managed Keys * Complete setup and access the admin portal By the end of onboarding, your instance will be ready with secure connectivity to begin data collection and governance operations. ## The Onboarding Process Onboarding overview page Activate your Azure subscription to provision the foundational Entegrata infrastructure. This automated process creates the initial resources needed to bootstrap your instance. Choose and configure your network connectivity method. Select between Virtual Network (VNet) Peering for Azure-based resources or Site-to-Site VPN for on-premises or hybrid environments. Configure optional Customer Managed Keys (CMK) for encryption. This setting can be configured now or added later after onboarding. Review your configuration and complete the onboarding process. You'll be redirected to the admin portal where you can begin configuring users, data pipelines, and access controls. ## Prerequisites During the onboarding process, you will need: * Access to the Entegrata Admin Portal at [https://admin.entegrata.com](https://admin.entegrata.com) * Azure subscription permissions to create and manage resources * Network information for your environment (CIDR ranges, VNet details, or VPN configuration) * For self-hosted instances, Databricks Account ID and Service Principal details * Optional Customer-Managed Key (CMK) in unencrypted PEM format if you want to use your own encryption keys * Azure Entra ID configured if you plan to use single sign-on (SSO) The onboarding process can be paused and resumed at any time. Your progress through the steps is automatically saved, though any unsaved form data within a step will need to be re-entered when you return. You can gather the necessary information as you progress through each step. ## Ready to Begin? You can start the onboarding process and gather the required information as needed for each step. Begin with Step 1: Activate your Azure subscription and provision infrastructure Need help during onboarding? Visit our [support portal](https://support.entegrata.com) or contact your Entegrata representative. # Step 1: Subscription Activation Source: https://support.entegrata.com/administration/instance/onboarding-step-1 Provision the foundational Azure infrastructure for your Entegrata instance. Activating your Azure subscription provisions the core infrastructure required for your Entegrata instance. This process is automated and typically completes within a few minutes. ## What Gets Created When you activate your subscription, Entegrata automatically provisions the following Azure resources: * **Resource Group**: Container that organizes all resources for your Entegrata instance * **Managed Identity**: Service identity enabling secure authentication between Entegrata components * **Service Bus**: Messaging infrastructure for reliable communication between services * **Container App Environment**: Platform for running containerized Entegrata services All Azure resources are created in accordance with Microsoft's security best practices and are configured to meet enterprise security and compliance requirements. ## Activating Your Subscription Click the "Activate Subscription" button to begin the provisioning process. Subscription activation screen Once you click the button, you'll see a loading screen indicating that your Azure infrastructure is being created. This process typically takes 2-5 minutes. Loading screen during subscription provisioning When provisioning is complete, you'll see a confirmation card displaying your Subscription Name, Owner, and ID. The Subscription ID uniquely identifies your subscription and will be used for support and billing purposes. Completed subscription showing subscription ID Make note of your Subscription ID for future reference. You can also find this ID later in the instance details. ## Self-Hosted Instances Only: Configure Azure Databricks Access *This section is only required for self-hosted instances.* To enable Entegrata to manage access and settings for your subscription, you'll need to assign the User Access Administrator role to the managed identity and add it as an Account Admin in your Azure Databricks account. ### Assign User Access Administrator Role to the Managed Identity 1. Sign in to the **Azure Portal** 2. Navigate to **Subscriptions** and select your Entegrata Subscription 3. In the left-hand menu, click **Access control (IAM)** 4. Click **+ Add > Add role assignment** 5. In the **Role** dropdown, select **User Access Administrator** 6. In the **Assign access to** section, choose **Managed identity** 7. Click **Select members**, then search for and select the managed identity (the name will be displayed in the interface) 8. Click **Select**, then **Review + assign** to complete the role assignment ### Retrieve the Application (Client) ID and Add as Databricks Account Admin #### Find the Application (Client) ID: * In the Azure Portal, go to **Microsoft Entra ID > Enterprise Applications** * In the **Application type** dropdown, select **All Applications**, then search for the managed identity name * Select the managed identity from the results * On the **Overview** page, copy the **Application (client) ID** #### Add the Managed Identity as a Databricks Account Admin: * Sign in to the **Azure Databricks Account Console** * In the top-right corner, click your username and select **Manage Account** * Go to **User management** * Choose the **Service Principals** tab * Click **Add Service Principal** and choose the **Microsoft Entra ID managed** option * In the **New service principal display name** field, put the managed identity name * In the **Microsoft Entra application ID** field, put the client ID that you copied above * Click **Add Service Principal** * Click on your newly added service principal * Click the **Roles** tab at the top * Toggle **Account Admin** on * You can now exit out of this console ## Troubleshooting If the subscription activation fails or takes longer than expected: * Ensure you have the necessary Azure permissions to create resources in your subscription * Verify that your Azure subscription has sufficient quota for the required resources * Check your network connection * If the issue persists, [open a support ticket](https://mysupport.entegrata.com/portal/new) with your Subscription ID ## Next Steps Once your subscription is successfully activated, you're ready to configure your network connection. Return to the onboarding overview Proceed to Step 2: Network Connection Setup # Step 2: Network Connection Setup Source: https://support.entegrata.com/administration/instance/onboarding-step-2 Establish secure connectivity between Entegrata and your environment. After activating your subscription, establish a secure network connection between Entegrata and your data sources. This connection enables Entegrata's collectors to access your systems for data collection while maintaining security and network isolation. After onboarding, network connections can be managed through [Network Connections](/administration/instance/network-connections) in your instance settings. ## Connection Type Selection You have three options for establishing network connectivity: Best for Azure-based resources. Uses cross-tenant Virtual Network peering for direct, private connectivity. Best for on-premises or hybrid environments. Creates an encrypted VPN tunnel between networks. Best for dedicated, high-bandwidth private connectivity via an Azure ExpressRoute circuit. ## When to Use VNet Peering Choose VNet Peering if: * Your data sources are hosted in Azure * You want the lowest latency and highest bandwidth * You prefer a simpler network configuration * You have existing Azure Virtual Networks ## Configuration Steps *This section is only shown for self-hosted instances.* Configure your Databricks workspace connection details. **Required fields:** * **Databricks SPN Application ID** - The Application (Client) ID of your Databricks Service Principal. Found in Azure Portal under Enterprise Applications. * **Databricks SPN Object ID** - The Object (Principal) ID of your Databricks Service Principal. Found in Azure Portal under Enterprise Applications. * **Databricks Account ID** - Your Databricks Account ID (UUID format). Found in the Databricks Account Console. These fields will be disabled after the external configuration is complete and cannot be modified later. Specify **three non-overlapping /23 CIDR ranges** reserved exclusively for Entegrata. **Required fields:** * **CIDR Range 1** - First /23 CIDR block (e.g., 10.0.0.0/23) * **CIDR Range 2** - Second /23 CIDR block (e.g., 10.0.2.0/23) * **CIDR Range 3** - Third /23 CIDR block (e.g., 10.0.4.0/23) CIDR ranges section CIDR ranges must not overlap with each other or with your existing network. Once saved, they cannot be changed. Before proceeding, gather the following information from your Azure environment: 1. Navigate to the Azure Portal 2. Go to **Virtual Networks** → Select your VNet 3. In the Properties section, copy the **Resource ID** The Resource ID format should be: ``` /subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.Network/virtualNetworks/{vnet-name} ``` You'll need this in the next section. Configure the peering connection from Entegrata to your Virtual Network. **Required field:** * **Virtual Network Resource ID** - The full Azure Resource ID of your Virtual Network Virtual Network Resource ID input field Example format: ``` /subscriptions/12345678-1234-1234-1234-123456789012/resourceGroups/myResourceGroup/providers/Microsoft.Network/virtualNetworks/myVNet ``` After submitting, Entegrata will initiate the peering connection. You'll need to accept it on your side in the next step. After Entegrata initiates the peering, you need to accept it in your Azure environment. #### For Self-Hosted Instances To allow Entegrata to establish a virtual network peering to your Azure environment, you need to grant the Network Contributor role to the managed identity created for your subscription. The managed identity name will be dynamically generated and resides within the Entegrata subscription in a corresponding resource group. **Steps to grant Network Contributor role:** 1. Sign in to the **Azure Portal** 2. Navigate to Subscriptions or Resource Groups, depending on where you want to assign the role. Make sure the virtual network you want to peer is within the source group or subscription: * For the entire subscription, select Subscriptions > Your subscription * For a specific resource group, select Resource Groups > Your resource group 3. In the left-hand menu, click **Access control (IAM)** 4. Click **+ Add > Add role assignment** 5. In the Role dropdown, select **Network Contributor** 6. In the Assign access to section, choose **Managed identity** 7. Click **Select members**, then search for and select the managed identity (the name will be displayed in the interface) 8. Click **Select**, then **Review + assign** to complete the role assignment 9. Verify the assignment by checking the **role assignments** tab in the Access control (IAM) section #### For EntegrataOne Managed Instances To allow Entegrata to establish a virtual network peering to your Azure environment, you need to download and apply a Lighthouse template that grants the necessary permissions. Customer peering setup instructions **Steps to apply the Lighthouse template:** 1. Download the Lighthouse template provided in the interface 2. Follow the instructions shown in the interface for where to apply this template in your Azure portal 3. The template will automatically configure the necessary permissions for the peering connection The peering connection will show as "Connected" once both sides have completed configuration. ## Validation After configuration, Entegrata will automatically: * Verify the peering connection is established * Test network connectivity * Validate routing configuration * Confirm DNS resolution ## Troubleshooting **This appears when:** The VNet Resource ID format is incorrect **Solutions:** * Ensure the ID starts with `/subscriptions/` * Verify all required path segments are present * Check for typos in the resource group or VNet name * Copy the exact Resource ID from Azure Portal → Virtual Networks → Properties **This appears when:** CIDR ranges don't meet requirements **Solutions:** * Ensure all three ranges use `/23` subnet mask (not /24, /22, etc.) * Format must be `X.X.X.X/23` (e.g., `10.0.0.0/23`) * Check for typos in IP addresses * Verify ranges don't overlap with your existing networks **This appears when:** You've entered the same CIDR range twice **Solutions:** * Use three different /23 ranges * Example: `10.0.0.0/23`, `10.0.2.0/23`, `10.0.4.0/23` * Ranges must not overlap or be identical **This means:** The peering hasn't been established yet **Next steps:** * Check Azure Portal for pending peering request * Accept the peering on your VNet side * Verify Network Security Group rules * Wait a few minutes for propagation, then refresh the status ## When to Use S2S VPN Choose Site-to-Site (S2S) VPN if: * Your data sources are on-premises * You have a hybrid cloud environment * You have existing VPN infrastructure * You cannot use VNet peering due to policy restrictions ## Configuration Steps *This section is only shown for self-hosted instances.* Configure your Databricks workspace connection details. **Required fields:** * **Databricks SPN Application ID** - The Application (Client) ID of your Databricks Service Principal * **Databricks SPN Object ID** - The Object (Principal) ID of your Databricks Service Principal * **Databricks Account ID** - Your Databricks Account ID (UUID format) These fields will be disabled after the external configuration is complete and cannot be modified later. Ensure you have the following information from your VPN device: **Required Information:** * Public IP address of your VPN device * Local network address spaces * Preferred encryption and authentication methods * Pre-shared key (PSK) for authentication Specify **three non-overlapping /23 CIDR ranges** reserved exclusively for Entegrata. **Required fields:** * **CIDR Range 1** - First /23 CIDR block (e.g., 10.0.0.0/23) * **CIDR Range 2** - Second /23 CIDR block (e.g., 10.0.2.0/23) * **CIDR Range 3** - Third /23 CIDR block (e.g., 10.0.4.0/23) CIDR ranges configuration CIDR ranges must not overlap with each other or with your existing network. Once saved, they cannot be changed. Configure the pre-shared key for VPN authentication. **Shared Key** - Pre-shared key (minimum 25 characters if provided). Used for mutual authentication between VPN endpoints. Shared key input with show/hide toggle **PSK Best Practices:** * Use at least 25 random characters * Include uppercase, lowercase, numbers, and symbols * Store securely and rotate periodically * Never share over insecure channels **Generate a Strong PSK:** ```bash theme={null} openssl rand -base64 32 ``` Configure the IPsec/IKE parameters to match your VPN device. ### IKE Version * **IKE Protocol** (Required) - Select IKEv1 or IKEv2 (IKEv2 recommended) ### Phase 1 (IKE) Settings * **IKE Encryption** (Required) - Encryption algorithm: AES128, AES192, AES256, GCMAES128, GCMAES256 * **IKE Integrity** (Required) - Integrity algorithm: SHA1, SHA256, SHA384, MD5 * **DH Group** (Required) - Diffie-Hellman group: DHGroup1, DHGroup14, DHGroup2, DHGroup2048, DHGroup24, ECP256, ECP384, None ### Phase 2 (IPsec) Settings * **IPsec Encryption** (Required) - IPsec encryption: None, AES128, AES192, AES256, DES, DES3, GCMAES128, GCMAES192, GCMAES256 * **IPsec Integrity** (Required) - IPsec integrity: GCMAES128, GCMAES192, GCMAES256, MD5, SHA1, SHA256 * **PFS Group** (Required) - Perfect Forward Secrecy group: None, PFS1, PFS2, PFS14, PFS24, PFS2048, ECP256, ECP384 * **SA Lifetime (Seconds)** (Required) - SA lifetime in seconds (300-2147483647, default: 27000) * **SA Lifetime (Kilobytes)** (Optional) - SA lifetime in kilobytes (0 or 1024-2147483647, default: 0) IPsec policy configuration **Recommended Configuration:** * IKE Version: IKEv2 * Encryption: AES256 * Integrity: SHA256 * DH Group: DHGroup14 * PFS: PFS2048 Configure your VPN gateway connection details. **Required fields:** * **Gateway IP Address** - Public IP address of your VPN device (must be static) * **Local Network Addresses** - List of on-premises network address spaces (CIDR format) Gateway configuration **Examples:** * Single network: `192.168.1.0/24` * Multiple networks: `192.168.1.0/24`, `192.168.2.0/24`, `10.0.0.0/16` * Host addresses: `192.168.1.10/32` Ensure your firewall allows IPsec traffic (UDP 500, UDP 4500, ESP) from Entegrata's public IPs. ## VPN Tunnel Establishment After saving, Entegrata will: 1. Provision Azure VPN Gateway (15-30 minutes) 2. Configure IPsec policies 3. Initiate VPN connection 4. Validate tunnel establishment and connectivity VPN gateway provisioning in progress ## Post-Configuration Once the VPN tunnel is established: 1. Configure your on-premises VPN device with the provided settings 2. Ensure routing is properly configured on both sides 3. Update firewall rules to allow required traffic 4. Test connectivity from both directions VPN tunnel status showing connected state ## Troubleshooting Common VPN Issues After configuration, check the connection status on this page and in your VPN device logs. The following are common VPN configuration issues and their solutions. **Possible causes and solutions:** * Verify the pre-shared key matches exactly on both sides * Ensure all IPsec parameters are identical between devices * Check that your gateway's public IP is correct and reachable * Confirm your firewall allows IPsec traffic (UDP 500, UDP 4500, ESP) **These settings must match on both sides:** * IKE version (both using IKEv1 or both using IKEv2) * Encryption and integrity algorithms * Diffie-Hellman group * Pre-shared key (check for extra spaces or typos) **These settings must match on both sides:** * IPsec encryption and integrity algorithms * PFS group setting * Local and remote network address definitions * SA lifetime settings (must be within compatible ranges) **Check these common causes:** * Routing tables on both sides have correct routes * Network Security Group rules allow your application traffic * No overlapping IP addresses between networks * Test with ping in both directions to verify basic connectivity ## When to Use ExpressRoute Choose ExpressRoute if: * You require dedicated, private connectivity with guaranteed bandwidth * You have an existing Azure ExpressRoute circuit provisioned by a connectivity provider * You need lower latency than internet-based connections * Your compliance requirements mandate private network paths ## Prerequisites Before configuring an ExpressRoute connection, ensure you have: * An **Azure ExpressRoute circuit** provisioned and in a "Provisioned" state * An **Authorization Key** generated by the circuit owner (single-use) * The **Circuit Resource ID** from the Azure portal Authorization keys are single-use. Each new connection requires a new key from the circuit owner. If the connection needs to be recreated, a new authorization key must be generated. ## Configuration Steps Specify **three non-overlapping /23 CIDR ranges** reserved exclusively for Entegrata. **Required fields:** * **CIDR Range 1** - First /23 CIDR block (e.g., 10.0.0.0/23) * **CIDR Range 2** - Second /23 CIDR block (e.g., 10.0.2.0/23) * **CIDR Range 3** - Third /23 CIDR block (e.g., 10.0.4.0/23) CIDR ranges configuration CIDR ranges must not overlap with each other or with your existing network. Once saved, they cannot be changed. Configure your ExpressRoute connection details. **Required fields:** * **Circuit Resource ID** - The full Azure Resource Manager ID of your ExpressRoute circuit ``` /subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.Network/expressRouteCircuits/{circuit-name} ``` You can find this in the Azure Portal under your ExpressRoute circuit's **Properties** page. * **Authorization Key** - A single-use token from the circuit owner that grants permission to connect ExpressRoute configuration fields **Optional fields:** * **Address Prefixes** - IP address ranges advertised over the circuit (e.g., `10.100.0.0/24`). These are used for CIDR overlap validation. ExpressRoute uses Border Gateway Protocol (BGP) for dynamic routing, so these are informational. The Circuit Resource ID cannot be changed after the connection is created. To use a different circuit, delete the connection and create a new one. ## Connection Establishment After saving, Entegrata will: 1. Provision an ExpressRoute Virtual Network Gateway (15-30 minutes) 2. Create the connection to your ExpressRoute circuit using the authorization key 3. Establish BGP peering 4. Validate connectivity Gateway provisioning can take 15-30 minutes. The system will display a loading screen during this process. Do not refresh or navigate away. ## Troubleshooting **This appears when:** The Circuit Resource ID doesn't match the expected format **Solutions:** * Ensure the ID starts with `/subscriptions/` * Verify it contains `/providers/Microsoft.Network/expressRouteCircuits/` * Copy the exact Resource ID from Azure Portal → ExpressRoute Circuits → Properties * Check that the subscription ID is a valid GUID **This appears when:** No authorization key was provided for a new connection **Solutions:** * Generate a new authorization key from the ExpressRoute circuit in the Azure Portal * Navigate to your circuit → Authorizations → Add * Copy the key and paste it into the form **Possible causes:** * ExpressRoute gateway is still provisioning (can take up to 30 minutes) * The authorization key has already been used * The ExpressRoute circuit is not in a "Provisioned" state **Solutions:** * Wait up to 30 minutes for initial provisioning * Verify your circuit status in the Azure Portal * If the key was already used, generate a new one and recreate the connection * Contact Entegrata support if the issue persists after 30 minutes ## Security Considerations Regardless of connection type, Entegrata implements multiple security layers: **All traffic is encrypted in transit - Network isolation between customers - Regular security audits and compliance certifications - Azure-native security features (NSG, Azure Firewall)** ## Save Configuration Once you've entered all required information for your chosen connection type, click the **Save** button to apply your configuration. The system will validate your settings and begin establishing the network connection. Network configuration changes can take 5-15 minutes to fully propagate. The system will display a loading screen during this process. ## Next Steps Return to Subscription Activation Proceed to Additional Settings # Step 3: Customer-Managed Key (Optional) Source: https://support.entegrata.com/administration/instance/onboarding-step-3 Upload a Customer-Managed Key for instance-level encryption. ## Overview Configure an optional Customer-Managed Key (CMK) during onboarding for enhanced data encryption control. This setting can also be managed later through [Data Encryption](/administration/instance/data-encryption) in your instance settings. This step is optional. You can click Save without uploading a key and Entegrata will generate a Customer-Managed Key for you. ## Customer-Managed Keys Customer-Managed Keys (CMK) allow you to control the encryption keys used to protect your data, providing enhanced security and compliance capabilities beyond Azure's default encryption. **Key Benefits of CMK:** - Control and ownership of encryption keys - Meet compliance requirements (SOC 2, HIPAA, PCI-DSS, FedRAMP) - Flexible key rotation on your schedule - Enhanced security for sensitive data ### Uploading Your CMK Ensure your key meets these requirements: * RSA private key (minimum 2048-bit, 4096-bit recommended) * PEM format with `.pem` extension * Begins with `-----BEGIN RSA PRIVATE KEY-----` * Unencrypted (no password protection) These requirements will be validated by the server when you upload the file. If the file doesn't meet requirements, you'll receive an error message. Click "Choose File" and select your private key file, then click "Save" to proceed. CMK file upload interface After clicking Save, the system will process your key. You'll see a loading screen with the message "Applying additional settings to your instance." This may take several minutes. The PEM file must be an unencrypted private key. Store it securely with restricted access and maintain encrypted backups. Never share this key file with unauthorized parties. ### If You Do Not Upload Your Own CMK If you don't upload your own key, Entegrata will generate a Customer-Managed Key for you: * AES-256 encryption for data at rest * Key stored in Azure Key Vault * Key managed by Entegrata To proceed without uploading your own CMK, simply click "Save" without selecting a file. For detailed CMK configuration and management after onboarding, see [Data Encryption](/administration/instance/data-encryption). ## Completing This Step Click "Save" to proceed. You can upload your own CMK file or let Entegrata generate one for you. ## Next Steps Return to Network Connection Setup Finish instance onboarding # Instance Administration Source: https://support.entegrata.com/administration/instance/overview Manage core infrastructure, networking, security, and system settings for your Entegrata instance. The Instance Administration module allows administrators to manage the core infrastructure settings that power your Entegrata deployment. These settings control how your instance connects to external systems, secures data, and stays up to date. Changes made in Instance Administration directly modify Azure infrastructure and may take several minutes to apply. ### Instance Infrastructure Instance Administration manages four critical areas of your Entegrata infrastructure: Manage secure connectivity between Entegrata and your environment. See [Network Connections](/administration/instance/network-connections). * Configure Site-to-Site VPN or Virtual Network (VNet) Peering * View connection status and health * Monitor connectivity for data collection Control outbound network access from your Entegrata instance. See [Firewall Rules](/administration/instance/firewall-rules). * Define allowed fully qualified domain names (FQDNs) and IP addresses * Rules are grouped into: * System (Managed by Entegrata) * Collector * Custom * Restrict access to only required external services and APIs Manage encryption keys used to protect your data in Azure. See [Data Encryption](/administration/instance/data-encryption). * Upload and manage Customer-Managed Keys (CMK) * Monitor key expiration * Rotate keys as needed to meet security and compliance requirements View and manage your Entegrata platform version. See [System Versions](/administration/instance/system-versions). * See the currently deployed version * View available updates * Initiate platform upgrades when new releases are available ### Infrastructure State All instance configuration changes trigger infrastructure updates in Azure. Understanding the infrastructure state is crucial: The infrastructure is stable and ready for new operations. All administrative functions are available when the system is in IDLE state. Infrastructure changes are being applied. During transitions, administrative functions are disabled to prevent conflicts. The system automatically polls for completion and notifies you when operations finish. While the infrastructure is in TRANSITIONING state, you cannot make additional configuration changes. Wait for the current operation to complete before initiating new changes. ### Azure Infrastructure Integration Entegrata runs on Azure infrastructure provisioned specifically for your firm. Instance Administration settings directly modify Azure resources including: * Azure Firewall rules * VPN Gateways and Virtual Network configurations * Key Vault encryption keys Changes are applied through infrastructure-as-code (IaC), ensuring consistent and repeatable deployments. ## What's Next? Configure VPN and VNet peering for secure connectivity Manage outbound access rules for your instance Upload and manage customer-managed encryption keys View and upgrade your Entegrata platform version # System Versions Source: https://support.entegrata.com/administration/instance/system-versions Manage Entegrata platform versions and perform system upgrades ## Overview System Versions allows you to view your current Entegrata platform version, check for available updates, and upgrade your instance to newer releases. Keeping your system up to date ensures access to the latest features, security patches, and performance improvements. System Version Card Entegrata uses semantic versioning (e.g., v2.4.1) to clearly communicate the nature and impact of each release. All upgrades are performed through automated infrastructure-as-code, ensuring consistent and reliable deployments. ## Understanding System Versions ### Version Numbering Scheme Entegrata follows semantic versioning (SemVer) to provide clear information about each release: Significant platform changes that may include: * New core functionality or architectural changes * Breaking changes to APIs or data structures * Major feature additions requiring infrastructure updates * Potential migration steps or configuration changes **Example**: `v2.0.0` → `v3.0.0` Feature additions and enhancements that are backward compatible: * New features and capabilities * Enhanced existing functionality * Performance improvements * New integrations or data sources * No breaking changes to existing functionality **Example**: `v2.3.0` → `v2.4.0` Bug fixes and small improvements: * Bug fixes and stability improvements * Security patches * Minor performance optimizations * Documentation updates * No new features or breaking changes **Example**: `v2.4.0` → `v2.4.1` ### Understanding Status Badges The System Version card displays status badges to communicate your update status at a glance: **Badge**: **Blue "Upgrade Available"** **Meaning**: A newer version of the Entegrata platform is available for your instance. **Action**: You can upgrade to the latest version at your convenience. When you see this badge, review the release notes for the newer version to understand what features and improvements are available. **Badge**: **Green "Up to Date"** **Meaning**: Your instance is running the latest available version of the Entegrata platform. **Action**: No action needed. Your system is current. Even when up to date, the system continues to check for new releases automatically. ## Checking for Available Updates ### Automatic Update Checks The Entegrata platform automatically checks for new versions: * **Frequency**: Checks occur when you access the System Version settings * **Caching**: Results are cached for 12 hours to optimize performance * **Background**: Checks happen without requiring manual intervention The 12-hour cache means you may not immediately see new versions within 12 hours of their release. You can manually trigger a check to bypass the cache. ### Manual Update Check To check for updates immediately: Navigate to **Instance Settings** → **Entegrata Version** card and click **Manage Versions & Updates**. Click the **Check For Updates** button in the version management dialog. Version Options This bypasses the 12-hour cache and queries for the latest available versions immediately. The system retrieves the latest version information from Entegrata's release servers. * If newer versions exist, the status updates to **"Upgrade Available"** * If you're on the latest version, the badge remains **"Up to Date"** Version Man Click **Manage Versions** to see all available versions you can upgrade to. Man Versions If you're expecting a specific version release, use the **Check For Updates** button to immediately see if it's available rather than waiting for the cache to expire. ### Downtime and Impact Entegrata version upgrades are designed for minimal disruption: **Minimal Impact Operations:** * ✅ Viewing dashboards and reports * ✅ Accessing historical data * ✅ Reading documentation and settings * ✅ Monitoring system status **Temporarily Unavailable:** * ❌ Making instance configuration changes * ❌ Creating or modifying firewall rules * ❌ Uploading customer-managed keys * ❌ Modifying network connections * ❌ Initiating additional upgrades For mission-critical environments, schedule upgrades during maintenance windows to avoid any potential impact on business operations. ### Upgrade Duration Typical upgrade times by version type: | Version Type | Typical Duration | What's Happening | | :- | :- | :- | | **Patch** (X.X.1) | 5-10 minutes | Service restarts, minor updates | | **Minor** (X.1.0) | 10-20 minutes | Feature rollout, database updates | | **Major** (1.0.0) | 20-45 minutes | Significant infrastructure changes | Actual upgrade times depend on your instance size, infrastructure complexity, and Azure region. The loading screen provides real-time status updates throughout the process. ## Rollback Considerations ### When to Consider Rollback In rare cases, you may need to revert to a previous version: Version rollbacks are not self-service and require coordination with Entegrata Support. Rollbacks should only be considered for critical issues that cannot be resolved through configuration or patches. Database rollbacks may not be possible if the newer version included schema changes or data migrations. Entegrata support will advise on feasibility during the assessment. ## Troubleshooting ### Common Version Management Issues **Symptoms**: System shows "Up to Date" but you've confirmed a newer version is released **Possible causes:** 1. 12-hour cache hasn't expired yet 2. New version not yet rolled out to your region 3. Your instance has a version hold (rare, support-initiated) **Solutions:** * Click **Check For Updates** to bypass cache * Wait 12 hours for automatic cache expiration * Verify version release on Entegrata support site * Contact support if issue persists beyond 24 hours of release New versions may take several hours to propagate to all Azure regions after release. **Symptoms**: Cannot click the Upgrade button in version selection modal **Possible causes:** 1. Infrastructure is in **Transitioning** state 2. Another administrator is performing an operation 3. Insufficient permissions 4. Network connectivity issue **Solutions:** * Check if infrastructure state is **Idle** (not **Transitioning**) * Wait for any ongoing operations to complete * Verify you have administrative privileges * Refresh the page and try again * Check browser console for JavaScript errors **Symptoms**: Upgrade initiated but status doesn't change for extended period **Possible causes:** 1. Complex infrastructure changes taking longer than usual 2. Azure service delays 3. Network connectivity issue with status polling 4. Backend service issue **Solutions:** * Wait at least 30-45 minutes for complex upgrades * Refresh the browser to ensure you're seeing latest status * Check the loading screen for any error messages * Verify network connectivity * If stuck beyond 1 hour, contact Entegrata support The system polls status every 30 seconds. If you don't see updates for several minutes, try refreshing your browser. **Symptoms**: Upgrade succeeded but features don't work as expected **Possible causes:** 1. Browser cache serving old JavaScript/CSS 2. New feature requires configuration 3. Breaking change requires updates to integrations 4. Bug in new version **Solutions:** * Hard refresh browser (Ctrl+Shift+R or Cmd+Shift+R) * Clear browser cache and cookies * Review release notes for configuration changes needed * Check Entegrata support site for known issues * Try accessing from incognito/private browsing mode * Contact support with specific behavior descriptions Always hard refresh your browser after completing a version upgrade to ensure you're running the latest client code. **Symptoms**: Version card shows upgraded version but new features aren't visible **Possible causes:** 1. Features require feature flags to be enabled 2. Features specific to certain instance types 3. Permissions don't grant access to new features 4. Incomplete upgrade (partial rollout) **Solutions:** * Review release notes to verify feature availability * Check if features require additional configuration * Verify your user role has permissions for new features * Contact support to enable feature flags if needed * Wait 24 hours and check again (some features activate gradually) **Symptoms**: Only current version is visible, no history of previous versions **Possible causes:** 1. Version history not tracked in UI (by design) 2. Looking in wrong location **Solutions:** * Current design only shows current and available future versions * Version history can be requested from Entegrata support * Check Azure infrastructure logs for deployment history * For audit purposes, contact support for upgrade history report The System Version interface focuses on forward-looking version management. Historical version data is maintained in backend systems and available through support. **Symptoms**: Upgrade process returned an error and didn't complete **Possible causes:** 1. Pre-requisite check failed 2. Infrastructure resource conflict 3. Database migration issue 4. Network or connectivity problem **Solutions:** * Note the exact error message displayed * Verify infrastructure health in Azure portal * Check if instance was in healthy state before upgrade * Review recent changes to instance configuration * Contact Entegrata support with error details and timestamp **Information to provide support:** * Exact error message * Source version and target version * Timestamp of upgrade attempt * Recent changes to instance configuration * Screenshots of error messages ## Related Documentation Learn about instance administration concepts and infrastructure state management Manage firewall rules that may be updated during version upgrades Configure network connectivity for your instance Upload and manage customer-managed encryption keys # Overview Source: https://support.entegrata.com/administration/user/overview Learn how to manage users in your Entegrata instance ## User Overview Understand how user accounts work in Entegrata and how they integrate with Azure Entra ID. ## Key Concepts ### User Record Each user in Entegrata has the following core attributes: The display name of the user as it appears throughout the system The primary email address used for authentication and login An alternative email address for communications (defaults to account email if not specified) The current state of the user account: * **Active** - User can log in and access the system * **Inactive** - User account is temporarily disabled * **Pending** - User has been invited but not yet activated * **Deleted** - User has been soft-deleted and cannot access the system ### User Lifecycle A user typically moves through four stages: **Creation → Active Use → Modification → Deactivation or Deletion.** Deactivated or deleted users lose access but their data and audit history remain for compliance. ## Azure Entra ID Integration Entegrata integrates with **Azure Entra ID** (formerly Azure Active Directory) for enterprise SSO. Create users with the same email address they use in Entra ID to ensure correct mapping. ## Navigation To access User Management: 1. Log into the Entegrata Admin Portal ([https://admin.entegrata.com](https://admin.entegrata.com)) 2. Select **Users** from the left sidebar User Management Navigation ## What's Next? Learn how to view, add, edit, deactivate, and delete users Common issues and solutions for user management # Troubleshooting Source: https://support.entegrata.com/administration/user/troubleshooting Common issues and solutions for user management in Entegrata ## Common Issues This guide helps you resolve common problems encountered when managing users in Entegrata. ## Login and Access Issues ### User Cannot Log In **Symptoms:** * User receives "Invalid credentials" error * User account exists in admin panel * Status shows as Active **Possible Causes & Solutions:** 1. **Password not set** * Verify password setup process was completed * Send password reset instructions * Check if SSO/Azure Entra ID is properly configured 2. **Wrong email used** * Confirm user is using Account Email, not Contact Email * Check for typos in email address * Verify no extra spaces in email field 3. **Browser cache issues** * Clear browser cache and cookies * Try incognito/private browsing mode * Test with different browser **Symptoms:** * User could log in before * Account email was recently changed * Old credentials no longer work **Solutions:** 1. Verify new email address is correct 2. Ensure user was notified of email change 3. User must log in with new account email 4. Clear saved passwords in browser 5. Update any SSO configurations **Symptoms:** * User status shows as Deleted * All login attempts fail * User not visible in active list **Resolution:** * Soft-deleted users cannot log in * Requires database admin to restore * Contact support for user recovery * Consider creating new user if appropriate ## Search and Display Issues ### Users Not Appearing in List **Possible Causes:** 1. **Page not refreshed** * Refresh browser (F5 or Cmd+R) * Navigate away and back to user list 2. **Pagination issue** * Check if on correct page * New user might be on different page * Sort by creation date to find recent users 3. **Search filter active** * Clear search box * Remove any active filters * Check URL for query parameters **Troubleshooting Steps:** 1. **Check search syntax** * Remove special characters * Try partial name matches * Search by email domain 2. **Timing issues** * Wait for 500ms debounce * Don't type too quickly * Let search complete 3. **Case sensitivity** * Search is case-insensitive * Try different capitalizations anyway * Remove accents from names ## Form and Validation Errors ### Cannot Save User Changes **Check for:** * All required fields filled * Valid email format * No trailing spaces * Name field not empty * Network connectivity **Solutions:** * Tab through all fields to trigger validation * Copy/paste to remove hidden characters * Check browser console for errors **Common Issues:** 1. **Invalid format** * Must be `user@domain.com` format * No spaces allowed * Special characters limited to `.` `-` `_` 2. **Duplicate email** * Email already in use * Check deleted users (via database) * Try different email 3. **Copy/paste problems** * Remove any invisible characters * Type email manually * Check for extra spaces ## Performance Issues ### Slow Loading Times **Optimization Steps:** 1. **Reduce page size** * Currently fixed at 20 users/page * Cannot be changed in UI * Use search to filter results 2. **Network optimization** * Check internet connection speed * Try wired connection * Reduce concurrent browser tabs 3. **Browser performance** * Clear browser cache * Update to latest browser version * Disable unnecessary extensions **Improvements:** * Be more specific with search terms * Wait for debounce (500ms) to complete * Avoid rapid consecutive searches * Use exact email if known ## Data Integrity Issues ### Inconsistent User Data **Diagnostic Steps:** 1. **Verify save completed** * Look for success toast * Check for error messages * Refresh page to confirm 2. **Concurrent editing** * Another admin might be editing * Changes might be overwritten * Coordinate with team 3. **Cache issues** * Hard refresh (Ctrl+Shift+R) * Clear browser data * Try incognito mode **Resolution Process:** 1. **Identify duplicates** * Compare email addresses * Check creation dates * Review user IDs 2. **Clean up** * Delete duplicate entries * Keep most recent/complete record * Transfer any unique data 3. **Prevention** * Search before creating * Establish naming conventions * Document user creation ## Integration Problems ### Azure Entra ID Issues **Check:** * Entra ID field populated * Correct tenant configuration * User exists in Azure AD * Proper permissions granted **Actions:** * Verify Azure configuration * Check authentication logs * Test with different user * Contact identity team **Common Causes:** * Manual process (no auto-sync) * Field mapping issues * Permission problems * Network connectivity **Solutions:** * Manually update user records * Verify API permissions * Check Azure audit logs * Plan manual sync schedule ## Error Messages Reference ### Common Error Messages and Solutions | Error Message | Cause | Solution | | - | - | - | | "User not found" | User deleted or doesn't exist | Verify user ID, check if deleted | | "Email already exists" | Duplicate email address | Use different email or find existing user | | "Invalid email format" | Malformed email address | Check format: [user@domain.com](mailto:user@domain.com) | | "Required field missing" | Empty required field | Fill in all required fields | | "Network error" | Connection issue | Check internet, retry operation | | "Permission denied" | Insufficient privileges | Verify admin access rights | | "Session expired" | Login timeout | Log in again | | "Concurrent modification" | Another user editing | Refresh and retry | ## Browser-Specific Issues ### Supported Browsers For best experience, use latest versions of these officially supported browsers: * Chrome (recommended) * Edge (Chromium-based) Support for Firefox & Safari will come soon. Admin App features *may* work in Firefox & Safari, but those are not officially supported browsers as of March 2026. ### Browser Troubleshooting **Common Fixes:** * Clear cache: Settings → Privacy → Clear browsing data * Disable extensions: chrome://extensions * Reset settings: Settings → Advanced → Reset **Common Fixes:** * Clear cache: Settings → Privacy → Clear browsing data * InPrivate mode: Ctrl+Shift+N * Reset Edge: Settings → Reset settings **Common Fixes:** * Clear cache: Settings → Privacy & Security → Clear Data * Safe mode: Help → Restart with Add-ons Disabled * Refresh Firefox: about:support → Refresh Firefox **Common Fixes:** * Clear cache: Develop → Empty Caches * Disable extensions: Preferences → Extensions * Reset Safari: Safari → Clear History ## Getting Help ### When to Contact Support Contact support if you encounter: * Database errors or corruption * Unable to access admin panel * Bulk operation failures * Data loss or recovery needs * Integration failures * Security concerns ### Information to Provide When contacting support, include: **Essential Information:** * User email/ID affected * Error messages (exact text) * Steps to reproduce * Browser and version * Screenshots if applicable * Time of occurrence * Recent changes made ### Self-Service Resources Before contacting support: 1. Check this troubleshooting guide 2. Review related documentation 3. Clear browser cache 4. Try different browser 5. Verify network connectivity 6. Check with other administrators ## Prevention Best Practices ### Regular Maintenance **Recommended routine:** * Weekly: Review user list for anomalies * Monthly: Audit deleted users * Quarterly: Clean up duplicates * Annually: Full user audit ### Documentation Maintain records of: * User creation/deletion reasons * Email change justifications * Access issues and resolutions * Common problems in your organization ## Related Documentation Understanding user management basics in the platform Learn how to view, add, edit, deactivate, and delete users # User Management Source: https://support.entegrata.com/administration/user/user-management Learn how to view, add, edit, deactivate, and delete users in your Entegrata instance. ## Creating Users Administrators can add new users from the **Users** section of the Admin Portal. ### **Step 1: Open the Create User Dialog** 1. Navigate to **Administration** → **Users** 2. Click the **New User** button in the top-right corner Create user button ### **Step 2: Enter User Information** Create user form ### Required Fields Enter the user's full name as it should appear in the system. The primary email address for user authentication. #### Optional Fields An alternative email for non-authentication communications. **Options:** * Leave blank to use the account email for all communications * Check "Use same as account email" to automatically copy the account email * Enter a different email for separate communication preferences **Behavior** * Users are created with an **Active** status by default. * There is **no invitation workflow** — notify new users manually after creation. * The **Account Email** must be unique and cannot be changed after activation. * If **Contact Email** is left blank, it defaults to the account email. **Limitations** * No bulk import or role assignment during creation * No password setup (handled via SSO or first-login reset) * No custom templates or attributes **Common Errors** * Duplicate or invalid email address * Missing required fields *** ## Viewing and Searching Users The **Users** list displays all active users in your instance. Inactive and deleted users are hidden by default. ### Status Indicators User status is displayed with color-coded badges for quick identification:
**Active** - User can access the system **Pending** - User invited but not activated **Inactive** - Account temporarily disabled **Deleted** - User has been removed
Currently, the interface displays only active users. Deleted and inactive users are filtered from the view. ### Search Use the search bar to find users by \*\*name \*\*or **email**. Results update in real time. Filtering by user status and bulk operations are not yet supported. ### Individual Actions Click a user row to open the edit view, or use the ⋮ actions menu for quick operations such as *Edit* or *Delete*. *** ## Editing Users Administrators can update user information directly from the list. All edits are logged automatically in the audit trail `updatedAt`, `updatedBy`). ## Accessing the Edit User Feature There are two ways to edit a user: ### **Method 1: Click on User Row** Click anywhere on a user's row in the user list table to open the edit modal. User list with row actions ### Method 2: Actions Menu 1. Click the actions menu (⋮) in the user's row 2. Select **Edit** from the dropdown User row actions menu **Editable Fields** * **Name** — updates display name throughout the system * **Account Email** — must remain unique; affects login credentials * **Contact Email** — optional; defaults to account email **Non-Editable Fields** User ID, Firm ID, Instance ID, Status, and Azure Entra ID. **Behavior** Edits take effect immediately upon Save and are recorded in the audit trail. The system does **not** automatically notify users — admins should communicate any critical changes manually. **Limitations** * Roles and permissions are not editable here * Status changes managed separately * No bulk edit or version history *** ## Deactivating and Deleting Users Deleting a user removes their access to Entegrata while preserving their historical data for audit purposes. Deleted users cannot log in or access the system, but their historical data and audit trails remain intact. ### Step 1: Access Delete Option Click the actions menu (⋮) in the user's row and select **Delete**. User row actions menu ### Step 2: Confirm Deletion A confirmation dialog will appear to prevent accidental deletions. Delete user confirmation dialog **Behavior** * Status changes to **Deleted**, ending all sessions and invalidating tokens * Audit logs and historical data remain intact * Deletions are recorded with who performed the action and when **Recovery** Deleted users cannot be restored through the UI. Contact Entegrata Support if recovery is required. **Best Practices** * Confirm deletion is appropriate and reassign any owned resources * Notify relevant team members and document the reason **Retention** Data for deleted users remains under your firm’s retention policy until manually purged. ## Related Documentation Understanding user management basics in the platform Common issues and solutions for user management # Default Permissions Source: https://support.entegrata.com/data/access_control/default-permissions Configure base-level access permissions for canonical objects in your Lakehouse ## Overview Default permissions define the base-level access control for each canonical object (entity type) in your Entegrata Lakehouse. These permissions apply to all resources of a specific entity type unless overridden by more specific access rules. Understanding and properly configuring default permissions is crucial for establishing a secure foundation for your data access control. **Critical Configuration Required:** All canonical objects default to **Deny (false)** permissions when first mapped. This means no users will have access to the data until you explicitly configure permissions. ## How Default Permissions Work Default permissions operate at the entity level and provide a baseline for access control: 1. **Entity-Level Application**: Each canonical object type (e.g., Client, Matter, Timekeeper) has its own default permission setting 2. **Global Scope**: Default permissions apply to all resources within an entity type across your entire instance 3. **Override Capability**: Specific access rules can override default permissions for targeted subjects and resources 4. **Permission Inheritance**: New resources automatically inherit the default permissions of their entity type Default permissions are the starting point for access control. They should be set based on your organization's general access philosophy - either "default open" (allow by default) or "default closed" (deny by default). ## Access Requirements To configure default permissions, you must have: * Administrative privileges in the Entegrata Admin Portal * Access to the Data Mapping configuration * At least one mapped entity in your system * Understanding of your organization's data access policies ## Configuring Default Permissions ### Step 1: Navigate to Entity Access Control 1. Go to **Data** → **Mapping** in the main navigation 2. Select the mapping configuration you want to modify 3. Click on the specific entity you want to configure 4. Navigate to the **Access Control** tab Entity mapping page with Access Control tab highlighted ### Step 2: View Current Default Permissions The Access Control tab displays a card showing the current default permission setting: Default permissions card showing current Allow/Deny status The permission level for viewing resources of this entity type. Can be set to: * **Allow**: All users can view resources by default * **Deny**: No users can view resources by default (requires explicit rules to grant access) ### Step 3: Modify Default Permissions 1. Click the **Configure** button on the Default Permissions card 2. The configuration modal will open Default permissions modal 3. Toggle the permission setting: * **Enable** (Green) = Allow access by default * **Disable** (Red) = Deny access by default 4. Review the impact statement that explains how this setting affects access 5. Click **Save** to apply the changes Default permissions modal change Changing default permissions affects all resources of this entity type immediately. Ensure you understand the impact before making changes, especially when switching from Allow to Deny. ### Step 4: Verify Changes After saving: 1. The default permissions card will update to reflect the new setting 2. The permission badge will show the current state (green for Allow, red for Deny) 3. The change is logged in the audit trail with timestamp and user information Default permissions card change ## Permission Precedence Understanding how default permissions interact with specific rules is crucial: When a user attempts to access a resource, the system first checks the default permission for that entity type The system then evaluates any specific rules that apply to the user (directly or through groups) and the resource More specific rules override default permissions. A rule targeting a specific user and resource takes precedence over entity-level defaults If multiple rules apply, an explicit Deny permission always takes precedence over Allow permissions ## Best Practices ### Choosing Your Default Strategy **For Law Firms:** Always use Default Deny (Restrictive) approach. Law firms handle sensitive client data that requires maximum security. Starting with permissive defaults poses unacceptable risks for client confidentiality and regulatory compliance. **When to use:** * **Always for law firms and legal data** * Sensitive or regulated data * Compliance requirements * Production environments with strict security **Advantages:** * Maximum security by default * No accidental data exposure * Explicit grants required for all access * Clear audit trail of permissions * Meets compliance requirements **Implementation:** * Set all canonical objects to Deny by default * Create explicit rules for necessary access * Use groups to manage access efficiently * Document all permission grants **Warning:** This approach is not recommended for law firms. **Limited use cases:** * Non-production test environments only * Public or non-sensitive data * Temporary development setups **Risks:** * Potential data exposure * Compliance violations * Client confidentiality breaches * Regulatory penalties **If used temporarily:** * Must transition to Default Deny before production * Requires extensive auditing * Document why this approach was chosen ### Configuration Guidelines **Start with Entity Classification**: Group your entities by sensitivity level and apply consistent default permissions to similar entity types. **Document Your Decisions**: Keep a record of why specific default permissions were chosen for each entity type to help future administrators. **Review Regularly**: Periodically review default permissions as your data and access requirements evolve. **Test Before Production**: Use the permission testing tools to verify that default permissions work as expected before applying them to production data. ## Common Scenarios ### Scenario 1: Public Reference Data For entity types containing non-sensitive reference data (e.g., office locations, practice areas): * Set default permission to **Allow** * No additional rules needed for basic access * Add deny rules only for specific exceptions ### Scenario 2: Client Confidential Data For entity types containing sensitive client information: * Set default permission to **Deny** * Create rules to grant access to authorized teams * Use group-based rules for easier management ### Scenario 3: Mixed Sensitivity Data For entity types with varying sensitivity levels: * Set default permission based on the most common case * Use rules to handle exceptions * Consider splitting into multiple entity types if complexity grows ## Impact on Other Features Default permissions affect several aspects of the system: * **Data Discovery**: Users can only discover resources they have permission to view * **Reporting**: Reports only include data the user has permission to access * **API Access**: API calls respect default permissions and rules * **Search Results**: Search returns only permitted resources ## Troubleshooting ### Users Can't Access Expected Data 1. Check the default permission for the entity type 2. Verify no deny rules are blocking access 3. Test permissions using the testing tools 4. Review group memberships if using group-based rules ### Users Have Unexpected Access 1. Review default permissions (may be set to Allow) 2. Check for overly broad allow rules 3. Audit group memberships 4. Use permission explorer to trace access path ### Changes Not Taking Effect 1. Allow 1-2 minutes for propagation 2. Clear user session cache if immediate update needed 3. Verify the change was saved successfully 4. Check for conflicting rules that might override defaults ## Related Topics * [Creating Access Rules](/data/access_control/managing-rules) - Override defaults with specific rules * [Testing Permissions](/data/access_control/testing-permissions) - Verify your configuration * [Managing Groups](/data/access_control/managing-groups) - Organize users for rule application * [Enabling Access Control](/data/access_control/enabling-access-control) - Initial setup steps # Enabling Access Control Ingestion Source: https://support.entegrata.com/data/access_control/enabling-access-control Import existing access controls from Intapp Walls, Aderant, and Microsoft Entra. ## Overview Access control ingestion is an **optional feature** that allows you to import and synchronize existing permission configurations from supported source systems. This feature is currently available only for **Aderant**, **Intapp Walls**, and **Microsoft Entra** data sources. **Important:** This feature is NOT required to use access control in Entegrata. You can create system rules and groups within the admin app without enabling ingestion from one of these data sources.. This feature specifically imports existing permissions from supported sources to provide centralized visibility and management. ## Supported Data Sources ### Currently Supported * **Intapp Walls** - Imports ethical wall configurations and restrictions * **Aderant** - Synchronizes security rules from the FMS database * **Microsoft Entra** - Imports groups and group memberships for use in access control ## Connection-Specific Requirements Before enabling access control ingestion, ensure your source system meets the specific requirements for your connection type: ### Aderant Connection Requirements The Aderant connection supports access control ingestion based on the **REVOKED flag** in the **CMS\_SECURE\_ROW** table. **What This Means:** Access control rules are ingested by reading the REVOKED flag from CMS\_SECURE\_ROW. This flag indicates whether a user's access to a specific Aderant table or a record in an Aderant table has been revoked. **Limitation:** Only access control rules based on the REVOKED flag are supported. Other security mechanisms in Aderant (if present) will not be synchronized at this time. **Requirements:** * The CMS\_SECURE\_ROW table and its parent CMS\_SECURE\_PLAN table must contain valid access control data * The REVOKED flag in the CMS\_SECURE\_ROW table must be maintained in your Aderant database ### Intapp Walls Connection Requirements The Intapp Walls connection requires the **Generic Extension** to be enabled in your Intapp Walls application. **Critical Requirement:** The Generic Extension must be enabled in Intapp Walls. Without this extension, access control ingestion will not function. **Why This Matters:** The Generic Extension populates the necessary access control tables in Intapp Walls that Entegrata uses for rule ingestion. This is the same extension used by other applications like NetDocuments to integrate with Intapp Walls. **Requirements:** * Generic Extension must be enabled in your Intapp Walls application **How to Verify:** * Check with your Intapp administrator to confirm the Generic Extension is enabled * Verify that third-party integrations (like NetDocuments) are working, as they depend on the same extension ### Microsoft Entra Connection Requirements The Microsoft Entra API provides **field-level governance** on its resources, and the set of activated fields is unique to each customer's configuration. Because of this, specific resources and fields must be activated **before** enabling access control data ingestion. **Critical Requirement:** Activate the resources and fields below before enabling access control ingestion. **Required Resources and Fields:** | Resource | Required Fields | | - | - | | `groups` | `id`, `displayName`, `description` | | `/groups/{group-id}/members` | `__sys_id`, `__ref_groups_id`, `id` | | `users` | `id`, `userPrincipalName` | **User Identification:** In order to properly identify users in the access control system, map the `userPrincipalName` field to the **username** trait on the **Employment** entity. See [Mapping Fields](/data/pipeline/mapping-fields) for details on configuring entity traits. **How to Verify:** * Review the resource configuration for your Entra connection and confirm each resource listed above is activated with the required fields * Confirm the `userPrincipalName` → username trait mapping exists on the Employment entity before enabling ingestion ## When to Use This Feature Enable access control ingestion when you: * Have Aderant, Intapp Walls, or Microsoft Entra as a data source * Want to import existing security configurations from these systems * Need centralized visibility of both source and custom permissions * Want to maintain synchronization with source system permissions Skip this feature if you: * Only need custom access rules created in Entegrata * Don't have existing permissions to import ## Prerequisites Before enabling access control ingestion, ensure: * You have administrative privileges in the Entegrata Admin Portal * Your data source is Aderant, Intapp Walls, or Microsoft Entra * Data mapping is complete for the connection * Initial data collection has been completed * You understand the access control model of your source system ## Enabling Access Control Ingestion ### Step 1: Navigate to Connection Settings 1. Go to **Connections** in the main navigation 2. Select the connection you want to configure 3. Click on the **Access Control** tab Connection list with Aderant highlighted ### Step 2: Find Access Control Tab on Supported Connection The Access Control page displays the sync status of the data source's access control resources: Connection list with Aderant highlighted If you do not see an Access Control tab for your connection, this feature is not available for that connection type. Contact support for information about future availability. ### Step 3: Enable Access Control Sync For supported data sources, you'll see the **Sync Access Control Resources** card: When enabled: * Access control resources from this connection will be collected during scheduled runs * Permission data will be synchronized with your Entegrata instance * Access control data will be available in data mappings and groups To enable: 1. Toggle the **Sync Access Control Resources** switch to the enabled position 2. The system will start configuring access control 3. Wait for the confirmation message Connection list with Aderant highlighted ### Step 4: Verify Resource Discovery Once enabled, the system automatically discovers access control resources: The resources table displays: * **Name**: Resource identifier from the source system * **Load Type**: How the resource data is collected (Full, Incremental, etc.) * **Unique Keys**: Primary keys used for data synchronization * **Last Collection**: When the resource was last synchronized ### Configuring Collection Settings Access control resources can be configured using the same collection settings as regular data resources. For detailed information on configuring collection settings, including load types, unique fields, and collection profiles, see the [Data Collection documentation](/data/collector/connections/collection-settings). When configuring access control resources, consider: * Using **Full Load** for permission data to ensure consistency * Setting appropriate collection frequencies based on how often permissions change * Selecting stable primary keys that won't change in your source system ### Troubleshooting Sync Issues Common issues and solutions: * Check connection credentials have sufficient permissions * Verify source system has access control data * Try manually refreshing the resource list * Verify source system connectivity * Check for permission changes in source system * Review error logs for specific issues * Contact support with connection ID and error details * Ensure all required resources are enabled * Check primary key configuration * Verify source system data completeness * Run full load instead of incremental ## Understanding Imported Rules and Groups When you enable access control for a data source, Entegrata imports existing access control configurations from that system. ### How Imported Items Appear Imported rules and groups are displayed alongside system-created items with clear visual distinctions: * **Blue background tint** - Easily identify imported items * **Source column** - Shows originating system (Aderant, Intapp, etc.) * **Disabled tooltips** - Disabled editing of access control items in Entegrata Imported rules and groups are read-only in Entegrata. To modify these configurations, you must make changes in the source system. The changes will be synchronized during the next collection run. ### Combining System and Imported Controls Entegrata allows you to layer your own access controls on top of imported configurations: 1. **View all controls** - See both imported and system rules/groups in one place 2. **Create additional rules** - Add custom rules that work alongside imported ones 3. **Build custom groups** - Create groups that complement imported group structures 4. **Test combined permissions** - Validate how all rules work together This hybrid approach provides: * Complete visibility across all access controls * Flexibility to extend source system permissions * Centralized testing and auditing capabilities * Preservation of source system configurations ## Impact of Enabling Access Control ### On Data Collection * Additional resources collected during scheduled runs * Slightly increased collection time * More storage used for permission data * Additional processing for permission evaluation * Import of existing rules and groups from source systems ### On System Performance * Initial sync may be resource-intensive * Ongoing syncs typically have minimal impact * Permission evaluation adds small latency to queries * Caching minimizes performance overhead ### On User Experience * Access control rules become available in mappings * Permission testing tools activated * Data filtered based on user permissions * Group management features enabled ## Best Practices **Plan Your Rollout**: Enable access control during a maintenance window to avoid disrupting users during initial sync. **Start with One Source**: If you have multiple connections, enable access control for one source at a time to manage complexity. **Document Source Permissions**: Understand how permissions work in your source system before enabling synchronization. **Test in Non-Production**: If possible, test access control configuration in a development environment first. ## Disabling Access Control If you need to disable access control: 1. Navigate to **Connections** → **\[Your Connection]** → **Access Control** 2. Toggle **Sync Access Control Resources** to disabled Disabling access control: * Stops collecting access control data * Deletes all ingested rules and groups for the connection ## Next Steps After enabling access control: Set base-level permissions for your entities Define specific permission rules Organize users for permission management Verify permissions work as expected ## Frequently Asked Questions Currently, access control is only available for Aderant, Intapp Walls, and Microsoft Entra. For most organizations, daily synchronization is sufficient. High-compliance environments may require more frequent updates. Existing data remains unchanged. Access control only affects how users view and access data going forward. Yes, you can configure which access control resources to collect in the resource configuration settings. # Managing Groups Source: https://support.entegrata.com/data/access_control/managing-groups Create and manage access control groups for efficient permission management ## Overview Access control groups allow you to organize users and apply permissions efficiently at scale. Instead of creating individual rules for each user, you can group users together and manage their permissions collectively. Groups support both static membership (manually selected users) and dynamic membership (automatically determined by filters), providing flexibility for various organizational structures. ## Understanding Groups ### Group Types in Entegrata Entegrata provides centralized visibility into two types of access control groups: **What are System Groups?** Groups created directly by administrators within Entegrata for custom access control management. **Characteristics:** * Created and managed entirely within Entegrata * Fully editable and deletable * Display with standard formatting (white background) * Show "System" in the Source column * Support static membership (manually selected users) * Support dynamic membership (filter-based automatic inclusion) * Can include both users and other groups **What are Ingested Groups?** Groups imported from connected data sources (Aderant, Intapp Walls) during access control synchronization. **Characteristics:** * Imported from external data sources * Read-only (cannot be modified in Entegrata) * Display with blue background tint for easy identification * Show source system name (e.g., "Aderant", "Intapp") in Source column * Membership is managed in the source system * Automatically updated during synchronization * Changes must be made in the originating system The combination of system and ingested groups provides complete visibility into your access control configuration. You can create custom groups in Entegrata while maintaining synchronization with groups defined in your source systems. ### Membership Models **Characteristics:** * Members manually selected * Explicit user and group lists * Membership changes require manual updates * Best for stable, well-defined teams **Use Cases:** * Project teams * Committees * Temporary access groups **Characteristics:** * Members determined by filter rules * Automatic membership updates * Based on user attributes * Self-maintaining groups **Use Cases:** * Department-based groups * Role-based groups * Location-based groups **Characteristics:** * Combines static and dynamic members * Manual selections plus filter rules * Maximum flexibility * Handles exceptions elegantly **Use Cases:** * Department plus consultants * Role-based with exceptions * Complex organizational structures ## Creating a Group ### Step 1: Navigate to Groups 1. Go to **Access Control** → **Groups** in the main navigation 2. The groups list displays all existing groups Groups list page showing mix of system and ingested groups ### Step 2: Initiate Group Creation Click the **New Group** button in the toolbar: New Group button highlighted ### Step 3: Configure Group Settings The Create Group modal opens with multiple tabs: Create Group modal - initial view #### General Information Provide a clear, descriptive name: * Good: "Legal Department", "Project Alpha Team" * Avoid: "Group1", "Test", "Temp" Explain the group's purpose and membership criteria Choose the membership configuration method (detailed in next section) ### Step 4: Configure Membership The Members modal provides three methods for defining membership: #### Users Tab - Static User Selection 1. Click the **Users** tab 2. Browse or search for users 3. Check boxes next to users to include 4. Selected users appear with checkmarks 5. Use pagination for large user lists Users tab with search and selection Use the search box to quickly find users by name or email. The search updates results in real-time. #### Groups Tab - Include Other Groups 1. Click the **Groups** tab 2. Search for existing groups to include 3. Select groups whose members should inherit access 4. Nested group members automatically included Groups tab with group selection Avoid circular references - a group cannot include itself directly or indirectly through other groups. #### Membership Filters Tab - Dynamic Membership 1. Click the **Membership Filters** tab 2. Click **Add Filter Condition** 3. Configure the filter: * Select attribute (Department, Role, Location, etc.) * Choose operator (equals, contains, starts with, etc.) * Enter comparison value 4. Add multiple conditions with AND/OR logic Membership Filters tab with filter builder Example filters: * `Department equals "Legal"` * `Role contains "Manager"` * `Location in ["New York", "Boston"]` * `StartDate after "2024-01-01"` ### Step 5: Save the Group 1. Review all selected members and filters 2. Click **Save** to create the group 3. Monitor save progress: * Compiling → Validating → Syncing The new group appears in the groups list immediately after creation. ### Viewing Group Details Click on any group row to view details: Group detail view The detail view shows: * Group metadata (name, description, source) * Current member count * Creation and modification information * Action buttons (available only for system groups) * Source indicator (System, Aderant, Intapp, etc.) ## Viewing Group Members ### Member List Click on a group to see its members: 1. Navigate to `/access-control/groups/{groupId}` 2. View the complete member list 3. See member details: * Member type (Direct Members/Groups/Dynamic Members) * Name Group members list page ### Understanding Membership Sources Members can be part of a group through: Explicitly selected in the Users tab Member of an included subgroup Automatically included by matching filter criteria ## Editing Groups ### Modifying Group Settings Only system groups (created in Entegrata) can be edited. Ingested groups from external sources are read-only and display with a blue background tint. To modify ingested groups, make changes in the source system. To edit a system group: 1. Click on the group row in the table 2. Click **Edit** in the detail view 3. The Edit Group modal opens 4. Modify settings as needed: * Update name or description 5. Click **Save** to apply changes Edit Group modal When editing a group, existing members and filters are preserved. Changes are additive unless explicitly removed. ### Managing Members To modify group membership: 1. Navigate to the group details page 2. Click **Manage Members** button 3. The Add Members modal opens with current selections 4. Make changes: * Add new users or groups * Remove existing members (uncheck) * Modify filter conditions 5. Save changes Managing existing group members ## Actor Group Membership ### Viewing an Actor's Groups To see which groups an actor belongs to: 1. Go to **Access Control** → **Actors** 2. Click on an actor row 3. The Actor Groups modal opens 4. View all groups containing this actor Actor Groups modal ### Adding Actors to Groups From the Actor Groups modal: 1. Click **Add to Groups** 2. Select groups to add the actor to 3. Save changes 4. The actor immediately gains group permissions Adding actor to groups ## Deleting Groups ### Deletion Process To delete a system group: 1. Find the group in the groups table 2. Click the **Delete** action button 3. Review the confirmation dialog 4. Confirm deletion Delete group confirmation dialog Deleting a group: * Immediately removes all permissions granted through the group * Cannot be undone * Does not delete the users, only the group container * May affect access rules referencing the group ### Pre-Deletion Checklist Before deleting a group: * Check for rules using the group * Verify alternative access exists if needed * Document the deletion reason * Consider disabling instead of deleting ## Group Management Best Practices ### Naming Strategies Establish clear naming conventions: * **Organizational**: "Dept-Legal", "Dept-Finance" * **Functional**: "Role-Partners", "Role-Associates" * **Project-Based**: "Project-Alpha-Team", "Project-Beta-Reviewers" * **Hierarchical**: "Region-NA", "Region-NA-East" ### Organizational Patterns **Mirror Organization Structure**: Create groups that reflect your actual organizational hierarchy for intuitive management. **Use Nested Groups**: Build complex permissions using group hierarchies rather than duplicating members across groups. **Separate Concerns**: Create distinct groups for different access purposes (read vs. write, different data domains). **Regular Audits**: Schedule periodic reviews of group membership to remove inactive users and update filters. ### Dynamic Filter Best Practices When using membership filters: 1. **Start Simple**: Begin with basic filters and add complexity as needed 2. **Test Thoroughly**: Verify filters capture intended members 3. **Document Logic**: Explain filter rules in the group description 4. **Monitor Changes**: Review membership after attribute updates 5. **Plan for Edge Cases**: Consider how filters handle null or missing values ## Advanced Group Patterns ### Pattern 1: Hierarchical Organization Create a hierarchy reflecting organizational structure: ``` All-Employees (filter: Status = "Active") ├── Legal-Department (filter: Dept = "Legal") │ ├── Legal-Partners (filter: Role = "Partner" AND Dept = "Legal") │ └── Legal-Associates (filter: Role = "Associate" AND Dept = "Legal") └── Finance-Department (filter: Dept = "Finance") ├── Finance-Managers (filter: Role contains "Manager" AND Dept = "Finance") └── Finance-Analysts (filter: Role = "Analyst" AND Dept = "Finance") ``` ### Pattern 2: Project-Based Groups Combine static and dynamic membership: * Base group: Project team (static members) * Filter: Include all managers from participating departments * Subgroups: Specific role-based access within project ### Pattern 3: Compliance Groups For regulatory compliance: * Restricted group: Users with conflicts (static) * Cleared group: Users with clearance (static + filtered) * Monitor group: Compliance officers (filtered by role) ### Pattern 4: Temporal Groups For time-based access: * Summer interns (filter: EndDate > TODAY AND Role = "Intern") * Active consultants (filter: ContractEnd > TODAY) * Probation period (filter: StartDate > TODAY-90) ## Monitoring Group Effectiveness ### Group Metrics Track group usage and health: * **Member Count**: Current number of members * **Growth Rate**: Membership changes over time * **Rule Usage**: Number of rules referencing the group * **Last Modified**: Recent changes indicator * **Empty Groups**: Groups with no current members ### Membership Auditing Regular auditing tasks: 1. Review groups with no members 2. Check for duplicate or redundant groups 3. Verify filter effectiveness 4. Validate nested group relationships 5. Confirm external group synchronization ## Troubleshooting Groups ### Common Issues * Verify user matches filter criteria * Check if user was explicitly excluded * Allow time for filter evaluation (1-2 minutes) * Verify user exists in the system * Review filter conditions for over-inclusion * Check nested groups for inheritance * Verify attribute values in user profiles * Look for OR conditions that may be too broad * Verify it's a system group (not external) * Check your administrative permissions * Ensure group isn't locked by another process * Try refreshing the page * Verify attribute names are correct * Check for typos in comparison values * Test with simpler filter first * Consider case sensitivity ## Integration with Rules Groups are most powerful when used with access rules: 1. **Create logical groups** based on access needs 2. **Build rules** targeting groups instead of individuals 3. **Maintain groups** as organization changes 4. **Test permissions** at the group level Example workflow: 1. Create "Legal-Matters-Team" group 2. Add legal department members 3. Create rule granting access to legal matters 4. Assign rule to the group 5. New legal team members automatically get access ## Next Steps Use your groups in access control rules Verify group member access works correctly See how group membership affects individual access Set baseline permissions for your entities # Managing Rules Source: https://support.entegrata.com/data/access_control/managing-rules Create and manage granular access control rules for precise permission management ## Overview Access control rules allow you to define specific permissions that override default settings for targeted subjects (users or groups) and objects (resources). Rules provide granular control over who can access what data, supporting both simple static assignments and complex dynamic filters for automatic permission management. ## Understanding Rules ### Rule Components Every access control rule consists of four key components: * **Name**: Descriptive identifier for the rule * **Description**: Explanation of the rule's purpose * **Permission**: Allow or Deny access The users or groups that the rule applies to: * **Specific Users**: Individual actors selected by name * **Groups**: Access control groups containing multiple users * **Dynamic Filters**: Attribute-based selection using filter conditions The resources that the rule governs: * **All Resources**: Rule applies to entire entity type * **Specific Resources**: Individual records selected by ID * **Filtered Resources**: Dynamically selected using attribute filters * **Entity Type**: The canonical object type (Client, Matter, etc.) * **Instance**: Applied within your Entegrata instance ## Creating a Rule ### Step 1: Navigate to Rule Creation Rules can be created by following these steps: 1. Go to **Data** → **Mapping** 2. Click the **Access Control** tab 3. Click **Create Rule** in the toolbar Access Rules table showing mix of system and ingested rules ### Step 2: Configure General Settings The Rule Modal opens with the General tab active: Rule Modal - General tab Provide a clear, descriptive name that explains the rule's purpose * Good: "Finance Team - Budget Access" * Poor: "Rule 1" or "Test" Write a detailed explanation of why this rule exists and what it accomplishes Toggle the permission setting: * **Allow** (Green): Grants access to specified resources * **Deny** (Red): Blocks access to specified resources The modal background changes color (green for Allow, red for Deny) to provide visual confirmation of the permission setting. ### Step 3: Define Subjects (Who Has Access) Click the **Subject** tab to specify who this rule applies to: Rule Modal - Subject tab You can add users (actors) directly to rules without creating groups. Groups are optional but recommended for managing multiple users efficiently. #### Option 1: Select Specific Users (Actors) Add individual users directly to the rule: 1. Click the **Users** dropdown 2. Search for users by name 3. Select users as needed 4. Selected users appear as tags This approach works well for: * Small numbers of users * Temporary or one-off access * User-specific exceptions User selection dropdown with search #### Option 2: Select Groups (Recommended for Multiple Users) Use groups to manage multiple users more efficiently: 1. Click the **Groups** dropdown 2. Search for access control groups 3. Select groups as needed 4. All group members inherit the rule permissions Benefits of using groups: * Easier to manage multiple users * Changes to group membership automatically update access * Cleaner rule configuration * Better organization of permissions Group selection dropdown #### Option 3: Use Dynamic Filters For automatic subject selection based on attributes: 1. Click **Add Filter Condition** 2. Select the attribute to filter on 3. Choose the operator (equals, contains, etc.) 4. Enter the comparison value 5. Add additional conditions with AND/OR logic Subject filter builder interface Dynamic filters automatically apply to new users matching the criteria, reducing maintenance overhead. ### Step 4: Define Objects (What They Can Access) Click the **Object** tab to specify which resources the rule covers: Rule Modal - Object tab #### Option 1: All Resources Leave the resource selection empty to apply the rule to all resources of the entity type. #### Option 2: Specific Resources 1. Click the **Resources** dropdown 2. Search for specific records by name 3. Select resources as needed 4. Selected resources appear as tags Resource selection dropdown #### Option 3: Filtered Resources For dynamic resource selection: 1. Click **Add Filter Condition** 2. Select the resource attribute 3. Choose the comparison operator 4. Enter the value to match 5. Combine conditions as needed Object filter builder interface Example filters: * `Status equals Active` * `Department contains Legal` * `CreatedDate after 2024-01-01` ### Step 5: Preview Rule Impact Click the **Explore** tab to preview which subjects and objects will be affected: The exploration shows: * **Affected Subjects**: Count and list of users/groups * **Affected Objects**: Count and list of resources * **Effective Permissions**: Resulting access levels Rule Modal - Explore subjects tab Rule Modal - Explore objects tab Always review the exploration results before saving to ensure the rule affects only the intended subjects and objects. ### Step 6: Save the Rule 1. Review all settings across tabs 2. Click **Save** to create the rule 3. Monitor the save progress: * Compiling → Validating → Syncing ## Understanding Rule Types ### System (Native) vs Ingested Rules Entegrata displays two types of rules in your access control configuration: **What are System Rules?** System rules are created directly by administrators within Entegrata. These are fully manageable and form your custom access control layer on top of any ingested permissions. **Characteristics:** * Created by administrators in Entegrata * Fully editable and deletable * Display with standard formatting (white background) * Show "System" in the Source column * Can be modified at any time **Management:** 1. Click on the rule row to open for editing 2. Modify any settings as needed 3. Save changes or delete if no longer needed 4. Changes take effect immediately **What are Ingested Rules?** Ingested rules are imported from your connected data sources (Aderant, Intapp Walls) during access control synchronization. These represent the permissions configured in your source systems. **Characteristics:** * Imported from external data sources * Read-only (cannot be edited in Entegrata) * Display with blue background tint for easy identification * Show source system name (e.g., "Aderant", "Intapp") in Source column * Automatically updated during background sync operations **Viewing:** 1. Click on the rule row to view details 2. Review the configuration (Only General (read only) and Explore tabs are available) 3. Close modal when done 4. To modify these rules, changes must be made in the source system The visual distinction (blue background tint) helps you quickly identify which rules are managed in Entegrata versus those synchronized from external systems. ## Editing Rules ### Making Changes To edit an existing system rule: 1. Navigate to the rule list 2. Click on the rule you want to edit 3. The Rule Modal opens with current settings 4. Make necessary changes 5. Click **Save** to apply updates Changes to rules take effect immediately but may require 1-2 minutes to fully propagate across the system. ## Deleting Rules To delete a system rule: 1. Find the rule in the rules table 2. Click the **Delete** action button 3. Confirm deletion in the dialog 4. The rule is removed immediately Delete confirmation dialog Deleting a rule immediately removes all permissions it granted. Ensure alternative access is available if needed. ## Rule Management Best Practices ### Naming Conventions Establish consistent naming patterns: * `[Department] - [Resource Type] - [Access Level]` * `[Project] - [Team] - [Permission Type]` * `[Compliance] - [Regulation] - [Restriction]` Examples: * "Legal - Active Matters - Full Access" * "Project Alpha - Development Team - Read Only" * "SOX - Financial Data - Restricted" ### Organization Strategies **Use Groups**: Create rules for groups rather than individuals to simplify management and reduce rule count. **Layer Rules**: Start with broad rules, then add specific exceptions. This creates a clear permission hierarchy. **Document Purpose**: Always fill in the description field to explain why the rule exists and when it should be reviewed. **Regular Reviews**: Schedule periodic reviews of rules to remove obsolete ones and update as needed. ### Filter Best Practices When using dynamic filters: 1. **Test Thoroughly**: Use the Explore tab to verify filter results 2. **Avoid Over-Complexity**: Simple filters are easier to understand and maintain 3. **Consider Performance**: Complex filters may impact evaluation speed 4. **Document Logic**: Explain filter conditions in the rule description ## Advanced Rule Patterns ### Pattern 1: Department-Based Access **Scenario**: Grant department members access to their department's data **Implementation**: * Subject Filter: `Department equals "Finance"` * Object Filter: `OwnerDepartment equals "Finance"` * Permission: Allow ### Pattern 2: Time-Based Access **Scenario**: Temporary access for consultants **Implementation**: * Subject: Specific consultant group * Object Filter: `ProjectEndDate after TODAY` * Permission: Allow * Review regularly to remove expired access ### Pattern 3: Hierarchical Access **Scenario**: Managers access their team's data **Implementation**: * Subject Filter: `Role contains "Manager"` * Object Filter: `TeamId in [managed_teams]` * Permission: Allow ### Pattern 4: Compliance Restrictions **Scenario**: Implement ethical walls **Implementation**: * Subject: Restricted user group * Object: Specific conflicted matters * Permission: Deny * Higher precedence than allow rules ## Rule Precedence and Conflicts When multiple rules apply to the same subject and object: Rules targeting specific users override group rules Rules targeting specific resources override filtered resources If both Allow and Deny rules apply at the same specificity level, Deny wins Default permissions only apply when no specific rules match ### Conflict Resolution Example Given these rules for User A accessing Resource X: 1. Default: Allow (entity level) 2. Rule 1: Deny for Group B (User A is member) 3. Rule 2: Allow for User A specifically **Result**: User A has access (specific user rule overrides group rule) ## Troubleshooting Rules ### Common Issues * Allow 1-2 minutes for propagation * Verify subject and object selections * Check for conflicting higher-precedence rules * Test using permission explorer * Review filter conditions carefully * Check group membership if using groups * Verify attribute values in source data * Test filters with different operators * Look for deny rules with higher precedence * Check if default permissions changed * Verify user group memberships * Use permission testing to trace the denial source * Simplify complex filter conditions * Reduce number of OR conditions * Consider splitting into multiple simpler rules * Contact support for optimization assistance ## Next Steps Use permission testing tools to verify rules work correctly Create groups to simplify rule management Set appropriate default permissions as rule foundation Understand how rules affect overall access patterns # Access Control Overview Source: https://support.entegrata.com/data/access_control/summary Comprehensive guide to managing data access permissions in Entegrata ## Overview Entegrata's access control system works hand-in-hand with our data mapping platform to secure your data. Access control data can be ingested from supported data sources into our standardized models, or you can manage new controls directly in our platform. This centralized management and observability gives you a single pane of glass view into data security across your entire data footprint, making it easy to audit permissions, troubleshoot access issues, and ensure the right people see the right data in the edges supported by Entegrata (i.e Power BI, Operational Data Store, AI projects, and more). Access control rules extend to any endpoint of the lakehouse via tables that contain the rules & who the rules apply to (by individual person).  Access Control rules are deliberately stored & shipped in tables so they can be provided to any downstream system - Power BI, AI, a SQL server, etc - and are not beholden to those systems being capable of integrating with Unity Catalog. ## Key Features Set entity-level default access permissions that apply to all resources of a specific data type Create granular access rules with subject and object targeting for precise permission control Organize users into groups with static or dynamic membership for efficient access management Test and explore permissions to verify access configurations before deployment ## Availability and Prerequisites Access control features require specific prerequisites to be met before this feature can be used. This section outlines those pre-requisites. ### Top-Level Requirement **Production Pipeline Required:** Access control only works when the "main" production pipeline exists. Below, find additional prerequisites per feature area. ### Supported Entities Access control rules (both System and Ingested) are available for the following canonical data mappings: Client entity data mappings Matter entity data mappings Person entity data mappings ### Prerequisites by Feature **Prerequisites for managing access control groups and actors:** To access the Groups and Actors pages: * The **Person** entity must be mapped and deployed To properly identify actors in your entity OBTs (One Big Tables), the **Employment** entity must be deployed with the **Entra** data source as a source that maps Entra's **userPrincipalName** field to the canonical **username** field. **Prerequisites for managing access control rules on entities:** To create and manage access control rules: 1. The **Person** entity must be mapped, deployed, and have completed at least one pipeline run (required for subject data) 2. The target entity (Client, Matter, or Person) must be mapped and deployed 3. The main pipeline must have completed at least one run after the target entity was deployed **Rules cannot be created without subject data.** The Person entity must be mapped, deployed, and have completed at least one pipeline run before any rules can be created. This is required because all rules need subjects (actors or groups) which are derived from Person data. Rules become available for a target entity once it has been deployed and processed through the main pipeline. **Prerequisites for enabling access control data ingestion from data sources:** **Actor Resolution Requirement:** For any actor resolution to take place on ingested rules and groups, each data source must be mapped in the **Person** entity in addition to mapping in Client and/or Matter entities. Without a Person entity mapping for the data source, the system cannot resolve which actors the ingested permissions apply to. **Required entity mappings:** * Person entity must be mapped and deployed * Employment entity must be mapped and deployed **Mapping Expectations:** * Map Person entity from the Entra users table * Map Employment entity from the Entra users table **Important:** Entra should map **userPrincipalName** to Employment **username** for OBT (One Big Table) usage. Otherwise, the principal name's in the OBTs will fall back to the **email** field on the Person entity. **Required entity mappings:** * Person entity must be mapped and deployed * Client entity must be mapped and deployed * Matter entity must be mapped and deployed **Additional requirement:** The CMS\_SECURE\_ROW table with the REVOKED flag and its parent CMS\_SECURE\_PLAN table must contain valid access control data. **Mapping Expectations:** * Map Person entity from the Aderant HBM\_PERSNL table * Map Client entity from the Aderant client table (typically HBM\_CLIENT) * Map Matter entity from the Aderant matter table (typically HBM\_MATTER) **Required entity mappings:** * Person entity must be mapped and deployed * Client entity must be mapped and deployed * Matter entity must be mapped and deployed **Additional requirement:** The Generic Extension must be enabled in your Intapp Walls application. **Mapping Expectations:** For each entity, configure the Intapp Walls data source with proper key matching to your primary source (typically Aderant): **Person Entity Mapping:** * Set **EntityId** as the Intapp Walls primary key * Join on your primary Person source (e.g., HBM\_PERSNL) * Match Intapp Walls identifiers (**FinancialSystemId**, **EntityCustomData**) to your primary source fields (**EMPL\_UNO**, **EMAIL**) Person entity mapping for Intapp Walls **Client Entity Mapping:** * Set **EntityId** as the Intapp Walls primary key * Join on your primary Client source (e.g., HBM\_CLIENT) * Match Intapp Walls **EntityRemoteSystemId** to your primary source client identifier (**CLIENT\_NUMBER**) Client entity mapping for Intapp Walls **Matter Entity Mapping:** * Set **EntityId** as the Intapp Walls primary key * Join on your primary Matter source (e.g., HBM\_MATTER) * Match Intapp Walls **FinancialSystemId** to your primary source matter identifier (**CLIENTS\_MATTER\_NUMBER**) Matter entity mapping for Intapp Walls ### Rule Types **Available for: Client, Matter, and Person entities** Once prerequisites are met for an entity: * Create custom access rules directly in Entegrata * Build groups with static or dynamic membership * Define granular permissions for your data * Layer controls on top of any existing permissions System rules provide full control and can be edited or deleted at any time. **Available for: Entra, Aderant, and Intapp Walls data sources** For supported data sources with prerequisites met, you can additionally: * Import existing access control configurations * Synchronize security groups from source systems * Maintain visibility of source-defined permissions * Combine ingested rules with custom system rules These appear as read-only (blue tinted) in the interface and must be modified in the source system. **Critical:** All canonical objects default to **Deny (false)** permissions. After mapping data sources, you must review and adjust default permissions to ensure users have appropriate base-level access. ## Core Concepts ### Permission Model The Entegrata Access Control system uses a hierarchical permission model: 1. **Default Permissions** - Base-level permissions applied to all resources of an entity type 2. **Rule-Based Permissions** - Specific rules that override defaults for targeted subjects and objects 3. **Permission Precedence** - More specific rules take precedence over general ones ### Key Terminology An individual user in the system who can be granted access permissions An actor or group that permissions are applied to (who has access) The data entity or specific record being accessed (what is being accessed) A canonical data type in your Lakehouse (e.g., Client, Matter, Timekeeper) An access control policy that defines permissions for specific subjects and objects A collection of actors that can be managed together for access control purposes ## How Access Control Works Ensure your main production pipeline is deployed and has completed at least one successful run. Access control features are only available after this prerequisite is met. Complete data mapping configuration for Client, Matter, and Person entities. Deploy these entity mappings and ensure the main pipeline processes them successfully. Check and configure default permissions for each entity. **Important:** All entities default to Deny (false), so this step is critical to ensure proper base-level access. For Aderant or Intapp Walls data sources, optionally enable access control synchronization to import existing rules and groups from these systems (requires specific entity prerequisites). Organize users into groups for efficient permission management, using either static membership or dynamic filters. Requires Person entity to be deployed. For best actor identification, also map Employment with Entra for UPN-to-username mapping. Create system rules to implement your access control requirements, layering custom permissions on top of defaults and any ingested rules. Requires Person entity to be deployed and run for subject data. Use the comprehensive testing tools to verify that permissions work as expected before rolling out to users. Use exploration features to review access patterns, audit permissions, and refine your configuration over time. ## Common Use Cases ### Legal Data Security * Restrict matter access to assigned attorneys and staff * Implement ethical walls for conflict management * Control access to sensitive client information ### Compliance Requirements * Enforce data residency and jurisdiction rules * Maintain audit trails of access permissions * Implement need-to-know access policies ### Organizational Hierarchy * Grant department-level access to relevant data * Implement role-based access control (RBAC) * Manage temporary access for consultants ## Best Practices **Start with Deny, Grant Explicitly**: Law firms require strict data security. Begin with restrictive defaults (Deny) and explicitly grant access where needed. This ensures no accidental data exposure. **Use Groups Effectively**: Create logical groups that reflect your organizational structure to simplify permission management. **Test Before Production**: Always use the permission testing tools to verify access configurations before applying them to production data. **Document Your Rules**: Use descriptive names and descriptions for rules to make their purpose clear to other administrators. ## Next Steps Learn how to enable access control for your data sources Set up default permissions for your entities Build custom access rules for specific scenarios Use testing tools to understand permission flows ## Need Help? If you encounter issues or have questions about access control configuration: * Contact [Entegrata Support](https://support.entegrata.com) * Review the [troubleshooting guide](/data/access_control/testing-permissions#troubleshooting) * Schedule a consultation with your Customer Success Manager # Testing Permissions Source: https://support.entegrata.com/data/access_control/testing-permissions Test, explore, and validate access control configurations to ensure proper permissions ## Overview The permission testing and exploration tools allow you to validate your access control configuration, understand permission flows, and troubleshoot access issues. These tools provide visibility into how permissions are evaluated and help ensure users have appropriate access to resources. ## Testing Tools Available Entegrata provides three main testing interfaces: Test specific subject-object permission combinations at the entity level View all permissions for a specific user across all resources Test permissions for group members and understand group access patterns ## Test Permissions ### When to Use Test Permissions Use this tool to: * Verify if a specific user can access a specific resource * Understand which rules determine access decisions * Troubleshoot unexpected access grants or denials ### Accessing Test Permissions Test Permissions can be accessed from the entity level page: **From Entity Access Control:** 1. Navigate to **Data** → **Mapping** 2. Select your mapping and entity 3. Go to the **Access Control** tab 4. Click **Test Permissions** in the toolbar Test permissions button ### Using Test Permissions The Test Permissions modal provides a visual interface for permission testing: Test Permissions modal interface Choose who you're testing access for: * Start typing to search users or groups * Select from the dropdown list * Both actors and groups appear in the combined list * Selected subject appears on the left side Choose what resource to test: * Leave empty to test entity-level permissions * Select specific resource from dropdown * Search by resource name or ID * Selected resource appears on the right side Click the **Play** button between subject and resource: * The test executes real time evaluation of permissions * Connection lines visualize the relationship * Permission result displays in the center The result shows: * Permission badge (green Allow / red Deny) * Which rules contributed to the decision * Rule precedence and evaluation order * Final permission determination Test permissions results ### Understanding Test Results The results panel displays: * **Allow** (green badge): Subject has view access * **Deny** (red badge): Subject is blocked from access Shows all rules that affected the decision: * Rule name and description * Permission type (Allow/Deny) * Precedence level * Why the rule applies Explains how the final permission was determined: * Default permission consideration * Rule evaluation order * Precedence conflicts resolution * Final decision reasoning ## Explore Actor Access ### Purpose Actor exploration provides a comprehensive view of an individual's access to all resources across your entire system. ### Starting Actor Exploration 1. Go to **Access Control** → **Actors** 2. Click **Explore Actor** in the toolbar 3. The Actor Explorer modal opens Explore actor modal Search and select the user to explore: * Type to search by name * Select from the dropdown * Click **Explore** to proceed The exploration shows all entity types: * Entity type and name * Aggregated permission level * Click an entity that shows an exception count Click an entity to see resource details: * List of resource exceptions * Individual resource permissions * Navigation breadcrumb to return ### Actor Access Views #### Entity Level View Explore actor results The entity view displays: * **Entity Type**: Canonical object type * **Resource Name**: Number of accessible resources * **Exception Count**: Resources with access opposite of entity level * **Permission Badge**: Aggregated access level Navigation: * Click any entity row to see resources * Use pagination for large entity lists * Sort by name or resource count #### Resource Level View Explore actor resource results The resource view shows: * **Breadcrumb**: Path back to entity list * **Entity Summary**: Overall entity permissions * **Resource List**: Individual resources * **Resource Details**: Name and permissions Features: * Paginated resource list * Individual permission badges * Back navigation to entity view ### Interpreting Actor Exploration Common patterns to look for: * **Full Access**: Allow badge on all entities * **Limited Access**: Mix of Allow and Deny badges * **No Access**: All Deny or no entities shown * **Exceptions**: High exception counts indicate complex rules ## Group Permission Testing ### Accessing Group Tests To test group-level permissions: 1. Navigate to **Access Control** → **Groups** 2. Click on a group to view details 3. Click **Test Permissions** button Test group permissions button ### Testing Options The Group Permissions Test modal offers two modes: Group test permissions results Tests permissions that apply to the group as a whole: * Select "Group Permissions" as the subject * Choose entity and optionally resource * Run test to see group-level access * Results show rules targeting the group Tests permissions for individual group members: * Select specific member from dropdown * Shows member name * Choose entity and resource to test * Results include both individual and group rules ### Group Test Workflow Choose between: * Group Permissions (group as a whole) * Specific member (individual in context of group) Pick the entity type to test: * Loads all available entities * Shows entity names and types For specific resource testing: * Available resources load based on entity * Leave empty for entity-level test * Search to find specific resources Click play button to run test: * Visual flow shows test configuration * Results appear immediately * Detailed rule explanation provided ## Exploring Rules ### Rule Impact Preview When creating or editing a rule, use the Explore tab to preview impact: Rule explore view The exploration shows: * **Affected Subjects**: Users and groups covered by the rule * **Affected Objects**: Resources the rule applies to * **Permission Impact**: How the rule changes access Always review the Explore tab before saving rules to ensure they affect only intended subjects and objects. ### Understanding Rule Exploration The Explore tab provides counts and samples: * Total count of affected subjects * List of affected subjects * Groups included * Total count of affected resources * List of affected resources * Resource type breakdown ## Best Practices for Testing ### Testing Strategy **Test Early and Often**: Test permissions during configuration, not after deployment. **Use Representative Examples**: Test with actual users and resources that represent common use cases. **Document Test Cases**: Keep a record of test scenarios for regression testing after changes. **Test Edge Cases**: Include tests for new users, archived resources, and boundary conditions. ### Systematic Testing Approach Verify default permissions work as expected: * Test with user having no specific rules * Confirm entity-level defaults apply * Document baseline behavior Validate individual rules: * Test each rule in isolation * Verify subject selection works * Confirm resource targeting is correct Check rule interactions: * Test overlapping rules * Verify precedence works correctly * Confirm deny overrides allow Validate group-based access: * Test group membership inheritance * Verify nested groups work * Check dynamic membership filters Use actual use cases: * Test common user workflows * Verify department access patterns * Validate compliance requirements ## Troubleshooting Access Issues ### Common Problems and Solutions **Diagnosis steps:** 1. Use Test Permissions with user and resource 2. Check if default is Deny 3. Look for blocking Deny rules 4. Verify user group memberships 5. Check resource exists and is active **Common fixes:** * Add Allow rule for user/group * Remove conflicting Deny rule * Add user to appropriate group * Adjust default permissions **Diagnosis steps:** 1. Use Actor Explorer to see all access 2. Test specific concerning resources 3. Review applied rules 4. Check for overly broad Allow rules 5. Verify group memberships **Common fixes:** * Add specific Deny rule * Narrow existing Allow rules * Remove user from groups * Change default from Allow to Deny **Diagnosis steps:** 1. Wait 1-2 minutes for propagation 2. Re-test permissions 3. Check if changes were saved 4. Verify no errors during save **Common fixes:** * Clear browser cache * Refresh the page * Re-save the configuration * Contact support if persists **Diagnosis steps:** 1. Verify testing correct user 2. Check testing correct resource 3. Confirm test environment matches production 4. Review any system-level overrides **Common fixes:** * Ensure consistent testing parameters * Check for environment differences * Verify no cached permissions * Review audit logs ## Advanced Testing Scenarios ### Scenario 1: Multi-Level Inheritance Test complex permission inheritance: ``` User → Group A → Group B → Rule ``` Verify permissions flow correctly through nested groups. ### Scenario 2: Time-Based Access Test temporal permissions: * Filter-based rules with date conditions * Verify access changes over time * Test expiration handling ### Scenario 3: Conflicting Rules Test precedence with conflicts: * User-specific Deny * Group-level Allow * Default Allow * Verify Deny wins ### Scenario 4: Dynamic Membership Test filter-based groups: * Change user attributes * Verify group membership updates * Test resulting permission changes ## Next Steps Create and manage access control rules Organize users into access control groups Configure entity-level default permissions Return to access control overview ## Getting Help If you encounter issues with permission testing: 1. **Check Documentation**: Review relevant sections for your scenario 2. **Contact Support**: Include test parameters and results 3. **Provide Context**: Share rules, groups, and expected behavior 4. **Include Screenshots**: Capture test configurations and results # About Connectors Source: https://support.entegrata.com/data/collector/catalog Types of data connectors Entegrata supports and their general requirements ## Overview Entegrata connectors are the integrations that move data between your source systems and the Lakehouse. They fall into a few broad types — each with its own setup style, authentication options, and operational characteristics. This page covers **connector types** and the **general requirements** that apply to all of them. For the full, card-by-card catalog of every specific system Entegrata integrates with, see the Data Connectors page. Full catalog of specific systems Entegrata can collect from and emit to, with per-connector setup guides ## Connector Types ### SQL Connectors SQL connectors read directly from relational databases. This includes generic engines (Microsoft SQL Server, Oracle, Microsoft Fabric SQL / Warehouse / Lakehouse) as well as purpose-built legal platforms whose data lives in SQL Server (Aderant, 3E, Iridium, Intapp, Foundation, and others). **Key features:** * **Direct database access** — query tables and views in the source with no middleware * **Schema discovery** — tables, views, columns, and constraints are discovered automatically * **Incremental extraction** — pull only changed rows using a cursor column * **Sync & Delete Operations** — most SQL connectors can detect row-level changes without a cursor and soft-delete records removed from the source (see [Sync Operation](/data/collector/resources/sync-operations)) ### API Connectors API connectors integrate with SaaS platforms through their REST APIs — DealCloud, Zoom, ADP, UKG, Microsoft Entra, Dynamics CRM, and others. They handle the operational complexity of REST automatically: authentication, pagination, rate limiting, and retries. **Key features:** * **OAuth, API key, or bearer-token authentication** * **Automatic pagination and rate-limit handling** * **Schema discovery** from API responses (or OpenAPI specs where available) * **Scheduled collection** with incremental updates ### Delimited File Connectors Delimited file connectors ingest CSV / TXT files from a directory on SFTP, Azure Blob Storage, or a local drive. They read flat-file data plus a separate **schema definition file** that tells the collector how to interpret each file. #### Supported File Types **Data files**: * `.csv` * `.txt` Only flat directories are processed — subdirectories are skipped during directory expansion. **Schema definition files**: * `.csv` * `.xlsx` (detected automatically by file contents) #### Schema Modes The schema file format depends on the `Headers in Schema` parameter set on the connection: * **`true`** (Full Schema mode): The schema definition file contains the full column definition for every table — column names, data types, and primary key flags. Use this when you need explicit control over data types or when your data files do not include a header row. * **`false`** (Inferred Schema mode): The schema definition file only maps tables to files. Column names are read from the first row of each data file, and data types are inferred from the first 100 rows. Use this when your data files have headers and you want quick setup. #### Required Schema File Headers The schema file must contain these columns in the header row: | Column | Required | Description | | - | - | - | | `TABLE_NAME` | Yes | Name of the table. Rows with the same `TABLE_NAME` are grouped into a single resource. | | `COLUMN_NAME` | Yes | Field name in the data file. | | `DATA_TYPE` | Yes | SQL data type (e.g. `varchar(50)`, `int`, `bigint`, `date`, `timestamp`, `decimal`, `bool`). | | `PRIMARY_KEY` | No | Whether the column is part of the primary key. Accepts `Y/N`, `true/false`, `1/0`, `y/n`, `yes/no`. Empty = not PK. | | `FILE_PATH` | Yes | Path to the data file, relative to the connection's directory. Each table needs at least one non-empty value. | **Copy-paste header row**: ```csv theme={null} TABLE_NAME,COLUMN_NAME,DATA_TYPE,PRIMARY_KEY,FILE_PATH ``` **Example**: ```csv theme={null} TABLE_NAME,COLUMN_NAME,DATA_TYPE,PRIMARY_KEY,FILE_PATH employees,employee_id,varchar(12),Y,employees.csv employees,first_name,varchar(50),N, employees,last_name,varchar(50),N, employees,hire_date,date,N, departments,department_id,varchar(10),Y,departments.csv departments,department_name,varchar(100),N, ``` `FILE_PATH` only needs a value on one row per table — subsequent rows for the same table can leave it blank to avoid repetition. To load multiple files into the same table (e.g. yearly partitions), add additional rows with the same `TABLE_NAME` and different `FILE_PATH` values. The schema file only maps tables to files: | Column | Required | Description | | - | - | - | | `TABLE_NAME` | Yes | Name of the table. | | `FILE_PATH` | Yes | Path to the data file, relative to the connection's directory. | | `PRIMARY_KEY` | No | Comma-delimited list of primary key column names (e.g. `"order_id,customer_id"`). Leave blank to auto-generate a key. | **Copy-paste header row**: ```csv theme={null} TABLE_NAME,FILE_PATH,PRIMARY_KEY ``` **Example**: ```csv theme={null} TABLE_NAME,FILE_PATH,PRIMARY_KEY employees,employees.csv,employee_id departments,departments.csv,department_id orders,orders.csv,"order_id,customer_id" ``` In Inferred mode, the first row of each data file must be a header row. Column types are inferred by sampling the first 100 rows. ## General Requirements ### Network Access Every connector needs network reachability from Entegrata to the source system. For databases and file servers, that usually means firewall or VPN rules allowing Entegrata's egress addresses; for public APIs, it's handled for you. See [Instance Administration](/administration/instance/overview) for your instance's network settings. ### Credentials Entegrata stores credentials **encrypted at rest** and uses them only to authenticate to the source. Credentials are never exposed back to Entegrata users or printed in logs. Grant read-only access whenever possible. For collector connections Entegrata never needs write, delete, or schema-modification permissions on source systems. ### Credential Rotation For connectors using time-bound credentials (API tokens, client secrets, certificates), set the **Secret Expiration** field on the connection so Entegrata notifies you before the credential expires. ## Need a New Connector? If you need a connector that isn't in the [catalog](/data/collector/data-connectors/overview), contact your Entegrata Customer Experience Manager or [submit a support ticket](https://support.entegrata.com). # Collection Settings Source: https://support.entegrata.com/data/collector/connections/collection-settings Configure connection-level schedules, throttling, and collection behavior ## Overview Connection collection settings define default behavior for all resources within a connection. Individual resources can override these defaults with their own settings. Collection settings apply to all resources in the connection unless overridden at the resource level. ## Accessing Collection Settings Click on a connection to view its Resources page Click the **Settings** icon (gear) in the Collection section The Collection Settings modal opens ## Schedule Configuration Choose how and when resources in this connection collect data. ### Interval-Based Scheduling Resources collect on regular time intervals or at specific times. **Options**: * Specific times: "Daily at 2:00 AM", "Weekly on Sunday at 3:00 PM" * Regular intervals: "Every 6 hours", "Every 30 minutes" Use interval scheduling for most use cases - it's simple and reliable. ### Timing: Static vs Rolling For schedules that run **more than once a day** (for example every 2 hours or every 30 minutes), you can choose how each run is timed: The next collection is scheduled for one full interval **after the previous one finishes**. This is simple and reliable, but the exact run time drifts depending on how long each collection takes. *Example*: an "every 2 hours" collection that starts at 1:00 and takes 5 minutes finishes at 1:05. The next run is scheduled 2 hours **after it finished** — so it starts at 3:05 (not 3:00), finishes at 3:10, and the following run starts at 5:10, and so on. Each run drifts a little later by however long the previous one took. Collections run on a **fixed clock schedule** anchored to a **start time** you choose, regardless of how long each run takes. *Example*: "every 2 hours" with a start time of 6:15 always runs at 6:15, 8:15, 10:15, … — on the dot. When you choose **Static**, set a **Start time** — the time of day the schedule is anchored to, shown in your instance's timezone. Daylight saving is handled automatically, so runs stay at the same local time year-round. The Timing option only appears for sub-daily schedules (hourly or minute-level intervals). Schedules that run daily or less often already fire at a specific time of day, so there is nothing to anchor. Choose **Static** when you need predictable, evenly spaced runs (for example "every 2 hours on the dot"). **Rolling** is fine for most cases. ## Collection Profile Control collection speed to manage load on your source systems. Information can be found in the [Throttle Profile](/data/collector/connections/overview#throttle-profile) documentation. Profile settings affect API rate limiting and database query batching but don't change schedules. ## Active Status Enable or disable collection for the entire connection. **Active** (Toggle ON): * Resources collect according to their schedules * Discovery runs automatically every 3 hours * Normal operation **Inactive** (Toggle OFF): * All data collection paused * Discovery still runs but no data is collected * Use for temporary suspension (e.g., maintenance windows) Setting a connection to Inactive stops ALL resource collection, even if individual resources are enabled. ## Saving Settings After making changes: Verify all settings are correct Click the **Save** button at the bottom of the modal Settings apply immediately to the connection and all resources using defaults ## How Settings Apply to Resources ### Default Behavior Resources without their own collection settings inherit from the connection: | Connection Setting | Applies to Resources | | - | - | | Schedule | Resources with no custom schedule | | Active Status | All resources (connection inactive = all paused) | | Collection Profile | All resources unless overridden | ### Resource Overrides Individual resources can override: * ✅ Schedule (intervals) * ✅ Active status (if connection is active) * ❌ Collection profile (always inherits from connection) See [Resource Collection Settings](/data/collector/resources/collection-settings) to configure resource-specific overrides. ## Best Practices Use simple interval schedules until you need more complex scheduling Set collection times when your source systems have low activity Only change to High or Low if you have specific performance needs Monitor the first few collections after changing schedules to verify timing ## Common Scenarios ### Scenario: Daily Overnight Collection **Goal**: Collect all connection data once per day at 2:00 AM **Configuration**: * Schedule Type: Interval * Schedule: "Daily at 2:00 AM" * Profile: Standard * Active: ON ### Scenario: Frequent Real-Time Updates **Goal**: Keep data as current as possible with hourly updates **Configuration**: * Schedule Type: Interval * Schedule: "Every 1 hour" * Profile: High (for maximum speed) * Active: ON ### Scenario: Maintenance Window **Goal**: Temporarily pause collection during source system maintenance **Configuration**: * Active: OFF * (Keep all other settings unchanged) * Re-enable Active when maintenance completes ## Troubleshooting ### Settings Not Applying **Symptoms**: Changes saved but collection behavior doesn't change **Possible causes**: * Resources have overriding settings * Connection is inactive * Schedule hasn't triggered yet **Resolution**: 1. Check individual resource settings for overrides 2. Verify connection Active status is ON 3. Wait for next scheduled time or trigger manual collection ### Override connection defaults for specific resources Track collection execution and verify schedules Resolve collection issues # Create Connections Source: https://support.entegrata.com/data/collector/connections/creating-connections Step-by-step guide to add new data source connections ## Overview Creating a new connection establishes the link between Entegrata and your data source. This process involves selecting a data source type, providing authentication credentials, and configuring connection parameters. Ensure you have the necessary credentials and network access before creating a connection. Test connectivity from your network if possible. ## Access Requirements To create connections, you must: * Be logged into the Entegrata Admin Portal * Have administrative privileges for your instance * Have valid credentials for the data source you're connecting to * Ensure the data source is accessible from Entegrata's network ## Creating a New Connection ### Step 1: Navigate to New Connection 1. Go to **Collectors** in the main navigation 2. Click the **New Connection** button in the top-right corner New Connection button ### Step 2: Select Data Source Type Choose your data source from the available options. Data source selection library #### Available Data Sources The full list of data connectors can be found at [https://www.entegrata.com](https://www.entegrata.com) Don't see your data source? Contact your Entegrata Customer Experience Manager to inquire about custom connectors or upcoming integrations. ### Step 3: Configure Connection Details After selecting a data source, you'll see a configuration form with fields specific to that source type. image.png #### Universal Fields A unique, descriptive name for this connection **Best practices**: * Include environment: "Production SQL - Finance" * Include purpose: "DealCloud - Deal Data" * Avoid generic names like "Connection 1" * Must be unique across all your connections If you enter a nickname that already exists, you'll see an error and need to choose a different name. When authentication credentials expire and need renewal **Why set this**: * Receive notifications before credentials expire * Plan credential rotation in advance * Avoid unexpected connection failures Secret expiration date picker #### Database Connection Fields For SQL databases (SQL Server, etc.): Server hostname or IP address Examples: * `sqlserver.company.com` * `192.168.1.100` * `sql.azure.microsoft.com` Database server port (defaults to standard port for each database type) Default ports: * SQL Server: 1433 Name of the database to connect to The specific database is required and is used to determine which resources can be discovered and collected. If you need to connect to multiple databases, you can create multiple connections. ### Required SQL Permissions To automatically throttle the collection of data based on the amount of data being collected, Entegrata needs the following permissions: Grant VIEW DATABASE STATE Grant VIEW SERVER STATE permissions for: * sys.dm\_exec\_sessions * sys.dm\_exec\_connections * sys.dm\_exec\_requests #### API Connection Fields For cloud platforms (DealCloud, Zoom, ADP, UKG, etc.): Base URL for the API (often pre-filled) Authentication token or API key for the platform Your organization or tenant identifier (if applicable) This is a non-exhaustive list of fields that may be required for each data source type. Your specific data source may require additional fields. Please contact Entegrata Support for assistance if you need help configuring your connection. ### Step 4: Configure Authentication Select the authentication method and provide credentials. #### SQL Authentication Standard username and password authentication: SQL Authentication selected Database login username Database login password (encrypted and stored securely) #### Windows Authentication (coming soon) Domain-based authentication for on-premise SQL Server: Windows domain name Domain username Domain password Windows Authentication requires network connectivity to your domain controller. #### Azure Entra ID (Managed Identity) Passwordless authentication for Azure-hosted environments: With Managed Identity, no credentials need to be entered. Authentication is handled automatically by your Azure tenant. Entegrata uses the managed identity assigned to the hosting environment to authenticate directly with your Azure data sources. Managed Identity is the recommended authentication method for Azure-hosted data sources. It eliminates the need to manage and rotate credentials, reducing operational overhead and improving security. #### Azure Entra Service Principal Application-based authentication using Azure AD app registration credentials: Azure Active Directory tenant identifier Application (client) ID from Azure AD app registration Client secret from Azure AD app registration Client certificate from Azure AD app registration Provide either a **Client Secret** or a **Client Certificate** — one of the two is required. Service Principal credentials (client secrets and certificates) expire and must be rotated. Always set the **Secret Expiration** field to track when rotation is needed. #### Selecting Authentication Type For data sources that support multiple authentication methods (like SQL Server), choose from the dropdown: Authentication type selector The form fields update based on your selection. ### Step 5: Test and Save Connection 1. Review all entered information for accuracy 2. Click **Connect** to test the connection and save it Connect button to save and test #### What Happens Next When you click Connect, Entegrata: Checks that all required fields are filled correctly Attempts to connect to your data source using the provided credentials Stores the connection details securely (credentials are encrypted) Begins discovering available resources (tables, views, endpoints) Takes you to the Resources page to view discovered resources ## Connection Success If the connection succeeds, you'll see a success message and be redirected to the Resources page. The Resources page shows: * All discovered resources * Discovery status * Options to enable resources for collection Discovery may take a few seconds to a few minutes depending on the size of your data source. ## Connection Failure If the connection fails, you'll see an error message explaining why. ### Common Error Messages **Cause**: Incorrect username, password, or authentication credentials **Resolution**: * Verify credentials are correct * Check for typos or extra spaces * Ensure the account hasn't been locked or disabled * Try logging in directly to the source system with the same credentials **Cause**: Data source is not accessible from Entegrata's network **Resolution**: * Verify the host/IP address is correct * Check firewall rules allow Entegrata's IP addresses * Ensure the database or API service is running * Test connectivity from another location on the internet **Cause**: User account lacks necessary permissions **Resolution**: * Grant READ permissions on required schemas/tables * For APIs, ensure the API key has appropriate scopes * Check database-level and schema-level permissions * Review the minimum permissions required for your data source type **Cause**: Specified database name doesn't exist or isn't accessible **Resolution**: * Verify the database name is spelled correctly * Check that the database exists on the server * Ensure the user account has access to that database **Cause**: Data source requires encrypted connection **Resolution**: * Enable SSL/TLS options if available in the connection form * Contact Entegrata Support if SSL options aren't visible * Ensure your data source is configured to accept SSL connections After 3 failed connection attempts, you may need to wait a few minutes before trying again to prevent account lockouts. ## Post-Creation Steps After successfully creating a connection: Allow automatic discovery to complete (usually under 5 minutes) Browse the resources found in your data source Toggle on the specific tables, views, or endpoints you want to collect See [Enabling Resources](/data/collector/resources/enabling-resources) for details Set schedules and collection behavior for the connection See [Collection Settings](/data/collector/connections/collection-settings) for details ## Best Practices Create dedicated service accounts with minimum necessary permissions, not personal accounts Keep track of which credentials are used for each connection in your organization's password manager Always set secret expiration dates to ensure proactive credential rotation Create test connections first to verify configuration before production deployments Grant only SELECT/READ permissions to Entegrata - write permissions are not needed Keep your firewall rules updated if Entegrata's IP addresses change ## Security Considerations ### Credential Storage * All credentials are encrypted at rest using industry-standard encryption * Passwords are never stored in plain text * Credentials are only used for connection and collection purposes ### Network Security * Connections use encrypted protocols (TLS/SSL) when supported * Data in transit is encrypted * IP allowlisting is recommended for database connections ### Permission Management Only grant the minimum permissions required for data collection. Entegrata never needs write, delete, or schema modification permissions. **Recommended permissions**: * **Databases**: SELECT on required schemas/tables * **APIs**: Read-only API scopes * **File systems**: Read access to required directories ## Troubleshooting If you encounter issues during connection creation, see the [Troubleshooting Guide](/data/collector/connections/troubleshooting) for detailed resolution steps. ## Next Steps Configure schedules and collection behavior Enable and configure resources for collection Learn how discovery finds your data Browse and search your connections # Delete Connections Source: https://support.entegrata.com/data/collector/connections/deleting-connections Remove connections and understand the impact on collected data ## Overview Deleting a connection permanently removes the connection configuration, authentication credentials, and all associated collection settings. However, data already collected to your data lakehouse is preserved. Connection deletion cannot be undone. Ensure you have backups of any configuration you may need to recreate the connection. ## Before You Delete Consider these alternatives to deletion: Temporarily pause collection without losing configuration Stop collection for individual resources while keeping the connection ## Access Requirements To delete connections, you must: * Be logged into the Entegrata Admin Portal * Have administrative privileges for your instance * Have confirmation that the connection is no longer needed ## What Gets Deleted When you delete a connection: **Removed**: * ❌ Connection configuration and credentials * ❌ All resource collection settings * ❌ Collection schedules and dependencies * ❌ Job history and execution records * ❌ Discovery metadata **Preserved**: * ✅ Data already collected in your data lakehouse * ✅ Downstream transformations and reports (may need data source updates) ## Deleting a Connection Go to the Connections page and click on the connection to delete Click the settings menu and select **Delete Connection** A confirmation dialog appears warning about the permanent action Review the warning and type the connection name to confirm Click **Delete** to permanently remove the connection The connection is immediately removed from your connections list ## Post-Deletion After deletion: 1. The connection disappears from the Connections page 2. All associated resources are no longer accessible via Entegrata 3. Scheduled collection jobs are canceled 4. Data in your lakehouse remains available for queries and reports If you need to reconnect to the same data source, create a new connection with the same credentials and configuration. ## Recreating a Deleted Connection If you accidentally delete a connection or need to recreate it: Follow the [Creating Connections](/data/collector/connections/creating-connections) guide Enter the same authentication and connection details as before Discovery will find resources again Re-enable resources and configure collection settings Settings are not preserved from the deleted connection - you must reconfigure manually ## Best Practices Document connection settings and resource configurations before deleting Confirm collected data is accessible in your lakehouse before deleting Review and update any mapping dependencies before deletion Inform teams using this data source before deleting the connection ## Next Steps Add a new connection to replace the deleted one Return to the connections list # Edit Connections Source: https://support.entegrata.com/data/collector/connections/editing-connections Update connection credentials and configuration settings ## Overview Edit existing connections to update expired credentials, change connection parameters, or modify authentication settings. Editing preserves all resource configurations and collection history. ## Access Requirements To edit connections, you must: * Be logged into the Entegrata Admin Portal * Have administrative privileges for your instance * Have the updated credentials or configuration details ## When to Edit a Connection Common scenarios for editing connections: * **Expired credentials** - Password or API keys have expired or changed * **Updated endpoints** - Host, port, or API endpoint has changed * **Authentication method change** - Switching from SQL to Windows authentication (coming soon) * **Secret expiration update** - Extending credential expiration dates * **Connection parameter changes** - Database name, organization ID, etc. Changing connection parameters (like database name or API endpoint) may affect existing resource configurations. ## Editing Connection Details ### Step 1: Navigate to Connection 1. Go to **Collectors** in the main navigation 2. Click on the connection you want to edit Click connection to view details ### Step 2: Access Edit Interface From the connection's Resources page, click the **Edit Connection** or settings icon. The connection configuration form opens with current settings pre-filled. ### Step 3: Update Fields Modify any of the following fields: The previously entered credentials will be empty as they are encrypted and stored securely and not fetched when editing the connection. Update the connection name (must remain unique) Update username, password, API keys, or tokens * Passwords are masked but can be replaced * New credentials are encrypted upon saving Modify host, port, database name, or API endpoints Changing these may require re-discovery of resources Update or extend credential expiration date ### Step 4: Save Changes 1. Review all modifications 2. Click **Update** to save and test the connection #### What Happens When You Save Entegrata validates the updated configuration Tests connectivity with the new credentials/parameters Saves the changes if the test succeeds May automatically trigger re-discovery if connection parameters changed ## Update Success If the update succeeds, you'll see a confirmation message. The connection immediately uses the new credentials for all future operations. In-progress collection jobs continue with the old credentials and complete normally. ## Update Failure If the update fails (e.g., invalid credentials), you'll see an error. The connection retains its previous configuration - changes are not saved. ### Handling Update Errors **Cause**: New credentials are incorrect **Resolution**: * Double-check username and password * Verify API keys or tokens are valid * Test credentials directly with the data source * Ensure no typos or extra spaces **Cause**: New host/endpoint is unreachable **Resolution**: * Verify host name or IP address * Check DNS resolution * Ensure firewall allows Entegrata's IP addresses * Test connectivity from another location **Cause**: New database name doesn't exist **Resolution**: * Confirm database name spelling * Ensure database exists on the server * Check user has access to the database ## Updating Authentication Type For connections that support multiple authentication methods (like SQL Server): Choose from the authentication type dropdown Authentication type dropdown Form fields update to match the selected authentication type Click **Update** to save the changes Changing authentication types (e.g., from SQL to Windows Authentication) requires entering completely new credentials. ## Updating Secret Expiration Extend or modify when credentials expire: Click the Secret Expiration date field Secret expiration date picker Choose a new expiration date (or clear it to remove expiration tracking) Click **Update** to save Set secret expiration dates a few days before actual expiration to receive advance notification. ## Impact of Editing Connections ### What's Preserved * All resource configurations (schedules, load types, filters) * Collection history and job records * Unique keys and primary key settings * Discovered resource catalog ### What May Change * **New resources discovered** if connection parameters change * **Resources removed** if they're no longer accessible with new credentials * **Schema changes detected** if switching databases or endpoints After editing connection parameters, review discovered resources to ensure expected tables/endpoints are still available. ## Best Practices Verify new credentials work by logging in directly before updating the connection Update credentials during low-activity periods to minimize impact Keep your internal documentation current with credential changes Manually run discovery after changing databases or endpoints to detect changes immediately Watch the first collection job after credential updates to ensure it succeeds ## Canceling Changes To discard changes without saving: 1. Click **Cancel** in the edit form 2. Changes are not applied 3. Connection retains its previous configuration ## Troubleshooting Edit Issues ### Changes Not Saving **Symptoms**: Click Update but changes don't persist **Possible causes**: * Network interruption during save * Validation error not displayed * Browser caching issue **Resolution**: 1. Refresh the page and try again 2. Check browser console for errors 3. Try a different browser 4. Contact support if issue persists ### Connection Goes to Error State After Edit **Symptoms**: Connection shows Error status after updating **Possible causes**: * New credentials are incorrect * Host/endpoint unreachable * Permissions insufficient with new credentials **Resolution**: 1. Edit the connection again 2. Verify all credentials are correct 3. Test connectivity to the data source 4. Check [Troubleshooting](/data/collector/connections/troubleshooting) for detailed steps ## Next Steps Update schedules and collection behavior Trigger discovery after connection changes Resolve connection issues Return to connections list # Overview Source: https://support.entegrata.com/data/collector/connections/overview Understand connections and how they link Entegrata to your data sources ## Overview Connections represent Entegrata's link to your data sources. Each connection securely stores authentication credentials and defines default collection settings for all resources within that data source. ## Key Concepts ### What is a Connection? A **Connection** provides the secure bridge between Entegrata and a data source (database, API, file system). Each connection includes: Secure credentials (username/password, API keys, tokens) used to access your data source Technical details needed to reach your data source (host, port, database name, API endpoint) Default schedule that determines when resources collect data (can be overridden per resource) Current state of the connection: * **Active** - Connection is operational and resources can collect * **Inactive** - Connection is paused; no data collection occurs ### Connection Lifecycle Add a new connection by providing data source type and authentication details Entegrata automatically discovers available resources (tables, views, endpoints) Set connection-level collection defaults (schedule, throttling, status) Resources collect data according to configured schedules Update credentials, adjust settings, or troubleshoot issues as needed Remove connection when no longer needed (preserves collected data) Each data source type has specific authentication requirements and available connection parameters. ## Connection vs. Resources It's important to understand the relationship: | Aspect | Connection | Resource | | - | - | - | | **Authentication** | Stored at connection level | Inherited from connection | | **Schedule** | Default for all resources | Can override connection default | | **Discovery** | Discovers all resources | Discovered within connection | | **Enable/Disable** | Determines if any resources can collect | Only affects specific resource | ## Authentication Methods ### SQL Authentication Username and password-based authentication for SQL databases: * Standard login credentials * Password expiration tracking * Connection string parameters ### Windows Authentication (coming soon) Domain-based authentication for on-premise SQL Server: * Uses Windows domain credentials * No password required (trusted connection) * Requires domain connectivity ### API Keys and Tokens Token-based authentication for cloud APIs: * API keys for various services * OAuth tokens for various platforms * Bearer tokens for REST APIs ### Azure Entra ID (Managed Identity) Passwordless authentication for Azure-hosted environments: * No credentials required — authentication is handled by your Azure tenant * Uses the managed identity assigned to the hosting environment * Recommended for Azure-hosted data sources ### Azure Entra Service Principal Application-based authentication for Azure services: * Requires Tenant ID, Client ID, and either a Client Secret or Client Certificate from an Azure AD app registration * Credentials must be rotated before expiration * Use when Managed Identity is not available ## Connection Settings ### Collection Schedule Determines when resources in this connection collect data: * **Interval-based**: Regular time intervals (minutely, hourly, daily) * **Time-based**: Specific times (e.g., "Daily at 2:00 AM") Individual resources can override the connection's default schedule. ### Throttle Profile Controls collection speed to manage source system load: * **Safe** - Slower collection to reduce impact on source systems (least resource intensive) * **Standard** - Normal collection speed for most use cases (default) * **Faster** - Faster collection for time-sensitive data (medium resource intensive) * **Fastest** - Fastest collection from read replicas (most resource intensive) ### Secret Expiration Track when authentication credentials expire: * Set expiration date for passwords and API keys * Receive notifications before expiration * Plan credential rotation in advance ## Connection Status ### Active * Collection is allowed * Resources can collect according to schedule * Discovery runs automatically every 3 hours ### Inactive * Collection is paused * All resources stop collecting (even if individually enabled) * Discovery does not run Setting a connection's collection status to Inactive pauses ALL resource collection within that connection, regardless of individual resource settings. ## Getting Started Step-by-step guide to add a new data source connection Browse and search your configured connections Update credentials and connection settings Configure schedules and collection behavior ## Next Steps Once your connection is created: Wait for automatic discovery or trigger it manually to find resources Enable the specific tables, views, or endpoints you want to collect Track collection execution and performance Configure individual resources within your connection Learn how Entegrata discovers resources # Troubleshooting Source: https://support.entegrata.com/data/collector/connections/troubleshooting Resolve connection authentication, connectivity, and configuration issues ## Overview This guide helps you diagnose and resolve common connection issues including authentication failures, network problems, and configuration errors. ## Authentication Failures ### Invalid Credentials **Symptoms**: "Authentication Failed" or "Login failed" errors **Causes**: * Incorrect username or password * Expired credentials * Account locked or disabled **Resolution**: 1. Verify credentials by logging in directly to the data source 2. Check for typos or case sensitivity 3. Ensure account is active and not locked 4. Update connection with correct credentials ### Permission Denied **Symptoms**: "Insufficient permissions" or "Access denied" errors **Causes**: * Service Account lacks READ permissions * Database/schema not accessible * API key has insufficient scopes **Resolution**: 1. Grant SELECT permissions on required tables/schemas 2. Verify database-level access 3. For APIs, check API key scopes and permissions 4. Test permissions by querying directly ## Connectivity Issues ### Cannot Reach Host **Symptoms**: "Connection timeout" or "Host unreachable" errors **Causes**: * Incorrect host/IP address * Firewall blocking Entegrata's IP addresses * Service is down **Resolution**: 1. Verify host name or IP address is correct 2. Check firewall rules allow Entegrata's IPs 3. Ensure database/API service is running 4. Test connectivity from another internet location ### SSL/TLS Errors **Symptoms**: "SSL handshake failed" or "Certificate error" **Causes**: * Source requires SSL but connection doesn't use it * Certificate issues * Protocol mismatch **Resolution**: 1. Enable SSL/TLS in connection settings if available 2. Verify source system SSL configuration 3. Contact Support for SSL troubleshooting ## Configuration Issues ### Database Not Found **Symptoms**: "Database does not exist" errors **Causes**: * Database name misspelled * Database doesn't exist on server * Service Account lacks access to database **Resolution**: 1. Verify exact database name (case-sensitive on some platforms) 2. Confirm database exists using database management tools 3. Check that Service Account has access permissions to the database ### Port Issues **Symptoms**: "Connection refused" on specific port **Causes**: * Incorrect port number * Service not listening on that port * Firewall blocking port **Resolution**: 1. Verify correct port for your database type (SQL Server: 1433) 2. Check service is running and listening on the port 3. Ensure firewall allows the port ## Best Practices Before creating/updating connection, verify credentials work directly Ensure Entegrata's IP addresses are allowlisted Dedicated service accounts with minimum permissions Set expiration dates and rotate credentials proactively ## Getting Help If issues persist after troubleshooting: 1. Gather error messages and screenshots 2. Note the connection ID and timestamp of failures 3. Contact Support with this information 4. Include details about your data source type and configuration ## Next Steps Update connection settings to resolve issues Start fresh with a new connection if needed # View Connections Source: https://support.entegrata.com/data/collector/connections/viewing-connections Browse, search, and filter your data source connections ## Overview The Connections page provides a centralized view of all your configured data sources. Use this page to browse connections, check their status, and access detailed configuration. ## Accessing the Connections Page 1. Navigate to **Collectors** in the main navigation 2. The Connections page displays by default Connections page in navigation ## Connections List The main connections table displays all your configured data sources with key information at a glance. Connections list showing configured data sources ### Column Information Unique identifier for the connection (hidden by default, for technical reference) Your custom nickname for this connection, making it easy to identify The type of data source with an icon (SQL Server, Aderant, DealCloud, etc.) When the connection configuration was last updated Current connection state: * **Active** (Green) - Operational and collecting data * **Inactive** (Gray) - Paused, not collecting * **Error** (Red) - Authentication or connectivity issues ## Searching Connections Use the search bar to quickly find specific connections by name or data source type. ### Using Search Find the search input at the top of the connections list Search bar above connections table Type any part of the connection name or data source type Examples: * "Production" - Finds all connections with "Production" in the name * "SQL" - Finds all SQL Server connections * "DealCloud" - Finds DealCloud connections The table updates automatically as you type, showing only matching connections Search results filtered Click the X icon or clear the search field to show all connections Search is case-insensitive and matches partial text, making it easy to find connections even if you don't remember the exact name. ## Sorting Connections Click column headers to sort the connections list. ### Sortable Columns * **Name** - Alphabetical A-Z or Z-A * **Data Source** - Alphabetical by source type * **Last Modified** - Most recent first or oldest first * **Status** - Grouped by status type Sortable column headers with sort indicators The sort indicator (up/down arrow) shows which column is currently sorted and in which direction. ## Pagination For accounts with many connections, the table uses pagination. See how many connections are displayed and total count at the bottom of the table Pagination showing X of Y connections Use the page navigation controls: * **Previous** - Go to previous page * **Page numbers** - Jump to specific page * **Next** - Go to next page Pagination controls Change how many connections appear per page (10, 20, 50, 100) ## Viewing Connection Details Click any connection row to view its resources and detailed configuration. Click anywhere on the connection row (except action buttons) Clickable connection row You'll be taken to the Resources page for this connection Connection resources page This page shows: * All discovered resources (tables, views, endpoints, files) * Resource collection status * Collection jobs and history Clicking a connection is the quickest way to start managing its resources and collection settings. ## Connection Status Indicators Understanding status badges helps you quickly identify connection health. ### Active Status Active connection badge **Meaning**: Connection is configured correctly and operational * Resources can collect according to schedule * Discovery runs automatically * No action needed ### Inactive Status Inactive connection badge **Meaning**: Connection is intentionally paused * No data collection occurring * Discovery still runs * Can be re-activated anytime ### Error Status Error connection badge **Meaning**: Connection has authentication or connectivity issues * Collection failing or cannot connect * **Action required**: Update credentials or check connectivity * See [Troubleshooting](/data/collector/connections/troubleshooting) for resolution steps ## Quick Actions From the connections list, you can perform quick actions without navigating away. ### Creating a New Connection Click **New Connection** in the top-right corner to add a new data source. New Connection button See [Creating Connections](/data/collector/connections/creating-connections) for detailed instructions. ### Accessing Connection Settings Some connection types may offer quick settings access directly from the list (implementation varies). ## Empty State If you haven't created any connections yet, you'll see an empty state with guidance. Click **New Connection** to add your first data source. ## Best Practices Name connections clearly (e.g., "Production SQL - Finance DB" instead of "SQL1") Review the connections list periodically to catch errors early Use consistent naming conventions to make searching easier Track when connections were last updated to identify stale configurations ## Next Steps Add a new data source connection Update existing connection settings Configure resources within your connections Resolve connection issues # Catalog Source: https://support.entegrata.com/data/collector/data-connectors/overview Systems Entegrata can collect data from ## Overview Entegrata ingests data from external systems through **connectors**. Each connector knows how to authenticate to a particular source system, discover its schema, and pull data into the Lakehouse. The pill on each card below opens the step-by-step setup guide for that connector. To write data back out to an external system, see the [Emitter catalog](/data/collector/data-emitters/catalog). ## Catalog Elite 3E Financial Management System Aderant Financial Management System ADP HR data — workers and time-off requests Anaplan module data — one table per module, typed by line item Asana work management — projects, tasks, subtasks, comments, attachments, and custom fields BigSquare Business Intelligence Cisco Meraki Dashboard — networks, devices, and telemetry CompuLaw docketing and calendaring CourtAlert court-case tracking feed Courtroom Insight judge, expert, and case-team intelligence Dayforce HCM — DDN entity APIs (employees and details) Dayforce Analytics — any ad hoc report built in Dayforce Analytics as a resource Dayforce Reports — any report defined in Dayforce as a resource DealCloud CRM - accounts, contacts, opportunities, and custom entry types CSV / TXT files in Azure Blob Storage CSV / TXT files served over SFTP Docket Alarm dockets, reports, and parties Elite Xcelerate budgeting & forecasting Microsoft Fabric Lakehouse Microsoft Fabric SQL endpoint Microsoft Fabric Warehouse Litera Foundation Export Database HubSpot CRM — contacts, companies, deals, activities, custom objects, and owners Ideate SQL Server database iManage Security Policy Manager - organization security, user activity, and compliance data iManage Work — libraries, workspaces, and document profiles Index.io matter classification and organization enrichment Intapp Conflicts — conflict checks, client entities, and relationship data Intapp Intake matter management Intapp Intake matter management via the REST API Intapp Terms — outside counsel guidelines, terms of business, and billing obligations Intapp Time — time entries, timekeeper, and billing data Intapp Walls ethical wall management LexisNexis Intelligize document analysis BigHand Intellistat financial and legal analytics LexisNexis InterAction CRM Introhive relationship intelligence — activities, contacts, companies, relationships LexisNexis IQ relationship intelligence Iridium BI data warehouse Lead Forensics B2B website-visitor identification — businesses, visits, pages, and KPI breakdowns Mailchimp campaigns, audiences, and list members Microsoft Dynamics 365 / Dataverse Microsoft Entra ID directory via Microsoft Graph CSV / TXT files in a user's OneDrive via Microsoft Graph Any Microsoft SQL Server database NetDocuments document search and metadata Nexl CRM - contacts, organizations, and relationships Oracle Database Piwik PRO analytics, tag manager, and account administration RubyLaw page data — attorneys, matters, offices, events, news Salesforce CRM — standard and custom SObjects via Bulk API 2.0 Sanity.io content management system SharePoint lists via Microsoft Graph Sitecore CMS content via the SSC API SurveyMonkey surveys, responses, and contacts Thomson Reuters Financial Insights benchmark and peer-group metrics UKG Pro REST API — HR, payroll, and time UKG Ready REST API — employees, cost centers, jobs UKG Ready reports — any saved or global report as a Lakehouse resource WordPress posts, custom post types, taxonomies, ACF fields, and revisions Workday Report-as-a-Service — any custom report exposed as a web service Zoom users, accounts, meetings, and phone data Don't see your system? Contact your Entegrata Customer Experience Manager to ask about custom connectors or upcoming integrations. ## Next Steps Step-by-step guide to add a new connection Configure load type, cursor field, unique keys, and filters per resource # Catalog Source: https://support.entegrata.com/data/collector/data-emitters/catalog Destinations Entegrata can emit data to ## Overview Entegrata writes data out to external systems through **emitters** — the mirror image of [collectors](/data/collector/data-connectors/overview). An emitter takes the curated output of a pipeline layer in the Lakehouse and ships it to a destination system you control. The pill on each card below opens the step-by-step setup guide for that destination. ## Catalog On-prem SQL Server, Azure SQL Database, Azure SQL Managed Instance, or RDS for SQL Server Microsoft Fabric SQL endpoint Microsoft Fabric Warehouse Don't see your destination? Contact your Entegrata Customer Experience Manager about adding emit support to an existing connector or building a new one. ## Next Steps How emitters work, properties, modes, and lifecycle Add an emission connection (write access) for a destination # Emitters Overview Source: https://support.entegrata.com/data/collector/data-emitters/overview How Entegrata writes data back out to external destination systems ## What is an Emitter? An **emitter** is the mirror of a [connector](/data/collector/data-connectors/overview): where a connector brings data **in** to the Lakehouse, an emitter pushes selected data **out** to an external system you control (your SQL Server, your Fabric Warehouse, etc.). Emitters use the same *integration types* the collector uses (Microsoft SQL Server, Microsoft Fabric SQL, and so on) — but each emitter requires its own [connection](/data/collector/connections/overview) to the destination system, separate from any collector connection. Emission connections need **write** credentials; collection connections only need **read**. You can't reuse a collector connection for an emitter, even if it points at the same system. See which systems Entegrata can emit to today ## How an Emitter Works Each emitter is bound to one **destination connection** and one **source layer** in the Lakehouse: * **Source layer** — typically the curated Silver layer of a pipeline you've already mapped. Emit takes whatever transformations you've defined in the pipeline and ships the result. * **Destination connection** — the credentials and network details for the system on the receiving end. * **Resource selection** — the specific tables/views from the source layer the emitter is allowed to write. On every emission run, Entegrata: Loads the latest contents of each selected resource from the Lakehouse. Writes rows into a staging table in the destination, then atomically swaps or merges into the target. Existing indexes, unique keys, and foreign keys are preserved across the swap. Logs which entities succeeded, which failed, row counts, and duration. ## Emitter Properties Every emitter has a set of properties you configure when you create it. Some apply to all destinations; others are specific to the destination type the emitter targets. ### Core Properties Display label for the emitter. Used everywhere the emitter appears in the admin portal. Optional human-readable description. The destination [connection](/data/collector/connections/overview) this emitter writes to. Only active emission connections are selectable. How emissions are kicked off. * **Manual** — run on demand from the emitter detail page. * **Event** — currently fires automatically when an associated pipeline run completes. More event sources may be added in the future. Event-driven emitters can still be triggered manually at any time from the emitter detail page — the trigger type controls automatic runs, not whether you're allowed to run it yourself. Swap primary-key and foreign-key columns to a numeric `sys_id` (bigint) instead of the default `id` (uuid) when the source resource has one. Use this when the destination prefers integer keys for indexing or downstream tooling. ### Destination-Specific Properties Beyond the core fields, each emitter exposes options that come from the destination type. These are shown on the **Options** tab when you create or edit an emitter, and the available fields are determined by the selected connection. Which Lakehouse layer the emitter pulls from. Currently only **Silver** is supported. **Gold** may be enabled as a source in the future. Schema inside the destination database where emitted tables land. Defaults vary by destination (`dbo` for SQL Server / Fabric, etc.). Dedicate a schema to Entegrata — see [Permissions Model](#permissions-model) below. For Silver emitters on platform v1.3.151+, this is set in the **Schemas** section, which lists the Databricks schemas in the pipeline's Silver catalog, such as **Main** and **Metric**: * When **Main** is the only schema, it's preselected with a target of `dbo`, which you can change. * When there are several, the section starts empty. Click **Add Schema** to pick each schema you want to emit, and enter its target schema. Schemas you leave out aren't emitted. Each schema can land in its own target schema (for example, **Main** to `dbo` and **Metric** to `metrics`), or several can share one. Schemas that share a target can't contain tables with the same name. Every schema you add needs a target before the emitter can be saved. The emission connection needs write access to every target schema you enter. How emitted table and column names are formatted. Default is `snake_case`. Configure to match the downstream tools that consume the replica. The exact set of destination-specific properties depends on the connector type. See the per-destination guide ([Microsoft SQL Server](/data/collector/data-emitters/mssql), [Fabric SQL](/data/collector/data-emitters/fabric-sql), [Fabric Warehouse](/data/collector/data-emitters/fabric-warehouse)) for the full list available for that destination. ## Emission Modes You choose the mode when you start an emission. Entegrata then makes a best-effort attempt to achieve that mode and falls back automatically if the underlying data doesn't allow it. Emit every selected resource from scratch. Entegrata loads all rows into staging tables and atomically swaps them with the targets. Use Full when you want a guaranteed clean state in the destination, after large schema changes, or for the first run. Emit only what has changed in the source layer since the last successful emission, applying inserts, updates, and deletes via a `MERGE` against the target's primary key. If a source table has been rebuilt or changed dramatically enough that an incremental merge would no longer be performant, Entegrata automatically falls back to Full for that table. The mode you selected is honored where possible — affected tables are upgraded to Full transparently. On platform v1.3.151+, if a destination table was emptied or dropped since the last emission, Entegrata reloads that table in full rather than merging only recent changes into it. Emit only a curated subset of resources you pick from the catalog. Useful for backfilling a single table, isolating a problem table, or shipping a fresh dataset without re-running everything. Entegrata automatically pulls in any tables your selection depends on via foreign keys, so referential integrity is preserved on the destination. ## Lifecycle Add the destination's credentials and network details. This is a **separate connection** from any collector connection pointing at the same system — emission connections need write access. Bind a source pipeline layer to the destination connection. The emitter inherits credentials and network rules from the connection. Choose the tables/views to emit. The picker surfaces row counts, recently updated flags, and foreign-key dependencies so you can see what will be included. Trigger the emitter on demand or let it run on its configured schedule, picking Full, Incremental, or Specific. Check the emitter detail page to see per-run status, per-entity outcomes, and row counts. ## Permissions Model Emitters need **write** access to the destination, unlike collectors which only need read. The service account or identity the emission connection uses typically needs: * `SELECT`, `INSERT`, `UPDATE`, `DELETE` on the target schema * `CREATE TABLE`, `ALTER`, `DROP` on the target schema (so Entegrata can create staging tables, apply schema changes, and atomically swap tables on Full) * `VIEW DEFINITION` on existing objects in the schema Entegrata manages the lifecycle of the tables it emits — it creates, drops, and swaps them as part of normal operation. Dedicate an emission schema rather than sharing one with tables maintained by other systems. The exact permissions vary by destination. See the per-destination guide for specifics. ## Supported Destinations On-prem SQL Server, Azure SQL Database, Azure SQL Managed Instance, or RDS for SQL Server Microsoft Fabric SQL endpoint Microsoft Fabric Warehouse Don't see your destination? Contact your Entegrata Customer Experience Manager about adding emit support to an existing connector or building a new one. ## Related Browse the systems Entegrata can collect data from Manage the connections emitters depend on Define the transformations whose output emitters ship Monitor emission runs alongside collection runs # Manual Discovery Source: https://support.entegrata.com/data/collector/discovery/manual-discovery Trigger discovery on-demand to find new resources ## Overview Manually trigger discovery to immediately scan for new resources or schema changes without waiting for the automatic 3-hour interval. ### Manual Discovery You can trigger discovery immediately when you need to: * Check for new resources right after adding them to the source * Refresh schema after making changes in the source system * Troubleshoot collection issues that may be schema-related * Verify connectivity after updating credentials ## Triggering Manual Discovery Go to the Resources page for your connection Click the **Discover Now**, **Refresh**, or **Disco** button (typically in the top-right corner or header) Discover now button in header A progress indicator shows discovery is running Discovery typically completes in: * Small data sources: 10-30 seconds * Medium data sources: 1-3 minutes * Large data sources: 5-10 minutes After discovery completes, the page refreshes to show: * Updated resource list * New resources discovered * Schema change notifications * Updated "Last Discovered" timestamp Understand how discovery works Track discovery job status and schema changes # Overview Source: https://support.entegrata.com/data/collector/discovery/overview Understand how Entegrata discovers and tracks your data sources ## Overview Discovery is Entegrata's automated process for scanning data sources to find available resources and tracking their schemas. ## How Discovery Works When discovery runs, Entegrata: Establishes a connection using your saved credentials Queries the data source for all available resources: * Database: Tables and views in accessible schemas * API: Available endpoints and objects * File System: Files and sheets in accessible locations For each resource, Entegrata examines: * Field names and data types * Primary keys and constraints * Relationships and hierarchies * Metadata and annotations Compares current schema with previously discovered schema to identify: * New resources added * Resources removed * Schema changes (fields added, removed, or modified) Updates Entegrata's catalog with the latest resource information ## Discovery Schedule ### Automatic Discovery Discovery runs **automatically every 3 hours** for all active connections. Automatic discovery ensures your catalog stays current with source system changes without manual intervention. ### Manual Discovery You can trigger discovery immediately when you need to: * Check for new resources right after adding them to the source * Refresh schema after making changes in the source system * Troubleshoot collection issues that may be schema-related * Verify connectivity after updating credentials ## Discovery Results ### Last Discovered Timestamp The Resources page displays when discovery last ran: Last discovered timestamp This timestamp helps you know how current your catalog is. ### New Resources When discovery finds new resources in your data source, they automatically appear in the Resources list: * **Default state**: Inactive (not collecting) * **Default settings**: Inherit connection-level configuration * **Ready to configure**: Enable and configure collection settings as needed After discovery finds new resources, review and enable the ones you want to collect. New resources don't collect automatically. ### Removed Resources If discovery detects that a resource no longer exists in the source system: * Resource is marked as **Removed** or **Deleted** * Collection automatically stops * Historical data remains in your data lakehouse * Configuration is preserved in case the resource returns If a resource was accidentally deleted from the source and later restored, re-enabling it in Entegrata will resume collection with existing settings. Manually trigger discovery immediately when you need to Track discovery job status and schema changes # Schema Changes Source: https://support.entegrata.com/data/collector/discovery/schema-changes Handling breaking and non-breaking schema changes ## Overview Discovery detects schema changes and categorizes them as breaking or non-breaking, helping you respond appropriately. ## Schema Change Detection Discovery detects two types of schema changes: ### Non-Breaking Changes Changes that don't affect existing collection: * **New fields added** - Automatically included in future collections * **Field metadata updates** - Descriptions, annotations, etc. * **New indexes or constraints** - Improve query performance Non-breaking changes are applied automatically. No action required. ### Breaking Changes Changes that may disrupt collection or downstream analytics: * **Fields removed** - Data previously collected is no longer available * **Field data type changed** - May cause collection or transformation errors * **Primary key changed** - Affects uniqueness and deduplication Breaking changes require review and may need manual intervention to resolve. Breaking schema change alert ### Responding to Breaking Changes When discovery detects breaking changes: Click the notification to see exactly what changed: * Which fields were removed or modified * Old vs. new data types * Impact on existing collection configuration Schema change details modal Determine how the change affects: * **Collection**: Will filters or incremental fields still work? * **Unique keys**: Are primary keys still valid? * **Downstream analytics**: Will reports or dashboards break? Adjust resource settings to accommodate the change: * Update filters if filtered fields changed * Change incremental load field if it was removed * Update unique keys if they changed * Notify downstream consumers of schema changes Trigger a manual collection to verify the updated configuration works Watch the first few scheduled collections to ensure stability Understand how discovery works Manually trigger discovery immediately when you need to # Error & Retry Handling Source: https://support.entegrata.com/data/collector/jobs/error-handling How Entegrata automatically retries transient errors during collection ## Overview Collections run against external systems — databases, APIs, and file stores — that occasionally return temporary errors. To keep collections resilient, Entegrata automatically retries errors it recognizes as **transient** and fails fast on errors it recognizes as **permanent**. This happens behind the scenes, so most temporary glitches never result in a failed job. This page explains, at a high level, what gets retried and what happens when it doesn't. ## Transient vs. permanent errors When a collection step hits an error, Entegrata classifies it before deciding what to do. Temporary conditions that usually clear on their own: * Rate limiting (too many requests) * Service temporarily unavailable * Expired authentication (a re-login is attempted automatically) * Request timeouts and deadlines * Network, DNS, or TLS interruptions * Database deadlocks and short-lived consistency lags Conditions that won't improve by trying again: * Bad request or invalid configuration * Forbidden / insufficient permissions * Resource not found * Unrecoverable errors from the source system Classification can be tuned per connector, so a small number of data sources may treat specific errors differently from the defaults above. ## How retries work When an error is transient, Entegrata retries the operation automatically: * **Multiple attempts.** Collection steps retry up to several times before giving up. * **Exponential backoff with jitter.** The wait between attempts grows with each retry, and a small random offset is added so many concurrent retries don't hammer the source system in sync. * **Respecting the source system.** When a source returns a rate-limit response with a `Retry-After` hint, Entegrata honors that wait (up to a reasonable cap) instead of retrying immediately. Retries are applied at each stage of collection independently — fetching data from the source, and loading it into the Lakehouse each have their own retry handling. ## When retries are exhausted If an operation keeps failing after all retry attempts, the job — or the specific resource within it — is marked as **Failed**, and the error reason is recorded on the job so you can review it. Two outcomes are treated as **not** failures: * **Clean early stops.** When a source signals there's no more data to collect (for example, the last page of an API), the job finishes normally. * **Resizing.** If a job needs more capacity than its assigned worker, it is automatically re-queued onto a larger worker rather than failing. ## Partial failures Collection is tracked **per resource**. If a connection collects several resources and one of them fails, the others continue and report their own outcomes — a single failing resource does not abort the rest. Entegrata also validates row counts when loading data. If the number of records loaded into the Lakehouse doesn't match what was collected (after an automatic refresh retry), that resource is failed rather than reported as a partial success. This prevents a job from silently completing with incomplete data. A job that failed after retries will show its final error reason in the job detail view. Include the Job ID when contacting support so the execution can be traced. Understand job statuses and what they mean Diagnose and fix collection job issues View and filter collection jobs Understand collection jobs and performance monitoring # Job Statuses Source: https://support.entegrata.com/data/collector/jobs/job-statuses Understanding job statuses and what they mean ## Overview Jobs progress through different statuses during execution. Understanding these statuses helps you monitor collection health. ## Understanding Job Statuses ### Running Jobs **Indicators:** * Blue "Running" status badge * Records count may increase as the job progresses * Duration timer shows how long it's been running **What to watch:** * Normal jobs complete in minutes to hours depending on data volume * If a job runs much longer than usual, investigate for: * Source system performance issues * Network connectivity problems * Larger-than-expected data volume ### Completed Jobs **Indicators:** * Green "Completed" status badge * Total records collected displayed * Final duration and performance metrics **What to check:** * Record count matches expectations * Duration is reasonable for the data volume * No warnings or partial failures Completed job in jobs table ### Failed Jobs **Indicators:** * Red "Failed" status badge * Error reason displayed (hover or click for details) * May show partial record count if failure occurred mid-collection **Common failure reasons:** * Authentication errors (expired credentials) * Network timeouts or connectivity issues * Source system errors (database down, API unavailable) * Schema changes breaking filters or configuration * Insufficient permissions ### Scheduled Jobs **Indicators:** * Gray "Scheduled" status badge * Future scheduled time displayed * No execution metrics yet **What it means:** * Job is queued to run at the scheduled time * If a job remains scheduled past its time, check: * Connection or resource is active * No conflicting jobs blocking execution * System capacity is sufficient ## Viewing Job Details Get detailed information about a specific job. Find the job in the jobs table Click the Job ID or details icon to open the job detail view The job detail view shows: * **Execution timeline** - Start time, end time, duration * **Data metrics** - Records collected, processing speed * **Error details** - Full error messages and stack traces (for failed jobs) * **Configuration snapshot** - Collection settings at time of execution * **Resource information** - Which resource was collected Copy the Job ID when reporting issues to support. It uniquely identifies the execution for troubleshooting. View and filter collection jobs Understand collection jobs and performance monitoring Troubleshooting collection jobs # Overview Source: https://support.entegrata.com/data/collector/jobs/overview Understand collection jobs and performance monitoring ## Overview Jobs represent individual collection executions. Each time Entegrata collects data from a connection or resource, it creates a job that tracks the execution status, performance metrics, and results. ## Next Steps View and filter collection jobs Understand job statuses and what they mean Troubleshooting collection jobs # Troubleshooting Source: https://support.entegrata.com/data/collector/jobs/troubleshooting Resolve collection job failures and performance issues ## Overview Diagnose and fix common job issues including failures, timeouts, and performance problems. ### Job Fails Immediately **Symptoms**: Job transitions from Scheduled to Failed within seconds **Common causes:** * Connection credentials invalid or expired * Source system unreachable * Network firewall blocking access **Resolution:** 1. Check connection authentication settings 2. Update credentials if expired 3. Test connectivity to the source system 4. Review firewall rules ### Job Times Out **Symptoms**: Job runs for an extended period then fails with a timeout error **Common causes:** * Data volume larger than expected * Source system query performance issues * Network latency or intermittent connectivity **Resolution:** 1. Check source system performance and query execution time 2. Consider using filters to reduce data volume 3. Switch to incremental load if using full load 4. Contact support if timeouts persist despite optimization ### Job Collects No Records **Symptoms**: Job completes successfully but records = 0 **Common causes:** * Filters too restrictive, excluding all data * Incremental cursor field already up-to-date (no new records) * Resource is empty in the source system * Permissions limiting visible data **Resolution:** 1. Review and test filters 2. For incremental loads, check the last collected timestamp 3. Verify resource has data in the source system 4. Check connection user permissions ### Job Collects Fewer Records Than Expected **Symptoms**: Job completes but record count is lower than anticipated **Common causes:** * Incremental load working correctly (only new records) * Filters excluding some records * Source data deleted or archived * Permissions limiting access to some records **Resolution:** 1. For incremental loads, verify the incremental field is working correctly 2. Review filters and their impact 3. Check source system for data changes 4. Verify connection user can see all expected records ### Job Collects More Records Than Expected **Symptoms**: Job complects but record count is higher than anticipated, possibly duplicates **Common causes:** * Unique keys not properly configured * Incremental cursor reset or not working * Source data has duplicates * Schema change affected uniqueness logic **Resolution:** 1. Verify unique keys are correct and cover all uniqueness dimensions 2. Check incremental load field is updating correctly in source 3. Query source system directly to confirm duplicates exist there 4. Review recent schema changes ### Inconsistent Job Performance **Symptoms**: Job duration varies significantly between runs **Common causes:** * Source system load varies (peak vs. off-peak hours) * Data volume fluctuates * Network conditions change * Entegrata system capacity varies **Resolution:** 1. Schedule collections during source system off-peak hours 2. Monitor data volume trends to identify growth 3. Consider adjusting collection frequency for large or small datasets View and filter collection jobs Understand job statuses and what they mean Understand collection jobs and performance monitoring # View & Filter Jobs Source: https://support.entegrata.com/data/collector/jobs/viewing-filtering-jobs Search and filter jobs by time, status, and resource ## Overview Use filters to find specific jobs, analyze patterns, and troubleshoot issues efficiently. ## Viewing Collection Jobs Jobs can be viewed at two levels: ### Connection-Level Jobs View all collection jobs across all resources in a connection. Click on a connection from the Connections page Click the **Collection** or **Jobs** tab Collection tab showing all jobs for connection ### Resource-Level Jobs View collection jobs for a specific resource. Click on a resource name from the Resources page Click the **Collection** tab to see jobs for this resource only ## Job Information Each job displays key information: ### Job Status Jobs have one of the following statuses: * **Running** (Blue) - Collection is currently in progress * **Completed** (Green) - Collection finished successfully * **Failed** (Red) - Collection encountered an error and stopped * **Scheduled** (Gray) - Job is queued but hasn't started yet Job status badges ### Job Metrics For each job, you can see: * **Resource** (connection-level view only) - Which resource was collected * **Timestamp** - When the job started * **Records** - Total number of records collected * **Records/s** - Collection speed (records per second) * **Duration** - How long the collection took * **Job ID** - Unique identifier for troubleshooting Jobs table with metrics ## Collection Performance Chart The Collection page includes a visual chart showing collection trends over time. Collection performance chart ### Chart Features * **Time series view** - See collection volume and frequency over time * **Multiple resources** - Compare performance across resources (connection-level view) * **Zoom and pan** - Focus on specific time periods * **Hover details** - See exact values for any data point ### Interpreting the Chart The chart helps you: * **Identify trends** - Is data volume growing or shrinking? * **Spot anomalies** - Sudden spikes or drops in record counts * **Verify schedules** - Are collections running when expected? * **Compare resources** - Which resources collect the most data? Use the chart to validate that collections are running according to your schedule and processing expected data volumes. ## Filtering and Searching Jobs Refine the jobs list to find specific executions. ### Time Range Filters Filter jobs by when they ran: Click the filter button or use the filter controls above the jobs table Jobs filter panel Choose from preset ranges or specify custom dates: * Last 24 hours (default) * Last 7 days * Last 30 days * Custom date range The jobs table and chart update to show only jobs within the selected range ### Status Filters Filter jobs by their execution status: Job status filter checkboxes * Show/hide Running jobs * Show/hide Completed jobs * Show/hide Failed jobs * Show/hide Scheduled jobs To troubleshoot issues, filter to show only Failed jobs. ### Resource Filters (Connection-Level Only) When viewing connection-level jobs, filter by specific resources: Select one or more resources to see only their jobs. ### Combining Filters All filters work together. For example: * **Last 7 days** + **Failed status** + **Specific resource** = See all failures for that resource in the past week ### Chart vs. Table Filters By default, the chart shows up to 1000 recent jobs for visualization, while the table is paginated. Both respect your filter settings. **Toggle Chart Resources** For connections with many resources, you can limit the chart to show: * **Top 10 resources** (default) - Most active resources by record count * **All resources** - Toggle on to see every resource Toggle to show all resources in chart ## Job Pagination Jobs are displayed in pages for easy navigation. Choose how many jobs to show per page (10, 20, 50, or 100) Jobs per page selector Use the pagination controls at the bottom of the table to move between pages Jobs pagination controls ## Job Performance Metrics ### Records Collected Total number of records processed in the job. Use this to: * Verify expected data volume * Track data growth over time * Identify anomalies (sudden spikes or drops) ### Records Per Second Processing speed showing how fast data was collected. Use this to: * Benchmark normal performance * Identify slow collections * Compare performance across resources ### Duration Total time from job start to completion. Use this to: * Estimate future collection times * Schedule jobs to avoid conflicts * Identify performance degradation ## Best Practices for Monitoring Jobs Configure notifications for job failures to respond quickly to issues Check failed jobs regularly and resolve issues promptly Track job duration and record counts over time to spot degradation Unusual record counts or durations may indicate data quality issues Manually trigger a collection after changing settings to verify it works Focus on recent jobs when troubleshooting active issues ## Next Steps Understand collection jobs and performance monitoring Understand job statuses and what they mean Troubleshooting collection jobs # Collector Overview Source: https://support.entegrata.com/data/collector/overview Understand how Entegrata's data collection system works and how to manage your connections ## What is the Collector? The Collector is Entegrata's core data ingestion system that automatically connects to your data sources, discovers available data, and synchronizes it to your data lakehouse. The Collector manages the entire data pipeline from source systems to your analytics-ready tables. ## Key Concepts ### Connections A **Connection** represents Entegrata's link to a data source. Each connection includes: * **Authentication credentials** - Secure access to your data source * **Collection schedule** - How often data is synchronized * **Connection-level settings** - Default collection behaviors for all resources ### Resources A **Resource** is a table, view, API endpoint, or file within a connection that Entegrata collects. Each resource can have: * **Individual collection settings** - Override connection-level defaults * **Load type** - Full load or incremental synchronization * **Unique keys** - Fields that identify unique records * **Filters** - Rules to limit what data is collected ### Resource Hierarchy Some resources have nested sub-resources (child resources). For example: * A database table might have related child tables * An API endpoint might have nested data structures * A file might contain multiple sheets or sections Sub-resources inherit collection settings from their parent and cannot be scheduled independently. ### Jobs A **Job** represents a single collection execution for a connection or resource. Jobs track: * Execution status (Running, Completed, Failed, Scheduled) * Records collected and processing speed * Duration and performance metrics * Error details if the job failed ### Discovery **Discovery** is the automated process where Entegrata: 1. Connects to your data source 2. Scans for available resources (tables, views, endpoints, files) 3. Analyzes schema and structure 4. Detects changes to existing resources Discovery runs automatically every 3 hours and can also be triggered manually. ## Collection Lifecycle Add a new connection by providing authentication credentials and connection details Entegrata automatically discovers all available resources in your data source Set up collection schedules, load types, and filters for your resources Data is automatically synchronized according to your configured schedules Track job status, performance, and data quality through the admin portal ## Collection Schedules Entegrata supports two scheduling approaches: ### Interval-Based Scheduling Collections run on a regular time interval (e.g., every 6 hours, daily at 2 AM). This is the most common approach for routine data synchronization. ## Load Types ### Full Load Copies all data from the source every time. Use when: * Source data is small * Historical changes aren't tracked * You need a complete snapshot each time ### Incremental Load Copies only new or changed data since the last collection. Requires: * An **incremental load field** (like `modified_date` or `updated_at`). An auto-incrementing integer ID also works, but it only picks up new records, not changes to existing ones — see [Incremental Load Field](/data/collector/resources/collection-settings#incremental-load-field) * The source system tracks when records change Incremental loads are faster and more efficient for large datasets. ### Time-Window Incremental Load Works like an Incremental Load, but re-scans a bounded **time window** on the cursor field every run instead of picking up where the last run left off. The window can be: * **Relative** — e.g., the last 3 months, sliding forward with each run * **Absolute** — a fixed start and end date (useful for backfills of a specific period) Use when the cursor field exists but isn't fully reliable — for example, when `modified_date` doesn't always update on every change. Re-scanning the window each run catches records whose cursor was set late or not at all, without the cost of a Full Load. ## Collection Status Connections and resources can be: * **Active** - Currently collecting data according to schedule * **Inactive** - Paused, not collecting data You can toggle status to temporarily stop collection without deleting configuration. ## Getting Started Learn how to add, update, and delete data source connections Configure resources, set schedules, and manage collection settings Track collection progress and troubleshoot issues Understand how Entegrata discovers and tracks your data # Collection Settings Source: https://support.entegrata.com/data/collector/resources/collection-settings Configure schedules, load types, unique keys, and filters for resources ## Overview Resource collection settings control how and when individual resources collect data. These settings can override connection-level defaults, allowing fine-grained control over each table, view, or endpoint. Click the **Config** button (three dots) in the last column for the resource Select **Collection Settings** from the dropdown menu The Collection Settings modal has multiple tabs: ### Schedule Tab **Interval-Based Schedule** Set a specific collection interval: * Time-based intervals (e.g., "Every 6 hours") * Cron expressions for custom schedules * Specific times (e.g., "Daily at 3:00 AM") Resource interval schedule configuration **Timing (Static vs Rolling)** For sub-daily intervals, choose **Rolling** (default — the next run starts one interval after the previous collection finishes) or **Static** (runs at a consistent clock time anchored to a start time you set). See [Timing: Static vs Rolling](/data/collector/connections/collection-settings#timing-static-vs-rolling) for details. **Active Status Toggle** Enable or disable collection for this resource ### Load Type Choose how data is collected: **Full Load** * Copies all data every time * Best for small datasets or when you need complete snapshots * No special source requirements **Incremental Load** * Copies only new or changed data * Requires an incremental load field, also called a cursor field (like `modified_date` or `updated_at`) * Much faster for large datasets **Time-Window Incremental Load** * Like Incremental Load, but re-scans a bounded time window on the cursor field every run instead of picking up where the last run left off * Best when the cursor field exists but isn't fully reliable (e.g., `modified_date` doesn't always update) — re-scanning the window catches records the cursor would otherwise miss * Requires both a cursor field and a configured time window (see below) **Change Data Load** (SQL sources only) * Detects and collects only records that have truly changed by comparing source and Entegrata * Used when a table does not have a reliable cursor field for incremental loads but has a stable date field (e.g., `created_date`, `transaction_date`) * More expensive than Incremental Load — choose Incremental whenever a reliable cursor field is available * See [Sync Operation](/data/collector/resources/sync-operations) for a full explanation Load type selector ### Incremental Load Field If you select Incremental or Time-Window Incremental load type, choose the field that tracks which records are new or changed, such as a last-modified timestamp or an ever-increasing ID: Incremental load field selector The incremental load field must: * Exist in the source data * Increase whenever a record is added or changed * Be a date, datetime, timestamp, or integer type (integer, small integer, or big integer) Integer fields require platform v1.3.144+ and are compared by numeric value, so `10` comes after `9`. They suit Incremental Load on sources with an auto-incrementing ID or row version. An ID only increases when a record is added, so it doesn't pick up updates to existing records. Float and double fields can't be used as the incremental load field. They store values with limited precision, so a stored value can differ slightly from the source value, and comparisons against the last collected value can't be guaranteed to be accurate. Records could be skipped or collected twice. Time-Window Incremental and Change Data Load need a date, datetime, or timestamp field, since their window is a date range. ### Time Window If you select Time-Window Incremental load type, configure the window that bounds each collection run: **Relative window** A sliding window anchored to "now", such as the last 3 months or last 30 days. The window moves forward with each run, so collection always covers the most recent period. **Absolute window** A fixed start and end date (e.g., `2025-01-01` to `2025-06-30`). The same range is scanned on every run — useful for backfilling a specific historical period. Use a relative window for ongoing collection of recent activity; use an absolute window when you're targeting a defined past period. ### Unique Key(s) Specify which field(s) uniquely identify each record: Unique keys selector **Source-Managed Keys** If the source system manages primary keys (e.g., database tables with PRIMARY KEY constraints), Entegrata uses those automatically and they cannot be changed. You'll see: "Primary keys are managed by the source and cannot be changed" **User-Defined Keys** For sources without explicit primary keys (e.g., views, API endpoints, files), select one or more fields that uniquely identify records: * Single field (e.g., `id`, `email`) * Multiple fields (composite key, e.g., `customer_id` + `order_date`) Unique keys are used to: * Detect duplicate records * Merge updates in incremental loads * Ensure data quality Add filters to limit what data is collected from this resource. ### Creating Filters 1. Select a field from the resource 2. Choose an operator (equals, greater than, contains, etc.) 3. Provide a value to filter on 4. Combine multiple filters with AND/OR logic ### Example Filters * Collect only active records: `status = 'active'` * Collect recent data: `created_date >= '2024-01-01'` * Exclude test data: `email NOT LIKE '%@test.com'` Filters are applied at the source before data is transferred, reducing collection time and storage costs. Click **Save** to apply the resource collection settings Turn on data collection for specific resources Manually start collection for testing Track collection execution and performance Resolve resource collection issues # Enable Resources Source: https://support.entegrata.com/data/collector/resources/enabling-resources Turn on data collection for specific resources ## Overview Enabling resources activates data collection for specific tables, views, or endpoints. Only enabled resources collect data according to their configured schedules. New resources discovered automatically default to **Inactive** status. You must explicitly enable them to begin collection. ## How Resource Enabling Works When you enable a resource: 1. The resource toggle turns ON (green) 2. Changes are staged (not yet saved) 3. Click **Save** to apply and begin collection 4. The resource starts collecting according to its schedule ## Enabling Single Resources Find the resource in the Resources table Click the toggle switch in the first column The toggle turns green (ON) Click the **Save** button in the floating save bar Changes are not applied until you click Save. You can toggle multiple resources before saving. The resource is now active and will collect on its next scheduled time ## Enabling Multiple Resources You can enable several resources at once before saving: Click the toggle for each resource you want to enable The floating save bar shows how many resources have pending changes Check the save bar to confirm the count Click **Save** to activate all toggled resources at once Batch enabling is efficient when setting up a new connection - enable all needed resources, then save once. ## Bulk Import Resources When a connection has many resources, toggling each one is slow. **Bulk Import Resources** enables many resources at once — either from a curated **preset** or from a **CSV file** you upload. To open it, go to the **Resources** page and select **Bulk Import Resources** from the actions menu (next to **Refresh Source Schema**). Bulk Import Resources dialog showing preset and CSV options Bulk import enables resources that have already been discovered on the connection. If a resource you expect isn't listed, run **Refresh Source Schema** first, then import. Presets are curated bundles of the resources most integrations need, offered for supported data sources (for example, 3E, Aderant, and Iridium). If presets exist for the connection's data source, the **From a preset** tab is selected by default. Choose one or more presets. Every resource in a selected preset is checked by default. Expand a preset to see its resources and uncheck any you don't want to enable. Click **Import**. The selected resources are enabled immediately. If a data source has no presets defined, only the CSV option is shown. Upload a CSV that lists the resources to enable — useful when you have a known set of tables to turn on. Select the **From CSV** tab (it's the only option when no presets are available). Click **Download CSV template** to get a correctly formatted starter file. Add a header column named `tables` (the headers `table` and `table_names` also work, in any case), with one resource name per row: ```csv template.csv theme={null} tables Matter Client Timecard ``` Select your file. The dialog lists the resources it found, each checked by default. Uncheck any you don't want. Click **Import** to enable the selected resources. If the file can't be read, or has no `tables` / `table` / `table_names` column, an error message explains what to fix. Resource names that don't match a discovered resource are skipped — the rest are still enabled. Unlike single and multiple toggles, bulk import applies as soon as you click **Import** — there's no separate **Save** step. Imported resources begin collecting on their next scheduled time (or trigger a manual collection to test). ## Disabling Resources The same process works in reverse to disable collection: Click the blue toggle to turn it gray (OFF) Click **Save** in the floating bar The resource stops collecting data * Scheduled jobs are canceled * Existing data in lakehouse is preserved Disabling a resource stops all future collections. Historical data remains but no new data is collected. ## Canceling Changes To discard toggled changes before saving: Click **Cancel** in the floating save bar All toggles return to their saved state ## Access Control Requirements Some resources cannot be enabled without access control configuration: **Indicators**: * Lock icon next to resource name * Toggle is disabled (grayed out) * Tooltip explains access control requirement **Resolution**: 1. Configure access control for the resource 2. See [Access Control](/data/access_control/overview) documentation 3. Return to enable the resource once access control is set Access control ensures only authorized users can query this resource's data in your lakehouse. ## Connection-Level Impact Remember that connection status affects all resources: | Connection Status | Resource Toggle ON | Will Resource Collect? | | - | - | - | | Active | Yes | ✅ Yes | | Active | No | ❌ No | | Inactive | Yes | ❌ No - Connection paused | | Inactive | No | ❌ No | Even if a resource is enabled, it won't collect if the connection is set to Inactive. ## First Collection After Enabling When you first enable a resource: Collection starts at the next scheduled interval * Connection schedule (if resource uses default) * Resource schedule (if custom override configured) Don't want to wait? Trigger a manual collection: * Click on the resource name * Click **Start a Collection** * See [Triggering Collection](/data/collector/resources/triggering-collection) Watch the first collection to verify: * Authentication works * Data is accessible * No configuration errors After enabling critical resources, trigger a manual collection to verify they work immediately rather than waiting for the schedule. ## Common Workflows ### Initial Setup: Enable Core Tables After creating a connection: Browse the discovered resources list Determine which tables/endpoints contain needed data Toggle ON the essential resources first (customers, products, transactions) Save changes and trigger manual collection to test After core resources work, enable supplementary data Setting up a large connection? Use [Bulk Import Resources](#bulk-import-resources) to enable a preset or a CSV list of tables in one step instead of toggling each resource. ### Temporary Pause: Disable During Maintenance To temporarily stop collection: Toggle OFF resources undergoing maintenance Apply the changes Complete source system work Toggle back ON and save For connection-wide pauses, set the connection to Inactive instead of disabling individual resources. ## Best Practices Enable a few critical resources first, verify they work, then expand Enable and test in a development connection before production Keep notes on why certain resources are enabled or disabled Regularly review enabled resources to ensure they're still needed ## Troubleshooting ### Toggle Won't Turn On **Symptoms**: Click toggle but it doesn't change **Possible causes**: * Access control required * Network issue * Browser cache problem **Resolution**: 1. Check for lock icon (access control required) 2. Refresh the page and try again 3. Check browser console for errors ### Changes Not Saving **Symptoms**: Click Save but resources don't activate **Possible causes**: * Network interruption * Server error * Validation failure **Resolution**: 1. Check for error messages 2. Refresh page and check resource status 3. Try enabling fewer resources at once 4. Contact support if issue persists ### Resource Enabled But Not Collecting **Symptoms**: Resource is active but no jobs appear **Possible causes**: * Connection is inactive * Schedule hasn't triggered yet **Resolution**: 1. Verify connection status is Active 2. Check schedule timing 3. Trigger manual collection to test 4. See [Troubleshooting](/data/collector/resources/troubleshooting) ## Next Steps Configure schedules and load types for enabled resources Manually start collection for testing Track collection execution and performance Resolve resource collection issues # Overview Source: https://support.entegrata.com/data/collector/resources/overview Understand resources and how they represent collectible data within connections ## Overview Resources are the individual tables, views, API endpoints, or files within a connection that Entegrata collects. Each resource represents a specific dataset you want to synchronize to your data lakehouse. ## Key Concepts ### What is a Resource? A **Resource** is a collectible data entity discovered within a connection: The resource identifier (table name, view name, API endpoint path, or file name) The kind of resource: * **Table** - Database table with data rows * **View** - Database view (virtual table) * **Endpoint** - API endpoint providing data * **File** - Data file or spreadsheet Collection state: * **Active** - Resource is collecting data * **Inactive** - Resource is not collecting Columns or properties within the resource (e.g., customer\_id, name, email) ## Resource Lifecycle Resources are automatically discovered when you create a connection or when discovery runs (every 3 hours) Enable resources and configure collection settings (schedule, load type, filters) Enabled resources collect data according to their configured schedules Track collection jobs, performance, and data quality Update settings, handle schema changes, or disable resources as needed ## Resource Types ### Tables Database tables containing rows of structured data: * Most common resource type * Have defined schemas with columns and data types * Support full and incremental loads * May have primary keys defined by the database ### Views Database views (virtual tables based on queries): * Derived from one or more tables * Schema defined by the view query * Typically use full load (incremental may not be supported) * Primary keys must be user-defined ### Endpoints API endpoints providing structured data: * REST API paths or GraphQL queries * Schema discovered from API responses * Support pagination and filtering * May have rate limits ### Files Files containing structured data: * Excel spreadsheets, CSV files, JSON documents * Schema inferred from file structure * May contain multiple sheets or sections * Typically use full load ## Resource vs. Connection Understanding the relationship: | Aspect | Connection | Resource | | - | - | - | | **Scope** | Entire data source | Single table/view/endpoint | | **Discovery** | Discovers all resources | Discovered within connection | | **Schedule** | Default for all resources | Can override connection default | | **Enable/Disable** | Affects all resources | Affects only this resource | | **Authentication** | Stores credentials | Inherits from connection | | **Collection** | May trigger all resources | Triggers single resource | ## Collection Settings Each resource can be configured with: ### supp ## Resource Status Indicators Resources display status information: ### Active Resource Active resource with toggle on * Toggle is ON (green) * Collects according to schedule * Jobs appear in collection history ### Inactive Resource Inactive resource with toggle off * Toggle is OFF (gray) * Does not collect data * No jobs created ## Load Type Indicators Resources show their configured load type: Load type badges * **Full** - Full load every collection * **Incremental** - Incremental load with change tracking Click the badge to quickly open collection settings. ## Getting Started with Resources Browse and search discovered resources Turn on data collection for specific resources Configure schedules, load types, and filters Manually start collection jobs ## Common Workflows ### Initial Setup After Creating Connection Allow automatic discovery to complete (1-5 minutes) Browse the discovered resources in the Resources table Toggle on the specific tables/endpoints you need Set load types, unique keys, and filters for each resource Save changes and watch the first collection jobs ### Ongoing Maintenance * **Monitor collection jobs** - Check for failures or performance issues * **Handle schema changes** - Respond to discovery alerts about schema modifications * **Adjust schedules** - Optimize timing based on data freshness needs * **Update filters** - Refine data selection criteria ## Best Practices Enable a few resources first, verify they work, then add more Incremental loads are faster and more efficient for large tables Always define unique keys to ensure proper deduplication Filter at the source to reduce data transfer and storage Watch initial jobs to catch configuration issues early Keep notes on why certain resources are enabled/disabled ## Next Steps Learn how to browse and search resources Understand how resources are discovered Track collection execution and performance # Sync Operation Source: https://support.entegrata.com/data/collector/resources/sync-operations Detect and pull only records that have changed between your source and Entegrata ## Overview The **Sync Operation** is a specialized collection mode that finds records whose data has drifted between the source system and Entegrata, and pulls only those changes. Unlike a full refresh (which copies everything) or an incremental load (which relies on a cursor date), sync compares the actual contents of records on both sides and only collects what's different. Sync is primarily designed for tables that **do not have a reliable cursor field** (such as `LAST_MODIFIED` or `UPDATED_AT`) to support incremental loads. When a table lacks such a field, incremental loads aren't possible — and running repeated full refreshes can be prohibitively expensive. Sync fills this gap by detecting changes through content comparison instead of a timestamp cursor. Sync operates in two modes — Full Sync and Change Data Load — with the right choice depending on whether your table has a reliable date field. Sync is a **much heavier operation** than a regular Full Load or Incremental Load. Only use sync when a table does not have a reliable cursor field for incremental loading. If your table has a dependable `LAST_MODIFIED` or `UPDATED_AT` field, an Incremental Load is almost always the better choice. Sync is currently only available for **SQL data sources**. Most APIs do not expose the kind of row-level checksum operations needed to perform sync efficiently, so sync is not available for API-based sources. ## How Sync Works At a high level, sync compares "fingerprints" of records between the source and Entegrata, then only pulls the records whose fingerprints don't match. Both the source and Entegrata partition records into buckets. Each bucket holds a subset of rows — for example, one bucket per day, or one bucket per range of primary keys. For each bucket, Entegrata computes a fingerprint (a combined hash of all record data) on both sides. Buckets with matching fingerprints are skipped — the data is already in sync. When a bucket's fingerprint differs, sync progressively narrows down to find the smallest subset of records where the difference lives (e.g., Year → Month → Day). The records in mismatched buckets are pulled from the source. Entegrata compares each collected record against the version already stored. Only records that are truly new or changed are written — unchanged records in the same bucket are discarded. Sync never deletes records. If a record has been removed from the source, use the separate Delete Operation to detect and handle it. ## Sync Modes Sync operates in two different modes depending on how it's scheduled and whether the resource has a date field available. ### Full Sync (Standalone Operation) Full Sync is a **standalone, on-demand operation**. It runs separately from your normal collection schedule and is typically triggered manually or on a less-frequent cadence (e.g., weekly). Full Sync uses the resource's primary key (or unique identifier) to group records into buckets and compares the entire table across the source and Entegrata. **Best for:** * Tables without a reliable cursor field AND without a reliable date field * Periodic verification that a full table is in sync * Reconciling after an outage or incident Full Sync uses a hash of the primary key to assign each record to a bucket. This means a record always falls in the same bucket, regardless of how its values change — keeping the comparison stable. ### Change Data Load (Scheduled Load Type) Change Data Load is a **load type** (selected on the resource's Collection Settings page, alongside Full Load and Incremental Load). Unlike standalone sync, Change Data Load runs on the resource's normal collection schedule — every time the resource collects, it performs a sync instead of a traditional full or incremental load. Change Data Load requires a date field on the resource (e.g., `created_date`, `transaction_date`) and groups records by Year → Month → Day. This is the most efficient sync mode because changes in real-world data tend to cluster around recent dates. **Best for:** * Tables without a reliable cursor field for incremental loading, but WITH a stable date field * Large historical tables where most records are older and unlikely to change * Scheduled catch-up on drift (configured once, runs automatically) **How it narrows down:** | Level | What it compares | | - | - | | Year | All records in each year | | Month | Within mismatched years, compares each month | | Day | Within mismatched months, compares each day | If years 2020–2023 match between source and Entegrata, those entire years are skipped — no month or day level work is needed for them. This makes Change Data Load extremely efficient for large historical tables. To use Change Data Load, open the resource's Collection Settings, choose **Change Data Load** as the load type, and select a date field. See [Collection Settings](/data/collector/resources/collection-settings) for details. ## When To Use Sync Sync is designed for tables where **Incremental Load is not viable** because the table lacks a reliable cursor field. ### Decision guide | Table has a reliable cursor field (`UPDATED_AT`, `LAST_MODIFIED`)? | Has a reliable date field (e.g., `created_date`)? | Recommended approach | | - | - | - | | Yes | — | **Incremental Load** (sync is unnecessary) | | Partially — cursor exists but isn't always updated | — | **Time-Window Incremental Load** (re-scans a bounded window each run to catch missed updates) | | No | Yes | **Change Data Load** (scheduled sync with date-based narrowing) | | No | No | **Full Sync** (standalone, run on-demand) + Full Load on a less frequent cadence | ### Typical scenarios When a SQL table has no `UPDATED_AT` or `LAST_MODIFIED` field, use Change Data Load or Full Sync instead of repeated full refreshes. Run Full Sync after a source system outage or Entegrata downtime to reconcile any gaps. Use Change Data Load on large tables with a stable date field — only recent partitions get compared in depth. Run Full Sync before important downstream processes to verify the data is fully in sync. ### When NOT to use sync * **Tables with a reliable cursor field** — use Incremental Load instead; it's dramatically lighter * **Small tables** — a Full Load is simpler and not much more expensive * **Tables that change constantly across all records** — most buckets will be flagged, and sync provides little benefit over a full refresh * **API-based data sources** — sync is only available for SQL data sources ## What Sync Does vs. Doesn't Do Sync finds records whose values differ between source and Entegrata Records that exist in the source but not in Entegrata are collected Sync compares every column — not just the cursor field — so it catches updates that incremental loads would miss Unchanged records that happen to share a bucket with changed ones are discarded before writing Records removed from the source require the separate Delete Operation Sync runs in addition to Full Load or Incremental Load — it does not replace them ## Performance Characteristics Sync is designed to be efficient on large tables by comparing fingerprints rather than pulling data upfront. | Table size | What sync does | | - | - | | Few changes | Only a handful of buckets are flagged; fast | | Many changes | More buckets are flagged; sync may approach the cost of a full refresh | | No changes | Sync completes quickly after the first comparison round with zero data pulled | For the best performance, use Change Data Load sync on tables with a date field. The Year → Month → Day narrowing means only the parts of the table with actual changes are examined closely — unchanged years are skipped entirely. ## Limitations A few data types and scenarios are not fully supported by sync: * **Binary columns** (`VARBINARY`, `BINARY`, `IMAGE`) are excluded from the comparison * **Non-ASCII Unicode text** in certain string columns may produce false positives; this is not an issue for data containing only standard characters * **Extremely large tables with widespread changes** — if most of the table has drifted, sync will flag most buckets and the operation becomes similar in cost to a full refresh ## Related Topics Configure load types, schedules, and filters for resources Manually run collection jobs on demand Track sync execution and performance Resolve resource collection issues # Trigger Collection Source: https://support.entegrata.com/data/collector/resources/triggering-collection Manually start collection jobs for resources ## Overview Trigger manual collection jobs to collect data immediately without waiting for the scheduled time. Useful for testing, recovering from failures, or urgent data refreshes. Go to the resource detail page or collection page Click the **Start a Collection** button in the top-right corner For resources with incremental load type, you can choose: * **Incremental Collection** - Collect only new/changed data (default) * **Full Refresh** - Collect all data, ignoring the incremental field Use Full Refresh to recover from data issues or when you suspect incremental collection missed records. The collection job starts immediately. Monitor progress in the Jobs section # Troubleshooting Source: https://support.entegrata.com/data/collector/resources/troubleshooting Resolve common resource collection issues ## Overview This guide helps you diagnose and resolve common issues with resource collection, configuration, and discovery. ### Resource Not Discovered If a resource doesn't appear after discovery: 1. Verify the resource exists in the source system 2. Check that the Service Account has READ permissions 3. Ensure the resource is in a schema/database the connection can access 4. Trigger a manual discovery to force a refresh ### Collection Failing If a resource collection fails: 1. Check the job error message in the Jobs tab 2. Verify source system connectivity and permissions 3. Review filters - overly restrictive filters may cause issues 4. Check for schema changes that broke the collection ### Incremental Load Not Working If incremental loads are collecting more data than expected: 1. Verify the incremental load field is updating when records change 2. Check the field data type is compatible (date/datetime/timestamp) 3. Review the last collected timestamp in job details 4. Consider using a Full Refresh to reset the incremental cursor ## Best Practices For large datasets, use incremental loads to improve performance and reduce costs Always define unique keys to ensure data quality and proper deduplication Filter data at the source to reduce transfer time and storage costs After changing schedules, monitor the first few collections to ensure timing is correct # Versioned History Source: https://support.entegrata.com/data/collector/resources/versioned-history Understand how Entegrata tracks changes to your data over time using versioned history ## What is Versioned History? Versioned history is a load strategy that preserves a complete history of changes to your data over time. Instead of overwriting records when they change, Entegrata keeps every version so you can see what a record looked like at any point. This strategy is based on a data warehousing pattern known as **Slowly Changing Dimension Type 2 (SCD2)**, which is the industry standard for tracking historical changes. Versioned history is automatically enabled when a resource is configured with **Full Load** and has **primary keys** defined. ## Variants Entegrata offers two variants of versioned history. Both keep a full version history of changes to existing records — they differ only in how they treat records that disappear from the source: | Variant | Tracks deletions? | Behavior when a record disappears from the source | | - | - | - | | **Versioned history** | Yes | Record is soft-deleted (`__sys_deleted_at` is set) and excluded from the current table | | **Versioned history (no deletes)** | No | Record is left untouched — its latest version remains active in both the main and current tables | Use **Versioned history (no deletes)** when the source system reuses, archives, or filters records in ways that would otherwise produce false deletions (for example, sources that only return "active" records, or APIs whose result set varies between collections). ## How It Works At a high level, each time data is collected: Entegrata collects a complete snapshot of the resource from your source system Each record is compared against its previous version using a fingerprint (a hash of all business data fields). If the fingerprint matches, the record hasn't changed Only records that have actually changed get a new version stored. Unchanged records are not duplicated Records that no longer appear in the source are soft-deleted — they are marked with a deletion timestamp rather than being permanently removed This step is skipped when the **Versioned history (no deletes)** variant is used. Records that disappear from the source remain active. A **current** table is rebuilt containing only the latest active version of each record, giving you a clean, deduplicated view of your data ## Two Tables: Main and Current When versioned history is enabled, your data is available in two tables: ### Main Table (Full History) The **main** table contains every version of every record that has ever been collected. This is your complete audit trail. | Ingestion Timestamp | Deleted At | Key | Name | Status | | - | - | - | - | - | | 2025-01-01 | — | A | Acme Corp | Active | | 2025-02-01 | — | A | Acme Corp | Inactive | | 2025-01-01 | — | B | Beta LLC | Active | | 2025-01-01 | 2025-03-01 | C | Gamma Inc | Active | In this example: * **Record A** changed between January and February (status went from Active to Inactive), so both versions are kept * **Record B** has never changed — only one version exists * **Record C** was removed from the source in March — it is soft-deleted rather than removed ### Current Table (Latest Only) The **current** table contains only the most recent active version of each record. This is the table you should use for most reporting and analytics. | Key | Name | Status | | - | - | - | | A | Acme Corp | Inactive | | B | Beta LLC | Active | Use the **current** table for day-to-day reporting and analytics. Use the **main** table when you need to analyze how data changed over time or perform historical audits. ## System Columns Versioned history relies on several system columns (prefixed with `__sys_`) to track record metadata. The `__sys_ingest_ts` and `__sys_deleted_at` columns are present on all collected resources regardless of load strategy, while `__sys_fingerprint` is specific to versioned history: The timestamp when this version of the record was collected. Use this to determine when a change was detected. The timestamp when the record was soft-deleted (removed from the source). A value of `NULL` means the record is still active. A hash of all business data fields, used for change detection. Two records with the same fingerprint have identical data. ## Change Detection Entegrata uses **fingerprinting** to efficiently detect which records have changed between collections. Here's how it works: 1. For each incoming record, a fingerprint is calculated by hashing all business data fields (excluding system columns) 2. The new fingerprint is compared against the fingerprint of the previous version of that record 3. If the fingerprints match, the record hasn't changed and the new copy is discarded 4. If the fingerprints differ, the record has changed and a new version is stored This means your main table only grows when data actually changes — not every time a collection runs. Fingerprints are calculated from all business data columns, sorted alphabetically for consistency. Even a single character change in any field produces a different fingerprint. ## Soft Deletes When a record that previously existed in the source no longer appears in a full load collection, Entegrata does **not** permanently delete it. Instead, the record is **soft-deleted** by setting the `__sys_deleted_at` column to the current timestamp. This means: * The historical record and all its versions remain in the **main** table for auditing * The record is excluded from the **current** table (only active records appear there) * You can query the main table to find when and which records were deleted ### Disabling Soft Deletes When the **Versioned history (no deletes)** variant is selected, Entegrata does not track deletions: * `__sys_deleted_at` is never set as a result of a record disappearing from the source * A record's most recent version stays in the **current** table indefinitely, even if the source stops returning it * Updates to records that *do* still appear in the source are versioned exactly as they are with standard versioned history With this variant you cannot rely on the main table to detect when records were removed from the source — the information is not captured. Choose this variant only when source-side absences are not meaningful deletions. ## Schema Changes When the data type of a field changes in your source system (for example, a field changing from text to a number), Entegrata handles this automatically: * **Non-breaking changes** (e.g., a small number type widening to a larger number type) are handled seamlessly — existing data and version history are preserved as-is * **Breaking changes** (e.g., text to number) trigger a schema migration where existing data is re-cast to the new type. Values that cannot be converted become `NULL`. A new version is created for all records at this point, effectively creating a "reset point" in the version history After a breaking schema change, existing historical versions lose their fingerprints. This means normal change detection resumes from that point forward, but older versions are preserved as permanent historical records. ## When to Use Versioned History Versioned history is ideal when you need to: * **Track changes over time** — See how records evolved (e.g., status changes, name updates) * **Perform historical audits** — Answer questions like "What did this record look like last month?" * **Detect deleted records** — Know when records were removed from the source system * **Comply with data retention requirements** — Maintain a full audit trail of data changes ## Requirements For versioned history to be active on a resource, the following conditions must be met: | Requirement | Details | | - | - | | **Load strategy** | Must be set to **Versioned History** or **Versioned History (no deletes)** | | **Load type** | Must be **Full Load** (incremental loads do not support versioned history) | | **Primary keys** | The resource must have primary keys defined (either source-managed or user-defined) | Primary keys are essential because they allow Entegrata to match records across collections and determine which records have changed, been added, or been removed. ## Best Practices Ensure your primary keys truly identify unique records. Incorrect keys lead to incorrect version tracking. The current table is optimized for queries — use it for dashboards and reports unless you specifically need historical versions. When querying the main table, filter by `__sys_ingest_ts` to view data as it existed at a specific point. Versioned history tables grow over time as changes accumulate. Monitor storage usage for high-change resources. ## Related Pages Configure load types and unique keys for resources Browse and search your discovered resources Learn how Entegrata handles schema evolution Track collection execution and performance # View Resources Source: https://support.entegrata.com/data/collector/resources/viewing-resources Browse, search, and filter resources within your connections ## Overview The Resources page displays all discovered tables, views, and endpoints within a connection. Use this page to browse available data, check collection status, and access configuration options. ## Accessing Resources From the Connections page, click on a connection The Resources page displays automatically Resources page for a connection ## Resources Table The table shows all discovered resources with key information: Resources table with multiple resources ### Column Information Enable/disable collection for this resource (first column) Resource name with type icon (Table, View, or Endpoint) Current load configuration (Full or Incremental) Fields that uniquely identify records When resource was last collected and the job status Quick access to collection settings (gear icon with three dots) ## Resource Type Icons Resources are visually identified by icons: * **Table icon** - Database table * **View icon** - Database view * **Endpoint icon** - API endpoint ## Searching Resources Find specific resources using the search bar. Type any part of the resource name in the search field Resource search bar The table updates to show only matching resources Search is case-insensitive and matches partial names. Try searching for common patterns like "customer", "order", or "trans". ## Filtering Resources Use filters to narrow down the resources list. ### Filter by Status Status filter dropdown * **All Resources** - Show everything * **Active Only** - Show enabled resources * **Inactive Only** - Show disabled resources ### Filter by Type Type filter dropdown * **All Types** - Show all resource types * **Tables** - Show only database tables * **Views** - Show only database views * **Endpoints** - Show only API endpoints ### Filter by Load Type Load type filter * **All Load Types** * **Full Load Only** * **Incremental Only** ## Pagination For connections with many resources: Resource pagination controls * View page information (e.g., "Page 1 of 10") * Navigate pages with Previous/Next buttons ## Last Discovery Timestamp See when resources were last discovered: Last discovered timestamp This indicates how current your resource catalog is. Discovery runs automatically every 3 hours. ## Clicking on Resources Click a resource name to view detailed information. Click anywhere on the resource name (blue link) Clickable resource name Navigate to the resource detail page showing: * All fields/columns * Field data types and metadata * Collection jobs for this resource * Resource-specific collection settings ## Load Type Badges Resources display their configured load type as a badge: Load type badges in table * **Full** - Copies all data every collection * **Incremental** - Copies only changed data Click the load type badge to quickly open collection settings and change it. ## Unique Keys Display The Unique Key(s) column shows which fields identify unique records: Unique keys in table * Single field: "id" * Multiple fields: "customer\_id, order\_date" Click the unique keys cell to quickly open collection settings and configure them. ## Last Collection Status See when each resource was last collected and whether it succeeded: Last collection with status * **Green checkmark** - Last collection succeeded * **Red X** - Last collection failed Click the timestamp to navigate to the connection's collection jobs page. ## Empty State If no resources are discovered: **Possible reasons**: * Discovery is still running * Connection has no accessible resources * User lacks permissions to see resources * Connection parameters are incorrect **Resolution**: 1. Wait a few minutes for discovery to complete 2. Trigger manual discovery 3. Check connection credentials and permissions 4. Verify database/schema names in connection settings ## Best Practices When you have hundreds of resources, search is faster than scrolling Focus on active resources when monitoring, or inactive when looking for new data to enable If you don't see expected resources, check when discovery last ran Click resource names to see full field lists and collection history ## Next Steps Turn on collection for resources Configure schedules, load types, and filters Learn how resources are discovered Resolve resource issues # Comparing Pipelines Source: https://support.entegrata.com/data/pipeline/comparing-pipelines Use Diff to see exactly what differs between two pipelines or two entities without changing anything ## Overview **Diff** answers one question: what's actually different between these two pipelines right now? Both Diff entry points are completely **read-only**. Nothing is written, nothing is published, and the dialog says so: > "Compares this entity's mappings against the target pipeline — nothing is changed." Diff uses the same comparison engine and the same walkthrough as the [Merge wizard](/data/pipeline/reviewing-sync-changes) — just with no decisions and no commit. What you see in a Diff is what a sync would act on. ## Comparing a Single Entity Open a pipeline, open a layer (for example **Silver**), and click the three-dot menu (⋮) on an entity row. Click **Diff** to open the comparison dialog for that entity. Choose the target pipeline and click **View diff**. ## Comparing Two Pipelines From the pipelines list, click the three-dot menu (⋮) on the pipeline you want to compare **from** and hover **Manage**. Click **Diff** to open the comparison dialog. Choose the pipeline to compare against. Click **View Diff**. You step through every differing entity, exactly like the Merge wizard, but with no toggles and no commit button. Entities that are identical on both sides are skipped, so everything you see is a real difference. ## When to Use Diff * **Before deciding whether to sync at all.** If nothing differs, there's nothing to sync. * **Before an Overwrite into production.** Overwrite deletes target-only mappings; a Diff shows you exactly what that means before you commit to it. * **To hand a colleague a review.** Walk the Diff together before a large sync. * **To confirm a sync landed.** Diff the two pipelines afterward — a clean comparison means the target matches the source. * **To measure drift.** Diff a long-lived test pipeline against main to see how far it has diverged before starting new work. ## Reading a Diff Each differing entity shows a one-line summary of what differs — a count such as `3 traits, 1 join`, or `only in source` / `only in target` for an entity that exists on just one side. Entities that depend on other entities list those as **depends on** chips. Differences are grouped by the part of the entity they belong to — the entity itself, or one of its maps — and tagged **Add**, **Update**, **Remove**, or **Unchanged**, from the perspective of applying the source to the target. **"Remove" Means Target-Only** A row tagged **Remove** exists in the target and not in the source. Under Overwrite it would be deleted; under Overlay it would be left alone and reported as **Skipped**; under Merge you'd decide. Diff itself never removes anything. **A Rename Reads as a Remove Plus an Add** Comparisons match items by their stable key, so something renamed on one side appears as one item removed and a different one added. If a Diff looks larger than you expected, check for renames before concluding something went wrong. ## Troubleshooting ### The Diff Shows More Differences Than Expected **Problem:** You changed three fields but the Diff lists many more. **Solution:** * Check for renames — they appear as a removal plus an addition. * Check whether the target has changed independently since you last looked. A Diff compares both sides as they are right now, not just your work. * Confirm you're comparing against the pipeline you intended. ## Related Topics Apply what the Diff showed you, one entity at a time Apply it across the whole pipeline The same walkthrough, with decisions How each strategy handles each difference # Configuring Data Sources Source: https://support.entegrata.com/data/pipeline/configuring-sources Learn how to add and configure primary and related data sources in your Entegrata data pipelines ## Overview Data sources are the foundation of your data pipelines. In Entegrata, you connect to various data sources, select the specific tables or datasets you need, and configure how they relate to each other. This guide covers everything you need to know about working with data sources in the mapping editor. ## Understanding Source Types ### Primary Source The **primary source** is the main source of data for your entity: * Determines the base set of records * All other sources are joined to this source * Each entity must have exactly one primary source * The first source you add becomes the primary source **Example**: For a Customer entity, your CRM system's customer table would be the primary source. ### Related Sources **Related sources** are additional data sources joined to enrich your primary data: * Can have multiple related sources per entity * Joined using one or more key fields (foreign key relationships) * Can come from different systems or databases **Example**: For a Customer entity, you might join: * Client table (to get customer client number) * Marketing engagement table (to get campaign responses) Canvas showing primary source with related sources and entity ## Adding a Primary Source Use the **Add Source Wizard** to add your first source. The wizard walks you through three steps: selecting a source, configuring a primary key, and setting up a record identifier. In the mapping editor, click the **+ Add Source** button in the Data Sources placeholder area. Mapping editor with Add Source placeholder Browse the **DataSourceBrowser** tree to find your source. Connections are listed at the top level with their logos, and each connection expands to show available resources (tables/datasets). The number next to each item shows the count of available fields. * Use the search bar to filter connections and resources * Click a connection to expand it and see its resources * Click a resource to select it (highlighted in teal with a checkmark) * Click **Next** to proceed Add Source Wizard Step 1 - Select Source with resource highlighted Select one or more fields that uniquely identify each record in this data source. * Use the multi-select dropdown to choose primary key fields * If the source has a known primary key, it may be auto-suggested * You can select multiple fields for composite primary keys * Click **Next** to proceed Add Source Wizard Step 2 - Primary Key selection The record identifier determines how records from this source are uniquely identified in the entity. For primary sources, you have two options: * **Use Primary Key as Identifier** (default, recommended): Check this box to use the same field(s) you selected as the primary key. * **Custom Identifier**: Uncheck the box to configure a custom identifier using functions like COALESCE, CONCAT, or HASH. The first source added becomes your **primary source** automatically. The primary source determines the base records. All other sources are joined to this. Click **Add Source** to finish. Add Source Wizard Step 3 - Record Identifier with Use Primary Key checked ## Adding Related Sources Related sources are joined to your primary source to enrich data. You can add a related source in two ways: * Click the **+ Add Related Source** placeholder below your existing sources * Open the source node's **three-dot menu** and select **Add Related Source** Add Related Source placeholder below existing source The wizard follows the same three steps, but Step 3 shows **Record Matching** instead of Record Identifier: Browse and select a resource from the DataSourceBrowser, the same as when adding a primary source. Select the field(s) that uniquely identify records in this source. Define how records from this source relate to the primary source: * **Source Identifier**: Select a field (or custom expression) on this source whose data relates to the primary source. * **Target Identifier**: Select the corresponding field on the primary source that matches. * Use the **Field/Custom** dropdown to switch between selecting a field directly or writing a custom expression. * Click **+ Add identifier field** to add additional matching conditions. Add Source Wizard Step 3 - Record Matching for related source **Join Best Practices** * Use exact match joins when possible (e.g., ID fields) * Our system preserves all primary source records, but not necessarily all related source records. * For complex or ambiguous cases, multiple fields can be used in combination to uniquely identify records. ## Editing Source Properties To edit an existing source's configuration, use the **three-dot context menu** on the source node and select **Edit Properties**, or click the **Edit Source** button on the source node. Source node context menu with Edit Properties, Add Related Source, and Delete The **Edit Source** modal has three tabs: ### Fields Tab Add or remove fields (columns) available from this source. Use the search bar to find fields, or click the **+** button to add a custom expression. Edit Source modal - Fields tab ### Properties Tab View and modify the source's primary key and record identifier settings. * **Primary Key**: Change the field(s) used to uniquely identify records * **Record Identifier**: Toggle "Use Primary Key" or configure a custom identifier Edit Source modal - Properties tab ### Filters Tab Add filter conditions to restrict which records are included from this source. This is equivalent to adding a WHERE clause. Edit Source modal - Filters tab ## Managing Multiple Related Sources When working with many related sources: ### Nested Joins Related sources can join to other related sources (not just the primary): Join a related source to your primary source using the Add Source Wizard. When configuring record matching, select the first related source instead of the primary source as the target. Set up the matching condition between the two related sources using the Source Identifier and Target Identifier fields. **Example**: * Primary: Customers * Related 1: Orders (joined to Customers) * Related 2: OrderItems (joined to Orders, not Customers) ## Troubleshooting ### Source Connection Failed **Issue**: Cannot connect to the data source. **Solutions**: * Verify the connection credentials are current * Check network connectivity * Ensure you have permission to access the source * Contact your administrator to refresh the connection ### No Tables Visible **Issue**: Can't see any tables or datasets in the source. **Solutions**: * Verify you have read permissions * Check if you're looking in the correct schema/database * Refresh the connection in the admin portal * Some sources may require specific catalog configuration ### Too Many Records After Join **Issue**: Join produces more records than expected. **Solutions**: * Check for duplicate values in join keys * Verify you're joining on the correct fields * Look for one-to-many relationships * Add additional join conditions to make relationship unique * Consider if this is actually correct (e.g., one customer, many orders) ## Best Practices **Source Configuration** * Always use filters at the source level when possible * Choose the most specific table/view available * Ensure you are joining to other tables with at most 1 record for each primary source record. **Data Quality** * Verify join keys have good cardinality * Check for nulls in join fields * Test edge cases (no matches, duplicates) * Validate against expected record counts ## Related Topics Learn about the visual mapping editor interface Map source fields to your entity fields Advanced transformations with multiple source fields Deploy your configured mappings to production # Creating Pipelines Source: https://support.entegrata.com/data/pipeline/creating-pipelines Step-by-step guide to creating new data pipelines in Entegrata ## Overview Creating a new pipeline is the first step in setting up your data integration workflow. This guide walks you through the process of creating a pipeline, from initial setup to configuring trigger settings. ## Prerequisites Before creating a pipeline, ensure you have: * Access to the Entegrata Admin Portal * Appropriate permissions to create pipelines * Understanding of what data you want to process * Knowledge of your source data systems (configured in Collector) ## Creating a Pipeline Log in to the [Entegrata Admin Portal](https://admin.entegrata.com) and click the **Pipelines** tab in the main navigation. In the pipeline list view, click the **Create New Pipeline** button in the top-right corner of the page. Pipeline list with Create button highlighted In the Create New Pipeline dialog, provide the following information: ### Pipeline Name (Required) Enter a descriptive name that clearly identifies the pipeline's purpose. **Examples:** * "Client Data Processing Pipeline" * "Daily ETL Pipeline" * "Historical Account Migration" * "Real-time Transaction Sync" Choose business-friendly names that non-technical users can understand. The name will appear in reports, logs, and monitoring dashboards. ### Pipeline Key (Auto-generated) The pipeline key is automatically generated from the name by converting it to lowercase and replacing spaces with underscores. **Example:** "Client Data Processing Pipeline" becomes `client_data_processing_pipeline` You can customize the key if needed, but it must be unique within your instance and contain only lowercase letters, numbers, and underscores. Create Pipeline dialog with name and key fields Provide a detailed description of what the pipeline does, what data it processes, and how it fits into your workflows. **Example:** "Processes client demographic and contact information from Salesforce CRM. Runs nightly to update Client and Contact data types with changes from the previous 24 hours. Includes validation rules for email addresses and phone numbers." Good descriptions help team members understand the pipeline's purpose and troubleshoot issues. Include: * Data sources involved * Data types processed * Business context * Any special considerations Choose how the pipeline should be executed: ### No Trigger (Default) Pipeline must be run manually by a user. Best for: * Development and testing * One-time data migrations * Ad-hoc processing needs ### Manual Explicitly configured for manual execution only. Same as "No Trigger" but more explicit. ### Event Driven Pipeline runs automatically when specific events occur. Best for: * Real-time or near-real-time processing * Event-based workflows * Triggered by external systems Review all the information you've entered: * Pipeline name is descriptive and unique * Description provides sufficient context * Trigger type matches your use case Click **Create Pipeline** to save the new pipeline. Create Pipeline dialog completed and ready to submit After successful creation, you'll see a success message and the pipeline will appear in the pipeline list. The pipeline is created in **Draft** status, meaning it's not yet deployed to production. You can now: * Add data type mappings * Configure data sources * Test the pipeline in dry-run mode ## After Creating a Pipeline Once your pipeline is created, follow these next steps: ### 1. Add Data Type Mappings Click on the pipeline name to open the mapping editor and start adding data types to process. Learn how to add and configure data types in your pipeline ### 2. Configure Data Sources For each data type, you'll need to connect to your Collector data sources and select which tables or resources to use. Learn how to add and configure data sources for your mappings ### 3. Map Fields Use the visual mapping editor to map source fields to data type fields, applying transformations as needed. Learn how to map fields between sources and data types ### 4. Test the Pipeline Before deploying to production, run the pipeline in dry-run mode to validate your mappings. Learn how to test and execute pipelines ## Pipeline Creation Tips **Start Simple, Then Expand** Begin with a single data type and validate it works before adding more. This makes troubleshooting easier and builds confidence in your configuration. **Use Consistent Naming Conventions** Establish a naming pattern for your pipelines that includes: * Data domain (e.g., "Client", "Portfolio", "Transaction") * Frequency (e.g., "Daily", "Weekly", "Historical") * Purpose (e.g., "Refresh", "Migration", "Sync") Example: "Client Daily Refresh Pipeline" **Document Thoroughly** Use the description field to document: * What data this pipeline processes * Why certain design decisions were made * Any dependencies or prerequisites * Who to contact for questions **Consider Data Volumes** When scheduling pipelines: * Estimate processing time based on data volume * Avoid scheduling during peak business hours * Allow buffer time between pipeline runs * Monitor execution times and adjust schedules as needed ## Troubleshooting ### Can't Create Pipeline **Problem:** The "Create New Pipeline" button is disabled or not visible. **Solution:** * Verify you have pipeline creation permissions * Check that you're logged in with the correct account * Contact your administrator if permissions are needed ### Pipeline Name Already Exists **Problem:** Error message indicating the pipeline name is already in use. **Solution:** * Choose a different, unique name for your pipeline * Check if a similar pipeline already exists that you could reuse or duplicate * Add qualifiers to make the name unique (e.g., "V2", "Test", or a date) ### Pipeline Created But Not Visible **Problem:** Pipeline was created successfully but doesn't appear in the list. **Solution:** * Refresh the page to reload the pipeline list * Check if pagination is hiding your pipeline (it may be on another page) * Use the search bar to find the pipeline by name * Clear your browser cache if the issue persists ## Related Topics Learn about pipeline lifecycle and management How to modify existing pipeline settings Create copies of existing pipelines Next steps: Configure your data mappings # Data Mapping Editor Source: https://support.entegrata.com/data/pipeline/data-mapping-editor Complete guide to using the visual data mapping editor interface in Entegrata ## Overview The Entegrata data mapping editor is a powerful visual interface for creating and managing data transformations. Using an intuitive node-based design, you can connect data sources, transform data, and map fields without writing code. This guide covers all features of the mapping editor, from basic navigation to advanced transformations. ## Interface Overview The mapping editor consists of several key areas: ### Main Canvas The central workspace where you visually build your data pipeline: * **Source Nodes**: Represent your data sources * **Field Nodes**: Individual fields from sources * **Transformation Nodes**: Data transformations and operations * **Entity Node**: Your target data type with its fields * **Connection Lines**: Show data flow between nodes Full mapping editor interface with labeled sections ### Toolbar Located at the bottom of the editor: * **Center View**: Re-center and fit all nodes in view * **Full-Screen**: Expand the editor to fill your screen * **Save / Discard**: Appear in a floating bar at the bottom center when you have unsaved changes Toolbar buttons: Center View and Full-Screen ### Edit Field Modal Click on any entity field to open the **Edit Field** modal with two tabs: * **Field Configuration**: Edit the field's name, description, type, format, foreign key, and tags * **Data Source**: Configure how the field gets its value — select a source field directly, or set up a multi-field operator (Use First Available, Join Text, Apply Conditions, etc.) Edit Field modal showing field properties ### Map Selection Dropdown located in the top left of the editor: * Select your current data map * Manage new and existing data maps Data map selection dropdown ## Navigating the Canvas ### Pan and Zoom * **Pan**: Click and drag on empty canvas space * **Zoom In/Out**: Use mouse wheel or zoom controls * **Fit to Screen**: Click the fit icon to center all nodes * **Reset Zoom**: Double-click empty canvas space ### Selecting Nodes * **Single Select**: Click on a node * **Entity Field Click**: Opens the Edit Field modal * **Context Menu**: Use the three-dot menu (⋮) on source nodes, or right-click on connection lines ### Source Node Actions Source nodes have a **three-dot context menu** (⋮) with the following options: * **Edit Properties**: Open the Edit Source modal to modify fields, primary key, identifier, or filters * **Add Related Source**: Launch the Add Source Wizard to add a related source * **Delete**: Remove the source from the mapping Source node context menu showing Edit Properties, Add Related Source, and Delete ## Working with Source Nodes ### Adding a Primary Source Click the **+ Add Source** button in the Data Sources placeholder to launch the **Add Source Wizard**. The wizard guides you through selecting a source, configuring a primary key, and setting up a record identifier. For detailed instructions, see [Configuring Data Sources](/data/pipeline/configuring-sources). ### Adding Related Sources Click the **+ Add Related Source** placeholder below your existing sources, or use the source node's context menu. The wizard walks you through selecting a source, configuring a primary key, and setting up record matching to join with the primary source. For detailed instructions, see [Configuring Data Sources](/data/pipeline/configuring-sources#adding-related-sources). ### Editing Source Properties Click the **Edit Source** button on any source node, or use the context menu's **Edit Properties** option. The Edit Source modal provides three tabs for managing fields, properties (primary key and identifier), and filters. For detailed instructions, see [Configuring Data Sources](/data/pipeline/configuring-sources#editing-source-properties). ## Working with Field Mappings ### Direct Field Mapping The simplest mapping: connect a source field directly to an entity field. #### Drag to Connect Click and drag from a source field to an entity field on the right side. A connection line appears. Alternatively, click on an entity field and select the source field from the dropdown. Direct field connection ### Unmapping Fields To remove a field mapping: * **Visual View**: Right-click the connection line and select **Delete** or select the line and press Delete key Context menu showing delete option ## Creating Combined Mappings ### Map Multiple Source Fields Drag a source field's handle to a field on the entity which is already mapped. This will create a new "logic" node in the middle of the mapping editor, allowing you to specify how the different fields should be used to set the value of the target entity field. Newly mapped "logic" node with details Click the logic node's three-dot menu and select **Edit Settings**, or click the entity field to open the **Edit Field** modal. On the **Data Source** tab, use the **Operator** dropdown to select: * **Use First Available (Coalesce)**: Use first non-null value from multiple fields * **Join Text (Concat)**: Combine multiple fields into one * **Apply Conditions**: Conditional logic (if-then-else) * **Generate Hash**: Combine & obfuscate multiple fields to produce a unique value * **Use Highest Value**: Take the highest value of the input fields * **Use Lowest Value**: Take the lowest value of the input fields See [Multi-Field Mapping](/data/pipeline/multi-field-mapping) for detailed guides on each. Operator dropdown in the Edit Field modal Configure the logic specific to the operator you've chosen (field priority order, text separator, case conditions, etc.). You can also connect additional source fields to the logic node on the canvas. ## Saving and Auto-Save ### Manual Save Click the **Save** button in the toolbar to save your changes: * Saves all mapping configuration * Creates a version history entry * Does not publish to production Save button and save status indicator ### Auto-Save The editor automatically saves your work on your browser (not the server): * Every change is persisted in a local state until saved. * Changes made only to your local browser can be discarded using the **Discard Changes** button in the toolbar. * Changes may disappear from your local browser if you clear your browser cache. ## Troubleshooting ### Editor Not Loading **Issue**: Canvas is blank or stuck loading. **Solutions**: * Refresh the browser page * Clear browser cache * Check network connectivity * Verify you have permission to view this entity ### Cannot Connect Fields **Issue**: Dragging a field doesn't create a connection. **Solutions**: * Check data type compatibility (see warning messages) * Ensure target field isn't already mapped (unmap first) * Verify source node is properly configured ### Changes Not Saving **Issue**: Save button doesn't work or changes are lost. **Solutions**: * Check for validation errors (fix before saving) * Verify you have edit permissions * Look for browser console errors * Try saving in smaller batches ## Related Topics Learn how to add and configure data sources Step-by-step field mapping procedures Advanced transformations with COALESCE, CONCAT, and CASE Deploy your mappings to production # Data Mapping Overview Source: https://support.entegrata.com/data/pipeline/data-mapping-overview Introduction to Entegrata data mapping capabilities and the visual mapping editor ## Introduction Data mapping is the core of Entegrata pipelines. It defines how source data from your Collector systems transforms into standardized data types in your warehouse. The visual mapping editor provides an intuitive interface for configuring these transformations without writing code. Data mapping in Entegrata uses a declarative, configuration-based approach. You define **what** the mapping should do, and the system generates optimized DLT (Delta Live Tables) scripts that execute the transformations. ## What is Data Mapping? Data mapping consists of configuring: 1. **Data Types (Entities)** - The standardized schemas you're populating (e.g., Client, Contact, Account) 2. **Data Sources** - The Collector tables or resources providing source data 3. **Field Mappings** - How individual source fields map to data type fields 4. **Transformations** - Logic applied during mapping (formatting, concatenation, conditional logic) 5. **Default Values** - Values used when source data is missing or null ## Key Concepts ### Data Types vs. Entities Throughout Entegrata, you'll see these terms used interchangeably: * **Data Type** - The standardized schema definition (preferred term in user interface) * **Entity** - Technical term for the same concept (used in backend systems) We use "data types" in documentation to align with business terminology. Think of them as the types of data your organization works with: Clients, Accounts, Transactions, etc. ### Primary vs. Related Sources Each data type mapping can use multiple sources: The main table or resource containing the core data for this data type. Every data type mapping must have exactly one primary source. Additional tables joined to the primary source to enrich data. You can have zero or more related sources. **Example:** * **Primary Source**: `CRM_Clients` table (contains client IDs, names, status) * **Related Source 1**: `CRM_Addresses` table (joined by client\_id for address information) * **Related Source 2**: `CRM_Industries` table (joined by industry\_id for industry details) ### Data Type Fields Each data type has a defined schema with fields that must be populated: Fields that must have a value (either from a mapping or default value). Pipeline cannot deploy if required fields are unmapped. Fields that can be left unmapped or null without blocking deployment. Fields automatically populated by Entegrata (e.g., record IDs, timestamps). Cannot be manually mapped. ## The Mapping Workflow A typical data mapping workflow follows these stages: Select which data type this pipeline should process (e.g., Client, Contact, Account). Use the **Add Source Wizard** to select a source from your Collector, configure its primary key, and set up a record identifier. Add additional sources through the wizard and configure record matching to define how they join to the primary source. Map all required fields from source to target, ensuring no required fields are left unmapped. Map additional fields as needed for your use case. Add business logic like concatenation, conditional mapping, or data formatting. Configure default values for fields that might be missing in source data. Run in dry-run mode to validate mappings without writing to production. Publish the mapping to generate DLT scripts and enable execution. ## Mapping Editor Interface The mapping editor is organized into several sections: ### Data Type List (Left Panel) Shows all data types configured in the current pipeline: * Click a data type to view/edit its mappings * Add new data types with the "+" button * See deployment status and configuration completeness Data type list showing configured entities ### Mapping Editor Primary interface for configuring data mappings per entity: * Configure sources and fields * Identify and configure relationships between sources * Map source fields to entity fields * Setup complex field mapping logic Source configuration panel ### Entity Field List Mapping A table view of all fields for an entity: * Quick view of all metadata for all fields on an entity * Shows all fields on an entity across all data maps Field mapping grid showing source-to-target mappings ### Entity Lineage A view of the sources and fields mapped to an entity: * A convenient, clutter-free way to view source to entity mappings Data Lineage view showing source columns connected to entity fields ### Entity Access Control A view of access controls rules applied to an entity: * Explore access control rules applied to an entity * Check source of access control rules applied to an entity Field mapping grid showing source-to-target mappings ## Mapping Capabilities ### Simple Direct Mapping Map a single source field directly to a data type field: **Example:** Map `CRM_Clients.client_name` to `Client.name` ### Multi-Field Mapping Combine multiple source fields into one target field: **COALESCE Example:** Use the first non-null value from multiple sources ``` COALESCE(email_primary, email_work, email_personal) ``` **CONCAT Example:** Combine multiple fields with a separator ``` CONCAT(first_name, ' ', last_name) ``` ### Conditional Mapping Apply different logic based on source data values: **CASE Example:** Map status codes to descriptive values ``` CASE WHEN status = 'A' THEN 'Active' WHEN status = 'I' THEN 'Inactive' ELSE 'Unknown' END ``` ### Type Conversion Automatically convert between data types: * String to number * String to date/timestamp * Number to string * Date format conversions ### Default Values Provide fallback values when source data is missing: **Static Default:** Use a fixed value (e.g., "Unknown", 0, current timestamp) **Expression Default:** Use a SQL expression (e.g., `CURRENT_DATE()`, `UUID()`) ## Data Quality and Validation Entegrata provides several data quality features: ### Required Field Validation * Ensures all required fields have mappings or defaults * Prevents deployment of incomplete configurations * Highlights missing mappings in the editor ### Type Compatibility Checking * Validates source and target data types are compatible * Warns about potential type conversion issues * Suggests appropriate type casting ### Null Handling * Configure how NULL values are handled * Use COALESCE for multiple fallback options * Set defaults for fields that can't be null ### Data Preview * See sample data from sources before mapping * Verify transformations work with real data * Identify data quality issues early Always use dry-run mode to validate mappings with actual data before deploying to production. ## Mapping Strategies ### Start-to-Finish Approach Best for new data types or simple mappings: 1. Map all required fields first 2. Map commonly used optional fields 3. Add transformations and defaults 4. Test and deploy ### Incremental Approach Best for complex data types or evolving requirements: 1. Map minimal required fields to get started 2. Deploy and test with real data 3. Add more fields incrementally 4. Redeploy after each addition ### Template-Based Approach Best for similar data types across multiple pipelines: 1. Create a reference mapping in one pipeline 2. Duplicate pipeline to new environments or regions 3. Adjust source connections and field specifics 4. Maintain consistency across deployments ## Common Mapping Patterns ### Full Name from Components Combine first, middle, and last names: ``` CONCAT_WS(' ', first_name, middle_name, last_name) ``` ### Email with Fallback Use primary email, falling back to secondary: ``` COALESCE(email_primary, email_secondary, 'noemail@company.com') ``` ### Status Normalization Convert various status codes to standard values: ``` CASE WHEN status IN ('A', 'Active', '1') THEN 'Active' WHEN status IN ('I', 'Inactive', '0') THEN 'Inactive' WHEN status IN ('P', 'Pending') THEN 'Pending' ELSE 'Unknown' END ``` ### Date Formatting Convert string dates to proper date types: ``` TO_DATE(date_string, 'YYYY-MM-DD') ``` ### ID Generation Create unique identifiers from multiple fields: ``` CONCAT(source_system, '_', source_id) ``` ## Performance Considerations ### Efficient Source Queries * Use filters in source configuration to reduce data volume * Select only needed columns from sources * Leverage indexes on join columns ### Transformation Complexity * Simple mappings perform better than complex CASE statements * Pre-aggregate data in sources when possible * Use materialized views for frequently joined related sources ### Incremental Processing * Configure change tracking to process only new/modified records * Use timestamp fields for incremental logic * Balance frequency vs. data volume ## Best Practices **Map Required Fields First** Always start by mapping all required fields. This ensures your configuration can deploy and prevents wasted effort on optional fields if required fields can't be satisfied. **Test with Real Data Early** Don't wait until all fields are mapped to test. Run dry-run mode early and often to catch: * Data quality issues in sources * Type conversion problems * Unexpected NULL values * Performance bottlenecks **Document Complex Transformations** For complex CASE statements or business logic: * Add comments in the pipeline description * Document the reasoning behind specific mappings * Include examples of input → output transformations * Note any business rules or regulatory requirements **Use Meaningful Source Aliases** When adding related sources, use clear aliases: * Good: `addresses`, `industry_lookup`, `contact_emails` * Poor: `table1`, `t2`, `rel1` This makes field mappings easier to understand and maintain. **Start Simple, Add Complexity** Begin with simple direct mappings and add transformations only when needed: 1. Direct field-to-field mappings first 2. Add COALESCE for NULL handling 3. Add CASE for conditional logic 4. Add CONCAT for combining fields This approach makes troubleshooting easier. ## Exploring Data Mapping Ready to start mapping data? Explore these detailed guides: Learn how to add and configure data types in your pipeline Connect primary and related data sources Map individual fields between source and target Use COALESCE, CONCAT, and CASE for complex mappings Understand data type field properties and requirements Configure fallback values for missing data ## Related Topics Deploy mappings to generate DLT scripts See which source columns feed each entity field Understand data type access controls Execute pipelines to process mapped data # Deleting Pipelines Source: https://support.entegrata.com/data/pipeline/deleting-pipelines How to safely delete pipelines and understand the impact ## Overview Deleting a pipeline permanently removes it from your instance, including all data type mappings, execution history, and configuration. This operation cannot be undone, so it's important to understand the implications before proceeding. Pipeline deletion is permanent and irreversible. Ensure you have backups or duplicates if you might need the configuration later. ## When to Delete a Pipeline Consider deleting pipelines in these situations: ### Pipeline No Longer Needed * Business process has changed and data is no longer required * Data source has been decommissioned * Pipeline was replaced by a newer version * Testing or development pipeline that served its purpose ### Duplicate or Incorrect Pipeline * Pipeline was created by mistake * Duplicate exists that serves the same purpose * Name or configuration is incorrect and easier to recreate than fix ### Cleanup and Organization * Removing old test pipelines * Consolidating redundant pipelines * Decluttering pipeline list for better management Before deleting a production pipeline, consider pausing it first and monitoring for a period to ensure it's truly not needed. ## What Gets Deleted When you delete a pipeline, the following are permanently removed: * **Pipeline Configuration** - Name, description, trigger settings, and metadata * **Data Type Mappings** - All configured data type relationships * **Field Mappings** - All source-to-target field mapping configurations * **Data Sources** - Primary and related source configurations * **Transformation Logic** - COALESCE, CONCAT, CASE expressions, and custom SQL * **Execution History** - All past run logs and status information * **Deployment** - DLT scripts are removed from warehouse ## What Is NOT Deleted Deleting a pipeline does NOT remove: * **Data in Warehouse** - Already-processed data remains in your data warehouse tables * **Data Types (Entities)** - The data type definitions remain available for other pipelines * **Source Systems** - Collector connections and source data are unaffected * **Other Pipelines** - No impact on other pipelines, even if they use similar data types Deleting a pipeline removes the processing logic but preserves the data that was already created. You may need to manually clean up warehouse tables if desired. ## Deleting a Pipeline Log in to the [Entegrata Admin Portal](https://admin.entegrata.com) and go to the **Pipelines** tab. Find the pipeline you want to delete using the search bar or by browsing the list. Double-check you've selected the correct pipeline. Pipeline names may be similar, especially test vs. production versions. Pipeline list with target pipeline Before proceeding, verify: * This is the correct pipeline (check name and description carefully) * No active jobs are running * Dependencies on this pipeline's data are understood * Stakeholders have been notified if needed * You have a backup or duplicate if you might need it later If the pipeline is currently running, wait for execution to complete before deleting. Click the three-dot menu (⋮) in the Actions column for the pipeline you want to delete. Pipeline actions menu From the actions menu, click **Delete** to open the deletion confirmation dialog. Delete option in actions menu The confirmation dialog displays: * Pipeline name being deleted * Warning that deletion is permanent * Information about what will be removed * Option to proceed or cancel Read the confirmation dialog carefully. Ensure you understand what will be deleted and that you're deleting the correct pipeline. Delete confirmation dialog If you're certain you want to delete the pipeline, click **Delete Pipeline** in the confirmation dialog. The deletion process begins and completes within a few seconds. After successful deletion: * Success confirmation message appears * Pipeline is immediately removed from the pipeline list * Pipeline no longer appears in search results * Associated DLT scripts are removed from warehouse If desired, manually remove warehouse tables that were populated by the deleted pipeline: 1. Connect to your data warehouse 2. Identify tables created by the pipeline (usually named after data types) 3. Drop or archive tables as needed 4. Update any downstream dependencies This is optional. Leaving tables in place won't cause issues but may consume storage space. ## Deletion Best Practices **Create a Backup First** Before deleting any pipeline: 1. Duplicate the pipeline with a name like " - Archive " 2. Update the duplicate's description to note it's a backup 3. Pause the backup so it doesn't run 4. Delete the original 5. Keep the backup for 30-90 days before deleting it This provides a safety net if you need to recover the configuration. **Document Why You're Deleting** Before deletion, document: * Why the pipeline is being deleted * What data it was processing * When it was last used * Who approved the deletion * Alternative solutions if the data is needed again Keep this documentation in your team wiki or ticketing system. **Communicate with Stakeholders** Before deleting production pipelines: * Notify anyone who consumes the pipeline's data * Provide advance notice (e.g., "Pipeline will be deleted in 1 week") * Offer alternatives or migration paths * Confirm no one objects to deletion This prevents surprises and broken dependencies. **Pause Before Deleting** For production pipelines: 1. Pause the pipeline (don't delete immediately) 2. Wait 1-2 weeks 3. Monitor for complaints or issues 4. If no problems arise, proceed with deletion This catches unexpected dependencies before permanent deletion. **Delete Test Pipelines Regularly** Schedule regular cleanup of test/development pipelines: * Monthly review of all Draft or Paused pipelines * Delete pipelines unused for 30+ days * Remove duplicate or failed test pipelines * Keep pipeline list focused on active work This maintains a clean, manageable pipeline list. ## Alternative to Deletion Instead of deleting, consider these alternatives: ### Pause the Pipeline If you're unsure about deleting: * Pause the pipeline to stop automatic execution * Keep configuration for future reference * Can be reactivated if needed * No permanent data loss Learn how to pause pipelines instead of deleting ### Archive with Naming Rename the pipeline to indicate it's archived: * Prefix name with "ARCHIVED - " * Add archive date to description * Pause to prevent execution * Keeps configuration accessible **Example:** "ARCHIVED - Legacy Client Pipeline (Archived 2024-10-02)" ### Duplicate for Backup Create an archive duplicate before deleting: * Duplicate the pipeline * Name it clearly as a backup * Pause the duplicate * Delete the original * Keep backup for a defined period (30-90 days) ## Impact of Deletion Understanding the impact helps avoid surprises: ### No Impact On * **Other Pipelines** - Deletion doesn't affect other pipelines * **Data Types** - Entity definitions remain available * **Existing Data** - Data already in warehouse is unaffected * **Source Systems** - Collector connections unchanged * **User Permissions** - Access controls remain the same ### Impacts * **Cannot Run** - Pipeline can no longer execute * **Configuration Lost** - Mapping logic is permanently deleted * **History Gone** - Execution logs and history are removed * **Scripts Removed** - DLT scripts deleted from warehouse * **Reports May Break** - If reports depend on real-time data from this pipeline If downstream reports or dashboards depend on data from this pipeline, they may show stale data after deletion (data stops updating). Update or disable affected reports. ## Recovery After Accidental Deletion If you accidentally delete a pipeline: ### Immediate Steps 1. **Don't panic** - The data in your warehouse is still there 2. **Check for backups** - Look for duplicate pipelines that might be backups 3. **Review documentation** - Check if mapping logic was documented elsewhere 4. **Contact team** - See if anyone else has a copy or remembers the configuration ### Recreation Options If no backup exists: 1. **Recreate from memory** - If recently worked with the pipeline 2. **Reverse engineer** - Examine existing warehouse data and infer mappings 3. **Contact support** - Entegrata support may have backup capabilities 4. **Restore from source control** - If pipeline configuration was versioned externally Entegrata does not provide built-in "undo" for pipeline deletion. Prevention through duplication and pausing is the best strategy. ## Troubleshooting Deletion ### Can't Delete Pipeline **Problem:** Delete option is not available or disabled. **Solutions:** * Verify you have delete permissions * Check if pipeline is currently running (wait for completion) * Ensure you're viewing the pipeline list (not inside pipeline editor) * Try refreshing the page ### Deletion Fails with Error **Problem:** Error message when attempting to delete. **Solutions:** * Verify no jobs are currently running * Check if pipeline has active locks or dependencies * Try pausing first, then deleting * Contact support if error persists ### Pipeline Still Appears After Deletion **Problem:** Pipeline shows in list after deletion confirmation. **Solutions:** * Refresh the page to update the list * Clear browser cache * Check if you're viewing a cached or filtered view * Verify deletion actually completed (check for error messages) ### Deleted Wrong Pipeline **Problem:** Accidentally deleted the wrong pipeline. **Solutions:** * Check if a duplicate or backup exists * Review team documentation for mapping details * Contact Entegrata support immediately for recovery options * Recreate pipeline from available documentation * Learn from mistake: always duplicate before deleting ## Deletion Checklist Before deleting any pipeline, use this checklist: * [ ] Verified this is the correct pipeline (checked name and description) * [ ] Confirmed pipeline is no longer needed * [ ] Notified stakeholders who consume this pipeline's data * [ ] Checked for downstream dependencies (reports, dashboards, other pipelines) * [ ] Created a duplicate backup if there's any uncertainty * [ ] Documented reason for deletion * [ ] Obtained necessary approvals (if required by organization) * [ ] Pipeline is not currently running * [ ] Reviewed warehouse table cleanup requirements * [ ] Scheduled follow-up to clean up warehouse tables (if needed) ## Related Topics Create backups before deleting Pause as an alternative to deletion Recreate deleted pipelines Back to pipeline management overview # Deploying Pipelines Source: https://support.entegrata.com/data/pipeline/deploying-pipelines How to publish pipelines to production and generate DLT scripts ## Overview Deploying a pipeline publishes all configured data type mappings to your production data warehouse, generating optimized Delta Live Tables (DLT) scripts that transform source data according to your mapping configurations. Deployment is required before a pipeline can execute with real data. Deployment is a one-way process that generates production-ready code. Always test your mappings thoroughly with dry runs before deploying. ## What Deployment Does When you deploy a pipeline, the system: 1. **Validates Configuration** - Checks that all mappings are complete and valid 2. **Generates DLT Scripts** - Creates optimized SQL/Python code for data transformations 3. **Publishes to Warehouse** - Deploys scripts to your Databricks environment 4. **Updates Status** - Changes pipeline status from Draft to Deployed 5. **Enables Execution** - Makes the pipeline available for running (manual or scheduled) ## Deployment Status Pipelines have one of these deployment statuses: Pipeline has been created but not yet deployed. Cannot execute with real data. Pipeline has been published to production and is ready to execute. After deployment, any changes to data type mappings require redeployment to take effect in production. ## Prerequisites Before deploying a pipeline, ensure: * **Mappings Are Complete** - All data types have configured field mappings * **Sources Are Configured** - Primary and related sources are properly connected * **Required Fields Are Mapped** - All mandatory fields have valid mappings or defaults * **Testing Is Done** - Dry runs have been performed and validated * **Permissions Are Granted** - You have deployment permissions Deploying overwrites any previous deployment of the same pipeline. Ensure you're ready to replace the existing configuration. ## Deploying a Pipeline Log in to the [Entegrata Admin Portal](https://admin.entegrata.com) and go to the **Pipelines** tab. Find the pipeline you want to deploy. Before deploying, click the pipeline name to review: * All intended data types are configured * Field mappings are complete * Sources are properly connected * Transformations are tested Run a dry run to validate mappings before deploying. This catches errors without impacting production. Navigate back to the pipeline list after verifying mappings. Click the three-dot menu (⋮) in the Actions column for the pipeline you want to deploy. Pipeline actions menu From the actions menu, click **Deploy** to open the deployment confirmation dialog. Deploy option in actions menu The deployment confirmation dialog shows: * Pipeline name * Number of data types that will be deployed * Warning about replacing existing deployment (if applicable) Deployment will deploy all mapped data types in this pipeline. Ensure you've reviewed all mappings, not just the ones you recently changed. Click **Deploy Pipeline** to proceed with deployment. The deployment process begins and may take several minutes depending on: * Number of data types being deployed * Complexity of field mappings * Size of transformation logic Monitor the deployment progress. During deployment: * A loading indicator shows deployment is in progress * You cannot modify the pipeline * Other operations on the pipeline are blocked Do not close the browser or navigate away during deployment. Wait for the confirmation message. Upon successful deployment, you'll see: * Success confirmation message * Pipeline status changes to **Deployed** * Last Modified timestamp is updated * Pipeline is now ready to run ## After Deployment Once a pipeline is deployed, you can: ### Run the Pipeline Execute the pipeline to process data using your deployed mappings. Learn how to execute deployed pipelines ### Activate for Scheduled Runs If the pipeline has a scheduled trigger, activate it to enable automatic execution. Learn how to activate pipelines for scheduled execution ### Monitor Execution Track pipeline runs and review execution logs to ensure everything works as expected. ### Make Updates If you need to change mappings after deployment: 1. Modify the mappings in the pipeline editor 2. Test with dry run 3. Deploy again to publish changes Changes to mappings are not reflected in production until you redeploy. Draft changes don't affect running pipelines. ## Redeployment Redeploying an already-deployed pipeline follows the same process as initial deployment. Use redeployment when: ### Mapping Changes After modifying field mappings, transformations, or data sources. ### Adding Data Types After adding new data type mappings to an existing pipeline. ### Fixing Errors After correcting mapping errors discovered during execution. ### Schema Updates After changes to source or target schemas that require mapping updates. Redeployment replaces the entire pipeline deployment. Test all mappings, not just changed ones, to avoid breaking working configurations. ## Deployment Validation The deployment process validates: ### Required Fields * All mandatory data type fields have mappings or default values * No orphaned field references ### Source Connections * All referenced sources are accessible * Source tables/resources exist ### Transformation Logic * SQL expressions are syntactically correct * Functions used are supported * Type conversions are valid ### Data Type Configuration * Data types are properly configured * No circular dependencies * Primary keys are defined If validation fails, deployment is aborted and you'll see specific error messages. Fix the errors and try deploying again. ## Deployment Best Practices **Test Thoroughly Before Deploying** Always perform these checks before deployment: 1. Run dry run to validate query logic 2. Review all field mappings visually 3. Check for any validation warnings 4. Verify source connections are active 5. Confirm required fields are mapped This prevents deploying broken configurations. **Deploy During Maintenance Windows** For production pipelines: * Deploy during scheduled maintenance windows * Avoid deployment during active pipeline runs * Notify team members of deployment timing * Have rollback plan ready if issues arise **Document Deployment Changes** After deploying, update the pipeline description to note: * What was changed in this deployment * Date of deployment * Reason for changes * Any known issues or considerations Example: "Deployed 2024-10-02: Added Account data type mapping, updated Client email field transformation" **Monitor First Run After Deployment** After deploying: 1. Run the pipeline manually first (don't rely on schedule) 2. Watch execution logs closely 3. Verify data quality in warehouse 4. Check row counts match expectations 5. Validate transformations worked correctly Only activate for scheduled runs after confirming success. **Version Control Significant Changes** Before major redeployments: * Duplicate the pipeline to create a backup * Name it with version number or date * Keep the backup until new version is validated * Allows quick rollback if needed ## Deployment Strategies ### Blue-Green Deployment For critical pipelines, use a blue-green strategy: 1. **Blue (Current)**: Keep existing deployed pipeline running 2. **Green (New)**: Create duplicate pipeline with changes 3. **Test Green**: Thoroughly test the new version 4. **Switch**: Deactivate blue, activate green 5. **Monitor**: Watch green for issues 6. **Cleanup**: Delete blue after green is validated ### Phased Deployment For complex changes, deploy in phases: 1. **Phase 1**: Deploy subset of data types 2. **Validate**: Test phase 1 thoroughly 3. **Phase 2**: Deploy additional data types 4. **Validate**: Test phase 2 thoroughly 5. **Continue**: Add remaining data types incrementally ### Rollback Strategy Always have a rollback plan: 1. **Before Deployment**: Duplicate pipeline as backup 2. **After Deployment**: Keep backup active initially 3. **If Issues Arise**: Reactivate backup pipeline 4. **Fix Issues**: Correct problems in new version 5. **Redeploy**: Try again after fixes ## Troubleshooting Deployment ### Deployment Fails with Validation Errors **Problem:** Deployment is blocked due to validation errors. **Solutions:** * Review error messages for specific issues * Check that all required fields are mapped * Verify source connections are accessible * Fix transformation syntax errors * Ensure data types are properly configured ### Deployment Hangs or Takes Too Long **Problem:** Deployment process doesn't complete or runs for excessive time. **Solutions:** * Wait at least 10-15 minutes for complex pipelines * Check network connection stability * Refresh page after 15+ minutes to see if it completed * Contact support if consistently hanging ### Deployment Succeeds But Pipeline Fails **Problem:** Deployment completes successfully but pipeline execution fails. **Solutions:** * Deployment validates configuration, not runtime execution * Run dry run to identify execution issues * Check source data availability and quality * Review transformation logic with actual data * Verify warehouse permissions for writing data ### Can't Deploy (No Deploy Option) **Problem:** Deploy option is not available in actions menu. **Solutions:** * Verify you have deployment permissions * Check if pipeline is already deployed (may need to redeploy instead) * Ensure pipeline has at least one data type mapping configured * Verify you're not in the middle of editing the pipeline ### Deployed But Changes Not Reflected **Problem:** After deployment, changes aren't visible in pipeline execution. **Solutions:** * Ensure you redeployed after making changes * Wait a few minutes for deployment to propagate * Run the pipeline fresh (not continuing previous run) * Clear any cached query plans in warehouse * Verify changes were saved before deploying ## Related Topics Execute deployed pipelines Activate deployed pipelines for scheduled runs Configure mappings before deployment Create backups before major deployments # Duplicating Pipelines Source: https://support.entegrata.com/data/pipeline/duplicating-pipelines How to create copies of existing pipelines with all data mappings and configurations ## Overview Duplicating a pipeline creates a complete copy including all data type mappings, field configurations, and transformation logic. This is useful for creating variations of existing pipelines, testing changes without affecting production, or setting up similar pipelines for different data sources. Duplication copies the pipeline structure and mappings but creates a new, independent pipeline. Changes to the duplicate won't affect the original, and vice versa. ## What Gets Duplicated When you duplicate a pipeline, the following are copied: * **Pipeline Metadata** - Name (with "Copy of" prefix) and description * **Data Type Mappings** - All configured data types in the pipeline * **Data Sources** - Primary and related source configurations * **Field Mappings** - All field-to-field mappings and transformations * **Default Values** - Any default values configured for fields * **Transformation Logic** - COALESCE, CONCAT, CASE expressions, and custom SQL ## What Is NOT Duplicated The following are NOT copied when duplicating: * **Execution History** - The duplicate has no run history * **Deployment Status** - Duplicate starts in Draft status (not deployed) * **Pipeline Key** - A new unique key is generated * **Created/Modified Dates** - Duplicate has its own timestamps ## Duplicating a Pipeline Log in to the [Entegrata Admin Portal](https://admin.entegrata.com) and go to the **Pipelines** tab. Locate the pipeline you want to duplicate using the search bar or by browsing the list. Pipeline list with source pipeline highlighted Click the three-dot menu (⋮) in the Actions column for the pipeline you want to duplicate. Pipeline actions menu with Duplicate option From the actions menu, click **Duplicate** to open the Duplicate Pipeline dialog. Duplicate option in actions menu The Duplicate Pipeline dialog opens with information from the source pipeline: **Pipeline Name:** Automatically prefixed with "Copy of " plus the original name * Example: "Copy of Client Data Processing Pipeline" **Pipeline Key:** Auto-generated from the new name * Example: `copy_of_client_data_processing_pipeline` **Description:** Copied from the source pipeline Duplicate Pipeline dialog with pre-filled fields Update the fields to customize your duplicate: ### Update Pipeline Name Change the name to something more meaningful than "Copy of...". Consider: * Purpose of the duplicate (testing, different environment, etc.) * Differences from the original * Version indicators if applicable **Examples:** * "Client Data Processing Pipeline - Test" * "Client Data Processing Pipeline v2" * "Client Data Processing Pipeline - UK Region" ### Update Pipeline Key The key auto-generates from the name, but you can customize it if needed. Ensure it's unique and follows naming conventions (lowercase, underscores, no special characters). ### Update Description Modify the description to explain: * Relationship to the source pipeline * Purpose of this duplicate * Key differences or customizations * Intended use case **Example Description:** "Testing version of Client Data Processing Pipeline. Used to validate new transformation logic before deploying to production. Do not activate for scheduled runs." Duplicate dialog with customized name and description Set the trigger type for your duplicate: * **No Trigger** - Manual execution only (recommended for test pipelines) * **Manual** - Explicitly manual (same as No Trigger) * **Event Driven** - Triggered by events After reviewing and customizing all fields, click **Create Pipeline** to duplicate. The duplication process may take a few moments as all mappings are copied. After successful duplication, you'll see a confirmation message. The new pipeline appears in the pipeline list with: * Your custom name * **Draft** status (not deployed) * No execution history * Current date as Created date Pipeline list showing both original and duplicate Click on the duplicate pipeline name to open it and verify that all data type mappings were copied correctly: * All data types present * Source configurations intact * Field mappings preserved * Transformation logic copied It's good practice to review critical mappings before deploying a duplicate, especially if you're planning to make modifications. ## Common Use Cases for Duplication ### 1. Testing Changes Duplicate a production pipeline to test modifications without risking the production configuration. **Workflow:** 1. Duplicate the production pipeline 2. Name it "Pipeline Name - Test" 3. Make your changes to the duplicate 4. Test thoroughly in dry-run mode 5. Once validated, apply the same changes to production 6. Delete or keep the test pipeline for future use ### 2. Environment-Specific Pipelines Create pipeline variations for different environments (Development, Staging, Production). **Example:** * Original: "Client Processing - Production" * Duplicate: "Client Processing - Staging" * Modify source connections to point to staging data sources ### 3. Regional Variations Duplicate pipelines for processing data from different regions or business units. **Example:** * Original: "Client Pipeline - US" * Duplicate: "Client Pipeline - EMEA" * Adjust source connections and any region-specific transformations ### 4. Version Control Create versioned copies when making significant architectural changes. **Example:** * Original: "Portfolio Pipeline v1" * Duplicate: "Portfolio Pipeline v2" * Implement new mapping approach in v2 while keeping v1 available for rollback ### 5. Historical Archive Duplicate a pipeline before major refactoring to preserve the original configuration. **Example:** * Original: "Legacy Client Pipeline" * Duplicate: "Legacy Client Pipeline - Archive 2024-Q3" * Refactor the original, keeping the archive for reference ## After Duplication Once you've duplicated a pipeline, typical next steps include: ### 1. Modify as Needed Make any necessary changes to the duplicate: * Update data sources to point to different systems * Adjust field mappings or transformations * Add or remove data types * Change business logic ### 2. Test Thoroughly Run the duplicate in dry-run mode to ensure it works as expected: Learn how to test and execute pipelines ### 3. Deploy When Ready Once tested and validated, deploy the duplicate to production: Learn how to deploy pipelines to production ### 4. Manage the Original Decide what to do with the original pipeline: * Keep both if serving different purposes * Delete the original if the duplicate replaces it * Archive the original for reference ## Duplication Best Practices **Use Clear Naming Conventions** When duplicating, use names that clearly indicate: * Purpose of the duplicate * Relationship to the original * Environment or version Examples: * "\[Original Name] - Test" * "\[Original Name] - v2" * "\[Original Name] - US-East" * "\[Original Name] - Archive 2024-01" **Document the Relationship** In the description field, note: * Which pipeline this was duplicated from * Why the duplicate was created * Key differences from the original * When the duplicate was created Example: "Duplicated from 'Client Data Processing Pipeline' on 2024-10-02. Used for testing new validation rules before production deployment." **Test Before Activating** Always test duplicated pipelines in dry-run mode before: * Deploying to production * Processing live data Duplication copies configurations, but issues can arise if source connections or data have changed. **Clean Up Unused Duplicates** Regularly review and delete duplicates that are no longer needed: * Test pipelines after changes are synced to production * Old versions after new versions are validated * Regional variations if business requirements change This keeps your pipeline list manageable and reduces confusion. ## Troubleshooting ### Duplication Fails **Problem:** Error message when trying to duplicate a pipeline. **Solution:** * Verify you have permissions to create pipelines * Check if the proposed pipeline name or key already exists * Ensure the source pipeline is not corrupted or partially configured * Try duplicating a different pipeline to see if the issue is specific to one pipeline ### Missing Mappings After Duplication **Problem:** Some data types or field mappings didn't copy to the duplicate. **Solution:** * Verify the source pipeline had complete mappings before duplication * Check if archived or deleted data types were excluded from duplication * Re-duplicate if mappings are critical, or manually recreate missing mappings * Contact support if duplication consistently misses mappings ### Duplicate Runs Against Wrong Data Source **Problem:** Duplicate pipeline is processing data from the original's sources instead of new sources. **Solution:** * Open the duplicate pipeline's mapping editor * For each data type, verify and update the source connections * Check both primary and related sources * Save changes and test in dry-run mode ### Can't Edit Duplicate After Creation **Problem:** Unable to modify the duplicate pipeline after creation. **Solution:** * Refresh the page to ensure the duplicate was fully created * Verify you have edit permissions * Check if the pipeline is locked by another operation * Try accessing the pipeline after a few moments (background processes may still be completing) ### Duplicate Has Same Key as Original **Problem:** Error about duplicate pipeline key. **Solution:** * The system should auto-generate a unique key, but if not: * Customize the pipeline key in the duplication dialog * Ensure the key is unique across all pipelines in your instance * Use underscores instead of spaces or special characters ## Related Topics Learn how to create pipelines from scratch How to modify pipeline metadata Learn how to delete pipelines safely Test and execute duplicated pipelines # Editing Pipelines Source: https://support.entegrata.com/data/pipeline/editing-pipelines How to update pipeline names, descriptions, and metadata in Entegrata ## Overview After creating a pipeline, you may need to update its name, description, or other metadata. This guide explains how to edit pipeline properties while preserving all configured data mappings and execution history. Editing a pipeline only updates its metadata (name and description). To modify data type mappings, trigger settings, or execution behavior, use the appropriate specialized operations described in related guides. ## What Can Be Edited When editing a pipeline, you can modify: * **Pipeline Name** - The display name shown in the pipeline list and throughout the interface * **Description** - The detailed explanation of the pipeline's purpose and function ## What Cannot Be Edited The following properties cannot be changed through the edit operation: * **Pipeline Key** - The unique identifier is set at creation and cannot be changed * **Trigger Type** - Must be configured through separate trigger management * **Data Type Mappings** - Managed through the mapping editor interface * **Created Date** - Historical metadata that cannot be modified If you need to change properties that cannot be edited, consider duplicating the pipeline and creating a new version with the desired configuration. ## Editing a Pipeline Log in to the [Entegrata Admin Portal](https://admin.entegrata.com) and go to the **Pipelines** tab. Find the pipeline you want to edit using the search bar or by browsing the list. You can search by pipeline name for faster access. Pipeline list with search functionality Click the three-dot menu (⋮) in the Actions column for the pipeline you want to edit. Pipeline actions menu From the actions menu, click **Edit** to open the Edit Pipeline dialog. Edit option in pipeline actions menu In the Edit Pipeline dialog, modify the **Pipeline Name** field as needed. ### Naming Guidelines * Use descriptive, business-friendly names * Include key information about purpose and scope * Follow your organization's naming conventions * Keep names reasonably concise (under 50 characters recommended) **Examples of Good Names:** * "Client Master Data Daily Sync" * "Portfolio Holdings Weekly Aggregation" * "Transaction History Incremental Load" Edit Pipeline dialog showing name field Modify the **Description** field to update or add details about the pipeline's function. ### Description Best Practices Include information about: * What data this pipeline processes * Which source systems are involved * How often it runs (if scheduled) * Any special business rules or transformations * Dependencies or prerequisites * Contact person or team responsible **Example Description:** "Processes daily client demographic updates from Salesforce CRM. Runs nightly at 2 AM EST to sync Client and Contact data types with changes from the previous 24 hours. Includes email validation and phone number formatting. Maintained by Data Integration Team." Edit Pipeline dialog showing description field Before saving, review your changes to ensure: * The new name is unique and descriptive * The description provides sufficient context * No critical information was accidentally removed Changing a pipeline name may impact documentation, runbooks, or scripts that reference the old name. Communicate name changes to your team. Click **Update Pipeline** to save your changes. The pipeline will be updated immediately, and you'll see a success confirmation message. After saving, verify the changes appear correctly in the pipeline list. The "Last Modified" timestamp will be updated to reflect when the edit was made. Pipeline list showing updated pipeline with new Last Modified date ## When to Edit a Pipeline Consider editing a pipeline when: ### Business Requirements Change Update the name or description to reflect new business processes or organizational changes. **Example:** "Client Onboarding Pipeline" becomes "Client and Account Onboarding Pipeline" after expanding scope. ### Improving Clarity Refine names and descriptions to make them more understandable to new team members. **Example:** "Daily Pipeline 1" becomes "Client Daily Incremental Refresh" ### Adding Documentation Enhance descriptions with additional context discovered during operations or troubleshooting. **Example:** Adding notes about special handling for international phone numbers. ### After Scope Changes Update pipeline metadata after adding or removing data types to keep documentation current. **Example:** After adding Account data type, update description to mention both Client and Account processing. ## Edit vs. Other Operations Understanding when to use Edit versus other pipeline operations: | Need to Change | Use This Operation | | - | - | | Pipeline name or description | **Edit Pipeline** | | Trigger type or schedule | Trigger configuration (future feature) | | Data type mappings | Open pipeline and use mapping editor | | Pipeline status (active/paused) | Pause/Activate action | | Create modified version | Duplicate Pipeline | | Deployment status | Deploy Pipeline | ## Impact of Editing Editing a pipeline has minimal impact on operations: ### What Is Affected * **Display Name** - Updated everywhere the pipeline name appears * **Description** - Updated in pipeline details and documentation * **Last Modified Date** - Updated to current timestamp * **Audit Log** - New entry recorded for the edit operation ### What Is NOT Affected * **Execution History** - Past runs remain associated with the pipeline * **Data Mappings** - All configured mappings remain unchanged * **Deployment Status** - Pipeline deployment status is preserved * **Scheduled Runs** - Pipeline continues running on schedule * **Performance** - No impact on execution speed or data processing Editing a pipeline does not require redeployment. Changes to metadata are reflected immediately without affecting the deployed DLT scripts. ## Editing Multiple Pipelines Currently, pipelines must be edited individually. To update multiple pipelines efficiently: 1. **Use Consistent Naming Patterns** - Easier to apply bulk updates later if needed 2. **Document Changes** - Keep a changelog of significant pipeline updates 3. **Plan Updates** - Group related pipelines and update them in batches 4. **Communicate Changes** - Notify team members of naming or description changes If you need to make the same description update across many pipelines, consider using copy-paste to maintain consistency. Prepare the updated description in a text editor first. ## Troubleshooting ### Can't Edit Pipeline **Problem:** Edit option is not available or disabled in the actions menu. **Solution:** * Verify you have edit permissions for pipelines * Check if the pipeline is locked by another user * Ensure you're viewing the pipeline list (not inside a pipeline) * Contact your administrator if permissions are needed ### Changes Not Saved **Problem:** After clicking "Update Pipeline," changes don't appear to be saved. **Solution:** * Refresh the page to reload the pipeline list * Verify the success message appeared after clicking Update * Check if there were any validation errors * Ensure you didn't accidentally click Cancel instead of Update ### Pipeline Name Already Exists **Problem:** Error message indicating the new name is already in use by another pipeline. **Solution:** * Choose a different, unique name * Add qualifiers to differentiate (version numbers, dates, environment) * Check if the existing pipeline can be deleted or renamed instead ### Lost Changes **Problem:** Accidentally closed the edit dialog without saving. **Solution:** * Changes are not saved unless you click "Update Pipeline" * Reopen the edit dialog and re-enter your changes * Consider copying your description text before opening the dialog to avoid retyping ## Best Practices **Version Your Pipeline Names** When making significant changes to a pipeline's configuration, consider creating a new version rather than editing the existing one: * Old: "Client Processing Pipeline" * New: "Client Processing Pipeline v2" This preserves the old configuration for rollback if needed. **Document Edit History** In the description field, maintain a brief change log for significant updates: ``` Last updated: 2024-10-02 - Added Account data type processing - Changed schedule from 2 AM to 3 AM - Updated to use new CRM connector ``` **Review Before Saving** Always review your changes before saving, especially for pipeline names that appear in: * Scheduled job logs * Email notifications * Monitoring dashboards * Team documentation ## Related Topics Learn how to create new pipelines from scratch Create copies of existing pipelines Back to pipeline management overview Learn how to safely delete pipelines # Entity Lineage Source: https://support.entegrata.com/data/pipeline/entity-lineage See which source tables and columns feed each field of an entity ## Overview **Data Lineage** is a read-only diagram showing which source columns feed each field of a single entity. Use it to answer "where does this field come from?" without opening every record's mapping in the editor. Data Lineage view showing source columns on the left connected to entity fields on the right ## Opening the Lineage View Open a pipeline, open a layer (for example **Silver**), and click an entity. In the breadcrumb, click the view menu (it shows **Editor** by default) and select **Data Lineage**. The diagram is rebuilt from the entity's current mappings each time you open it, so edits made in the editor appear right away. You don't need to publish first. ## Reading the Diagram * **Data Sources** (left) groups source columns by table. Each group shows the table name with its source system underneath. Tables used as a record's primary source and tables joined in as related sources both appear. * **Target Fields** (right) lists the entity's fields that have at least one mapped source column. * **Arrows** connect each source column to every entity field it feeds. A field built from several columns has several incoming arrows; a column used by several fields has several outgoing arrows. Use the controls in the corner, or scroll and drag, to zoom and pan. ## Tracing a Field Back to Its Source If an entity field has unexpected values, find it under **Target Fields** and follow its arrows back to the source columns. Check those columns' data in the source system, then open the record in the **Editor** to review how the columns are combined. ## Limitations * Lineage covers one entity and one hop: source columns to this entity's fields. It doesn't show downstream entities or reports that use the entity. * The diagram shows which columns feed a field, not how. Expressions, joins, and filters are visible only in the [editor](/data/pipeline/data-mapping-editor). * For fields mapped with an expression, only column names wrapped in backticks (for example `` `client_id` ``) are detected. Columns referenced without backticks don't appear. * Fields with no mapped source columns, such as constants, aren't shown. * Data Lineage isn't available when viewing an entity snapshot. ## Related Topics Edit the mappings that lineage displays Map source columns to entity fields Create and organize entities Set up the primary and related sources shown in lineage # Field Management Source: https://support.entegrata.com/data/pipeline/field-management Learn how to add, edit, and configure entity fields and their properties in Entegrata data pipelines ## Overview Entity fields define the structure of your data types. Each field has properties like name, data type, and validation rules that determine how data is stored and used. This guide covers everything you need to know about managing fields in your entities. ## Understanding Entity Fields ### What is an Entity Field? An entity field represents a single piece of information in your data type: * **Customer Entity**: Name, Email, Phone, Address fields * **Order Entity**: OrderID, OrderDate, Amount, Status fields * **Product Entity**: SKU, Name, Price, Category fields Each field has: * **Name**: The field identifier * **Description**: Any comments or documentation for the field * **Type**: What kind of data it holds (UUID, Text, Integer, Timestamp, etc.) * **Format**: Display formatting rule (Currency, Percent, Email, etc.) * **Foreign Key**: Link to another entity type * **Tags**: Custom labels for categorization Entity field list showing various fields and their properties ## Viewing Entity Fields ### Field List View Access the complete field list for your entity: Navigate to your entity in the data pipeline section. Entity navigation Navigate to the **Fields** tab at the top of the page. The field list appears in a table showing the following information: * Field name * Data type * Data Formatting Rule * Description * Tags * Status Field list panel Use the search box to quickly find fields by name. ### Visual Canvas View Fields also appear in the visual mapping editor: * On the entity node (right side of canvas) * With connection lines showing mappings Entity node showing fields in visual editor ## Adding New Fields Click **Add Custom Field** button in the entity node (right side). Provide a clear, descriptive name: * Use business-friendly terms * Avoid technical abbreviations * Use consistent naming conventions (camelCase or snake\_case) * Example: "CustomerName" not "cust\_nm" Choose the appropriate data type for your field. See [Data Types](#data-types) section below for details. Set additional properties: * **Description**: Document the field's purpose * **Format**: The formatting rule to be applied to values displayed for this field Provide a default value used when source data is null. See [Default Values](/data/pipeline/default-values) guide for details. ## Data Types ### Available Data Types Entegrata supports comprehensive data types for various data scenarios: #### String For any data you wish to be represented as text. | Type | Description | Example | Max Length | | - | - | - | - | | **shorttext** | Short Text | "John Smith" | 255 chars | | **text** | Text | "Some kind of descriptive text." | 65,535 chars | | **longtext** | Long Text | "This is used for a long body of text, with data too large for normal text fields..." | MAX | #### Numeric Types | Type | Description | Example | Range | | - | - | - | - | | **tinyint** | Tiny Integer | 0, 1, -65 | +/- 2^8 - 1 | | **smallint** | Small Integer | 42, -1,200, 653 | +/- 2^16 - 1 | | **integer** | Integer | 34, -532, 34,011 | +/- 2^32 - 1 | | **biginteger** | Big Integer | 17, -14,987,200, 9,450 | +/- 2^64 - 1 | | **shortdecimal** | Short Decimal | 123.45, -67.89 | Configurable precision | | **decimal** | Decimal | 123.4589, -67.8923 | Configurable precision | | **longdecimal** | Long Decimal | 123.45123409, -67.00194409 | Configurable precision | | **float** | Floating Point Number | 123.45, -67.89 | Configurable precision | | **double** | Large Floating Point Number | 123.45, -67.89 | Configurable precision | #### Date and Time Types | Type | Description | Example | Format | | - | - | - | - | | **Date** | Date only | 2024-01-15 | YYYY-MM-DD | | **Timestamp** | Precise timestamp | 2024-01-15T14:30:00.000Z | ISO 8601 | #### Boolean | Type | Description | Example | Values | | - | - | - | - | | **Boolean** | True/false | true, false | true, false | #### Complex Types | Type | Description | Example | Use Case | | - | - | - | - | | **JSON** | Structured data | `{"key": "value"}` | Complex objects | | **UUID** | Unique identifier | "123e4567-e89b-12d3-a456-426614174000" | System IDs | ### Choosing the Right Data Type **Selection Guidelines** * Match the source data type when possible * Use most specific type (Small Int vs. Int) * Consider downstream usage ## Editing Existing Fields Click on the field in the entity node on the canvas, or click a field in the Fields tab list. This opens the **Edit Field** modal. On the **Field Configuration** tab, update properties: * Name (with caution - may break existing mappings) * Description * Format * Tags **Changing Data Type**: Changing a field's data type may break existing mappings or cause data loss. Always preview data after type changes. On the **Data Source** tab, change how the field gets its value: * Select a different source field * Change the operator (Use First Available, Join Text, Apply Conditions, etc.) * Adjust field priority order or conditions ### Safe vs. Risky Changes **Safe Changes** (no impact on data): * Updating description * Changing display name **Risky Changes** (may break things): * Changing field name * Changing data type ## Deleting Fields You may delete custom fields on your entities if you no longer need them or have made breaking updates to your source data systems. Entegrata's canonical fields may not be deleted, but they can be unmapped. **Permanent Action**: Deleting a field removes it from the entity definition and all historical data. This cannot be undone. Click on the field you want to delete. Make sure this field is a custom field that you created. Non-custom fields may not be deleted. Right-click and select **Delete Field**. Type the field name to confirm deletion and click **Delete Permanently**. ### Alternatives to Deletion Consider these alternatives: * **Archive**: Hide field from views but preserve data * **Remove Mapping**: Keep field definition but stop populating it ## System Fields Entegrata automatically adds system fields to every entity. These fields are the minimum required fields for Entegrata to operate as intended. System fields cannot be edited or deleted. They're automatically maintained by Entegrata. ## Best Practices **Naming Conventions** * Use clear, business-friendly names * Be consistent (all camelCase or all snake\_case) * Avoid abbreviations unless universally understood * Use full words (e.g., "CustomerName" not "CustNm") **Data Types** * Choose most specific type available * Use validation rules to enforce quality * Document any special formats or requirements * Consider downstream system compatibility **Maintenance** * Review field usage regularly * Archive unused fields rather than delete * Document the purpose of custom fields * Keep field names stable (avoid frequent changes) ## Troubleshooting ### Cannot Add Field **Issue**: Add Field button is disabled or not working. **Solutions**: * Verify you have edit permissions * Check if entity is currently publishing (wait for completion) * Ensure entity is not archived * Try refreshing the page ### Field Changes Not Reflected **Issue**: Modified field properties don't seem to take effect. **Solutions**: * Clear browser cache and refresh * Verify changes were saved (look for save indicator) * Check if you need to republish entity * Try logging out and back in ### Data Type Mismatch Errors **Issue**: Errors when mapping source data to field after changing data type. **Solutions**: * Preview source data to verify compatibility * Add type conversion in mapping * Use CASE expression to handle edge cases * Consider reverting data type change * Check for null values causing issues ## Related Topics Map source data to entity fields Set default values for fields Advanced transformations with multiple fields Create and manage data types # Managing Data Types Source: https://support.entegrata.com/data/pipeline/managing-entities Learn how to create, edit, and archive data types (entities) in your Entegrata data pipelines ## Overview Data types (also called entities) are the foundation of your data pipeline. Each data type represents a business concept like customers, orders, or products. In Entegrata, you define data types and map them to your source data, creating a unified view across all your systems. This guide covers the complete lifecycle of managing data types in your Entegrata instance. ## Understanding Data Types A data type in Entegrata represents: * A unified business concept (e.g., "Customer", "Invoice", "Product") * The fields and structure of that concept * Mappings to one or more data sources * Access controls determining who can view the data * Lineage information showing data flow ## Creating a New Data Type From the admin portal navigation, select **Data Mappings** or **Pipelines** to access the data type management interface. Pipeline navigation menu showing Data Types section Click the **New Data Type** or **Create Entity** button in the top right corner of the data types list. Data types list view with Create button highlighted In the creation dialog, provide: * **Name**: A clear, singular name for your data type (e.g., "Customer", "Order") * **Description**: A brief explanation of what this data type represents * **Icon** (optional): Select an icon to help identify this data type visually * **Category** (optional): Group related data types together Create data type dialog with fields filled Click **Create** to save your new data type. You'll be taken to the entity list view. ## Editing an Existing Data Type ### Updating Basic Properties Find the entity you want to edit in the list view. Click the **Edit** option under the entity's option menu. Edit entity menu option Update any of the following: * Name * Description Edit entity details Click **Save** to apply your changes. These updates take effect immediately and don't require republishing. ### Modifying Field Structure To add, remove, or change fields in your data type, see the [Field Management](/data/pipeline/field-management) guide. ### Updating Mappings To change how source data maps to your data type fields, see the [Mapping Fields](/data/pipeline/mapping-fields) guide. ## Archiving Data Types Archiving removes a data type from active use without deleting historical data. Archiving a data type will: * Stop all data processing for this data type * Hide it from most users' views * Preserve historical data and lineage * Allow restoration if needed Archived data types can be unarchived, but the process may require reconfiguration. Navigate to the data type you want to archive. Click the **Archive** option in the options menu for the entity. Edit Options menu with Archive option Review the warning message and confirm that you want to archive this data type. Edit Archive confirmation modal Click **Archive** to complete the process. The data type will be moved to the archived status. ## Best Practices **Naming Conventions** * Use singular nouns (e.g., "Customer" not "Customers") * Be descriptive but concise * Avoid technical abbreviations * Use consistent casing (recommend Title Case) **Organization** * Group related data types with categories * Use clear, business-friendly descriptions * Maintain a logical hierarchy (e.g., Orders before OrderItems) * Document dependencies between data types **Lifecycle Management** * Start with draft status and test thoroughly * Archive instead of delete when possible * Document why data types are archived * Regularly review and clean up unused data types ## Troubleshooting ### Cannot Archive Data Type **Issue**: The archive option is grayed out or unavailable. **Solutions**: * Check if you have admin permissions * Verify no active dependencies exist * Ensure the data type is published (only published types can be archived) ### Archive Failed with Dependencies **Issue**: Archive fails with message about dependent objects. **Solutions**: * Check for reports or dashboards using this data type * Look for other data types with relationships to this one * Remove or update dependencies first, then archive ### Unarchive Not Restoring Data **Issue**: After unarchiving, no data appears in the data type. **Solutions**: * Verify source connections are still active * Check that credentials haven't expired * Republish the mapping to restart data processing * Review field mappings for any broken references ## Related Topics Learn how to add and configure fields in your data types Map source data to your data type fields Control who can view and edit your data types Deploy your data type mappings to production # Managing Pipeline Status Source: https://support.entegrata.com/data/pipeline/managing-pipeline-status How to pause, activate, and monitor pipeline execution status ## Overview Pipeline status controls whether a pipeline can execute automatically on its configured schedule. Managing status allows you to temporarily disable pipelines, control resource usage, and prevent unintended executions during maintenance or troubleshooting. ## Pipeline Status Types Pipelines can have several status states: ### Run Status Pipeline is deployed and enabled for automatic execution based on its trigger configuration. Scheduled pipelines will run at their configured times. Pipeline is deployed but disabled from automatic execution. Can still be run manually but won't execute on schedule. ### Deployment Status Pipeline has not been deployed to production. Cannot execute with real data. Pipeline has been published to production and is ready to execute. ### Execution Status Pipeline is currently executing a job. Last execution completed without errors. Last execution encountered errors. A pipeline can be Deployed and Paused simultaneously. This means it's ready to run but won't execute automatically until activated. ## Pausing a Pipeline Pausing a pipeline temporarily disables automatic execution while keeping the deployment intact. ### When to Pause Pause pipelines when: * **Maintenance Required** - Source systems are undergoing maintenance * **Data Quality Issues** - Source data has known problems that need resolution * **Resource Management** - Reducing processing load during peak times * **Troubleshooting** - Investigating errors without impacting schedules * **Temporary Disable** - Business process changes require temporary halt * **Cost Control** - Reducing compute costs during low-priority periods Log in to the [Entegrata Admin Portal](https://admin.entegrata.com) and go to the **Pipelines** tab. Find the pipeline you want to pause. Look for pipelines with **Active** status. Pipeline list showing active pipelines Click the three-dot menu (⋮) in the Actions column for the pipeline. Pipeline actions menu From the actions menu, click **Pause** to disable automatic execution. Pause option in actions menu The pipeline status immediately changes to **Paused**. You'll see: * Status badge updates to show Paused * Success confirmation message * Pipeline remains deployed and can be run manually Pipeline showing Paused status Pausing a pipeline does not stop a currently running execution. It only prevents future scheduled runs. If a pipeline is running when you pause it, that execution will complete normally. ## Activating a Pipeline Activating a pipeline enables automatic execution based on its trigger configuration. ### When to Activate Activate pipelines when: * **Initial Deployment** - After deploying and testing a new pipeline * **Resume After Maintenance** - Source systems are back online * **Issues Resolved** - Data quality or configuration problems are fixed * **Scheduled Enable** - Enabling during appropriate business hours * **After Updates** - Reactivating after redeployment with changes Go to the **Pipelines** tab in the [Entegrata Admin Portal](https://admin.entegrata.com). Find the pipeline you want to activate. Look for pipelines with **Paused** status. Pipeline list showing paused pipelines Before activating, ensure: * Pipeline is deployed (not Draft) * Mappings are complete and tested * Source systems are accessible * Trigger schedule is configured correctly * You're ready for automatic execution Run the pipeline manually first to verify it works before activating for scheduled runs. Click the three-dot menu (⋮) in the Actions column. Pipeline actions menu for paused pipeline From the actions menu, click **Activate** to enable automatic execution. Activate option in actions menu The pipeline status changes to **Active**. You'll see: * Status badge updates to show Active * Success confirmation message * Pipeline will now run automatically on schedule Pipeline showing Active status After activating: * Note the next scheduled execution time * Monitor the first automatic run closely * Verify execution completes successfully * Check data quality in warehouse The pipeline won't run immediately upon activation. It will execute at the next scheduled time according to its cron expression. ## Manual Execution During Pause Even when paused, you can still run pipelines manually: 1. Paused status only affects automatic/scheduled execution 2. Manual "Run" action remains available in the actions menu 3. Useful for testing or one-off processing needs 4. Does not change the Paused status This allows you to disable scheduled runs while maintaining the ability to run pipelines on-demand for testing or special situations. ## Status Management Best Practices **Communicate Status Changes** When pausing or activating pipelines: * Notify stakeholders who depend on the data * Document the reason in pipeline description * Set calendar reminders to reactivate if temporary * Update team dashboards or status pages This prevents confusion about why data isn't updating. **Test Before Activating** Before activating a pipeline for scheduled runs: 1. Run manually in dry-run mode 2. Execute a full manual run 3. Verify data in warehouse 4. Check execution time is acceptable 5. Confirm schedule timing is appropriate Only activate after successful validation. **Use Pause for Maintenance Windows** During planned maintenance: 1. Pause affected pipelines before maintenance starts 2. Perform maintenance work 3. Test pipeline manually after maintenance 4. Reactivate once confirmed working This prevents failed runs and error alerts during maintenance. **Monitor Active Pipelines** Regularly review active pipelines: * Check execution success rates * Monitor execution times * Review data quality * Verify schedules are still appropriate * Pause pipelines that are no longer needed This ensures active pipelines are healthy and necessary. **Document Pause Reasons** When pausing a pipeline, note why: * Update pipeline description with reason * Add expected reactivation date/condition * Document who to contact for reactivation * Link to related tickets or issues Example: "Paused 2024-10-02 due to source system migration. Reactivate after migration completes (est. 2024-10-15). Contact: [data-team@company.com](mailto:data-team@company.com)" ## Status Transition Rules Understanding valid status transitions: | Current Status | Can Change To | Action Required | | - | - | - | | Draft | Deployed | Deploy pipeline | | Draft | Active | Deploy pipeline, then Activate | | Deployed | Active | Activate | | Deployed | Paused | Pause (or was Active and Paused) | | Active | Paused | Pause | | Paused | Active | Activate | You cannot directly change a Draft pipeline to Active. You must first deploy the pipeline, then activate it. ## Monitoring Pipeline Status ### Status Dashboard The pipeline list provides an at-a-glance view of all pipeline statuses: * **Status badges** show current state with color coding * **Last Run** column shows when pipeline last executed * **Last Modified** column shows when configuration changed ### Filtering by Status To find pipelines with specific statuses: 1. Use the search bar with status keywords 2. Sort by status column 3. Look for color-coded status badges Create a spreadsheet or monitoring dashboard that tracks: * Which pipelines should be active * Expected execution times * Last successful run date * Status of dependencies Review regularly to catch unintended pauses or failures. ## Bulk Status Management Currently, pipelines must be paused or activated individually. For managing multiple pipelines: ### Planned Approach 1. **List Critical Pipelines** - Document which pipelines must remain active 2. **Group by Schedule** - Identify pipelines that run at similar times 3. **Coordinate Changes** - Update multiple pipelines in sequence 4. **Verify All Changes** - Check each pipeline after status update ### For Large-Scale Changes If you need to pause/activate many pipelines: * Plan during maintenance window * Document sequence of operations * Have team members assist with parallel updates * Verify critical pipelines first Contact your administrator if you regularly need to pause/activate many pipelines. Bulk operations may be possible through API or scripts. ## Troubleshooting Status Management ### Can't Pause Pipeline **Problem:** Pause option is not available or disabled. **Solutions:** * Verify pipeline is currently Active (can't pause if already Paused or Draft) * Check you have appropriate permissions * Ensure you're not inside the pipeline editor (must be in list view) * Refresh page and try again ### Pipeline Still Running After Pause **Problem:** Paused pipeline continues executing on schedule. **Solutions:** * Verify status actually changed to Paused (check badge) * Wait for current execution to complete (pause doesn't stop running jobs) * Check if multiple pipelines with similar names exist (paused wrong one) * Refresh page to see current status ### Can't Activate Pipeline **Problem:** Activate option not available or pipeline won't activate. **Solutions:** * Verify pipeline is Deployed (must deploy Draft pipelines first) * Check pipeline has trigger configuration (manual-only pipelines don't need activation) * Ensure you have activation permissions * Verify pipeline is not already Active ### Pipeline Activated But Not Running **Problem:** Active pipeline doesn't execute on schedule. **Solutions:** * Verify trigger type is Scheduled (not Manual or No Trigger) * Check cron expression is valid and next run time hasn't passed * Ensure schedule timezone is correct * Review execution logs for errors preventing scheduled start * Confirm source systems are accessible ## Related Topics Execute pipelines manually regardless of status Deploy before activating pipelines Configure triggers when creating pipelines Back to pipeline management overview # Mapping Fields Source: https://support.entegrata.com/data/pipeline/mapping-fields Step-by-step procedures for mapping source fields to entity fields in Entegrata data pipelines ## Overview Field mapping is the core of creating data pipelines in Entegrata. You define how data from your source systems maps to the standardized fields in your entities. This guide covers everything from simple direct mappings to understanding data type compatibility and handling common mapping scenarios. ## Understanding Field Mapping ### What is Field Mapping? Field mapping creates a connection between: * **Source Field**: A column from your data source (e.g., `customer_name` from CRM) * **Entity Field**: A standardized field in your entity (e.g., `Name`) When data flows through your pipeline, values are transformed from source fields to entity fields based on these mappings. Visual representation of field mapping from source to entity ## Direct Field Mapping (Visual View) ### Basic Connection Navigate to your entity and open the mapping editor. Ensure you have at least one data source configured. Mapping editor with source and entity nodes Find the source field you want to map. You can add new fields to your source node through the **Edit Source** modal's Fields tab, which lets you search and select from available columns. Managing source fields in the Edit Source modal On the right side, find the entity field you want to map to. Entity fields are shown with their data types. Entity node with field list Click and hold on the source field, then drag to the entity field. Release to create the connection. An animated line appears showing the mapping relationship. Dragging connection between source and entity field ### Alternative: Edit Field Modal Instead of dragging, you can configure mappings through the Edit Field modal: Click on the entity field you want to map. This opens the **Edit Field** modal. Switch to the **Data Source** tab in the modal. Use the source and field dropdowns to select where the data comes from. The mapping is applied when you click **Save**. ## Understanding Data Type Compatibility ### Automatic Type Conversion Entegrata automatically converts between compatible data types: | Source Type | Compatible Entity Types | | - | - | | String/Text | String, Integer (if numeric), Decimal, Boolean, Date | | Integer | String, Integer, Decimal, Boolean | | Decimal/Float | String, Decimal, Integer (rounded) | | Boolean | String, Boolean, Integer (0/1) | | Date/DateTime | String, Date, DateTime | | JSON | String, JSON | ## Mapping from Multiple Sources When you have related sources, you can map fields from any of them: Each source node is labeled. Fields show which source they come from. Multiple source nodes with color coding Connect entity fields to fields from either: * Primary source * Any related source When mapping from related sources (joined with LEFT JOIN), some records may have null values. Consider: * Using COALESCE to provide fallback values * Setting default values COALESCE example with multiple sources ## Common Mapping Patterns ### Name Fields Combining first and last name: **Option 1: Use CONCAT transformation** ``` Source: FirstName, LastName Combination: CONCAT(FirstName, ' ', LastName) Entity Field: Name ``` **Option 2: Map separate fields** ``` Source: FirstName -> Entity: FirstName Source: LastName -> Entity: LastName ``` ### Address Fields Mapping complete addresses: **Single Field Address:** ``` Source: full_address -> Entity: Address ``` **Multi-Field Address:** ``` Source: street_address -> Entity: Street Source: city -> Entity: City Source: state -> Entity: State Source: zip_code -> Entity: ZipCode ``` ### Date and Time Fields Common date/time mapping scenarios: **Date Only:** ``` Source: created_date (DateTime) -> Entity: CreatedDate (Date) (Time portion automatically truncated) ``` **Timestamp to Date and Time:** ``` Source: timestamp -> Entity: Date (extract date) Source: timestamp -> Entity: Time (extract time) ``` ### Boolean/Flag Fields Mapping yes/no or true/false values: **String to Boolean:** ``` Source: active (String: "Y"/"N") -> Entity: IsActive (Boolean) Condition: "Y" = true, "N" = false ``` **Integer to Boolean:** ``` Source: enabled (Integer: 1/0) -> Entity: IsEnabled (Boolean) Condition: 1 = true, 0 = false ``` **Complex Conditions:** Use CASE expression for complex logic (see [Multi-Field Mapping](/data/pipeline/multi-field-mapping)) Boolean field mapping configurations ### Numeric Fields Mapping numbers with precision: **Integer to Decimal:** ``` Source: quantity (Integer) -> Entity: Quantity (Decimal) (Automatically converted with .00 decimal places) ``` **Decimal Precision:** ``` Source: price (Decimal 10,4) -> Entity: Price (Decimal 10,2) (Automatically rounded to 2 decimal places) ``` ## Unmapping Fields To remove a field mapping: ### Visual View Click on the connection line between source and entity field. Selected connection line Press **Delete** key, or right-click the connection line and select **Delete**. Context menu with remove option If the field is required, you'll be asked to confirm removal. ## Remapping Fields To change an existing mapping: ### Visual View Remove the existing connection as described above. Drag from the new source field to the entity field. ## Troubleshooting ### Cannot Map Field **Issue**: Dragging a field doesn't create a connection. **Solutions**: * Check if entity field is already mapped (unmap first) * Verify data types are compatible * Ensure source node is properly connected ### Missing Fields in Dropdown **Issue**: Expected source field doesn't appear in dropdown. **Solutions**: * Refresh source schema * Verify you're looking at the correct source * Check if field name has special characters (may be escaped) * Ensure source connection is active * Ensure field is enabled on the resource ## Best Practices **Field Naming** * Use consistent naming conventions * Match business terminology, not technical names * Document unusual mappings with comments * Consider future maintainability **Performance** * Map only fields you need * Avoid complex transformations when simple mapping works * Use source-level filters to reduce data volume * Consider data types impact on storage ## Related Topics Learn about the visual mapping editor interface Advanced transformations with COALESCE, CONCAT, CASE Add, edit, and configure entity fields Set default values for entity fields # Multi-Field Mapping Source: https://support.entegrata.com/data/pipeline/multi-field-mapping Advanced data transformations using COALESCE, CONCAT, CASE, ARRAY, and JSON operators in Entegrata ## Overview Multi-field mapping allows you to combine or transform data from multiple source fields into a single entity field. Entegrata provides powerful operators that handle common data combination scenarios without requiring code. This guide covers all multi-field operators and how to use them effectively in your data pipelines. ## Available Operators ### Operator Overview | Operator (UI Label) | Purpose | Example Use Case | | - | - | - | | **Use First Available** (Coalesce) | Use first non-null value | Fallback from preferred to alternative fields | | **Join Text** (Concat) | Combine text values | Build full names, addresses, or composite keys | | **Apply Conditions** (Case) | Conditional logic | Transform values, categorize data, apply business rules | | **Generate Hash** | Create unique hashes | Generate uniquely random text strings by combining data from multiple fields | | **Use Highest Value** | Use highest value | Take the input whose value is the highest | | **Use Lowest Value** | Use lowest value | Take the input whose value is the lowest | Operator selection menu ## COALESCE Operator ### What is COALESCE? COALESCE returns the first non-null value from a list of fields. It's perfect for providing fallback values when preferred data might be missing. **Syntax**: `COALESCE(field1, field2, field3, ..., defaultValue)` ### Common Use Cases **Email Address Fallback:** ``` COALESCE(work_email, personal_email, "no-email@example.com") ``` **Phone Number Fallback:** ``` COALESCE(mobile_phone, home_phone, work_phone) ``` **Name Fallback:** ``` COALESCE(preferred_name, first_name, username) ``` ### Creating a COALESCE Transformation Drag a second source field to an entity field that is already mapped. This automatically creates a logic node on the canvas, defaulting to a **Use First Available (Coalesce)** operation. Alternatively, click the entity field to open the **Edit Field** modal and configure the operator on the **Data Source** tab. The order matters - COALESCE checks fields left to right. You can view and change the order of the inputs by dragging the fields listed in the **Field Priority Order** section. View and re-order source fields Check the **Default Value** checkbox and enter a fallback value to use if all input fields are null. ### COALESCE Best Practices **Ordering Inputs** * Put most reliable/preferred fields first * Put default or fallback fields last * Consider data quality when ordering If all fields are null and no default is provided, the entity field will be null. For required fields, always provide a default value. ## CONCAT Operator ### What is CONCAT? CONCAT combines multiple text values into a single string. You can add separators, prefixes, and suffixes. **Syntax**: `CONCAT(field1, separator, field2, separator, field3, ...)` ### Common Use Cases **Full Name:** ``` CONCAT(first_name, ' ', last_name) Result: "John Smith" ``` **Full Address:** ``` CONCAT(street, ', ', city, ', ', state, ' ', zip_code) Result: "123 Main St, Springfield, IL 62701" ``` **Composite Key:** ``` CONCAT(customer_id, '-', order_id) Result: "CUST001-ORD12345" ``` **Email Construction:** ``` CONCAT(username, '@', domain) Result: "john.smith@company.com" ``` ### Creating a CONCAT Transformation Drag a second source field to an entity field that is already mapped. This automatically creates a logic node on the canvas, defaulting to a **Use First Available (Coalesce)** operation. Alternatively, click the entity field to open the **Edit Field** modal and configure the operator on the **Data Source** tab. In the Edit Field modal's **Data Source** tab, select **Join Text (Concat)** from the **Operator** dropdown. Select the CONCAT option in the dropdown The order matters - CONCAT appends fields left to right. You can view and change the order of the inputs by dragging the fields listed in the **Text Combination Order** section. View and re-order source fields In the **Text Separator** field, enter the character(s) to insert between each field value (e.g., a space, comma, or dash). ## CASE Operator ### What is CASE? CASE provides if-then-else conditional logic. Transform values, categorize data, or apply complex business rules based on conditions. **Syntax**: ``` CASE WHEN condition1 THEN result1 WHEN condition2 THEN result2 ELSE default_result END ``` ### Common Use Cases **Status Categorization:** ``` CASE WHEN status = 'A' THEN 'Active' WHEN status = 'I' THEN 'Inactive' WHEN status = 'P' THEN 'Pending' ELSE 'Unknown' END ``` **Value Ranges:** ``` CASE WHEN revenue >= 1000000 THEN 'Enterprise' WHEN revenue >= 100000 THEN 'Mid-Market' WHEN revenue >= 10000 THEN 'SMB' ELSE 'Small Business' END ``` **Boolean Conversion:** ``` CASE WHEN active_flag = 'Y' THEN true WHEN active_flag = 'N' THEN false ELSE null END ``` **Multiple Conditions:** ``` CASE WHEN country = 'US' AND state = 'CA' THEN 'California' WHEN country = 'US' AND state = 'TX' THEN 'Texas' WHEN country = 'CA' THEN 'Canada' ELSE 'Other' END ``` ### Creating a CASE Transformation Drag a second source field to an entity field that is already mapped. This automatically creates a logic node on the canvas, defaulting to a **Use First Available (Coalesce)** operation. Alternatively, click the entity field to open the **Edit Field** modal and configure the operator on the **Data Source** tab. In the Edit Field modal's **Data Source** tab, select **Apply Conditions** from the **Operator** dropdown. Select the CASE option in the dropdown In the Edit Field modal's **Data Source** tab, click **Add case** to create a new condition: * **When**: Define the condition to evaluate (field, operator, and value) * **Then**: Specify the output value when the condition is true * Check **Default Case (ELSE)** to set a fallback value when no conditions match First CASE condition configuration Click **Add case** to add additional conditions. Cases are evaluated in order from top to bottom. Check **Default Case (ELSE)** at the bottom and enter the fallback value returned when no conditions match. Always provide a default case, especially for required fields. Without it, unmatched records will be null. ### Available Operators in CASE | Operator | Description | Example | | - | - | - | | **=** (equals) | Exact match | status = 'Active' | | **≠** (not equals) | Not equal to | country ≠ 'US' | | **>** (greater than) | Numeric/date comparison | revenue > 100000 | | **\<** (less than) | Numeric/date comparison | age \< 18 | | **≥** (greater or equal) | Inclusive comparison | score >= 90 | | **≤** (less or equal) | Inclusive comparison | quantity \<= 10 | | **CONTAINS** | String contains substring | name CONTAINS 'Corp' | | **STARTS WITH** | String starts with | email STARTS WITH 'admin' | | **ENDS WITH** | String ends with | filename ENDS WITH '.pdf' | | **IN** | Value in list | state IN ('CA', 'NY', 'TX') | | **IS NULL** | Field is null | middle\_name IS NULL | | **IS NOT NULL** | Field has value | phone IS NOT NULL | ### CASE Best Practices **Condition Ordering** * Put most specific conditions first * More general conditions go later * Always include an ELSE clause ## Troubleshooting ### COALESCE Returns Unexpected Nulls **Issue**: COALESCE still returns null even with multiple input fields. **Solutions**: * Verify input field order * Check that at least one input has non-null values in sample data * Add a literal default value as last input * Preview each input field separately to verify data ### CONCAT Produces Extra Separators **Issue**: Concatenated string has separators where fields were null. **Solutions**: * Enable "Skip nulls" option in CONCAT settings * Use COALESCE on inputs to convert nulls to empty strings * Consider using CASE to handle null scenarios differently ### CASE Always Returns ELSE Value **Issue**: No conditions seem to match. **Solutions**: * Verify field data types match comparison values * Check for whitespace in string comparisons * Use case-insensitive operators for text * Preview source data to see actual values * Check operator choice (= vs CONTAINS vs STARTS WITH) ## Related Topics Learn basic field-to-field mapping Configure entity field properties and data types Set default values for fields Complete guide to the mapping editor interface # Pipeline Management Source: https://support.entegrata.com/data/pipeline/overview Comprehensive guide to creating, managing, and monitoring Entegrata data pipelines ## Overview Pipeline management in Entegrata provides a centralized interface for organizing and controlling your data integration workflows. From this dashboard, you can create new pipelines, monitor their execution status, deploy changes to production, and manage the entire lifecycle of your data processing jobs. ## Accessing Pipeline Management To access the pipeline management interface: 1. Log in to the [Entegrata Admin Portal](https://admin.entegrata.com) 2. Navigate to the **Pipelines** tab in the main navigation 3. You'll see the pipeline list view with all pipelines in your instance ## Pipeline List View The pipeline list view displays all pipelines with the following information: ### Pipeline Columns The descriptive name of the pipeline (e.g., "Client Data Processing Pipeline") A brief description of what the pipeline does and what data it processes When the pipeline configuration was last updated When the pipeline was last executed (manually or automatically) Current pipeline status combining deployment status and run status: * **Draft** - Pipeline has not been deployed yet * **Deployed** - Pipeline is deployed and ready to run * **Active** - Pipeline is deployed and scheduled to run automatically * **Paused** - Pipeline is deployed but paused from automatic execution * **Running** - Pipeline is currently executing * **Failed** - Pipeline last run encountered errors * **Success** - Pipeline last run completed successfully Quick action menu for pipeline operations (see below) ### Pipeline Actions Menu Each pipeline has an actions menu (⋮) that provides quick access to common operations: * **Edit** - Modify pipeline name and description * **Duplicate** - Create a copy of the pipeline with all its mappings * **Run** - Execute the pipeline immediately (manual trigger) * **Dry Run** - Test the pipeline without writing data to production * **Deploy** - Publish all mapped data types to production * **Pause/Activate** - Toggle automatic execution on or off * **View Mappings** - Navigate to the data mapping editor * **Delete** - Remove the pipeline (requires confirmation) Deleting a pipeline is permanent and cannot be undone. Ensure you have backups or duplicates if needed before deleting. ## Pipeline Lifecycle Understanding the pipeline lifecycle helps you manage your data integration effectively: When first created, pipelines are in draft status. You can configure data type mappings, but the pipeline is not yet available for execution in production. After deploying, the pipeline's DLT scripts are generated and published to your data warehouse. The pipeline can now be executed but won't run automatically unless activated. Active pipelines are deployed and configured to run automatically based on their trigger settings (e.g., scheduled execution). Paused pipelines remain deployed but won't execute automatically. This is useful for temporarily disabling a pipeline without deleting it. ## Pipeline Trigger Types When creating or editing a pipeline, you can configure how it should be triggered: ### Manual Trigger Pipelines run only when explicitly triggered by a user through the "Run" action. Best for: * Ad-hoc data migrations * Testing and development * One-time data loads ### Event-Driven Trigger Pipelines run automatically when specific events occur (e.g., new data detected in source). Best for: * Real-time or near-real-time data processing * Event-based workflows * Triggered by external systems Event-driven triggers require additional configuration with your Collector and may not be available for all data sources. ### No Trigger (Manual Only) Pipeline has no automatic trigger configured and must be run manually. ## Searching and Filtering Pipelines Use the search bar at the top of the pipeline list to quickly find pipelines by name. The search supports: * Partial name matching * Case-insensitive search * Real-time filtering as you type The pipeline list is paginated to handle large numbers of pipelines efficiently. Use the page controls at the bottom to navigate through multiple pages. ## Pipeline Execution Monitoring Monitor pipeline execution through several indicators: ### Run Status Indicators * **Success** (Green) - Pipeline completed without errors * **Running** (Blue) - Pipeline is currently executing * **Failed** (Red) - Pipeline encountered errors during execution * **Pending** (Gray) - Pipeline is queued for execution ## Best Practices Follow these best practices for effective pipeline management: 1. **Use Descriptive Names** - Choose pipeline names that clearly indicate their purpose and data scope 2. **Monitor Regularly** - Check pipeline status after scheduled runs to catch issues early 3. **Start Small** - Begin with a single data type, validate it works, then add more mappings ## Common Operations ### Creating a New Pipeline Step-by-step guide to creating a new pipeline from scratch ### Modifying Existing Pipelines Update pipeline name, description, and settings Create a copy of an existing pipeline Move mapping configuration from one pipeline into another See what differs between two pipelines, read-only ### Execution and Deployment Execute pipelines manually or on schedule Publish pipelines to production Pause, activate, or monitor pipeline status ## Related Topics Learn how to configure data type mappings within pipelines Create and manage the data types your pipelines process ## Troubleshooting ### Pipeline Won't Run * Check that the pipeline is deployed (not in draft status) * Verify the pipeline is activated (not paused) * Ensure your Collector data sources are connected and accessible ### Pipeline Failing Consistently * Check source data quality and schema changes * Verify required fields have valid mappings or default values * Review data type field requirements ### Pipeline Running Too Long * Check for inefficient field mappings or complex transformations * Verify source queries are optimized with appropriate filters * Review data volumes and adjust processing strategies For additional support, contact your Entegrata administrator or submit a ticket through the [Support Portal](https://support.entegrata.com). # Publishing Mappings Source: https://support.entegrata.com/data/pipeline/publishing-mappings Learn how to publish data mappings as DLT scripts to Databricks and manage deployment workflows ## Overview Publishing is the process of deploying your data mappings to production. When you publish, Entegrata generates Delta Live Tables (DLT) scripts and deploys them to your Databricks environment, where they run as automated data pipelines. This guide covers the complete publishing process, from pre-publish validation to monitoring deployed pipelines. ## Understanding Publishing ### What Happens When You Publish? When you click **Publish**, Entegrata: 1. **Validates** your mapping configuration 2. **Generates** Delta Live Tables (DLT) Python code 3. **Deploys** the code to your Databricks workspace 4. **Creates or updates** the DLT pipeline 5. **Schedules** the pipeline to run automatically 6. **Starts** initial data processing ### Publishing Requirements Before publishing, your mapping must meet these requirements: **Required for Publishing:** * At least one data source configured * No validation errors * Valid join conditions (if using related sources) * Proper permissions to publish ## Publishing Your Mapping Click **Save** to ensure all changes are persisted. Publishing will fail if there are unsaved changes. Save button and saved indicator Click the **Publish** button in the top toolbar. When complete, you'll see a success message. ## Related Topics Learn the mapping editor before publishing Understand entity lifecycle and status See which source columns feed each entity field Control who can publish and access data # Reviewing Changes with Merge Source: https://support.entegrata.com/data/pipeline/reviewing-sync-changes Walk through every difference between two pipelines and approve changes one at a time before committing ## Overview Choosing **Merge (review changes)** and clicking **Review changes** opens a step-by-step wizard that walks you through every difference between the source and target pipelines. You decide what comes across, and **nothing is applied until you commit**. The wizard only shows entities that **actually differ**. Entities that are already identical on both sides are skipped entirely, so you're never clicking through dozens of no-op tables. If the two pipelines match completely, the wizard tells you there's nothing to sync. Merge is the only strategy that lets you mix decisions in a single pass — bring two fields across, leave a third, and remove a fourth. ## The Top Bar On the left, your position in the review: * **`{n}` of `{total}`**, followed by the entity's name. * A one-line summary of what differs — either a count such as `3 traits, 1 join`, or `only in source` / `only in target` for an entity that exists on just one side. * If the entity depends on other entities, those are listed as **depends on** chips. On the right, your controls, in order: A searchable dropdown of every differing entity, so you don't have to page through in order. The filter box reads **Filter entities…** How many entities you've looked at so far. Move between entities. Apply your decisions across all remaining entities at once. Apply just the entity you're looking at. Once committed, this becomes a green **Committed** chip. ## Making Decisions on an Entity Each entity has two views, toggled at the top: * **Summary** — a readable overview of what differs. * **Interactive** — the decision surface. In **Interactive**, every difference is a row tagged **Add**, **Update**, **Remove**, or **Unchanged**, grouped by the part of the entity it belongs to — the entity itself, or one of its maps. Toggle each row on to apply it, or off to leave the target as-is. ### Shortcuts * **Apply all** / **Ignore all** at the top affect every row on the entity. Each group also has its own pair. * **Reset** appears once you've excluded something, and puts everything back to the default. * The footer keeps a running count: `{n} add · {n} update · {n} remove · {n} ignore`. ### Entities That Exist on Only One Side When an entity exists on only one side, you get a single card instead of a row list: * **Ignore / Add** toggles for an entity that only the source has. * **Keep / Remove** toggles for an entity that only the target has. Removing an entity archives it in the target, so that option is flagged in red. Some entity settings are shown but not selectable, under the note *"Entity settings differ but aren't part of a merge."* Edit those on the entity directly — see [Managing Entities](/data/pipeline/managing-entities). ## Committing One Entity at a Time **Commit entity** applies just the entity you're looking at, and leaves the rest for later. The confirmation dialog states the scope plainly: > "This applies your decisions for Person into Main — just this one entity. The rest of the pipeline is left for later." The **Reason for Sync** field is pre-filled with `Merged from {source}` and you can edit it. There's also a **Publish `{target}` after committing** checkbox — see [Publishing After a Sync](/data/pipeline/sync-overview#publishing-after-a-sync). After it commits, you get a confirmation, the entity gets a green **Committed** chip, and the wizard moves on. **Commit Entity by Entity for Large Syncs** Committing incrementally lets you stop, verify, and resume. It also means a failure affects one entity instead of interrupting a batch partway through. ## Committing Everything at Once **Commit all (`{n}`)** applies your decisions across all remaining entities. The dialog summarizes exactly what's about to happen: * `{n}` changed entities merged * `{n}` added * `{n}` removed from target * `{n}` add/remove ignored If any entity is going to be archived from the target, a red callout says so explicitly and calls it destructive. **Entities you never opened are committed with all their changes accepted.** If you want an entity left alone, open it and use **Ignore all** — don't just skip past it. While it runs you'll see `Committing {done} of {total}…` and a note asking you not to navigate away. When it's done you'll see **Merge complete**, with a tally of committed, skipped, and failed entities, and any failures listed with a reason. **Done** takes you to the target pipeline. A batch commit is a per-entity loop, not a single transaction. If one entity fails partway through, the entities committed before it stay applied. The result screen tells you exactly which succeeded and which didn't — re-open the wizard and retry the failures. ## Troubleshooting ### Nothing to Sync **Problem:** The wizard opens and reports that there's nothing to sync. **Solution:** The two pipelines are already identical. There's nothing to review or commit. ### A Commit Was Rejected **Problem:** Committing fails with a message about a stale diff, or a batch failure reads *"diff went stale — re-open the wizard and retry."* **Solution:** Someone edited the target after you opened the wizard, so the commit was rejected rather than applied to something you didn't review. Click **Re-fetch diff**, review the affected entity again, and commit. ### Some Entities Failed in a Batch Commit **Problem:** **Merge complete** lists failures alongside successful commits. **Solution:** * The entities that succeeded are applied and don't need redoing. * Re-open the wizard, which will now show only the remaining differences, and retry the failures. * If the same entity fails repeatedly, check the reason listed and contact support if it isn't actionable. ### Removals Didn't Apply **Problem:** You expected fields to be deleted from the target and they're still there. **Solution:** Check that each removal row was toggled on before you committed. If the target should match the source exactly, use Overwrite instead. ## Related Topics When Merge is the right choice Getting to the Merge wizard The same walkthrough, read-only Edit entity settings that aren't part of a merge # Running Pipelines Source: https://support.entegrata.com/data/pipeline/running-pipelines How to execute and monitor pipeline jobs in Entegrata ## Overview Running a pipeline executes all configured data type mappings, pulling data from your Collector sources and loading it into your Entegrata data warehouse. Pipelines can run manually on-demand or automatically based on configured triggers. ## Execution Modes Entegrata supports two pipeline execution modes: ### Standard Run A standard run executes the pipeline and writes data to your production data warehouse. Use this for: * Production data processing * Scheduled automatic executions * Final data loads after validation ### Dry Run A dry run executes the pipeline logic without writing data to the warehouse. Use this for: * Testing new pipelines before deployment * Validating field mappings * Troubleshooting errors * Verifying source data quality Always perform a dry run before deploying a new pipeline or after making significant mapping changes. This helps catch errors without impacting production data. ## Running a Pipeline Manually Log in to the [Entegrata Admin Portal](https://admin.entegrata.com) and go to the **Pipelines** tab. Find the pipeline you want to run using the search bar or by browsing the list. Pipeline list with target pipeline Before running, verify the pipeline status: * **Draft**: Pipeline must be deployed before running * **Deployed**: Ready to run * **Running**: Already executing (wait for completion) * **Paused**: Can be run manually even when paused If the pipeline is already running, you cannot start another execution. Wait for the current run to complete. Click the three-dot menu (⋮) in the Actions column for the pipeline. Pipeline actions menu From the actions menu, choose: * **Run** - Execute with data writes (production run) * **Dry Run** - Execute without data writes (test run) Run options in actions menu After selecting Run or Dry Run, the pipeline execution begins immediately. You'll see: * Success message confirming the job was queued * Pipeline status changes to **Running** * Execution start time is recorded While the pipeline runs, monitor its progress: * Status badge shows "Running" with progress indicator * Click the pipeline name to view detailed logs * Refresh the page to see updated status Pipeline showing Running status Once execution completes, the status updates to: * **Success** (green badge) - Completed without errors * **Failed** (red badge) - Encountered errors The "Last Run" column shows the execution timestamp. ## Scheduled Execution Pipelines with scheduled triggers run automatically without manual intervention. ### Configuring Schedules When creating or editing a pipeline, select **Scheduled** as the trigger type and provide a cron expression: **Common Schedules:** * `0 0 * * *` - Daily at midnight * `0 2 * * *` - Daily at 2 AM * `0 */6 * * *` - Every 6 hours * `0 9 * * 1-5` - Weekdays at 9 AM * `0 0 1 * *` - First day of each month at midnight Scheduled pipelines must be in **Active** status to run automatically. Paused or Draft pipelines won't execute on schedule. ### Managing Scheduled Pipelines To control scheduled execution: * **Pause** - Temporarily disable automatic runs (pipeline remains deployed) * **Activate** - Re-enable automatic runs * **Edit Schedule** - Modify the cron expression through trigger settings Learn how to pause and activate pipelines ## Event-Driven Execution Event-driven pipelines run automatically when specific events occur, such as: * New data detected in source systems * External API calls or webhooks * Completion of upstream pipelines * Manual triggers from external systems Event-driven execution requires additional configuration with your Collector and may not be available for all data sources. Contact your administrator for setup assistance. ## Execution Monitoring ### Understanding Status Indicators Pipeline completed without errors. All data types processed successfully. Pipeline is currently executing. Check back for completion status. Pipeline encountered errors during execution. Review logs for details. Pipeline is queued for execution but hasn't started yet. ### Execution Metrics Key metrics to monitor: **Execution Time** * How long the pipeline took to complete * Helps identify performance issues or bottlenecks **Row Counts** * Rows read from each source * Rows written to each data type * Helps validate data volumes **Success Rate** * Percentage of successful vs. failed runs * Indicator of pipeline stability **Last Run Time** * When the pipeline last executed * Helps verify schedules are working ## Force Run vs. Incremental Run ### Force Run (Full Refresh) Processes all data from sources, regardless of when it was last processed. Use for: * Initial pipeline deployment * After structural changes to mappings * Recovery from errors * Data quality fixes Full refresh runs can be time-consuming and resource-intensive for large datasets. Use sparingly in production. ### Incremental Run (Default) Processes only new or changed data since the last successful run. Use for: * Regular scheduled executions * Efficient ongoing data synchronization * Minimizing processing time and costs Incremental processing requires proper configuration of change tracking or timestamp fields in your source data. ## Troubleshooting Pipeline Runs ### Pipeline Won't Start **Problem:** Clicking Run doesn't start execution. **Solutions:** * Verify pipeline is deployed (not in Draft status) * Check if pipeline is already running * Ensure you have run permissions * Verify Collector sources are connected and accessible ### Pipeline Fails Immediately **Problem:** Pipeline status changes to Failed within seconds. **Solutions:** * Review error logs for specific error messages * Verify source connections are active * Check for missing required field mappings * Ensure data types have valid configurations * Run in dry-run mode to isolate issues ### Pipeline Runs Too Long **Problem:** Pipeline takes much longer than expected. **Solutions:** * Check source data volumes (unexpected growth?) * Review field mappings for inefficient transformations * Verify source queries don't have missing filters * Consider breaking into smaller pipelines * Check for network or database performance issues ### Pipeline Succeeds But Data Is Wrong **Problem:** Pipeline completes successfully but data doesn't look right. **Solutions:** * Run in dry-run mode and examine query logic * Verify field mappings are pointing to correct source fields * Check transformation logic (COALESCE, CONCAT, CASE) * Review default values for unexpected overrides * Validate source data quality ### Dry Run Succeeds, Standard Run Fails **Problem:** Dry run works but standard run encounters errors. **Solutions:** * Check warehouse permissions and write access * Verify storage quotas haven't been exceeded * Review data type constraints in warehouse schema * Check for concurrent processes causing locks * Examine differences in dry-run vs. standard execution paths ## Best Practices **Test with Dry Run First** Always perform a dry run before: * Deploying a new pipeline * Making significant mapping changes * Processing in a new environment * Recovery after errors Dry runs catch issues without impacting production data. **Monitor First Runs Closely** When running a pipeline for the first time: * Watch execution logs in real-time * Verify row counts match expectations * Check data quality in the warehouse * Be prepared to pause or stop if issues arise **Schedule During Low-Usage Hours** For scheduled pipelines: * Run during off-peak hours (early morning, weekends) * Avoid business-critical hours * Consider time zones if processing global data * Allow buffer time before business day starts **Set Up Alerts** Configure monitoring alerts for: * Failed pipeline executions * Pipelines running longer than expected * Zero rows processed (may indicate source issues) * Repeated failures over multiple runs **Document Execution Patterns** Keep notes about: * Typical execution times * Expected row counts * Known seasonal variations * Dependencies on other systems This helps identify anomalies quickly. ## Execution Frequency Guidelines ### Daily Pipelines Best for data that changes frequently and needs to be current: * Client and account information * Transaction data * Daily metrics and KPIs ### Weekly Pipelines Best for less time-sensitive data or data-intensive processes: * Historical aggregations * Complex analytical calculations * Archive and cleanup operations ### Monthly Pipelines Best for periodic reporting data: * Month-end calculations * Historical trend analysis * Regulatory reporting data ### On-Demand Pipelines Best for ad-hoc or conditional processing: * Data migrations * Backfill operations * Testing and development * Manual data corrections ## Related Topics Learn how to deploy pipelines before running Control pipeline active/paused status Set up pipelines with proper triggers Configure mappings that pipelines execute # Searching and Filtering Pipelines Source: https://support.entegrata.com/data/pipeline/searching-pipelines How to efficiently find and filter pipelines in Entegrata ## Overview As your pipeline library grows, efficiently finding specific pipelines becomes essential. Entegrata provides search and filtering capabilities to help you quickly locate pipelines based on name, status, and other criteria. ## Search Functionality The pipeline list includes a search bar that allows you to filter pipelines by name in real-time. ### Accessing Search The search bar is located at the top of the pipeline list view, just below the page title and "Create New Pipeline" button. Pipeline list with search bar highlighted ### How Search Works The search functionality provides: * **Real-time filtering** - Results update as you type * **Partial matching** - Find pipelines containing your search term anywhere in the name * **Case-insensitive** - Search works regardless of capitalization * **Instant results** - No need to press Enter or click a search button Search currently filters by pipeline name only. Searching descriptions, keys, or other fields is not supported at this time. ## Using the Search Bar Go to the [Entegrata Admin Portal](https://admin.entegrata.com) and click the **Pipelines** tab. Find the search input field at the top of the pipeline list, below the page header. Empty search bar in pipeline list Click in the search bar and start typing your search query. Results filter automatically as you type. **Search Examples:** * "client" - Shows all pipelines with "client" in the name * "daily" - Shows all pipelines with "daily" in the name * "test" - Shows all test pipelines * "v2" - Shows all version 2 pipelines Search bar with active search query The pipeline list automatically updates to show only matching pipelines. All columns and actions remain available for filtered results. If no pipelines match your search, the list will be empty. Try a shorter or more general search term. ## Search Strategies ### Finding Specific Pipelines When you know the exact pipeline name or a unique part of it: **Example:** Searching for "Client Data Processing" * Type: "client data" or "processing" * Result: Finds the specific pipeline quickly ### Finding Groups of Related Pipelines Use common naming elements to find groups: **Example:** Finding all test pipelines * Type: "test" * Result: All pipelines with "test" in the name **Example:** Finding all daily pipelines * Type: "daily" * Result: All pipelines that run daily ### Finding Pipelines by Version If you use version numbers in pipeline names: **Example:** Finding all version 2 pipelines * Type: "v2" or "version 2" * Result: All pipelines marked as version 2 ### Finding Pipelines by Data Domain If you include data domains in names: **Example:** Finding all client-related pipelines * Type: "client" * Result: Client Data Pipeline, Client Migration, etc. **Example:** Finding all portfolio pipelines * Type: "portfolio" * Result: All pipelines processing portfolio data ## Search Best Practices **Use Consistent Naming Conventions** Establish naming patterns that make searching easier: * **Data Domain + Frequency + Type** * "Client Daily Refresh Pipeline" * "Portfolio Weekly Aggregation" * "Transaction Real-time Sync" * **Environment + Purpose** * "Production Client Pipeline" * "Test Migration Pipeline" * "Dev Data Quality Pipeline" * **Version Indicators** * "Client Pipeline v1" * "Client Pipeline v2" * "Legacy Client Pipeline" Consistent naming makes it easier to find related pipelines using search. **Start with Broad Searches** If you're not sure of the exact name: 1. Start with a general term (e.g., "client") 2. Review the results 3. Add more specific terms if needed (e.g., "client daily") This approach helps when you don't remember the full pipeline name. **Use Keywords from Descriptions** If you include keywords in pipeline descriptions, remember to use those same keywords in names for searchability. Since search only works on names, important identifying information should be in the name. **Example:** * Good: "Salesforce Client Daily Pipeline" * Less searchable: "Pipeline 1" (with "Salesforce Client Daily" only in description) **Document Search Terms** For team efficiency, document common search terms in your team wiki: * How to find test pipelines * How to find pipelines by data source * How to find pipelines by schedule * Naming conventions to use This helps team members find pipelines quickly. ## Pagination and Search The pipeline list supports pagination to handle large numbers of pipelines efficiently. ### How Pagination Works with Search * **Without Search** - Navigate through all pipelines using page controls * **With Active Search** - Navigate through filtered results only * **Page Reset** - Starting a new search resets to page 1 * **Results Per Page** - Typically 10 pipelines per page When you search, the page automatically resets to page 1 of the filtered results. This ensures you see all matching pipelines starting from the beginning. ## Combining Search with Sorting While viewing search results, you can still use column sorting: 1. Perform your search to filter pipelines 2. Click column headers to sort filtered results 3. Sort by: * Name (alphabetically) * Last Modified (most/least recent) * Last Run (most/least recent) * Status Combining search with sorting is powerful: * Search for "client" to find all client pipelines * Sort by Last Modified to see which were updated recently * Sort by Last Run to see which haven't run in a while ## Alternative Finding Methods Beyond search, use these methods to locate pipelines: ### Recently Modified Sort the full list by "Last Modified" column (click header) to see: * Pipelines you or others recently changed * New pipelines at the top * Oldest pipelines at the bottom ### Recently Run Sort by "Last Run" column to see: * Active pipelines that ran recently * Idle pipelines that haven't run in a while * Never-run pipelines (useful for finding orphaned test pipelines) ### Status-Based Organization Visually scan for status badge colors: * **Green** - Active, successful pipelines * **Yellow** - Paused pipelines * **Red** - Failed pipelines * **Gray** - Draft pipelines * **Blue** - Running pipelines ### Manual Browsing For smaller pipeline lists (under 50): * Scan the full list visually * Group mentally by naming patterns * Use browser find (Ctrl+F or Cmd+F) to search current page ## Troubleshooting Search ### No Results Found **Problem:** Search returns no pipelines. **Solutions:** * Check spelling of search terms * Try shorter or more general search terms * Remove extra spaces * Try searching for different parts of the pipeline name * Clear search and verify the pipeline exists in the full list ### Wrong Pipelines in Results **Problem:** Search returns pipelines you didn't expect. **Solutions:** * Search is partial match - "test" finds "Latest Client Pipeline" * Make search more specific * Use unique terms from the pipeline name * Consider if unexpected results share terminology ### Search Not Updating **Problem:** Typing in search bar doesn't filter results. **Solutions:** * Ensure you're typing in the correct search input field * Refresh the page * Clear browser cache * Check for JavaScript errors in browser console * Try a different browser ### Search Cleared Unexpectedly **Problem:** Search terms disappear or reset. **Solutions:** * Navigating between pages may reset search (by design) * Leaving and returning to Pipelines tab clears search * Page refresh clears search * This is expected behavior - search is session-based ## Future Search Enhancements While current search is name-only, potential future enhancements may include: * **Description Search** - Find pipelines by description content * **Multi-field Search** - Search across name, description, and keys * **Advanced Filters** - Filter by status, last run date, trigger type * **Saved Searches** - Save common search queries * **Search History** - Quick access to recent searches * **Regex Support** - Advanced pattern matching Feature requests for search improvements can be submitted through the [Support Portal](https://support.entegrata.com). ## Search Tips by Use Case ### Finding Test Pipelines for Cleanup **Goal:** Identify test pipelines to delete **Strategy:** 1. Search "test" 2. Sort by Last Run 3. Identify pipelines that haven't run in 30+ days 4. Review and delete as appropriate ### Finding Pipelines by Source System **Goal:** Find all pipelines using a specific data source **Strategy:** 1. Include source system name in pipeline names 2. Search for source system (e.g., "salesforce", "sap", "workday") 3. Review all pipelines using that source **Example Naming:** "Salesforce Client Pipeline", "SAP Finance Daily" ### Finding Pipelines to Update **Goal:** Find pipelines that need configuration updates **Strategy:** 1. Search for pipelines by version (e.g., "v1") 2. Or search for pipelines by data domain 3. Sort by Last Modified 4. Update outdated configurations ### Finding Failed Pipelines **Goal:** Identify pipelines needing attention **Strategy:** 1. Don't use search initially 2. Sort by Status to group failed pipelines 3. Or sort by Last Run and look for red badges 4. Address failures systematically ## Related Topics Back to pipeline management overview Name pipelines for easy searching Update names to improve searchability Remove pipelines found through search # Pipelines Overview Source: https://support.entegrata.com/data/pipeline/summary Comprehensive guide to Entegrata pipeline management and data mapping capabilities ## Introduction The Entegrata Pipeline system enables you to create, configure, and manage data pipelines that transform and load data from your Collector sources into the Entegrata platform. Pipelines provide a powerful framework for defining how your source data maps to standardized data types, allowing you to test configurations before deploying them to production. Pipelines are the foundation of your data integration strategy. They define the transformation logic that converts raw data from your sources into structured, queryable data types. ## What are Pipelines? Pipelines in Entegrata serve two primary functions: 1. **Data Orchestration** - Define when and how your data processing jobs execute 2. **Data Mapping** - Configure the transformation logic that maps source data to canonical data types Each pipeline contains one or more data type mappings that specify: * Which data sources to pull from * How fields map between source and target schemas * Transformation logic for data cleansing and enrichment * Default values and data quality rules ## Key Features Create, edit, duplicate, and delete pipelines with full version control and audit history Use the intuitive drag-and-drop interface to map source fields to data type fields Run pipelines in test mode to validate mappings without affecting production data Configure pipelines to run automatically on schedules or trigger them manually Map data from multiple sources into a single data type with primary and related sources Use COALESCE, CONCAT, CASE statements, and custom SQL for complex field mappings ## Pipeline Workflow The typical pipeline workflow follows these stages: Set up a new pipeline with a name, description, and trigger configuration Define which data types (entities) this pipeline will process Connect to your Collector data sources and select the tables or resources to use Use the visual editor to map source fields to data type fields, applying transformations as needed Run the pipeline in dry-run mode to validate your mappings and catch errors Publish your pipeline to production, generating optimized DLT scripts Track pipeline execution status, view logs, and troubleshoot issues ## Pipeline vs. Data Mapping It's important to understand the relationship between pipelines and data mappings: * **Pipeline** - The container that organizes and schedules data processing jobs. A pipeline can include multiple data type mappings. * **Data Mapping** - The specific configuration that maps one or more source tables to a single data type (entity). Think of a pipeline as a project that groups related data mappings together. For example, you might have a "Client Data Processing Pipeline" that includes mappings for Client, Contact, and Address data types. ## Getting Started Learn how to create and manage pipelines Explore the visual mapping editor and field configuration Step-by-step guide to creating your first pipeline Execute and monitor pipeline jobs ## Common Use Cases ### Daily Client Data Refresh Create a scheduled pipeline that runs nightly to pull updated client information from your CRM, mapping it to standardized Client and Contact data types. ### Historical Data Migration Build a one-time pipeline to migrate legacy data, using complex field mappings to handle data quality issues and schema differences. ### Multi-System Integration Configure pipelines that combine data from multiple sources (e.g., CRM, ERP, billing systems) into unified data types for comprehensive reporting. ### Incremental Updates Set up event-driven pipelines that process only new or changed records, ensuring your Entegrata data stays current with minimal processing overhead. ## Next Steps Ready to start building pipelines? Choose your path: Follow our quickstart guide Learn about data type management Explore transformation options # Sync Overview Source: https://support.entegrata.com/data/pipeline/sync-overview How Sync moves data mapping configuration from one pipeline into another, including terminology, scopes, and limits **Sync is a beta release feature.** If you don't see it in your subscription, contact your Entegrata Customer Experience Manager to ask about having it enabled. ## Overview **Sync** lets you take the data mapping work you've built on one pipeline and move it into another — for example, from a test pipeline into your production pipeline — without rebuilding any of it by hand. A **pipeline** is a working copy of your data mapping configuration: the entities, the fields on them, where each field comes from, and how they relate to each other. Most firms keep more than one: * A **production** pipeline (usually called **main**) that feeds real reporting. * One or more **test** pipelines where new mapping work gets built and validated safely. Sync solves the last mile. Once you've proven a change on a test pipeline, you need the same change in production — and redoing that work by hand is slow and easy to get wrong. Sync moves **mapping configuration only** — the definitions of your entities and fields. It never moves your data, and it never changes the pipeline you're syncing *from*. ## Key Terms The pipeline the configuration comes **from**. Never modified by a sync. The pipeline the configuration goes **into**. This is the one that changes. How much of the source you bring across, and how much say you get over the details. Three options — see [Sync Strategies](/data/pipeline/sync-strategies). ## The Two Directions Both directions are supported, and both are common: * **Test → main.** You built something new on a test pipeline and you want it live. * **Main → test (Sync from main).** Your test pipeline has drifted from production and you want to reset it before starting fresh work. ## Two Scopes Sync works at two scopes. They are not the same feature at different sizes — they differ in what they can add, what they can remove, and what they check. | | Single entity | Whole pipeline | | - | - | - | | **What moves** | Only the entity you picked | Every entity in the pipeline | | **Where you start** | Inside a pipeline layer, the **⋮** on an entity row | Pipelines list → **⋮** → **Manage** | | **Strategies** | All three, applied in place. **Merge requires the entity to already exist in both pipelines** | All three. Choosing **Merge** opens the review wizard rather than applying anything | | **Adding an entity the target doesn't have** | Overwrite and Overlay only | Yes — Overwrite and Overlay create it, and Merge offers it as an explicit choice | | **Removing an entity the source doesn't have** | **Never.** Single-entity sync replaces that entity's fields and sources; the entity itself is never deleted | Overwrite deletes it; Overlay leaves it and reports **Skipped**; Merge asks you | | **Entities that reference each other** | Not checked | Move together, so relationships stay consistent | Start with single-entity sync when you want a surgical change and you know the entity isn't entangled with others. Use whole-pipeline sync when entities are interrelated, or as your default when you're syncing a body of work rather than one change. Move just one entity Sync an entire pipeline into another ## Where to Find Sync Log in to the [Entegrata Admin Portal](https://admin.entegrata.com) and open **Data Pipelines** from the left sidebar. You'll land on a page listing your pipelines, split into production and test tables. ### On an entity Open a pipeline, then open a layer — for example **Silver** — and click the three-dot menu (⋮) on any entity row: * **Sync** — copy just this entity into another pipeline. * **Diff** — compare just this entity against another pipeline, changing nothing. ### On a pipeline Find the pipeline you want to sync **from**, click the three-dot menu (⋮) at the right of its row, and hover **Manage**: * **Duplicate** — make an independent copy of this pipeline (not part of Sync). * **Diff** — compare this pipeline against another and change nothing. * **Sync** — push this pipeline's mappings into another pipeline. * **Sync from main** — the reverse: pull production's mappings *into* this pipeline. The main pipeline's own actions menu has no **Manage** submenu. To sync *into* main, start from the pipeline you're syncing *from*. ## Publishing After a Sync **A sync updates mapping configuration. It does not change what your pipeline is currently running.** Until the target pipeline is **published**, it keeps running the mappings it had *before* your sync. The new configuration is saved, safe, and visible in the UI — but it isn't in effect. Two consequences are worth planning around: * If a **scheduled run** fires between your sync and your publish, it runs the **old** configuration. * A target that was **never** published won't run at all, and may report that it has no pipeline nodes. You have two ways to handle it: 1. **Tick the publish checkbox in the Sync dialog** — **Publish the target pipeline after syncing** (or **Publish `{target}` after committing** in the Merge wizard). It's off by default, and publishes immediately after the sync is saved. 2. **Publish manually afterward**, using the **Publish** action on the target pipeline. This is the right choice when you want to sync several changes, verify them together, and publish once. If the sync succeeds but the publish fails, the sync is **still saved** — nothing is rolled back. The banner says so, and names the failing entities: > "The sync was saved, but publishing failed." > > "`{target}` still runs the mappings it had before the sync. Publish it again to pick up these changes." **Publishing again is the fix. You do not need to re-sync.** Resolve whatever the failing entities report — usually a validation problem in the mapping itself — and publish the target again. What publishing does and what it requires Activate a pipeline for scheduled runs ## Limits and What Sync Does Not Do Being explicit so nothing is a surprise: * **Source and target must be in the same firm and environment.** You cannot sync across firms. * **The source is never modified.** Sync is strictly one-directional per run. * **Data is never moved** — only mapping configuration. * **Sync does not publish unless you ask it to.** * **Access control entities are never synced.** They're deliberately excluded so a target can't be left in an inconsistent access state. * **Merge decisions are per item, not per attribute.** You accept or reject a whole field, not individual attributes of it. * **Renaming can look like a delete plus an add.** Sync matches items by their stable key, so if something was renamed on one side, the comparison sees one item removed and a different one added rather than a rename. * **A batch Merge commit is not all-or-nothing.** Entities are committed one at a time, so if one fails partway through, the ones before it stay applied. ## Best Practices **Diff Before You Sync** Especially the first few times, and especially before an Overwrite into production. [Comparing Pipelines](/data/pipeline/comparing-pipelines) is completely read-only — use it to answer "what's actually different between test and production right now?" before deciding whether to sync at all. **Match the Strategy to Your Intent** * **Overlay** when you're adding to production and don't intend to remove anything. It's the strategy that can't surprise you. * **Overwrite** for resets — pulling main down onto a drifted test pipeline is exactly what it's for. * **Merge** when both pipelines have moved on. If real work has happened on production *and* on test, neither Overwrite nor Overlay expresses what you want. **Write Reasons a Stranger Could Read** Every sync requires a reason, and it's saved to the target's change history. Write something a colleague would understand in three months: "Syncing new billing-attorney fields validated on test." **Decide About Publishing Before You Click Sync** Sort this out up front rather than discovering it after a scheduled run has already used the old configuration. **Sync Whole Pipelines When Entities Are Interrelated** Single-entity sync is precise but doesn't check the entities that reference the one you're moving. ## Related Topics Overwrite, Overlay, and Merge compared Review differences without changing anything Approve differences one at a time Create an independent copy instead of syncing # Sync Strategies Source: https://support.entegrata.com/data/pipeline/sync-strategies Understand Overwrite, Overlay, and Merge, and choose the right strategy for each sync ## Overview Choosing a strategy is the one decision in a sync that matters most, so it's worth understanding before you click anything. Sync compares the source and the target **as they are right now**. It has no memory of who changed what, or when — so it can't guess your intent. Your strategy choice *is* the intent. Every difference Sync finds falls into one of three buckets, and each strategy handles those buckets differently: | The difference | Overwrite | Overlay | Merge | | - | - | - | - | | In the source, **not** in the target | Added | Added | You choose | | In both, but **different** | Target takes the source's value | Target takes the source's value | You choose | | In the target, **not** in the source | **Deleted** | Left alone | You choose | ## Overwrite *"Make the target match the source."* The target ends up an exact copy of the source's mapping configuration. Anything in the target that doesn't exist in the source is **deleted**. **Overwrite is destructive.** Mappings in the target pipeline that don't exist in the source will be deleted. Use it when the source is unambiguously the truth — resetting a test pipeline from main, or syncing a test pipeline you know is complete. Don't use it on production if production contains mapping work that isn't also on your test pipeline. Overwrite is the default strategy when you open the Sync dialog. ## Overlay *"Add and update, never delete."* Everything new or changed in the source lands in the target. Anything the target has on its own stays exactly where it is. This is the safe choice for the common case: you added three fields on a test pipeline and want them in production, and you don't want to risk anything else. Overlay **cannot express a deletion.** If you deliberately removed a field on your test pipeline and want it gone from production too, Overlay won't do it — use Merge or Overwrite. When Overlay finds an entity that only the target has, it leaves it alone and reports it as **Skipped** in the sync result. ## Merge *"Show me every difference and let me decide."* Merge walks you through the differences one entity at a time, and within each entity, one item at a time. For each one you either accept the suggested change or leave the target alone. **Nothing is applied until you commit.** This is the precise option — slower, but you see and approve everything. It's also the only strategy that lets you mix decisions: bring two fields across, leave a third, and remove a fourth, all in a single pass. Walk through the Merge review wizard step by step ## Choosing a Strategy | Your situation | Use | | - | - | | Test pipeline is complete and should replace production wholesale | **Overwrite** | | Resetting a drifted test pipeline from main | **Overwrite** | | Adding new fields or entities to production, removing nothing | **Overlay** | | You're not sure what's in the target and don't want to risk deletions | **Overlay** | | Real work has happened on *both* pipelines | **Merge** | | You need to delete something from the target, but not everything | **Merge** | | You want to review every difference before anything is written | **Merge** | When you select a strategy in the Sync dialog, a matching banner confirms what it will do — read it before you continue. It's the last checkpoint before a destructive change. ## Related Topics What Sync is and where to find it Apply a strategy to one entity Apply a strategy to a whole pipeline See the differences before choosing # Syncing a Single Entity Source: https://support.entegrata.com/data/pipeline/syncing-entities Copy one entity’s mapping configuration into another pipeline without moving the whole pipeline ## Overview You don't have to move a whole pipeline. Single-entity sync copies just one entity's mapping configuration into another pipeline — useful when you want a surgical change and you know the entity isn't entangled with others. Single-entity sync uses the same target picker, the same three strategies, the same required reason, and the same publish checkbox as a whole-pipeline sync. It's the pipeline dialog narrowed to one entity. ## Syncing an Entity Open the pipeline you're syncing **from**, then open a layer — for example **Silver**. Click the three-dot menu (⋮) on the entity row. You'll see: * **Sync** — copy just this entity into another pipeline. * **Diff** — compare just this entity against another pipeline, changing nothing. Click **Sync** to open the dialog. Choose where the entity is going. The source pipeline is never offered as its own target. The same three strategies apply, scoped to this one entity. The Overwrite warning reads: > "Mappings on this entity in the target pipeline will be replaced with the source's." **Reason for Sync** is required for Overwrite and Overlay. It's saved to the entity's change history in the target. The publish checkbox is off by default. Until the target is published, it keeps running the mappings it had before the sync — see [Publishing After a Sync](/data/pipeline/sync-overview#publishing-after-a-sync). Click **Sync**. The result reports the outcome for this entity, and the target's change history records a new version. ## What Single-Entity Sync Can and Can't Do **Merge requires the entity to exist in both pipelines.** To add a brand-new entity to the target, or to remove one, use Overwrite or Overlay — or use the pipeline-wide [Merge wizard](/data/pipeline/reviewing-sync-changes), which handles whole-entity add and remove as explicit choices. * **Overwrite never deletes the entity itself.** It replaces that entity's fields and sources. Other entities in the target are untouched. * **Entities can depend on each other.** If the entity you're syncing is referenced by others through relationships, syncing it alone can leave those references pointing at something that changed. Sync does not check this at the entity scope. * **Only the entity you picked moves.** Nothing else in the target is added, updated, or removed. **When in Doubt, Sync the Whole Pipeline** Whole-pipeline sync moves related entities together, so relationships stay consistent. Reach for single-entity sync when you're confident about the entity's isolation. ## After Syncing an Entity 1. **Check the entities that reference it.** Open the target pipeline and review any entity with a relationship to the one you just synced. 2. **Publish the target** if you didn't tick the checkbox, otherwise the target keeps running its previous configuration. 3. **Review the change history** on the entity in the target to confirm the new version landed with your reason attached. ## Best Practices **Diff the Entity First** The entity-level **Diff** action is read-only and shows exactly what a sync would change. Use it before an Overwrite into production. **Prefer Overlay for Additive Changes** If you added fields to an entity on test and want them in production, Overlay brings them across without touching anything else on that entity. ## Troubleshooting ### The Entity Doesn't Exist in the Target **Problem:** You want to add a new entity to the target and Merge won't let you. **Solution:** Merge requires the entity on both sides. Use Overwrite or Overlay to create it, or use the pipeline-wide Merge wizard, which offers whole-entity additions as an explicit choice. ### Related Entities Broke After Syncing **Problem:** After syncing one entity, entities that reference it behave unexpectedly. **Solution:** * Single-entity sync doesn't check dependent entities. Open each entity that references the synced one and verify its relationships and sources. * If several entities are entangled, re-sync at the pipeline scope so they move together. ### The Target Still Runs the Old Mappings **Problem:** The sync succeeded but reports and scheduled runs still reflect the old configuration. **Solution:** The target hasn't been published. See [Publishing After a Sync](/data/pipeline/sync-overview#publishing-after-a-sync). ## Related Topics Overwrite, Overlay, and Merge compared Move every entity at once Diff an entity without changing it See which source columns feed each entity field # Syncing Pipelines Source: https://support.entegrata.com/data/pipeline/syncing-pipelines How to sync an entire pipeline into another pipeline, including Sync from main and reading the results ## Overview A whole-pipeline sync moves every entity's mapping configuration from a source pipeline into a target pipeline in a single operation. Because related entities move together, relationships stay consistent — which is why this is the recommended default over syncing entities one at a time. Nothing in the source pipeline changes. A sync only writes to the target. ## Syncing a Pipeline From the pipelines list, click the three-dot menu (⋮) on the pipeline you're syncing **from**, hover **Manage**, and click **Sync**. A dialog opens, titled **Sync `{source}` to `{target}`**. A blue panel labeled **Syncing** shows **From** (the pipeline you clicked) and **To** (the target). Read this before anything else — it's the fastest way to catch a backwards sync. The **Target pipeline** dropdown defaults to your main pipeline, shown as `{name} (main)`. Change it if you're syncing somewhere else. The source pipeline is never offered as its own target. Three cards are available: **Overwrite**, **Merge (review changes)**, and **Overlay**. Selecting one shows a banner describing exactly what it will do. Overwrite deletes anything in the target that doesn't exist in the source. If you're unsure what the target contains, choose Overlay or Merge. The **Reason for Sync** box is **required** for Overwrite and Overlay — the **Sync** button stays disabled until you fill it in. Your reason is saved to the target's change history, so write something a colleague would understand in three months: > "Syncing new billing-attorney fields validated on test." If you chose Merge, you'll enter the reason later, at commit time. The **Publish the target pipeline after syncing** checkbox is **off** by default. Until the target is published, it keeps running the mappings it had before the sync. This is the step people miss — see [Publishing After a Sync](/data/pipeline/sync-overview#publishing-after-a-sync). Click **Sync** to apply. If you chose Merge, the button reads **Review changes** instead, and takes you into the review wizard rather than applying anything. A full-screen overlay reads `Syncing "{source}" to "{target}"...` while it runs. When it finishes you'll get a notification summarizing the outcome, for example: ```text theme={null} Synced "Test Pipeline" to "Main": 12 changed, 0 unchanged, 0 skipped. Published "Main". ``` If anything needs your attention — a skipped entity, a failed publish, or nothing having changed — a **Sync Result** window also opens. ## Sync from Main **⋮** → **Manage** → **Sync from main** opens the same dialog with the direction already reversed: **main is the source**, and the pipeline whose menu you opened is already selected as the target. Everything else works identically. This is the "reset my test pipeline to match production" shortcut. Overwrite is usually what you want here. **Sync from Main Before Starting New Work** Starting a change on a test pipeline that has drifted from production means you're building on a configuration that no longer reflects reality. Resetting first keeps your eventual sync clean. ## Reading the Results Every sync reports a per-entity outcome. A notification always appears; the **Sync Result** window opens on top of it when something is worth a closer look. The result table has three columns — **Entity**, **Outcome**, and **Reason**: | Outcome | What it means | | - | - | | **Changed** | The entity was updated in the target, and a new version was written to its change history. | | **Unchanged** | The entity was considered, but there was nothing to bring across. | | **Skipped** | The strategy deliberately left it alone. In practice this means Overlay found an entity that only the target has, and Overlay never deletes. The **Reason** column will say so. | If you asked to publish, a banner reports how that went separately from the sync itself — green (*"Published `{target}` — it now runs the synced mappings."*) or red, with the failing entities listed. ### Where to Look Later Every entity a sync changed gets an entry in the target's **change history**, stamped with your reason, your user, and the time. That history is a full snapshot of the entity, so a sync is reviewable — and revertible — after the fact. This is why the reason field is required. ## Best Practices **Diff First, Sync Second** Run **Manage → Diff** against the same target before syncing. It's read-only, it takes a minute, and it turns "I think this is right" into "I know what's about to change." **Verify the Target After Syncing** Open the target pipeline and spot-check the entities you expected to change. Confirm the publish banner was green before assuming the new configuration is live. ## Troubleshooting ### The Sync Button Is Disabled **Problem:** You've selected a target and strategy but **Sync** stays greyed out. **Solution:** * Fill in the **Reason for Sync** field — it's required for Overwrite and Overlay. * Confirm a target pipeline is selected in the dropdown. ### Nothing Changed **Problem:** The result reports `0 changed` and every entity as **Unchanged**. **Solution:** * The two pipelines already match for the strategy you chose. Run a Diff to confirm. * If you expected deletions to be applied, check whether you used Overlay — Overlay never deletes. Use Merge or Overwrite instead. * Confirm you saved your work on the source pipeline before syncing. ### Entities Reported as Skipped **Problem:** One or more entities show **Skipped** in the Sync Result window. **Solution:** * This is Overlay behaving as designed: it found entities that only exist in the target, and Overlay never deletes. Read the **Reason** column to confirm. * If you intended to remove those entities, re-run with Merge and explicitly turn on the removals. ### The Target Still Runs the Old Mappings **Problem:** The sync succeeded but reports and scheduled runs still reflect the old configuration. **Solution:** The target hasn't been published. See [Publishing After a Sync](/data/pipeline/sync-overview#publishing-after-a-sync). ### No Manage Submenu **Problem:** The pipeline's three-dot menu doesn't show **Manage**. **Solution:** The main pipeline has no **Manage** submenu by design. To sync *into* main, start from the pipeline you're syncing *from*. ## Related Topics Move just one entity instead Choose between Overwrite, Overlay, and Merge Approve differences one at a time Scopes, limits, and publishing # Entegrata User Group Source: https://support.entegrata.com/eug-info Interested in joining? Contact your Entegrata Customer Experience Manager for help; Entegrata does not manage the group and will facilitate an introduction to the group's cochairs. ## Mission The Entegrata User Group is a community for professionals actively implementing, developing, and adopting Entegrata at their firms. The group brings together peers across information technology, data architecture, knowledge management, innovation, and business development to share what's working, what isn't, and what comes next. ## Commitment One hour per quarter, plus optional conversation on LinkedIn between meetings. Engage at the level that fits your schedule. ## What You Get Direct access to peers at other firms solving similar implementation and adoption challenges A candid forum for sharing what's working, what isn't, and the lessons that don't make it into vendor documentation The benefit of collective experience across firms at different stages of their Entegrata journey A first call for problem-solving before you open a support ticket A seat at the table shaping the Entegrata product roadmap — your real-world experience influences what gets built next ## Interested in joining? Contact your Entegrata Customer Experience Manager for help; Entegrata does not manage the group and will facilitate an introduction to the group's cochairs. # Welcome to Entegrata Source: https://support.entegrata.com/index Your comprehensive guide to the Entegrata legal data platform ## What is Entegrata? Entegrata is a comprehensive legal data platform that enables law firms to collect, transform, secure, and analyze their critical business data. Built on Azure infrastructure, Entegrata provides a centralized data lakehouse that integrates seamlessly with your existing systems while maintaining the highest standards of security and compliance. Entegrata transforms disparate legal data sources into a unified, analytics-ready platform that powers business intelligence, reporting, and advanced analytics for modern law firms. ## Platform Documentation There are five core areas outlined in the documentation: ### Data Collection Automatically ingest data from your SQL databases, practice management systems, and cloud applications into a centralized lakehouse. Connect to Microsoft SQL Server, Iridium, Aderant, DealCloud, UKG, and other data sources. Discover resources, configure collection schedules, and monitor data synchronization jobs. ### Data Transformation Map and transform source data into standardized canonical models that power your analytics and reporting. Create data pipelines that map source fields to canonical entities. Configure transformations, manage data quality, and deploy pipelines to production. ### Data Emission Write curated, transformed data back out to the external systems your firm runs on — the inverse of collection. Ship Lakehouse data to Microsoft SQL Server, Fabric SQL, or Fabric Warehouse. Configure emitters, pick the resources to emit, and run full, incremental, or specific emissions on demand or on a schedule. ### Data Security Implement fine-grained access control to ensure users only see the data they're authorized to access. Define default permissions, create access rules for specific subjects and objects, manage user groups, and test permissions before deployment. ### Platform Administration Manage users, configure infrastructure, and maintain your Entegrata instance. Manage users with Azure Entra ID integration, configure network connectivity, set up firewall rules, manage encryption keys, and maintain system versions. ## How Entegrata Works Use the Collector to establish secure connections to your practice management systems, databases, and cloud applications. Entegrata automatically discovers available resources and manages data synchronization. Create pipelines that transform source data into standardized canonical entities like Client, Matter, Timekeeper, and Invoice. Define field mappings and transformations to ensure data consistency. Configure access control rules to implement your firm's data governance policies. Set default permissions, create granular rules, and organize users into groups for efficient permission management. Push curated Lakehouse data back out to external systems with emitters. Bind a pipeline layer to a destination, pick the resources you want to ship, and run full, incremental, or specific emissions. Your transformed, secured data is now ready for business intelligence tools, custom applications, and advanced analytics. Connect Power BI, Tableau, or other tools to your Entegrata lakehouse. Your transformed, secured data is now ready for business intelligence tools, custom applications, and advanced analytics. Connect Power BI, Tableau, or other tools to your Entegrata lakehouse. ## Key Features Automatically discover tables, views, and API endpoints from your data sources Efficient incremental loading for large datasets with change tracking Intuitive visual interface for field mapping and transformation configuration Row-level and column-level access control with dynamic group membership Native Azure Entra ID authentication and Azure infrastructure deployment Track collection jobs, pipeline runs, and data quality metrics in real-time ## Getting Started New to Entegrata? Start with these essential guides: Set up your Entegrata instance with network connectivity, firewall rules, and encryption Connect to your first data source and start collecting data Transform source data into canonical models Add users to your instance with Azure Entra ID integration ## Support & Resources Get help from the Entegrata support team Listen to insights on legal data and analytics Stay updated on Entegrata announcements and features Learn more about Entegrata's platform and services # Navigation Refresh Source: https://support.entegrata.com/overview/navigation-refresh What changed in the Admin Application navigation, and where to find the things you used before The Admin Application has a new navigation model. The pages and data are the same, but the page furniture around them moved: titles and tab strips gave way to a **dropdown breadcrumb** at the top of every page, and details that used to live on their own pages now open in a **side panel** without taking you away from your list. This page shows the old and new side by side, so you can find anything that moved. Nothing was renamed in your data. Connections, pipelines, entities, groups, and settings all keep their names and values. The changes here are to layout and navigation only, except where explicitly noted. ## The short version | If you used to… | Now… | | - | - | | Read the page title to know where you are | Read the breadcrumb at the top left | | Click an underlined tab strip | Click the last breadcrumb crumb and pick from its menu | | Go back to a list to switch connection, pipeline, or group | Click that name in the breadcrumb and pick another, with a search box | | Open a pipeline's **Statistics** page | Click **Insights** on the entities list | | Open a pipeline's **History** page | Click **History** on any pipeline page | ## Sidebar labels Two sidebar items were shortened, and the Pipelines icon changed. Sidebar with Data Connections and Data Pipelines Sidebar with Connections and Pipelines * **Data Connections** is now **Connections** * **Data Pipelines** is now **Pipelines** ## Connections ### The connections list The page title and the tab strip are gone. Search moved up into the same row as **New Connection**, which leaves noticeably more room for your connections. Connections list with a large title, breadcrumb, and tab strip Connections list with a breadcrumb and a single search row The **Collection**, **Emission**, and **Schedules** tabs are now in the breadcrumb. Click **Collection** to switch between them. Collection breadcrumb menu open showing Collection, Emission, and Schedules ### Connector health at a glance New: click **Overview** in the top right for a summary of your whole connection fleet, one bar per connector, coloured by health, with counts and a short summary underneath. Overview panel showing 46 connectors with active, error, and inactive counts The panel opens beside your list rather than on top of it, so you can keep working with the table while it is open. ### Searching connections Search works the same way and still matches both the connection name and its data source. Two things improved: results no longer re-query on every keystroke, and the term in the address bar is now a short, shareable `?search=` value. Connections list filtered to five Aderant results Connections list filtered to the same five results ### A single connection The connection name is no longer a heading; it is the middle crumb in the breadcrumb. The five tabs became a menu on the last crumb, and the actions moved into the top-right corner. Connection detail with a title and five underlined tabs Connection detail with breadcrumb menus and top-right actions Tables breadcrumb menu open showing all five connection views Views that do not apply to a connection are now left out of the menu instead of appearing greyed out. If a connection is inactive, you will not see **Access Control** listed. ### Jumping between connections New: the connection name in the breadcrumb is a searchable switcher. You no longer need to return to the list to open a different connection. Each entry shows a status dot, and the data source in parentheses where it differs from the name. Connection switcher listing connections with status dots Connection switcher filtered to two iManage connections ### Jobs and settings without leaving the page Two new panels open from the top-right of a connection. **Jobs** shows what is running now or what ran last, counts by state, and the most recent runs. **View all jobs** opens the full job history. Jobs panel with completed, errored, scheduled, and running counts **Settings** summarises the collection configuration as read-only text, so you can check a value without opening the editor. **View full settings** opens the editor when you do want to change something. Settings panel showing type, data retention, deletion, sync, and rate limit ### Finding a table or field quickly The old **Quick navigate to resource** box in the header is now a search palette. Open it with the magnifier in the top right, or press ⌘K (Ctrl+K on Windows). Resource search palette with keyboard hints Palette showing two invoice tables as results The palette searches the entire schema, not just the page you are looking at, so you can jump straight to a table that is several pages deep. Your filters and search on a connection's resource and field lists are now remembered when you switch tabs and come back. ## Pipelines ### The pipelines list Same change as connections: no page title, no tab strip, and search sits with **Create Pipeline**. Switch between **Pipelines** and **Catalog Queries** from the first crumb. Pipelines list with a title and a two-tab strip Pipelines list with a breadcrumb and search row ### Opening a pipeline goes straight to its entities Opening a pipeline used to land on a screen where you picked Silver before you could do anything. That step is gone: you now land directly on the Silver entities list. Pipeline landing page with a Silver card Silver entities list with Insights, History, and Run Pipeline The breadcrumb now shows which pipeline and layer you are in, so you always have a way back up without using the browser button. Old links still work. A bookmark to a pipeline, or to `/silver`, now forwards to the entities list. ### Running a pipeline **Run Pipeline** used to live on the Statistics page, which meant opening a pipeline, choosing a layer, and then switching tabs before you could start a run. It is now in the top right of the entities list, the page you land on. It also asks for confirmation before starting, naming the pipeline it is about to run. Run pipeline dialog naming the Main pipeline with Cancel and Run Pipeline ### Statistics is now Insights The Statistics page became the **Insights** panel, which opens beside your entity list instead of replacing it. The four counters and the pipeline status block carry over unchanged, and recent runs are summarised with a link to the full job history. Statistics page with counters, pipeline status, and an entity table Insights panel with the same counters and pipeline status ### History Pipeline history moved into a **History** panel available from any pipeline page, and it is aware of where you are: on a list it shows the pipeline's history, and on an entity it shows just that entity's history. History panel showing recent changes with a View all changes button Reverting works as before; the confirmation steps now happen inside one window rather than a chain of dialogs. ### Entity pages On an entity, the **Editor**, **Fields**, **Data Lineage**, and **Access Control** tabs are now a menu on the last crumb, and the entity name itself is a searchable switcher for moving between entities. The entity's **History** tab is now the History panel described above. ## Instance Settings ### The settings cards The cards are unchanged. Three of them, **Data Encryption**, **Timezone**, and **Log Analytics Retention**, now open in a window over the grid instead of replacing the page. Instance settings with a large title Instance settings with a breadcrumb These three now have their own links, so you can bookmark or share one directly. It opens with the card grid still behind it. Timezone dialog open over the instance settings grid ### Firewall settings All firewall functionality is unchanged. The **Data Sources** and **System** tabs moved into the breadcrumb, **Apply Changes** moved to the end of the breadcrumb row, and **Add Data Source** now sits beside a full-width search box. Firewall settings with a title, tab strip, and two header buttons Firewall settings with breadcrumb crumbs and a wide search box Which tab you are on is now part of the address, so you can link straight to the System tab. ### Network Connections Network Connections is in the same place, reached from the **Network Connections** card in Instance Settings. It picked up the same chrome as the rest of the application: the page title gave way to a breadcrumb, and search moved into the row with **Add Connection**. Network connections list showing a VNet peering connection The **Network Connections** crumb is a menu, so you can jump straight to Firewall Settings from here. Your search term is now part of the address as well. ## Users The **Users** heading became a breadcrumb, and the generic search box became a labelled **Search users** box next to **New User**. Users list with a large heading and a generic search box Users list with a breadcrumb and a labelled search box Searching now updates the address bar properly, so your browser's back button steps back through searches and a link to a filtered list opens already filtered. ## Access Control ### Groups and Users The **Groups** and **Actors** tabs became a menu on the last crumb. **Actors** is now labelled **Users** throughout this area, including the empty states and dialogs that used to say "actor". Groups list with breadcrumb and Search groups box View menu open showing Groups and Users ### A single group The group name is a searchable switcher, the member tabs became a menu, and the two tools in the corner now open beside the member list instead of covering it. Group detail with a title, tab strip, and header buttons Group detail with breadcrumb menus and corner actions **Test Permissions** and **Explore Access** were windows that covered the page. They are now panels, so you can read the member list and your test results at the same time. Test Permissions panel with subject, entity, and resource fields ## Things to know Most pages no longer show a large heading. The breadcrumb in the top left is now what identifies the page, which is why lists start higher up the screen and show more rows. Your browser tab still shows the page name. Where a tab used to appear greyed out with an explanation on hover, it is now simply absent from the menu. If you expect an option and cannot find it, check whether the connection is active and whether its prerequisites are met. On a wide screen, Insights, History, Jobs, Settings, and the Access Control panels narrow the page rather than covering it, so your list stays readable. On a narrower screen they slide over the content instead. They close when you navigate. Ctrl+Space jumps to the search box on any list. On a connection's resource pages, ⌘K opens the resource search palette. ## Questions If something you relied on is not where you expect, let us know through the support portal. Tell us what you were looking for and which page you expected to find it on. # Build Custom Reports Source: https://support.entegrata.com/power-bi/build-custom-reports Three ways to build your own Power BI reports on the data Entegrata delivers to your Databricks environment ## Overview You can build your own Power BI reports on the data Entegrata delivers to your Databricks environment. This page covers three ways to do it and which one to choose. ## Prerequisites * **Power BI Desktop.** [Download Power BI Desktop](https://www.microsoft.com/en-us/download/details.aspx?id=58494) if you do not have it. * **Build permission.** For option 1 you need **Build** permission on the semantic model. Ask the semantic model's owner or a workspace admin if you are not sure. * **License.** To publish and share reports, you need a Power BI Pro or Premium Per User license. Who can view them depends on your workspace; see [Licensing Patterns](/power-bi/licensing). * **Connection details.** For options 2 and 3 you need your Databricks connection details. Ask your Entegrata Customer Experience Manager for them. See [Set up your connection](#set-up-your-connection). ## Choose an Option | Option | Use it when | Trade-off | | - | - | - | | 1. Connect to an existing model | Your firm, or Entegrata, has already published a semantic model with the data you need. | Recommended. You start from a trusted model and can begin building straight away. | | 2. Connect to a Databricks view | You need a new model, for example one you plan to share with others. | Next best. You still build on curated, governed views. | | 3. Run a Databricks query | You are prototyping, and no view gives you what you need. | Most flexible, and the biggest risk to governance and stability. Use it only when options 1 and 2 cannot work. | For Microsoft's general guide to the Databricks connection, see [Connect Power BI Desktop to Azure Databricks](https://learn.microsoft.com/azure/databricks/partners/bi/power-bi/desktop). ## Option 1: Connect to an Existing Model You can build on a published semantic model in the Power BI service or in Power BI Desktop. Either way, the report uses a live connection to the model, so the model itself does not change. * **In the Power BI service:** create a new report from your workspace, pick the published semantic model, and select **Create a blank report**. Entegrata recommends this over **Auto-create report**. See [Microsoft Learn: Create reports based on semantic models from different workspaces](https://learn.microsoft.com/power-bi/connect-data/service-datasets-discover-across-workspaces). * **In Power BI Desktop:** select **Get data**, then **Power BI semantic models**, and connect to the model. See [Microsoft Learn: Connect to semantic models in the Power BI service from Power BI Desktop](https://learn.microsoft.com/power-bi/connect-data/desktop-report-lifecycle-datasets). ## Option 2: Connect to a Databricks View This option uses a shared query called **Common Schema**. It holds your connection details in one place, so every table you add uses the same connection. In Power BI Desktop, start a blank report and open **Transform data**. Create the four connection parameters described in [Set up your connection](#set-up-your-connection). Right-click in the Queries pane and select **New Query**, then **Blank Query**. Name it `Common Schema`, open the **Advanced Editor**, paste in the code below, and select **Done**. Then right-click the query and turn off **Enable load**. New Query menu with Blank Query selected ```powerquery theme={null} let Source = Databricks.Catalogs( DatabricksHost , DatabricksHTTPPath , [ Catalog = "" // Does not force a specific catalog , Database = "" // Does not force a specific database , EnableQueryResultDownload = "0" , Implementation = "2.0" ] ) , ref_database = Source{[ Name = Catalog , Kind = "Database" ]}[Data] , ref_schema = ref_database{[ Name = Schema , Kind = "Schema" ]}[Data] in ref_schema ``` Common Schema context menu with Enable load For each table or view you want in your model, right-click **Common Schema** and select **Reference**. In the new query, select the **Table** link in the Data column for the table or view you want. Common Schema context menu with Reference Common Schema query results with a Table link for each table in the Data column Make any adjustments each table needs, for example setting data types. ## Option 3: Run a Databricks Query This option uses a shared function called **Common Query Function**. You paste a SQL query into it and it returns the result as a table. Create the four connection parameters described in [Set up your connection](#set-up-your-connection), if you have not already. Create a blank query, open the **Advanced Editor**, paste in the code below, and name the query `Common Query Function`. ```powerquery theme={null} let Source = Databricks.Query( DatabricksHost , DatabricksHTTPPath , [ EnableQueryResultDownload = "0" , Implementation = "2.0" ] ) in Source ``` Databricks SQL Query function with SQL Query box and Invoke button Paste the query you want to run into the **SQL Query** box of the function and select **Invoke**. If the query runs, the results appear. Query results from the Common Query Function, with the query list on the left Repeat for each result set you want in your model. You can then load them as if they were views in Databricks. Always use fully qualified names in your queries: `catalog.schema.table`. ## Set Up Your Connection ### Connection Details Ask your Entegrata Customer Experience Manager for these four values. | Parameter | What it is | Example | | - | - | - | | `DatabricksHost` | The URL of your Databricks instance. | `adb-{workspace-id}.{number}.azuredatabricks.net` | | `DatabricksHTTPPath` | The part of the URL that points to your SQL warehouse. | `/sql/1.0/warehouses/{warehouse}` | | `Catalog` | The Unity Catalog that holds your schemas. | `{yourfirm}_gold` | | `Schema` | The schema inside the catalog that holds your tables and views. | `main` | ### Parameters Create one Power BI parameter for each value, using the names in the table above. The code in options 2 and 3 uses those names. See [Microsoft Learn: Using parameters](https://learn.microsoft.com/power-query/power-query-query-parameters). Four connection parameters in the Queries pane ### Required Connection Options The code above already sets both of these. Keep them if you write your own queries. Both are described in [Microsoft Learn: Azure Databricks connector](https://learn.microsoft.com/power-query/connectors/databricks-azure). * `EnableQueryResultDownload = "0"` handles firewall restrictions. * `Implementation = "2.0"` uses the newer ADBC driver instead of the ODBC driver, which is being retired. The ADBC driver is in preview. Databricks lists ADBC support for Power BI as a public preview, although Power BI uses it by default for new Databricks connections. See [Microsoft Learn: Transition from ODBC to ADBC drivers](https://learn.microsoft.com/power-query/transition-to-adbc). ## Sign In Power BI or Power BI Desktop may ask you to sign in, and you may need to complete two-factor authentication. Use your EntegrataOne account if you have been issued one. Otherwise, use your organization's credentials. ## Learn More * [Microsoft Learn: Power BI report creation](https://learn.microsoft.com/power-bi/create-reports/) * [Tour the report editor](https://learn.microsoft.com/power-bi/create-reports/service-the-report-editor-take-a-tour) * [Report view in Power BI Desktop](https://learn.microsoft.com/power-bi/create-reports/desktop-report-view) * [Interact with a report in Editing view](https://learn.microsoft.com/power-bi/create-reports/service-interact-with-a-report-in-editing-view) * [Add a filter to a report](https://learn.microsoft.com/power-bi/create-reports/power-bi-report-add-filter) * [Create Apply All and Clear All slicer buttons](https://learn.microsoft.com/power-bi/create-reports/buttons-apply-all-clear-all-slicers) * [Drillthrough in Power BI reports](https://learn.microsoft.com/power-bi/create-reports/desktop-drillthrough) * [Add visualizations to a report](https://learn.microsoft.com/power-bi/visuals/power-bi-report-add-visualizations) # Connect Excel Source: https://support.entegrata.com/power-bi/connect-excel Analyze your Entegrata data in Excel with a live PivotTable connected to your Power BI semantic model ## Overview You can analyze your Entegrata data in Excel with a live PivotTable that refreshes from the same semantic model your reports use. You can start from Excel, or from the Power BI service with **Analyze in Excel**. ## Prerequisites * A Power BI license. * **Build** permission on the semantic model. Ask the semantic model's owner or a workspace admin if you are not sure. * Your Power BI administrator must turn on the tenant setting **Users can work with semantic models in Excel using a live connection**. Entegrata recommends it; see [Recommended Tenant Settings](/power-bi/tenant-settings). ## Connect Microsoft documents each way to connect, and keeps the steps current as Excel changes: * [Microsoft Learn: Connect Excel to Power BI semantic models](https://learn.microsoft.com/power-bi/collaborate-share/office-integration/service-connect-power-bi-datasets-excel) covers every option. * [Microsoft Learn: Power BI semantic model experience in Excel](https://learn.microsoft.com/power-bi/collaborate-share/office-integration/service-connect-excel-power-bi-datasets) covers starting in Excel for the web or Excel desktop. * [Microsoft Learn: Create Excel workbooks with refreshable Power BI data](https://learn.microsoft.com/power-bi/collaborate-share/office-integration/service-analyze-in-excel) covers starting in the Power BI service. ## Security in Excel Row-level security applies in Excel the same way it does in your reports. People see only the data they are allowed to see in Power BI. # Data Freshness Source: https://support.entegrata.com/power-bi/data-freshness How your data reaches your Entegrata Power BI reports each day, and when to schedule your refresh ## Overview Your Entegrata reports show data that has passed through three steps. Knowing when each one runs tells you how current your reports are, and when to schedule your Power BI refresh. ## How Data Reaches Your Reports Entegrata connects to your firm's source systems and brings that data into your firm's lakehouse, on a schedule your firm sets. You can change that schedule at any time. See [Collection Settings](/data/collector/resources/collection-settings). At least once a day, Entegrata's automated jobs build the Gold layer that your Power BI reports are based on. This is usually early morning Eastern Time, but your Entegrata Customer Experience Manager can help you understand your firm's current time and update it as needed. Your semantic model refreshes on the schedule set in your Power BI workspace. Your firm manages this schedule. ## When to Schedule Your Refresh Schedule your Power BI refresh **after** a Gold layer build has finished. A refresh that runs before the build finishes only reloads the data from the previous build. Leave some margin after the build time. If your source systems are slow to deliver data, schedule the refresh later in the day. ## If a Source Update Fails If data from one of your sources could not be updated, the build still runs, but that source's data may not be current. Entegrata notifies your firm if this happens. ## Learn More * [Microsoft Learn: Data refresh in Power BI](https://learn.microsoft.com/power-bi/connect-data/refresh-data) * [Microsoft Learn: Configure scheduled refresh](https://learn.microsoft.com/power-bi/connect-data/refresh-scheduled-refresh) # Executive Dashboards Source: https://support.entegrata.com/power-bi/executive-dashboards What each page of the Entegrata Executive Dashboards shows, and who it is built for ## Overview The Executive Dashboards give managing partners and firm leadership a view of the firm's finances, people, and performance. Each page answers a different question. ## Home **The firm's financial pulse.** Tracks the full revenue pipeline, from fees worked to billed, collected, and net income, each against the prior period and budget. Collection health is the focus: turnover, days to bill, days to pay, and cash receipts. Leaderboards show top collectors and unapplied cash, and alert tiles flag aging AR and WIP that may need attention. *Built from:* revenue, receipts, timesheets, proformas and prebills, invoices, WIP, AR, clients, matters, timekeepers, offices, and practice groups. ## Inventory **Where WIP and AR are piling up, and who is holding them.** Breaks WIP and AR down by aging bucket, with month-over-month trend lines. Leaderboards rank attorneys, clients, and matters by outstanding balances. A separate view shows unreturned proformas. *Built from:* WIP, AR, proformas and prebills, invoices, revenue, clients, matters, timekeepers, offices, and practice groups. ## Productivity **Attorney utilization and hours.** Shows billable hours against targets by practice group and title, with FTE headcount trends for context. Includes time entry velocity, average billable hours per FTE per day, and days to release time. A leaderboard ranks attorneys by hours and target attainment, so leadership can see who is on pace and where capacity is short. *Built from:* timesheets, timekeepers, revenue, targets, offices, and practice groups. ## Rates **Rate realization and pricing health.** Tracks effective rates and realization over time, comparing standard, effective, and collected rates by title. A rate opportunity view finds timekeepers billing well below the firm average and sizes the revenue at stake. Alert tiles flag clients and matters with high write-downs or discounts. *Built from:* timesheets, write-offs and write-downs, revenue, timekeepers, rates, clients, matters, offices, and practice groups. ## Client **One client in depth.** Select a client to see their full financial profile: AR and WIP aging, collections against the prior year, realization, and leverage. Includes invoice history, open matters, timekeeper activity, and how long it has been since the last time entry, invoice, or payment. Built for partners preparing for a client conversation or reviewing account health. *Built from:* clients, revenue, WIP, AR, invoices, timesheets, timekeepers, offices, and practice groups. ## Matter **One matter in depth.** The Client page's view, scoped to a single matter. Shows WIP aging, collections, and realization, along with timekeeper activity, invoice history, and matter details such as practice group, office, and open date. Built for billing attorneys managing matter economics. *Built from:* clients, matters, revenue, WIP, AR, invoices, timesheets, timekeepers, offices, and practice groups. ## Timekeeper **One attorney's performance profile.** Shows a timekeeper's fees worked, billed, and collected over time, billing realization by year, and hours against target. Compares their utilization and realization with the firm average, and shows their WIP and AR aging, write-downs, top clients and matters, the partners they bill time to, and the timekeepers they work with most. Built for performance reviews and partner self-service. *Built from:* timekeepers, revenue, WIP, AR, timesheets, offices, practice groups, and diversity metrics. ## Performance **How new hires are tracking.** Lists recent lateral and associate hires by title, with annual hours and projections, salary, merit bonus, and total compensation. Shows leadership whether new investments in talent are on track for their revenue targets. *Built from:* timekeepers, revenue, WIP, AR, timesheets, offices, and practice groups. ## Headcount **Practice group growth.** Compares headcount and production across practice groups year over year, and flags groups with significant growth or decline. Helps leadership see where the firm is gaining or losing capacity. *Built from:* timekeepers and all other employees, offices, departments, and practice groups. # Licensing Patterns Source: https://support.entegrata.com/power-bi/licensing Common Power BI licensing patterns for building and sharing Entegrata reports ## Overview Your firm licenses Power BI directly from Microsoft, and the right mix depends on how your firm works. These are the patterns Entegrata sees work well for building and sharing Entegrata reports. Entegrata does not sell or manage Power BI licenses. Confirm your firm's licensing with your Microsoft account team or licensing partner. ## Building and Sharing Reports People who build, publish, and share reports need a **Power BI Pro** or **Premium Per User** license. A Pro license gives the right to create and share content. It does not, on its own, let people with a free license view that content. Who else needs a paid license depends on where the report lives. In a Pro workspace, viewers need Pro or Premium Per User. In a Premium Per User workspace, viewers need Premium Per User. On a Fabric capacity of F64 or higher, people with a free license can view. ## Sharing with Free-License Users To share reports with many people who only view them, place the reports in a workspace on a **Fabric capacity of F64 or higher**. People with a free Power BI license can then view the reports, while the people who build them keep their Pro or Premium Per User licenses. Smaller Fabric capacities require a Pro or Premium Per User license for every viewer. See [Microsoft Learn: Microsoft Fabric licenses](https://learn.microsoft.com/fabric/enterprise/licenses#licensing-scenario-summary). ## Choosing a Pattern | Pattern | Who needs a paid license | Suits | | - | - | - | | Pro workspace | Everyone who builds or views reports | A small group of report users | | Premium Per User workspace | Everyone who builds or views reports needs Premium Per User | A small group that needs Premium features | | Fabric capacity, F64 or higher | Only the people who build and publish reports | Sharing reports with many viewers across the firm | ## Learn More * [Microsoft Learn: Power BI license types and capabilities](https://learn.microsoft.com/power-bi/fundamentals/service-features-license-type) * [Microsoft Learn: Microsoft Fabric licenses](https://learn.microsoft.com/fabric/enterprise/licenses) * [Microsoft Learn: Power BI Premium to Microsoft Fabric migration FAQ](https://learn.microsoft.com/power-bi/support/premium-migration-faq) # Power BI Overview Source: https://support.entegrata.com/power-bi/overview What Entegrata delivers in Power BI, and how your firm benefits from Power BI on top of your lakehouse ## What is Power BI? Power BI is Microsoft's business intelligence and data visualization platform. Entegrata serves data to Power BI in your firm's tenant, using the Gold layer of your lakehouse, and refreshes data into Power BI on a schedule of your firm's choosing. Entegrata's Starter Pack Power BI reports and semantic models are built on the Gold layer and can serve as an accelerant to your firm's journey into Power BI. ## Why Use Power BI on Your Lakehouse * **A jumping-off point.** Entegrata's Starter Pack dashboards cover firm finances and performance, matter pricing, and presence. Once they are connected to Entegrata's Gold layer in your lakehouse, your firm can customize their measures and visuals to fit how it works. * **One source for every report.** Every Entegrata report reads the same Gold layer, so figures agree from one report to the next. * **Data in your firm's terms.** Data is already organized around topics like clients, matters, timekeepers, and invoices. * **Current on your schedule.** Your firm sets when source data updates and when Power BI refreshes, and your Customer Experience Manager can help you update when the Gold layer builds. See [Data Freshness](/power-bi/data-freshness). * **Room to grow.** Your team can build their own reports on the same data, or analyze it in Excel, without starting from scratch. ## How It Works At least once a day, Entegrata builds the Gold layer in your lakehouse that your Power BI reports are based on. Entegrata's semantic models and reports read the Gold layer. Your team views the reports, builds their own, and connects from Excel. ## In This Section The Power BI tenant settings Entegrata recommends for firms running Entegrata reports. How your data reaches your reports each day, and when to schedule your refresh. Common Power BI licensing patterns for building and sharing reports. Three ways to build your own Power BI reports on your Entegrata data. Analyze your Entegrata data in Excel with a live PivotTable. What each Entegrata dashboard shows, and who it is built for. # Pricing Dashboard Source: https://support.entegrata.com/power-bi/pricing-dashboard What the Entegrata Pricing Dashboard shows, and who it is built for ## Overview The Pricing Dashboard helps partners and pricing teams price new work by looking at how comparable past matters actually turned out. Filter the firm's matters down to the ones that look like the work in front of you, then see what they cost, how long they took, and who worked on them. ## Find Comparable Matters Narrow the portfolio by the attributes that make matters alike: * **Matter profile:** practice group and specialty, type of matter and work, fee arrangement, stage, and status. * **Client:** client industry and whether the client is public or private. * **Transactions:** transaction type, deal value, deal status, structure, buyer type, countries involved, and key dates. * **Litigation:** jurisdiction, trial type, judge, opposing counsel, and outcome. * **Financial ranges:** hours worked, fees, production value, gross profit, and realization. ## See How They Performed For the matters you select, the dashboard shows: * **Financials:** fees worked, billed, and collected, amounts written off, gross profit and margin, realization, and leverage. * **Portfolio ranges:** where matters fall across the top 10, middle 80, and bottom 10 percent of the portfolio. * **Staffing:** the number of timekeepers per matter, the mix of hours by timekeeper type, average rates by title, and where hours came from by office and practice. * **Duration:** how long comparable matters ran. * **Open balances:** WIP and AR on the selected matters. ## Drill Into a Matter Select any matter to see its details, billing attorney, and the attorneys who worked on it. *Built from:* matters, clients, timesheets, invoices, receipts, write-offs, WIP, AR, timekeepers, rates, offices, and practice groups. Transaction and litigation filters depend on the matter attributes your firm records. # RTO Dashboard Source: https://support.entegrata.com/power-bi/rto-dashboard What the Entegrata Return to Office Dashboard shows, and who it is built for ## Overview The RTO (Return to Office) Dashboard shows firm leadership and office managers how people are using the office, and how that compares with the firm's in-office policy. ## What It Shows * **Policy compliance:** average in-office policy compliance, and how it is trending over time. * **Days in the office:** average days in the office per week, and average attorney days per month. * **Busiest days:** in-person attendance by day of the week. * **Summary:** a practice, department, and office summary of attendance and compliance. * **Side-by-side comparison:** compare two practice areas next to each other. Filter by office, department, practice area, title, or individual timekeeper. *Built from:* timekeepers and employees, presence data, firm holidays, PTO and leave, offices, departments, and practice groups. The RTO Dashboard needs presence data, your firm's holiday calendar, and your firm's in-office policy. # Recommended Tenant Settings Source: https://support.entegrata.com/power-bi/tenant-settings The Power BI tenant settings Entegrata recommends for firms running Entegrata reports ## Overview These are the Power BI tenant settings Entegrata recommends for firms running Entegrata reports. No tenant settings are required to install Entegrata Power BI reports. The one setting marked **Required** below is needed only to use Excel. Unless your firm has a business or technical reason to differ, Entegrata recommends the settings below. ## Prerequisites You need Power BI administrator access. To open the admin portal, see [Microsoft Learn: What is the admin portal?](https://learn.microsoft.com/fabric/admin/admin-center) ## Tenant Settings For what each setting does, see [Microsoft Learn: About tenant settings](https://learn.microsoft.com/fabric/admin/about-tenant-settings). ### Workspace Settings | Setting | Recommended | Why | | - | - | - | | Create workspaces | Enabled for specific users | Keeps workspaces from multiplying without an owner. Let a small group of people create them for the firm. | | Use semantic models across workspaces | Enabled for all users | Lets report builders reuse the Entegrata semantic model from their own workspace instead of copying it. See [Build Custom Reports](/power-bi/build-custom-reports). | ### Information Protection Skip this section if your firm does not license Microsoft Purview Information Protection. An error mentioning Microsoft Purview when you change these settings means it is not licensed. | Setting | Recommended | Why | | - | - | - | | Allow users to apply sensitivity labels for content | Enabled for all users | Lets people mark reports and files with the firm's confidentiality labels. | | Apply sensitivity labels from data sources to their data in Power BI | Enabled for all users | Carries labels from the source data into Power BI, so confidential data stays marked. | | Automatically apply sensitivity labels to downstream content | Enabled for all users | Passes a label on to reports, exports and copies built from labeled content. | | Allow workspace admins to override automatically applied sensitivity labels | Enabled | Lets a workspace admin correct a label that was applied automatically but is wrong. | ### Export and Sharing Settings | Setting | Recommended | Why | | - | - | - | | Users can see guest users in lists of suggested people | Enabled for all users | If you already have guest users, people can find them when sharing, instead of mistyping addresses. | | Users can work with semantic models in Excel using a live connection | Enabled for all users | Required to analyze Entegrata data in Excel. See [Connect Excel](/power-bi/connect-excel). | | Certification | Enabled for specific groups | Lets a trusted group mark reports and models as certified, so people know which ones to rely on. | | Users can set up email subscriptions | Enabled for all users | Lets people receive reports by email on a schedule, instead of exporting and forwarding them. | | Install Power BI app for Microsoft Teams automatically | Enabled for all users | Puts Power BI where people already work, so reports are easy to find and share in Teams. | | Enable Power BI add-in for PowerPoint | Enabled for all users | Lets people present live report pages in PowerPoint instead of pasting screenshots that go out of date. | ### Power BI Visuals | Setting | Recommended | Why | | - | - | - | | Allow visuals created using the Power BI SDK | Enabled for all users | Allows custom visuals, which some Entegrata reports use. The next setting limits them to certified ones. | | Add and use certified visuals only (block uncertified) | Enabled for all users | Limits custom visuals to ones Microsoft has certified, which are checked for safe data handling. Entegrata reports use only certified custom visuals, so this does not affect them. | ### Audit and Usage Settings | Setting | Recommended | Why | | - | - | - | | Usage metrics for content creators | Enabled for all users | Shows report owners how often their reports are used. | | Per-user data in usage metrics for content creators | Enabled for all users | Shows which people use a report, so owners can plan training and retire unused reports. | ### Dashboard Settings | Setting | Recommended | Why | | - | - | - | | Web content on dashboard tiles | Enabled for specific groups | Web content tiles can show outside web pages on a dashboard. Limit them to people you trust. | ### Developer Settings | Setting | Recommended | Why | | - | - | - | | Service principals can call Fabric public APIs | Enabled for specific groups | Lets approved automation, such as deployment or refresh tools, work with Power BI without a named user. Limit it to specific security groups. | ### Template App Settings | Setting | Recommended | Why | | - | - | - | | Install template apps | Enabled for specific groups | Template apps come from third parties. Limit who can install them. | | Install template apps not listed in AppSource | Enabled for specific groups | Apps outside AppSource have not been through Microsoft's review. Limit who can install them. | ## Power BI Desktop Preview Features Turn these on for each person who deploys Entegrata reports from their own computer. In Power BI Desktop, go to **File**, **Options and settings**, **Options**, then **Preview features**. | Preview feature | Why | | - | - | | Enhanced DAX Time Intelligence | Entegrata semantic models use it for date calculations. | | Store semantic model using TMDL format | Entegrata semantic models are stored in this format. Turn it on if it appears in your version of Power BI Desktop. | Without these, Power BI Desktop may not open or deploy Entegrata reports correctly. Power BI Projects, the enhanced report format (PBIR), and DAX user-defined functions are generally available, so they no longer need a preview feature. Keep Power BI Desktop up to date to have them. ## Workspaces No specific workspace settings are required. Entegrata reports can be published to Pro, Premium Per User, or Fabric capacity workspaces. For long-term use, Entegrata recommends Fabric capacity. See [Licensing Patterns](/power-bi/licensing). ## Stay on the Current Power BI Release Entegrata always develops against the latest monthly Power BI release. To avoid version mismatches, Entegrata strongly recommends keeping your firm on the current release rather than a delayed release schedule. # Release Notes Source: https://support.entegrata.com/release-notes/release-notes Latest updates and features for the Entegrata Platform and Admin Application. ## October 2026 ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Asana data connector** *(requires platform v1.3.151+)* - New collector plugin for Asana. ## Emitter *Data Emission* ### New Features * **Emit multiple source schemas** *(requires platform v1.3.151+)* - You can now emit multiple source schemas within one catalog to different schemas in the same destination. ### Improvements * **Fully reload emptied destination tables** *(requires platform v1.3.151+)* - A destination table that was emptied or dropped since the last emission is reloaded in full. ## Collector *Data connections, configuration, and collection monitoring* ### Improvements * **Use OpenAPI spec for RubyLaw connector discovery** *(requires platform v1.3.142+)* * **Use new OpenAPI spec location in Mailchimp connector** *(requires platform v1.3.143+)* * **Incremental collection on integer fields** *(requires platform v1.3.144+)* - You can now run incremental collections using an integer field, such as an auto-incrementing ID, as the incremental load field. * **Improve UKG Pro connector discovery** *(requires platform v1.3.144+)* * **Retry Fabric Warehouse collections while Fabric capacity is updating or paused** *(requires platform v1.3.144+)* * **Add more error policies to Dayforce connector** *(requires platform v1.3.146+)* * **Add more error policies to Elite 3E connector** *(requires platform v1.3.148+)* * **Add more error policies to ADP connector** *(requires platform v1.3.149+)* * **Workday Reports URLs are now an editable list** *(requires platform v1.3.147+)* - Report URLs appear in the connection form as individual entries you can add or remove, instead of a hidden field you had to retype in full to change a single report. Existing connections keep collecting unchanged; their URLs show as empty until you enter and save them once. ## Emitter *Data Emission* ### Improvements * **Improve reliability of Fabric emissions** *(requires platform v1.3.144+)* - Emissions wait and retry while Fabric capacity is updating, and schema changes that need it trigger a full table reload. ### Bug Fixes * **Fix SQL Server emissions to schemas whose names need quoting** *(requires platform v1.3.144+)* * **Fix emissions failing on text primary keys with no known length** *(requires platform v1.3.144+)* ## September 2026 ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Anaplan data connector** *(requires platform v1.3.135+)* - New collector plugin for Anaplan, the connected planning platform. Each module becomes a table with one row per dimension combination and a typed column per line item, and each saved view becomes a table of its own. * **HubSpot data connector** *(requires platform v1.3.136+)* - New collector plugin for HubSpot CRM. Contacts, companies, deals, activities, and custom objects are discovered from your account with a column per property, along with the HubSpot users who own records. Supports incremental loads and filtering. * **Dayforce Analytics Reports data connector** *(requires platform v1.3.137+)* - New collector plugin for the ad hoc reports built in Dayforce Analytics, which the existing Dayforce Reports connector doesn't cover. Each analytics report becomes a table, collected as it is saved in Dayforce. ### Improvements * **Add more error policies to Microsoft Dynamics CRM connector** *(requires platform v1.3.137+)* * **Use new OpenAPI specs in Zoom connector** *(requires platform v1.3.138+)* * **Avoid overflows on large future dates when collecting incrementally** *(requires platform v1.3.139+)* * **Increase request timeout for Zoom discovery** *(requires platform v1.3.140+)* ## Emitter *Data Emission* ### Improvements * **Improve reliability of finalizing large SQL Server emissions** *(requires platform v1.3.129+)* ## Collector *Data connections, configuration, and collection monitoring* ### Improvements * **Improve collection job locking** *(requires platform v1.3.125+)* * **Support new zoom OpenAPI spec** *(requires platform v1.3.126+)* * **WordPress collects null for fields expected to return invalid values** *(requires platform v1.3.127+)* — Only those fields are collected as null, and only for the records where the field's value is invalid. Valid values collect as before. ## Emitter *Data Emission* ### Improvements * **Retry Fabric emissions while Fabric capacity is updating or inactive** *(requires platform v1.3.128+)* ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Broken source schemas are surfaced before they cost you a collection** — When a source schema changes in a way that pauses collection, the connections list marks the affected connection and every screen for that connection carries a banner naming the paused resources, linking to each one, and saying whether a cursor field was dropped or field types changed. The warning clears once you review the change. * **Secrets can be explicitly cleared** — Saved connection secrets now offer a Clear value option that empties and disables the field and confirms it will be cleared on save. A blank secret means remove it, rather than leave it as it was. * **Resource dependency tooltips name the mappings** — Hovering the info icon on a locked resource or field now lists each dependent mapping as a clickable path through pipeline, layer, entity, and dataset, taking you straight to that entity instead of showing a generic message. ### Improvements * **Improve logging around error retries** *(requires platform v1.3.118+)* * **Improve azure blob storage writes during collection process** *(requires platform v1.3.121+)* * **Emitters can be edited from the emitter page** — An Edit button in the header replaces navigating back to the list to reach the 3-dot menu. * **Resource detail pages load faster** — The page no longer re-fetches the connection list it already had. * **Errored connections offer both actions** — A connection in an error state can be disabled from the actions menu, and offers retry and disable separately instead of assuming which you wanted. ### Bug Fixes * **A failed schema refresh says so** — Refreshing a connection's schema reported success whenever the discovery lock cleared, even when the run had failed. It now reports the actual failure, only for your own refresh rather than an unrelated one, and tells you to reload if the resource list could not be refreshed afterward. * **Trigger Emission works from the connection route** — The button rendered but did nothing on the emitter detail page reached from a connection, while working correctly from the pipeline route. * **Dropdown labels follow the selected source** — When two sources shared a column name, switching between them left the previous source's labels on screen. This affected searchable dropdowns across the app. ## Pipelines *Data mapping, transformations, and pipeline execution* ### New Features * **Mapping validation results are detailed** — A validation failure now opens a detailed results experience instead of a bare failure message. * **Add every field in the source dialog at once** — Add all fields adds every column shown, or only those matching your search, with the count in the button label. Columns whose name is already taken by a custom formula are skipped and listed rather than overwritten. * **Per-source target maps on multi-field foreign keys** — Foreign key attributes built with coalesce, greatest, or least now give each contributing field its own Target Map control, so each can point at a specific map of the related entity instead of all sharing one setting. * **Unmapped canvas columns survive save and reload** — Source columns placed on the canvas but not yet wired to an attribute were discarded on every save and reload. They now persist, deleting one sticks, and discarding or resetting an entity correctly drops them. ### Improvements * **Unset target maps are visibly unset** — An unset target map reads as a dashed placeholder instead of a pill labelled Auto, so you can tell which fields you have actually configured. * **Unmapped source rows are dimmed** — Source field rows not yet mapped to an attribute are dimmed as a cue for remaining work. Primary key and identifier rows are never dimmed. ### Bug Fixes * **Queries with a trailing comma before FROM are rejected on save** — This previously passed local validation and failed later at execution time. * **Long entity names keep Add Custom Field reachable** — A long name pushed the button out of view on the mapping canvas. The name now wraps to two lines, with the full name on hover. * **Archive is hidden on standard catalog entities** — Standard entities were never archivable, but offered the action anyway. Only custom entities show it. * **Target Map survives an operator switch** — Changing a multi-field key's operator mid-edit hid the control until the dialog was reopened. * **Jobs views no longer stack error toasts** — A failing backend added a new never-dismissing toast every ten seconds on pipeline and emitter job views. Repeat failures now replace a single toast, which clears once data loads again. * **Activating an entity returns a complete response** — Fields were silently missing from the activation result. ## Administration *System configuration, monitoring, and administrative workflows* ### Improvements * **Pages load faster across the app** — The signed-in user's profile is fetched once per request rather than repeatedly. ### Bug Fixes * **Logging in from the alternate admin address returns you there** — Sign-in bounced you to the primary address instead. ## Collector *Data connections, configuration, and collection monitoring* ### Improvements * **Add more error policies to Courtroom Insights connector** *(requires platform v1.3.113+)* * **Improve logging around SQL collection flow** *(requires platform v1.3.114+)* * **Add more error policies to Dayforce connectors** *(requires platform v1.3.115+)* * **Retry on individual chunks instead of retrying whole collection when there are errors collecting from SQL sources** *(requires platform v1.3.117+)* ### Bug Fixes * **Workday Reports String Array fields are no longer null** *(requires platform v1.3.116+)* ## August 2026 ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Intelligize data connector** *(requires platform v1.3.105+)* — New collector plugin for Intelligize. Import SEC filings, regulatory documents, and compliance data from Intelligize. ### Improvements * **Add more error policies to RubyLaw connector** *(requires platform v1.3.105+)* ### Bug Fixes * **Enabling a connection now tests it first** — Enabling a connection runs a connection test against the data source and only activates it if the test passes. A failed test leaves the connection as it was and shows the reason the data source gave, rather than switching it on and leaving you to find out later. * **Primary key selection is kept when saving collection settings** — Saving collection settings from the connections panel discarded the primary keys you had chosen. Selections now persist, incomplete filter rows are dropped rather than saved, and an incremental save that references a field that no longer exists fails instead of saving a broken configuration. * **Creating a connection no longer reports a false error** — Connection creation could show an error even though the connection had been created successfully. * **Enable and disable no longer act on stale clicks** — Repeated clicks on disable no longer send duplicate requests, and closing the enable dialog while its connection test is running cancels the test instead of leaving it going in the background. * **North Central US is available for lakehouse IP access** — The region is now selectable in the lakehouse IP access region list. ## Pipelines *Data mapping, transformations, and pipeline execution* ### New Features * **Entities flagged for review can be reviewed in place** — An entity that has been flagged now shows a banner on its detail page with the reason, letting you activate it directly instead of hunting for its status elsewhere. * **Remove fields from a function source without leaving the dialog** — Function source fields can be deleted from within the field editing dialog. * **Reuse a custom field across an entity's mappings** — A custom field defined on one mapping can now be added to the entity's other mappings from a picker, instead of being recreated by hand on each map. The picker lists the entity's existing custom fields with each one's type, lets you add several at once, and rejects names that would duplicate a field already on the entity. ### Improvements * **Publishing a large pipeline no longer times out** — Publishing now runs in the background and reports its result when it finishes, instead of failing on pipelines big enough to exceed the request timeout. * **Save can't be double-submitted** — The save bar in the mapping editor ignores repeated clicks while a save is in flight. * **Clearer status wording** — "Flagged for review" is now "Flagged" in change history. ### Bug Fixes * **Publishing an entity that fails validation says so** — A publish rejected for validation errors failed silently. It now shows the error. * **Unique constraints confirm they saved** — Adding or removing a unique constraint now confirms the result, and a failed add keeps the form open with your input rather than discarding it. * **Foreign key targets load on pipelines with many entities** — The Target Map selector failed to load on pipelines with more than 200 entities. Targets now load on demand, and the selector also appears for hash and concatenated multi-field foreign keys, which previously offered no target. * **Boolean CASE conditions keep their Then value** — Switching a boolean condition from Custom to Value cleared the Then dropdown. * **Editing a source opens straight from adding one** — Adding a source now moves directly into its edit dialog instead of closing and requiring a second click. * **Record-scoped pipeline access is honored in the editor** — Users granted access to specific pipelines were treated as read-only in the mapping editor and emission views despite having edit rights on those pipelines. ## Access Control *Permissions and security* ### Bug Fixes * **Create Group reopens after being dismissed** — Closing the Create Group dialog with X or Escape prevented it from opening again without a page reload. * **Only rows with exceptions are clickable** — Rows showing no exceptions were clickable and led nowhere. * **Granted-permissions detail is readable on the first row** — The popover listing granted permissions was clipped on the first row of the matrix, and has been restyled to a light surface to match the rest of the page. * **The permissions page scrolls correctly** — Focusing a permission checkbox jumped the page back to the top, and tall permission pages could not be scrolled to the bottom. ## Administration *System configuration, monitoring, and administrative workflows* ### Improvements * **Pages no longer flicker on load** — Parts of the interface briefly rendered in one state and then switched once configuration finished loading. Pages now render in their final state on first paint. * **Faster job lists** — Job views no longer request up to 10,000 rows when filtering to the last 24 hours. ### Bug Fixes * **Switching accounts from the error page signs you out properly** — Choosing to switch accounts left the underlying identity provider session active, so you were signed back into the same account. It now signs out fully, and a failed sign-out reports the failure instead of appearing to succeed. * **Instance setup keeps your settings across screens** — The settings panel lost its state when moving between onboarding screens mid-setup. ## Collector *Data connections, configuration, and collection monitoring* ### Improvements * **Add more error policies to Entra and Sitecore connectors** *(requires platform v1.3.100+)* * **Support lookup fields for SharePoint Lists connector** *(requires platform v1.3.101+)* * **Support `"` characters in CourtAlert CSV responses** *(requires platform v1.3.102+)* * **Add error policy to retry Entra collections when batches expire** *(requires platform v1.3.103+)* ## Collector *Data connections, configuration, and collection monitoring* ### Improvements * **Add more error policies to Salesforce connector** *(requires platform v1.3.94+)* * **Cast strings to their length when collecting from sql server sources** *(requires platform v1.3.95+)* * **Iridium view collection optimizations** *(requires platform v1.3.96+)* * **Change UKG Pro date fields to timestamp type** *(requires platform v1.3.97+)* * **Change Courtroom Insights date fields to timestamp type** *(requires platform v1.3.99+)* ### Bug Fixes * **Data source logos keep their shape in the connections list** — Logos in the Data Source column are no longer stretched or squashed to fit a square, so wide and tall logos now display correctly. * **UKG Pro connector: Fix nameSuffixCode being set to null in bronze when it is populated in the API** *(requires platform v1.3.98+)* * **Connection type is only asked for when it applies** — The Connection Type selector on new connection setup appeared for every data source and offered an "Any" option that filtered identically to Collection. The selector now appears only for data sources that support emission, and "Any" has been removed. Existing connections created as "Any" keep their type when edited, and requesting an emitter on a source that does not support one is now rejected rather than created. ## Pipelines *Data mapping, transformations, and pipeline execution* ### New Features * **Failed jobs say why they failed** — A pipeline job can fail at the workflow level even when every individual entity finishes, which previously left the results table showing no error at all. Entity Results now shows a banner on jobs that errored, timed out, or were terminated, stating the reason directly rather than leaving you to infer it from an empty table. ### Improvements * **Pipelines open directly to their entities** — Opening a pipeline now takes you straight to the Silver entities view instead of an intermediate page of navigation cards. Statistics and change history have moved into side panels reachable from any pipeline view, and the duplicate tab strips have been replaced by the breadcrumb. Running a pipeline is now a button in the pipeline breadcrumb, including the Full Refresh option where the pipeline supports it. * **"Affected Records" is now "Records Processed"** — The records column on the pipeline jobs table has been renamed to describe what it actually counts. ## Administration *System configuration, monitoring, and administrative workflows* ### New Features * **Refreshed navigation with in-page search and side panels** — Home, Connections, Pipelines, Users, and Instance Settings now share a new header built around a dropdown breadcrumb, so you can move between related views without returning to a list first. List pages have gained an in-page header with a search box, result counts, filters, and that page's primary action; the keyboard shortcut for search still works. Connections and Pipelines also gain panels that slide in from the right — connector fleet health, and per-connection Jobs and Collection Settings on Connections; pipeline Insights and change History on Pipelines — so you can check something without leaving the page you are on. Panels close automatically when you navigate. ### Improvements * **Simplified Navigation Naming** — "Data Connections" is now "Connections" and "Data Mapping" is now "Pipelines". * **Network connections have moved to Instance Settings** — Network connection management now sits under Instance Settings alongside the other instance-level infrastructure settings, rather than under Connections. Existing links and bookmarks redirect automatically. * **Copy buttons confirm the copy** — Copy actions now show a checkmark briefly after copying. Copying from inside a table row no longer also opens that row, and a copy that fails no longer reports success. ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **UKG Ready Reports connector** *(requires platform v1.3.92+)* — New collector plugin for UKG Ready reports. Collect saved and global reports into the Lakehouse, with incremental loading for reports that expose a date range selector. ### Improvements * **More job statuses available as filters** — The status filter on collection jobs and emitter jobs now offers Waiting, Deferred, Timeout, and Terminated alongside Completed, Failed, Running, and Scheduled, so you can isolate jobs in those states directly. * **Add more error policies to Workday Reports connector** *(requires platform v1.3.93+)* ### Bug Fixes * **Introhive collections no longer error on blank key fields** *(requires platform v1.3.90+)* - Introhive resources identified by a combination of fields errored when one of those fields came back empty. Each row now carries a generated identifier, so those rows collect successfully. ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **ExpressRoute network connections** — ExpressRoute is an initial release and is not yet Generally Available. It joins VNet Peering and Site-to-Site as a network connection type. You can add one during onboarding or from the network connections list, and it appears in the list with its own type filter. Setup asks for the circuit's Azure Resource Manager ID and a single-use authorization key from the circuit owner, plus optional address prefixes used for CIDR overlap validation. The connection detail page includes a Circuit Information section explaining where to find each value in the Azure portal, with links to Microsoft's documentation. On an existing connection the circuit ID becomes read-only and the authorization key is hidden, since each key can only be used once. * **Schedule start times and fixed schedules** — Collection schedules now accept an explicit start time, and can be set to run on a fixed configuration rather than a recurring interval. The schedule summary text reflects the start time in your instance's time zone. * **Connection field validation as you type** — Connection parameters that have a required format now validate live while you fill them in, so a malformed value is flagged in the form instead of failing when you save. ### Improvements * **Automated update schedules use your instance time zone** — The automated-update schedule in Instance Settings is now read and written in the instance's configured time zone instead of the browser's. Schedule edits are also held as a draft until the save succeeds, so a failed save no longer leaves the form showing a value that was never stored. * **Faster collection charts** — Collection and emitter job charts now fetch their record counts in a single bucketed request instead of one request per job, which noticeably speeds up loading for connections with a lot of run history. * **Double-precision fields supported in resource fields** — Resource field tables now recognize and display the double numeric type. * **Page size and cursor field selectors render correctly** — The page-size and cursor-field dropdowns in Collection Settings now populate their options; previously they could render empty. * **Improve reliability of large Courtroom Insights collections** *(requires platform v1.3.78+)* - Rate-limited requests now retry automatically with backoff, large exports have a longer download timeout, and big tables stream in batches to keep memory flat and avoid out-of-memory failures. ### Bug Fixes * **Passwords accept the full character set** — Password and secret fields no longer restrict input to ASCII-only characters, so credentials containing other characters can be entered as-is. * **Collection start time displays correctly** — The interval picker now shows the configured start time instead of defaulting to a placeholder value. * **Resource rows stay on screen when you cancel navigation** — Declining the unsaved-changes prompt while leaving the Resources page no longer leaves the table blank behind two stranded spinners; your rows stay rendered. * **Collect invalid Courtroom Insights date values as null instead of failing** *(requires platform v1.3.78+)* - Unparseable date and time values (such as all-zero dates) in nullable columns are now stored as null rather than erroring the collection. ## Pipelines *Data mapping, transformations, and pipeline execution* ### New Features * **Sync mapping configuration between pipelines** — Pipeline Sync is an initial release and is not yet Generally Available. You can now move a pipeline's mapping configuration into another pipeline instead of rebuilding it by hand. From the pipelines list, **⋮ → Manage → Sync** pushes the selected pipeline's entities, fields, sources, and joins into a target pipeline of your choice. The source pipeline is never modified, and only mapping configuration moves — never data. * **Three sync strategies** — Every sync asks how much of the source to bring across. **Overwrite** makes the target an exact copy of the source and deletes target-only mappings. **Overlay** applies the source's additions and changes and never deletes. **Merge** walks you through every difference so you can accept or reject each one individually. * **Single-entity sync** — The **⋮** menu on any entity row inside a pipeline layer can sync just that entity into another pipeline, using the same target picker, strategies, and reason field. Single-entity sync replaces that entity's fields and sources and never deletes the entity itself. * **Merge review wizard** — Choosing Merge opens a step-by-step review that shows only the entities that actually differ, one at a time. Every difference is a toggleable row tagged Add, Update, Remove, or Unchanged, grouped by the part of the entity it belongs to. Nothing is applied until you commit — either one entity at a time or all at once — and a searchable **Jump to entity** picker plus a reviewed-count means you can work through a large sync without paging in order. * **Sync from main** — **⋮ → Manage → Sync from main** opens the same dialog with the direction reversed: main is the source and the pipeline you started from is the target. This is the one-step way to reset a test pipeline that has drifted from production. * **Read-only entity and pipeline diff** — **Manage → Diff** on a pipeline, and **Diff** on an entity row, compare against a target and change nothing. Both step through the same per-entity walkthrough as the Merge wizard, so you can answer "what's actually different between test and production right now?" before deciding whether to sync at all. * **Optional publish after syncing** — A sync updates mapping configuration but does not change what the target is currently running. The dialog now carries a **Publish the target pipeline after syncing** checkbox (off by default) that publishes immediately after the sync is saved, with the tradeoff stated inline. If the sync succeeds but the publish fails, the sync is still saved and the banner says so explicitly — publishing again is the fix, with no need to re-sync. * **Per-entity sync results and change history** — Every sync reports each entity as **Changed**, **Unchanged**, or **Skipped**, with a reason, and opens a **Sync Result** window when something needs attention. A required **Reason for Sync** is recorded against every changed entity in the target's change history alongside your user and timestamp, so a sync is reviewable and revertible after the fact. * **Required sources** — The Add Source wizard and Source Edit dialog now offer a Required option, so you can mark a source as mandatory for the entity mapping. * **Custom filter on the entity list** — The entity list now has a Custom column and filter, so you can separate custom entities from standard ones at a glance. ### Bug Fixes * **View button appears for catalog queries** — The View button for a catalog query now reads its SQL from the runtime cache, so it appears when it should instead of staying hidden. * **Pipeline name validation explains the problem** — Creating or duplicating a pipeline with a name that generates an invalid key (one that starts with a number, or is too long) now shows exactly what's wrong and what the generated key would be, instead of failing on save. * **Add Source wizard resets cleanly** — Closing the Add Source wizard now clears the Required setting, so it doesn't carry over the next time you open it. ## Administration *System configuration, monitoring, and administrative workflows* ### Improvements * **Faster page loads on long-lived sessions** — Sessions that had been open for a while were triggering a token refresh on nearly every request, adding noticeable latency to each page load. Refreshed credentials are now cached server-side, bringing response times back to normal. ### Bug Fixes * **Client log payloads sanitized** — Diagnostic logs sent from the browser now have their payloads sanitized and are stamped with a verified identity before being recorded. ## July 2026 ## Collector *Data connections, configuration, and collection monitoring* ### Improvements * **Optimize parallelism when there are no chunking keys (SQL collections)** *(requires platform v1.3.70+)* * **Change Dayforce date fields to timestamp type** *(requires platform v1.3.71+)* * **Change Dayforce Reports date fields to timestamp type** *(requires platform v1.3.72+)* * **Change `__sys_ingest_ts` value to reflect when the record got written to bronze table** *(requires platform v1.3.75+)* ### Bug Fixes * **Convert invalid date values to null instead of erroring collection** *(requires platform v1.3.74+)* ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **SurveyMonkey data connector** *(requires platform v1.3.62+)* - New collector plugin for SurveyMonkey, the online survey platform. Collect surveys, survey responses and designs, collectors, and contacts into the Lakehouse, with incremental loading for responses. ### Improvements * **Change DealCloud float fields to double type** *(requires platform v1.3.63+)* * **Ensure Intapp Intake API discovery returns resources with unique names** *(requires platform v1.3.66+)* * **Lower concurrency for Intapp Intake API/add error policies** *(requires platform v1.3.68+)* * **Ensure all Dayforce sub-resources have a \_\_sys\_id field (system generated primary key)** *(requires platform v1.3.69+)* ## Collector *Data connections, configuration, and collection monitoring* ### Improvements * **Introhive data connector now supports incremental loading** *(requires platform v1.3.54+)* * **Change DealCloud date fields to timestamp type** *(requires platform v1.3.55+)* * **Connection parameters get merged on connection update** *(requires platform v1.3.58+)* ### New Features * **Intapp Terms data connector** *(requires platform v1.3.54+)* - New collector plugin for Intapp Terms, Intapp's client-obligation management platform. Collect outside counsel guidelines, client terms of business, and billing obligations into the Lakehouse. * **Ideate data connector** *(requires platform v1.3.54+)* - New collector plugin for collecting tables and views from an Ideate SQL Server database into the Lakehouse. ## Collector *Data connections, configuration, and collection monitoring* ### Improvements * **Add matter id to SALI values (via Index.io connector)** *(requires platform v1.3.53+)* ## June 2026 ## Collector *Data connections, configuration, and collection monitoring* ### Improvements * **Add hierarchies to NAICS/SALI values (via Index.io connector)** *(requires platform v1.3.44+)* * **Retry transient errors in Lead Forensics test connection** *(requires platform v1.3.47+)* * **Use OData protocol for SiteCore connector pagination** *(requires platform v1.3.49+)* * **Add more error policies to SiteCore and DealCloud connectors** *(requires platform v1.3.50+)* * **Add support to SQL connections with spaces in the username/password** *(requires platform v1.3.51+)* ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Emitter** *(requires platform v1.3.39+)* - Emitter is now available to everyone. It can be found in pipelines under the emission tab. * **OneDrive data connector** *(requires platform v1.3.37+)* — New collector plugin for integrating with Microsoft OneDrive. * **Intapp Conflicts data connector** *(requires platform v1.3.38+)* - New collector plugin for integrating with Intapp Conflicts. ### Improvements * **Improve error handling for Dayforce Reports connector** *(requires platform v1.3.33+)* * **Discover Media endpoint in RubyLaw connector** *(requires platform v1.3.35+)* * **Ensure "null" strings get collected as null for CourtAlert connector** *(requires platform v1.3.36+)* * **Resource name shown on the Collection Settings header** — The Collection Settings dialog now shows the resource name and key (e.g. `BLT_BILLM`) in its header, so you can confirm at a glance which resource you're editing. * **New Cisco Meraki endpoints supported** *(requires platform v1.3.40+)* — A new version of the OpenApi spec is being used by the Meraki connector, more endpoints are available for collection. ## Pipelines *Data mapping, transformations, and pipeline execution* ### Improvements * **Clearer error messages when a save can't be completed** — When the mapping editor can't save your changes (invalid entity name, leading or trailing spaces, duplicates, etc.), the dialog now shows the specific reason — for example, "A pipeline with that name already exists" — instead of a generic "Save failed" message that hid what went wrong. ### Bug Fixes * **Clear notice when a resource has no available fields** — In the Add Source wizard and Source Edit dialog, picking a resource that doesn't have any fields yet now shows an inline amber notice with a link to that connection's Fields page, and the Continue / Confirm button is disabled until the issue is resolved. Previously the dialog would accept the empty selection and the mapping would fail later on Save. * **NULL option for Boolean values in the CASE editor** — When building a CASE expression for a Boolean trait, the THEN result and default-value pickers now offer NULL alongside True and False, so you can express a tri-state outcome without a workaround. * **Removing a source from a field clears it completely** — Removing the source on a mapped field now also clears any leftover references that pointed to it. Previously, a leftover empty reference could remain behind and cause the next Save to fail with a confusing validation error. ## Emitter *Data Emission* ### Improvements * **Single Resource Emission** *(requires platform v1.3.39+)* - You may now choose specific resources to emit ### Bug Fixes * Addressed an issue where an emission would incorrectly send an empty table to the destination ## Administration *System configuration, monitoring, and administrative workflows* ### Improvements * **Time zone list sorted by UTC offset** — The time zone selector in Instance Settings now orders entries by UTC offset (UTC-12 → UTC+14) instead of alphabetically, so you can find your offset faster. ### Bug Fixes * **Onboarding sends you to the step you're actually on** — If you open an onboarding URL for a step that's ahead of where your account actually is, the app now redirects you to your current step instead of stranding you on an out-of-order page. * **Revoked sessions are sent back to the sign-in screen** — If your session has been revoked (for example, an admin signed you out remotely), the next page load in your browser will now send you to the sign-in screen instead of continuing with a stale signed-in view. Signing out also now invalidates your session on the server side, so a copy of your session cookie can't be used to make further requests after you've logged out. ### New Features * **Lead Forensics data connector** *(requires platform v1.3.26+)* — New collector plugin for integrating with Lead Forensics. ### Improvements * **Improve error handling in collector system** *(requires platform v1.3.31+)* * **Improve OpenAPI spec discovery for Dayforce connector** *(requires platform v1.3.32+)* ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Intapp Time data connector** *(requires platform v1.3.25+)* - New collector plugin for integrating with Intapp Time. * **iManage SPM data connector** *(requires platform v1.3.25+)* - New collector plugin for integrating with iManage SPM. ### Improvements * **Collector infrastructure change** *(requires platform v1.3.27+)* - Use container app jobs for data collection process (instead of container apps). * **UKG Pro updates** *(requires platform v1.3.25+)* - Added support for multiple v2 endpoints. ## Collector & Emitter *Data connections, configuration, and collection monitoring* ### New Features * **Workday Reports data connector** *(requires platform v1.3.19+)* - New collector plugin for integrating with Workday Report-as-a-Service (RaaS). * **Intapp Intake API data connector** *(requires platform v1.3.15+)* - New collector plugin for integrating with Intapp Intake API. * **Password / Passwordless auth option on connector setup** — Connection setup forms now expose a Password / Passwordless choice for connectors whose authentication template includes the Password option, letting users opt into passwordless flows where the connector supports it. * **Sortable column headers on jobs tables** — Collection and pipeline jobs tables now support click-to-sort on Timestamp, Total Records, Total Bytes, Records/sec, Duration, and Resource columns. The active sort is preserved across auto-refresh, page changes, and filter updates so the order doesn't snap back to server order under the user. * **Empty-state label on the users table** — The users list now shows a "No users found" empty state instead of an empty table when there are no users to display. ### Improvements * **Use correct silver queries in index.io data connector** *(requires platform v1.3.23+)* * **Improve error handling on Oracle connector** *(requires platform v1.3.14+)* * **Faster page loads across the Connections area** — The connections list, connection detail, Auth, Collection, Emission, Resources, and Fields pages all load noticeably faster. * **Filters and search stick around as you navigate between Resources and Fields** — On a connection's Resources and Fields pages, your active filters and search query are now preserved when you switch between the Nested Resources and Fields tabs, paginate, or navigate away and come back. Each resource keeps its own field-level filters so they don't leak between resources, and the URLs are shorter. * **Settings menu now works on resources awaiting input** — On a brand-new connection, the three-dot collection-settings menu on each resource row is now usable for resources in an Input-Needed state, instead of appearing greyed out until the connection is fully set up. * **Collection Settings modal protects unsaved edits from accidental dismissal** — Clicking outside the Collection Settings modal no longer silently closes it and discards your pending edits. Closing now goes through the Close (X) button, which prompts you to save or discard if there are unsaved changes. ### Bug Fixes * **"Total Rows Affected" chart title on the Collection page no longer shows "NaN"** — When the time-range query in the URL was malformed (e.g. an old bookmark or a hand-edited URL), the chart title on a connection's Collection page previously read "Total Rows Affected (Last NaN Days)". The title now falls back gracefully instead. * **"Collect" button no longer flashes on screens that can't collect** — When navigating to a nested resource, the action button no longer briefly appears for the previous parent before its data resolves. * **Type-diff modal closes cleanly and the banner dismisses with it** — After acting on type changes, the modal now shows a success state with a Close button, the yellow "needs input" banner on the resource hides for the session, and reopening the modal works as expected. * **Inactive connections appear in the connections list** — The connections list and search now include connections with an Inactive status alongside Active and Error, instead of silently filtering them out. ## Pipelines *Data mapping, transformations, and pipeline execution* ### New Features * **"Any" toggle and clearable foreign key on Target Map** — The field/trait editor's foreign-key Target Map now has a clearable selector and a new "Any" toggle. Turning Any on lets the field match any map of the related entity instead of requiring one specific map. For new fields with a foreign key, Any is on by default. * **Choose a direct mapping source while creating a new field** — The New Field dialog now offers an inline searchable picker to wire the field directly to one of the canvas's mapped source fields at creation time, eliminating the separate drag step. Existing multi-source and expression paths are unchanged. * **Catalog query Key surfaced as a read-only field** — Create and Edit catalog query dialogs now display the auto-generated Key as a visible disabled input with an explicit note that it cannot be changed after creation. * **Warning when editing a catalog query with dependent entities** — Editing a catalog query that has dependent entities now shows a yellow callout reminding the user that downstream entities will need to be republished. * **"Use Sequence ID" toggle on emitter setup** — The New Emission and Edit Emission dialogs now expose a Use Sequence ID toggle (on by default). When on, the emitter swaps primary-key and foreign-key columns to the source's bigint sequence ID instead of the entity UUID, so outbound records carry the smaller integer key where applicable. ### Improvements * **"expression" label on custom-expression source nodes in the editor canvas** — Source nodes carrying a custom expression now show an italic "expression" label in place of the underlying type, making custom-expression refs visually distinct from direct field refs in the mapping editor. * **Cleaner mapping error message for label vs. name collisions** — When the mapping backend rejects a save because a value is already in use, the surfaced message now specifies whether the collision is on display label or attribute name, instead of one ambiguous "duplicate" message. * **Long-text cells clamp consistently and only show a tooltip when truncated** — Description and other long-text columns across pipelines, schedules, catalog queries, and entity lists now share one clamped-cell behavior: two-line truncation, no tooltip when the text fully fits, a width-capped tooltip when it doesn't, and proper horizontal-overflow detection. The pipeline list also stays column-aligned between Production and Test tables. * **Tables scroll horizontally on small viewports instead of clipping** — The shared table now scrolls horizontally when content exceeds the viewport rather than silently cutting off the right-most columns. ### Bug Fixes * **Mapping editor "Unsaved changes" bar accuracy** — The floating Unsaved changes bar now reflects actual unsaved state in every flow: it no longer reappears on hard reload when nothing has changed, reopening the entity edit modal after Save now shows the latest description instead of stale text, and reverting an edit back to its original value now clears the bar in the same session without requiring a reload. * **Expired sessions in pipeline routes redirect to sign-in instead of showing a 500** — Authentication errors thrown deep in the call stack are now rethrown through the auth-redirect helper before reaching the generic 500 logger, so an expired session redirects to login instead of producing an opaque server error. ## Administration *System configuration, monitoring, and administrative workflows* ### Improvements * **Real-time validation on the S2S Gateway IP** — The Gateway IP field on Site-to-Site VPN setup now validates as you type, not only on blur, so format errors appear immediately. CIDR-style entries (e.g. `1.1.1.1/32`) are now correctly rejected for this field. * **Site-to-Site Local Network Addresses require CIDR notation** — The Local Network Addresses field on Site-to-Site VPN connections no longer accepts bare IP addresses; entries must use CIDR notation (e.g. `192.168.1.1/32`). The placeholder and error message have been updated to reflect this. * **SA Lifetime validation aligned across S2S surfaces** — The IPsec SA Lifetime fields on the post-setup S2S edit form now apply the same range rules as the onboarding form: SA Lifetime (KB) accepts 0 or values between 1,024 and 2,147,483,647, and SA Lifetime (seconds) requires a value between 300 and 2,147,483,647. * **Managed identity name updates when the instance key loads** — On the subscription-setup step, the `id-{instance}-{env}-core` managed-identity name now reactively rebuilds when the instance key arrives from the live API call. Previously, sessions with a blank key alias rendered a stale `id--p-core`. ### Bug Fixes * **"Previous step" no longer spins during onboarding** — Clicking "Previous step" during onboarding (notably from the Settings step) now navigates cleanly. The background revalidation poll is paused during the navigation so it can no longer preempt the in-flight transition and trap users on the current page. ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Wordpress data connector** *(requires platform v1.3.8+)* - New collector plugin for integrating with Wordpress. * **Piwik Pro data connector** *(requires platform v1.3.8+)* - New collector plugin for integrating with Piwik Pro. * **Introhive data connector** *(requires platform v1.3.7+)* - New collector plugin for integrating with Introhive. ### Improvements * **Introhive data connector optimizations** *(requires platform v1.3.9+)* - Optimize collections for Introhive data connector. * **Introhive data connector incremental load** *(requires platform v1.3.10+)* - Fix incremental load for Introhive data connector. * **RubyLaw incremental loads** *(requires platform v1.3.7+)* - Added support for incremental collection in the RubyLaw data connector, reducing load times and resource usage for ongoing data collection. ## May 2026 ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Site-to-Site VPN configuration** *(requires platform v1.3.0+)* - When creating or editing an S2S connection, choose the IKE version (IKEv1 or IKEv2), configure IKE Phase 1 / IPsec policy details (encryption, integrity, DH group, PFS group), set the gateway IP, and manage address prefixes with per-entry CIDR validation. * **VNet peering options** *(requires platform v1.3.0+)* - VNet connections support a multi-entry Remote Address Space CIDR list with inline duplicate and CIDR validation, plus Allow Forwarded Traffic, Allow Gateway Transit, and Use Remote Gateways toggles, now visible as read-only in view mode. * **Manage network connections after setup** - A new Network Connections area lets you view, add, edit, and delete VNet peering and Site-to-Site (S2S) VPN connections to your instance, filter the list by connection type, and see a live status badge (Connected, Disconnected, Pending, Not Found) for each. Reachable from a card in instance settings and a dedicated connections page. ### Improvements * **Lower parallelism for data collection of SQL views** *(requires platform v1.2.180+)* - reduced database load during SQL view collection. * **Deployment progress feedback** - Saving, creating, or deleting a network connection now shows a "Deploying Changes" screen and polls for completion in the background, so you can see when infrastructure changes are still applying. * **Accurate per-connection status** - Connection status is matched to the specific connection, and unknown statuses render with a clearer badge. ### Bug Fixes * At least one connection is now required; deleting the last remaining connection is blocked with an explanation. * Connection save failures now show the specific server error instead of a generic message. * VNet connections now require at least one valid Remote Address Space CIDR before saving. * Deleting a connection now targets the correct instance, fixing failed deletions. ## Collector *Data connections, configuration, and collection monitoring* ### Improvements * Change Created and Updated SiteCore fields to timestamp *(requires platform v1.2.174+)*. ### New Features * **IQ data connector** *(requires platform v1.2.176+)* - New collector plugin for integrating with LexisNexis IQ to collect relationship intelligence — contact scores, interaction history, and the signals IQ derives from your firm's communications. * **InterAction data connector** *(requires platform v1.2.176+)* - New collector plugin for integrating with LexisNexis InterAction CRM to collect contacts, companies, activities, and relationships. ## Pipelines *Data mapping, transformations, and pipeline execution* ### New Features * **Production mapping edit access** — Production pipeline mappings can now be made editable for specific users without requiring full admin privileges. ### Improvements * **Pipeline key length validation** — Creating or duplicating a pipeline now shows a clear inline error when the auto-generated key exceeds 20 characters, preventing unclear backend rejections. * **"Display Fields" label** — The Governance tab button previously labeled "Display Traits" now reads "Display Fields" to match the modal it opens. ### Bug Fixes * Fixed saving trait details on a multi-ref field silently destroying the multi-ref source configuration. * Fixed the "Update Expression" button remaining clickable during the validation debounce window. * Fixed mapping filter scoped property being lost when filters were cloned or grouped. * Fixed emission trigger hanging until the 45-second timeout — triggering an emission now shows immediate feedback. ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Salesforce data connector** *(requires platform v1.2.167+)* — New collector plugin for integrating with Salesforce CRM. Collects standard and custom SObjects via Bulk API 2.0. * **Versioned History (No Deletes) load strategy** *(requires platform v1.2.167+)* — A new data retention option that preserves historical versions of records without tracking deletions. Ideal for sources that reuse, archive, or filter records in ways that would otherwise produce false deletions. ### Improvements * **Separate connection and collection status columns** — The connections list table now shows two distinct status columns: "Connection Status" and "Collection Status", making it clear whether the connection itself is active versus whether data collection is running. * **Collection scheduling uses instance timezone** — Collection schedule times are now calculated and displayed using the instance timezone instead of the browser's local timezone, with correct DST adjustments for future occurrences. * **Access Control tab visible on inactive connections** — The Access Control tab is now shown (grayed out with a tooltip) on inactive connections instead of being hidden entirely, so users know the feature exists. * **Inactive connection safeguards** — Inactive connections now redirect to the auth page, disable the "Refresh Source Schema" button, and block discovery API calls. The Resources tab is disabled while the Collection tab remains accessible so users can re-activate collection. * **Connector setup documentation link** — The new connection page now shows a link to connector-specific setup documentation for all connection types. ### Bug Fixes * Fixed a platform issue where collection queries could fail during authentication token refresh *(requires platform v1.2.167+)*. ## Access Control *Fine-grained data access management* ### Improvements * **Subject filter entity selector** — Access control rule filters now include an entity type selector on Subject filters with a confirmation prompt when switching entity types on rules that already have conditions. ## Administration *System configuration, monitoring, and administrative workflows* ### New Features * **Instance timezone setting** — A new Timezone card on the Instance Settings page lets administrators set the instance-wide timezone used for collection scheduling and other time-sensitive displays. ### Improvements * **Default dataset protected from deletion** — The default dataset row in the Manage Maps modal no longer shows a delete button, preventing accidental removal. ### Bug Fixes * Fixed a server-side rendering crash (500 error) on the onboarding setup wizard Step 1 page. * Fixed action dropdown menus (connections, emitters, schedules, access control) that could get stuck open or fail to close. * Fixed stale page titles persisting when navigating between pages. * Fixed mapping editor polling silently failing — after 3 consecutive sync failures, a warning banner now appears and saves are blocked until the page is reloaded. ## Collector and Emitter *Data connections, configuration, and collection/emission monitoring* ### Improvements * Set default to XS Databricks Compute for Emissions *(requires platform v1.2.157+)* ## April 2026 ## Collector and Emitter *Data connections, configuration, and collection/emission monitoring* ### New Features * **Intellistat data connector** *(requires platform v1.2.152+)* - New collector plugin for integrating with BigHand Intellistat to collect financial performance metrics, matter analytics, and utilization data. * **Activate / deactivate connections** — Each connection in the connections list now has a three-dot menu with Activate and Deactivate actions, and inactive connections are now shown in the list rather than hidden. ### Improvements * Validate parquet files before loading into bronze table *(requires platform v1.2.154+)* * Increase job timeouts for v2 pipeline scripts *(requires platform v1.2.153+)* * Support encryption on SQL connections *(requires platform v1.2.151+)* ## Pipelines *Data mapping, transformations, and pipeline execution* ### New Features * **Dataset count badge on the mapping editor's dataset selector** — The dataset selector in the mapping editor now shows a count badge for the number of additional datasets mapped to the entity, making multi-dataset coverage visible at a glance without opening the dropdown. * **Entity change history tab** — Entity detail pages now have a Change History tab showing a paginated, action-tagged audit trail of edits to that entity (including a new Reset action). * **Required change comments** — Saving an entity change now prompts for a brief change note that is recorded alongside the history entry. ### Bug Fixes * **Case builder preview: type-aware literals** — Boolean and numeric values in WHEN/THEN preview lines are no longer wrapped in quotes; the preview now respects the trait's inferred type. * **Case builder preview: dialect prefix stripped** — Raw dialect prefixes are no longer shown in the WHEN/THEN preview lines of case-mapped fields. * **Stale primary-key identifier line on rebuild** — Rebuilding an entity no longer leaves behind a ghost primary-key identifier annotation from the previous build. ## Administration *System configuration, monitoring, and administrative workflows* ### Improvements * **Dismissible toast notifications** — Success and error toasts can now be dismissed manually instead of waiting for the auto-close timeout. ## Collector and Emitter *Data connections, configuration, and collection/emission monitoring* ### New Features * **Oracle data connector** *(requires platform v1.2.146+)* - New collector plugin for integrating with Oracle databases. ### Improvements * Added more logs to SiteCore data connector *(requires platform v1.2.143+)* ## Collector and Emitter *Data connections, configuration, and collection/emission monitoring* ### Improvements * Updated Emitter to more gracefully clean up failed jobs ### Bug Fixes * **Resource deletion mode default** — Resource-level collection settings now default the deletion mode dropdown to "Inherit from Connection" rather than appearing empty when no explicit value is set. Added a "Select deletion mode" placeholder for clarity. ## Collector *Data connections, configuration, and collection monitoring* ### Improvements * Updated Emitter to limit emission of global id tables to actual changes * **UKG Ready field type changes** *(requires platform v1.2.137+)* - Changed ID field types from INT to BIGINT. ### Bug Fixes * Fixed an issue where emission records were skipped if they were previously emitted, deleted, and re-emitted. ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Cursor field change detection** — When a resource's incremental cursor field is removed from the source schema, a new modal prompts users to select a replacement cursor field or switch to full load. ### Improvements * **Nexl OData pagination** *(requires platform v1.2.135+)* - The Nexl data connector now uses OData nextLink pagination. * **Data source browser height** — The data source browser in the Add Source wizard now fills the full available modal height, making it easier to browse large source trees. ### Bug Fixes * Fixed duplicate breadcrumb entries when navigating connection resource hierarchies. * Fixed flat resources opening an empty nested view instead of navigating to the resource detail page. * Fixed unnecessary re-discovery requests when returning from collection or authentication pages. * Fixed nested resource loading completing prematurely before child data was fetched. ## Pipelines *Data mapping, transformations, and pipeline execution* ### New Features * **Emitter details modal** — View emitter metadata including source schema, trigger type, and plugin parameters from the emission page. * **Full refresh pipeline runs** — v2 engine pipelines now have a "Full Refresh" option in the run dropdown menu. * **Entity list filters** — Filter entities by status (Active, Archived, Needs Review) and deployment status (Deployed, Not Deployed). The list now defaults to Active and Needs Review. ### Improvements * **YAML save error details** — YAML mapping save errors now display the specific error message instead of a generic failure notice. * **Emitter chart padding** — The emitter jobs chart now includes right-side padding so the most recent data point is fully visible. * **Emitter action permissions** — Emitter action menus are now permission-gated: read-only users see only Details, while edit users see all actions. * **Node status copy** — Node and job status badges now display a "Click to copy" hint and copy the status to clipboard on click. ### Bug Fixes * Fixed entity lineage graph not updating when switching to the Lineage tab after making changes in the mapping editor. * Fixed validation error "Locate" not navigating to the correct source or trait node in the mapping editor. * Fixed validation modal appearing for internal-only errors not relevant to users. * Fixed TagInput crash when duplicate tag values are entered. * Fixed dual loading flash on the pipelines list screen. * Fixed tables across the app not resetting to page 1 after data changes, search, or sort. ## Access Control *Data access permissions and security* ### Bug Fixes * Fixed the Test Permissions modal not loading entities or resources correctly. Entity selection is now filtered to relevant entity types, and resources load with proper pagination. ## Administration *System configuration, monitoring, and administrative workflows* ### Bug Fixes * Fixed duplicate firewall policies being allowed. Adding a policy with the same name and type as an existing one now shows an error. * Fixed login URL construction failing on malformed data. * Fixed form data extraction treating empty string values as missing required fields. * Fixed cookie parsing crashes on malformed session data. ## March 2026 ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Fabric Lakehouse Plugin** *(requires platform v1.2.134+)* - New collector plugin for integrating with Microsoft Fabric Lakehouse. ### Improvements * **Support foreign key discovery for SQL data sources** *(requires platform v1.2.133+)* ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Microsoft Dynamics CRM plugin** *(requires platform v1.2.131+)* - New collector plugin for integrating with Microsoft Dynamics CRM. ### Bug Fixes * Fixed an arithmetic overflow issue in SQL connections. * Fixed a datatype mismatch in Emitter. ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Mailchimp plugin** *(requires platform v1.2.124+)* - New collector plugin for integrating with Mailchimp. * **Entra Service Principal authentication** - Azure SQL Server and Fabric connections now support Entra Service Principal as an authentication method, with options for Client Secret or Client Certificate credentials. ### Improvements * **Support for Collection from Fabric** - *(requires platform v.1.2.129+)* Added support for collection from Fabric SQL and Fabric Warehouse connections. * **UKG Ready plugin changes** *(requires platform v1.2.126+)* - Added more resources to UKG Ready collector plugin. * **VNet Check Connection always available** - The Check Connection button is now shown for VNet connections even when status is already Connected, allowing re-validation at any time. ## Pipelines *Data mapping, transformations, and pipeline execution* ### New Features * **Mapping experience v2** - The mapping editor has been redesigned with a guided source setup, inline editing, and improved visualization. * **Source wizard** - Step-by-step guided flow replaces the previous source setup panel * **Inline entity and trait editing** - Edit entities and traits directly from the canvas via click or right-click * **Visual identifier links** - Visual indicators show how sources are linked through shared identifiers * **Composite key visualization** - Composite key relationships are now visualized in the mapping graph * **Prefill trait name from source field** - When mapping a source field to a new custom trait, the trait name is automatically prefilled from the source field name. ### Improvements * **Consistent "trait" terminology** - Standardized UI naming from "field" to "trait" across the mapping editor for consistency. * **Emitter job sorting** - Emitter jobs are now sorted by affected records count for easier prioritization. * **Scheduled job filtering** - Collection jobs can now be filtered by scheduled date with future date ranges (Next 24h, Next 7d) and a clear button on search. ### Bug Fixes * Fixed resource tab switching between nested resources and fields. * Fixed a crash when managing nested resources. ## Access Control *Data access permissions and security* ### Bug Fixes * Fixed incorrect access control tooltip message. ## Administration *System configuration, monitoring, and administrative workflows* ### Improvements * **Improved network connection language** - Updated Site-to-Site casing, improved whitelisting descriptions, and added View Details for VNet and Site-to-Site connections when CIDR ranges are configured. * **VNet Gateway IPs in tunnel config** - VNet Gateway IPs are now displayed directly in the Site-to-Site tunnel configuration dialog. * **Firewall Policies now generally available** - The Firewall Policies page is now available to all users. The legacy firewall rules tab has been removed. ### Bug Fixes * Fixed API endpoints returning incorrect error responses for invalid requests. * Fixed save bar not centering correctly on screen. ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Delimited File - AZ Blob plugin** *(requires platform v1.2.123+)* - New Collector plugin for reading files from Azure Blob storage. * **3E Xcelerate plugin** *(requires platform v1.2.119+)* - New collector plugin for integrating with 3E Xcelerate. ### Improvements * **Zoom plugin field type changes** *(requires platform v1.2.122+)* - Changes 'meeting\_id' and 'id' fields to 'big int' type for zoom resources. * **Emitter update** *(requires platform v1.2.120+)* - Fix for an issue where Emitter would timeout after retrying. * **Improved timeout handling during container resize** - Timeout behavior has been updated to better handle scenarios where compute containers are actively resizing, reducing job failures caused by transient resize delays. ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Name filter search** - Added name filter search for connections, resources, and collections. ### Improvements * **Connection parameter upserts** *(requires platform v1.2.115+)* - Added id to connection parameter for upserts. * **Cross table incremental pulls for Aderant tables** *(requires platform v1.2.114+)* - Added logic to incrementally collect data from Aderant using other tables' LAST\_MODIFIED value. * **Log ADP correlation-ID on failures** *(requires platform v1.2.113+)* ### Bug Fixes * Fixed duplicate resource loads occurring before filters restore. * Fixed connection update failing after a previous failed attempt. ## Pipelines *Data mapping, transformations, and pipeline execution* ### New Features * **Auto Pipeline Engine** *(requires platform v1.2.113+)* - The Pipelines platform now supports a new "Auto" pipeline which greatly enhances pipeline execution. This new pipeline engine is now live, but will be slowly rolled out as needed to existing pipelines. * **Faster Pipeline Execution** * **Fewer Intermediate Tables and Steps** * **Enhanced Observability into Pipeline Runs** * **Intelligent Compute Selection for Improved Performance & Stability** * **Mapping validation UI** - Added mapping validation with modal and visual feedback for improved error visibility. * **Custom expressions in case statements** - Case statements now support custom SQL expressions for more flexible data transformations. * **Centralized SQL expression helpers** - SQL expression helpers are now centralized with alias suffix support. * **Custom query sources in mapping editor** - Custom query sources are now visible and editable directly in the mapping editor. ### Improvements * **Updated View SQL button** - Replaced View SQL icon with outlined text button and updated modal title for better clarity. * **Improved suggested fields UX** - Cleaned up suggested fields experience in custom expression modals. * **Decouple Emissions tab from Statistics tab** - Emissions tab now operates independently from Statistics tab logic. * **Enhanced and specific validation errors in the mapping editor** - Validation errors for data mappings are now specified and made easily addressable in the mapping editor UI. * **Pipeline job details & history** - View the execution details of specific jobs running in data pipelines, tracking when they ran, for how long, and how much data was processed. * **Faster pipeline deployment** - The pipeline deployment process has been optimized, yielding noticeably reduced times to deploy mappings. ### Bug Fixes * Fixed expression editor updates not saving correctly. * Fixed read-only default value display for custom expression sources. * Fixed dropdown stacking order issue in edit field modal. * Fixed custom entities not available for foreign key selection in mapping. * Fixed single quote escaping in SQL expression helpers. * Fixed spaces being allowed in expression alias field. * Replaced unreliable Monaco decoration placeholder with HTML overlay. ## Access Control *Data access permissions and security* ### New Features * **Prerequisite checks for access control** - Added prerequisite validation checks for access control features. * **Read-only access control entities** - Access control entities are now read-only and cannot be deleted for improved data integrity. ## Administration *System configuration, monitoring, and administrative workflows* ### Improvements * **Streaming skeleton loading states** - Added streaming skeleton loading states on homepage and instance settings for improved perceived performance. * **Homepage performance optimization** - Optimized homepage performance with lazy cookie reading and component reuse. ### Bug Fixes * Fixed multiple 3-dot menus opening simultaneously. * Fixed repeated prefetch in CreateGroup modal. * Resolved Svelte 5 reactive loops in MultiRef components. * Fixed MultiRef reactivity issue showing stale data. ## February 2026 ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **UKG Pro plugin** *(requires platform v1.2.111+)* - New collector plugin for integrating with UKG Pro HCM. * **Nexl plugin** *(requires platform v1.2.110+)* - New collector plugin for integrating with Nexl CRM. ### Improvements * **Improve SQL to arrow timestamp conversion** *(requires platform v1.2.110+)* - Date and time SQL types now convert more accurately, resulting in more reliable date and time data across pipelines. ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Dayforce plugins** *(requires platform v1.2.107+)* - New collector plugins for integrating with Dayforce and Dayforce Reports. ### Improvements * **Microsoft SQL Server error detection improvements** *(requires platform v1.2.108+)* - Added support for more Microsoft SQL Server error codes. * **Databricks compute** *(requires platform v1.2.106+)* - Databricks serverless startup and shutdown times have been adjusted to further optimize Databricks serverless cost for the Lakehouse. ### Bug Fixes * Fixed an issue where collection jobs processing no records on first load could fail unexpectedly ## Pipelines *Data mapping, transformations, and pipeline execution* ### New Features * **Custom expressions in source identifiers** - Identifier fields in the mapping editor now support custom SQL expressions in addition to simple field selections. Users can mix and match field picks and custom expressions across multiple identifier rows. ### Improvements * **Connection highlighting in the mapping graph** - Hovering over a node in the mapping editor now highlights all of its connections while dimming unrelated elements. Clicking a node or edge locks the highlight in place, making it easy to trace data flows in complex mappings. * **Editor-level data catalog caching** - The data catalog is now fetched once when the mapping editor opens and shared across all components, eliminating redundant loading when working with multiple sources in the same session. * **Pipeline, emitter, and collection charts upgraded** - All job history charts have been migrated to a new charting library with cleaner rendering, full date/time tooltips, and better performance. Removes visual glitches from the previous charting library. * **Simplified field selector loading** - The source field selector modal now uses the shared catalog cache, resulting in faster load times when opening field selectors for sources you've already worked with. * **Cleaner identifier field management** - The primary and target identifier sections now use a row-based UI with explicit add/remove controls, replacing the previous multi-select dropdowns. ## Administration *System configuration, monitoring, and administrative workflows* ### New Features * **Unified charts**: Consolidated chart components to a single Chart.js implementation across collections, pipelines, and emitter jobs. ### Bug Fixes * Fixed white screens/blank pages when sessions expire — users are now silently re-authenticated and returned to their original page. * Fixed memory leaks and timeouts in onboarding polling components; extended compilation timeout from 30s to 90s. * Fixed UI getting stuck on nested tabs when parent resource fetch fails. ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **CompuLaw plugin** *(requires platform v1.2.104+)* - New collector plugin for integrating with Aderant CompuLaw ### Improvements * **Incremental load support for Microsoft Graph plugin** *(requires platform v1.2.103+)* ## Pipelines *Data mapping, transformations, and pipeline execution* ### Improvements * **Databricks Metastore Management** *(requires platform v1.2.105+)* - Subscription data plane Azure Managed Identity can now effectively manage Entegrata pipeline creation, monitoring and maintenance within the Azure Databricks workspace. ## Collector *Data connections, configuration, and collection monitoring* ### New Features * **Visual indicator for inactive connection schedules** - Connection schedules now display a clear indicator when they are inactive, making it easier to identify paused or disabled collection jobs at a glance. * **Last discovered date display** - The auto-update toggle now shows the last discovered date below it, providing visibility into when resources were last refreshed. * **Courtroom Insights plugin** *(requires platform v1.2.98+)* - New collector plugin for integrating with courtroom insights data systems. * **NetDocuments plugin** *(requires platform v1.2.98+)* - New collector plugin for integrating with NetDocuments document management systems. * **Iridium natural key support** *(requires platform v1.2.98+)* - Enhanced data identification using natural keys for Iridium data sources. * **SharePoint plugin** *(requires platform v1.2.102+)* - New collector plugin for integrating with SharePoint. ### Improvements * **Schema refresh timeout extended** - Increased the schema refresh polling timeout from 60s to 120s, improving reliability for larger data sources with complex schemas. * **Collection chart enhancements** - Improved the collection chart with better empty state handling and proper respect for filter time ranges. * **Case-insensitive SFTP file extensions** *(requires platform v1.2.98+)* - SFTP file path matching now ignores case, improving flexibility when collecting files with varying extension formats. * **Field type modifiers** *(requires platform v1.2.98+)* - More granular control over data type handling in source fields during collection. * **Increased message handling capacity** *(requires platform v1.2.98+)* - Better response times through optimized message processing in the infrastructure layer. * **Improved the way natural keys are identified** *(requires platform v1.2.99+)* * **Ensure primary keys are required fields** *(requires platform v1.2.100+)* - Adds extra checks to ensure primary key fields are set to be required. ### Bug Fixes * Fixed connection profile not refreshing after status checks. * Fixed arithmetic overflow issues in data processing during collection jobs *(requires platform v1.2.98+).* * Various stability and reliability improvements across platform services *(requires platform v1.2.98+).* ## Pipelines *Data mapping, transformations, and pipeline execution* ### New Features * **Source field selector with multi-select** - Added a new source field selector modal with multi-select support, allowing you to add multiple fields to mappings in a single action. * **Type modifiers support** - Added support for type modifiers on data source fields, providing more granular control over data type handling. * **Prevent publishing empty entities** - The system now validates that entities have at least one data source before allowing publish, preventing accidental deployment of incomplete configurations. * **Emitter service** *(requires platform v1.2.98+)* - New backend infrastructure service enabling data emission from pipelines to external systems. Emitters allow you to send transformed pipeline data to external destination systems, with support for manual, scheduled, or event-driven triggers. ### Improvements * **Consistent table row heights** - Fixed mapping table row heights to remain consistent regardless of user permissions, preventing UI shifts. ### Bug Fixes * Fixed an issue with mapping hierarchy navigation. * Added badge for manual pipelines. ## Administration *System configuration, monitoring, and administrative workflows* ### Improvements * **Internal alerting system migration** *(requires platform v1.2.101+)* ### Bug Fixes * Fixed an issue where analytics initialization could delay application load. * Various interface improvements including spacing and alignment fixes.