Documentation Index
Fetch the complete documentation index at: https://support.entegrata.com/llms.txt
Use this file to discover all available pages before exploring further.
Overview
Resource collection settings control how and when individual resources collect data. These settings can override connection-level defaults, allowing fine-grained control over each table, view, or endpoint.Open Settings
Click the Config button (three dots) in the last column for the resourceSelect Collection Settings from the dropdown menu
Configure Schedule
The Collection Settings modal has multiple tabs:
Schedule Tab
Interval-Based ScheduleSet a specific collection interval:- Time-based intervals (e.g., “Every 6 hours”)
- Cron expressions for custom schedules
- Specific times (e.g., “Daily at 3:00 AM”)
Configure Settings (Config Tab)
Load Type
Choose how data is collected:Full Load- Copies all data every time
- Best for small datasets or when you need complete snapshots
- No special source requirements
- Copies only new or changed data
- Requires an incremental load field, also called a cursor field (like
modified_dateorupdated_at) - Much faster for large datasets
- Like Incremental Load, but re-scans a bounded time window on the cursor field every run instead of picking up where the last run left off
- Best when the cursor field exists but isn’t fully reliable (e.g.,
modified_datedoesn’t always update) — re-scanning the window catches records the cursor would otherwise miss - Requires both a cursor field and a configured time window (see below)
- Detects and collects only records that have truly changed by comparing source and Entegrata
- Used when a table does not have a reliable cursor field for incremental loads but has a stable date field (e.g.,
created_date,transaction_date) - More expensive than Incremental Load — choose Incremental whenever a reliable cursor field is available
- See Sync Operation for a full explanation
Incremental Load Field
If you select Incremental or Time-Window Incremental load type, choose the field that tracks when records were last updated:
Time Window
If you select Time-Window Incremental load type, configure the window that bounds each collection run:Relative windowA sliding window anchored to “now”, such as the last 3 months or last 30 days. The window moves forward with each run, so collection always covers the most recent period.Absolute windowA fixed start and end date (e.g.,2025-01-01 to 2025-06-30). The same range is scanned on every run — useful for backfilling a specific historical period.Unique Key(s)
Specify which field(s) uniquely identify each record:
You’ll see: “Primary keys are managed by the source and cannot be changed”
- Single field (e.g.,
id,email) - Multiple fields (composite key, e.g.,
customer_id+order_date)
Configure Filters (Filters Tab)
Add filters to limit what data is collected from this resource.
Creating Filters
- Select a field from the resource
- Choose an operator (equals, greater than, contains, etc.)
- Provide a value to filter on
- Combine multiple filters with AND/OR logic
Example Filters
- Collect only active records:
status = 'active' - Collect recent data:
created_date >= '2024-01-01' - Exclude test data:
email NOT LIKE '%@test.com'
Filters are applied at the source before data is transferred, reducing collection time and storage costs.
Enabling Resources
Turn on data collection for specific resources
Triggering Collection
Manually start collection for testing
Monitoring Jobs
Track collection execution and performance
Troubleshooting
Resolve resource collection issues