Skip to main content

​
Overview

Resource collection settings control how and when individual resources collect data. These settings can override connection-level defaults, allowing fine-grained control over each table, view, or endpoint.
1

Open Settings

Click the Config button (three dots) in the last column for the resourceSelect Collection Settings from the dropdown menu
2

Configure Schedule

The Collection Settings modal has multiple tabs:

​
Schedule Tab

Interval-Based ScheduleSet a specific collection interval:
  • Time-based intervals (e.g., “Every 6 hours”)
  • Cron expressions for custom schedules
  • Specific times (e.g., “Daily at 3:00 AM”)
Resource interval schedule configuration
Active Status ToggleEnable or disable collection for this resource
3

Configure Settings (Config Tab)

​
Load Type

Choose how data is collected:Full Load
  • Copies all data every time
  • Best for small datasets or when you need complete snapshots
  • No special source requirements
Incremental Load
  • Copies only new or changed data
  • Requires an incremental load field (like modified_date or updated_at)
  • Much faster for large datasets
Load type selector

​
Incremental Load Field

If you select Incremental load type, choose the field that tracks when records were last updated:
Incremental load field selector
The incremental load field must:
  • Exist in the source data
  • Update whenever a record changes
  • Be a date, datetime, or timestamp type

​
Unique Key(s)

Specify which field(s) uniquely identify each record:
Unique keys selector
Source-Managed KeysIf the source system manages primary keys (e.g., database tables with PRIMARY KEY constraints), Entegrata uses those automatically and they cannot be changed.
You’ll see: “Primary keys are managed by the source and cannot be changed”
User-Defined KeysFor sources without explicit primary keys (e.g., views, API endpoints, files), select one or more fields that uniquely identify records:
  • Single field (e.g., id, email)
  • Multiple fields (composite key, e.g., customer_id + order_date)
Unique keys are used to:
  • Detect duplicate records
  • Merge updates in incremental loads
  • Ensure data quality
4

Configure Filters (Filters Tab)

Add filters to limit what data is collected from this resource.

​
Creating Filters

  1. Select a field from the resource
  2. Choose an operator (equals, greater than, contains, etc.)
  3. Provide a value to filter on
  4. Combine multiple filters with AND/OR logic

​
Example Filters

  • Collect only active records: status = 'active'
  • Collect recent data: created_date >= '2024-01-01'
  • Exclude test data: email NOT LIKE '%@test.com'
Filters are applied at the source before data is transferred, reducing collection time and storage costs.
5

Save Configuration

Click Save to apply the resource collection settings

Enabling Resources

Turn on data collection for specific resources

Triggering Collection

Manually start collection for testing

Monitoring Jobs

Track collection execution and performance

Troubleshooting

Resolve resource collection issues