Overview
Resource collection settings control how and when individual resources collect data. These settings can override connection-level defaults, allowing fine-grained control over each table, view, or endpoint.Open Settings
Click the Config button (three dots) in the last column for the resourceSelect Collection Settings from the dropdown menu
Configure Schedule
The Collection Settings modal has multiple tabs:
Schedule Tab
Interval-Based ScheduleSet a specific collection interval:- Time-based intervals (e.g., “Every 6 hours”)
- Cron expressions for custom schedules
- Specific times (e.g., “Daily at 3:00 AM”)
Configure Settings (Config Tab)
Load Type
Choose how data is collected:Full Load- Copies all data every time
- Best for small datasets or when you need complete snapshots
- No special source requirements
- Copies only new or changed data
- Requires an incremental load field (like
modified_dateorupdated_at) - Much faster for large datasets
Incremental Load Field
If you select Incremental load type, choose the field that tracks when records were last updated:
Unique Key(s)
Specify which field(s) uniquely identify each record:
You’ll see: “Primary keys are managed by the source and cannot be changed”
- Single field (e.g.,
id,email) - Multiple fields (composite key, e.g.,
customer_id+order_date)
Configure Filters (Filters Tab)
Add filters to limit what data is collected from this resource.
Creating Filters
- Select a field from the resource
- Choose an operator (equals, greater than, contains, etc.)
- Provide a value to filter on
- Combine multiple filters with AND/OR logic
Example Filters
- Collect only active records:
status = 'active' - Collect recent data:
created_date >= '2024-01-01' - Exclude test data:
email NOT LIKE '%@test.com'
Filters are applied at the source before data is transferred, reducing collection time and storage costs.
