Overview
Resources are the individual tables, views, API endpoints, or files within a connection that Entegrata collects. Each resource represents a specific dataset you want to synchronize to your data lakehouse.Key Concepts
What is a Resource?
A Resource is a collectible data entity discovered within a connection:The resource identifier (table name, view name, API endpoint path, or file name)
The kind of resource:
- Table - Database table with data rows
- View - Database view (virtual table)
- Endpoint - API endpoint providing data
- File - Data file or spreadsheet
Collection state:
- Active - Resource is collecting data
- Inactive - Resource is not collecting
Columns or properties within the resource (e.g., customer_id, name, email)
Resource Lifecycle
Discovery
Resources are automatically discovered when you create a connection or when discovery runs (every 3 hours)
Resource Types
Tables
Database tables containing rows of structured data:- Most common resource type
- Have defined schemas with columns and data types
- Support full and incremental loads
- May have primary keys defined by the database
Views
Database views (virtual tables based on queries):- Derived from one or more tables
- Schema defined by the view query
- Typically use full load (incremental may not be supported)
- Primary keys must be user-defined
Endpoints
API endpoints providing structured data:- REST API paths or GraphQL queries
- Schema discovered from API responses
- Support pagination and filtering
- May have rate limits
Files
Files containing structured data:- Excel spreadsheets, CSV files, JSON documents
- Schema inferred from file structure
- May contain multiple sheets or sections
- Typically use full load
Resource vs. Connection
Understanding the relationship:| Aspect | Connection | Resource |
|---|---|---|
| Scope | Entire data source | Single table/view/endpoint |
| Discovery | Discovers all resources | Discovered within connection |
| Schedule | Default for all resources | Can override connection default |
| Enable/Disable | Affects all resources | Affects only this resource |
| Authentication | Stores credentials | Inherits from connection |
| Collection | May trigger all resources | Triggers single resource |
Collection Settings
Each resource can be configured with:supp
Resource Status Indicators
Resources display status information:Active Resource
- Toggle is ON (green)
- Collects according to schedule
- Jobs appear in collection history
Inactive Resource
- Toggle is OFF (gray)
- Does not collect data
- No jobs created
Load Type Indicators
Resources show their configured load type:
- Full - Full load every collection
- Incremental - Incremental load with change tracking
Getting Started with Resources
Viewing Resources
Browse and search discovered resources
Enabling Resources
Turn on data collection for specific resources
Collection Settings
Configure schedules, load types, and filters
Triggering Collection
Manually start collection jobs
Common Workflows
Initial Setup After Creating Connection
Ongoing Maintenance
- Monitor collection jobs - Check for failures or performance issues
- Handle schema changes - Respond to discovery alerts about schema modifications
- Adjust schedules - Optimize timing based on data freshness needs
- Update filters - Refine data selection criteria
Best Practices
Start Small
Enable a few resources first, verify they work, then add more
Use Incremental When Possible
Incremental loads are faster and more efficient for large tables
Set Unique Keys
Always define unique keys to ensure proper deduplication
Apply Filters Early
Filter at the source to reduce data transfer and storage
Monitor First Collections
Watch initial jobs to catch configuration issues early
Document Decisions
Keep notes on why certain resources are enabled/disabled