Error & Retry Handling

​

Overview

Collections run against external systems — databases, APIs, and file stores — that occasionally return temporary errors. To keep collections resilient, Entegrata automatically retries errors it recognizes as transient and fails fast on errors it recognizes as permanent. This happens behind the scenes, so most temporary glitches never result in a failed job. This page explains, at a high level, what gets retried and what happens when it doesn’t.

​

Transient vs. permanent errors

When a collection step hits an error, Entegrata classifies it before deciding what to do.

​

How retries work

When an error is transient, Entegrata retries the operation automatically:

• Multiple attempts. Collection steps retry up to several times before giving up.
• Exponential backoff with jitter. The wait between attempts grows with each retry, and a small random offset is added so many concurrent retries don’t hammer the source system in sync.
• Respecting the source system. When a source returns a rate-limit response with a Retry-After hint, Entegrata honors that wait (up to a reasonable cap) instead of retrying immediately.

​

When retries are exhausted

If an operation keeps failing after all retry attempts, the job — or the specific resource within it — is marked as Failed, and the error reason is recorded on the job so you can review it. Two outcomes are treated as not failures:

• Clean early stops. When a source signals there’s no more data to collect (for example, the last page of an API), the job finishes normally.
• Resizing. If a job needs more capacity than its assigned worker, it is automatically re-queued onto a larger worker rather than failing.

​

Partial failures

Collection is tracked per resource. If a connection collects several resources and one of them fails, the others continue and report their own outcomes — a single failing resource does not abort the rest. Entegrata also validates row counts when loading data. If the number of records loaded into the Lakehouse doesn’t match what was collected (after an automatic refresh retry), that resource is failed rather than reported as a partial success. This prevents a job from silently completing with incomplete data.