Integrations fail at the edges

Every integration demo works. Two systems exchange a request and a response, the data appears in the right place, and everyone moves on to the next requirement.

Then the integration meets production, and the problems arrive from the edges: a timestamp without a time zone, a retry that creates a duplicate, a token that expires during a long batch.

After enough implementations, the pattern becomes predictable. The middle of an integration is rarely the hard part. The edges are.

Time is not a detail#

A surprising share of integration defects involve time. Not because time is complicated in principle, but because two systems usually disagree about it.

Common variations include:

• A timestamp with no offset, interpreted differently by each system.
• Local time in a database, UTC in an API, and a display layer that assumes a third thing.
• A "business day" defined by one team and assumed by everyone else.
• Daylight saving transitions that create an hour that does not exist or happens twice.

The fix is not clever. Pick one representation, write it down, and convert at the boundary of each system rather than in the middle of a workflow. Then test the awkward dates explicitly: the daylight saving boundary, the last day of the month, and the end of a billing period.

Retries need a contract#

Retries are usually described as a robustness feature. In practice, a retry is a new request that may create a second side effect.

If an integration creates an invoice, sends an email, or opens a ticket, retrying on a timeout can duplicate that action. The system may have processed the request and failed before responding. The client cannot tell the difference between "not processed" and "processed, response lost".

The practical answers are not glamorous:

• Give each request an idempotency key generated by the caller.
• Make the receiver return the existing result for a repeated key.
• Separate "read" retries from "write" retries in the client.
• Log the request id on both sides so a failure can be traced rather than guessed.

If the receiving system cannot support idempotency, then retries need to be manual or explicitly bounded. Pretending otherwise just moves the duplicate somewhere less visible.

Identity is an edge by itself#

Authentication is where integrations acquire surprising coupling.

A service account that works in a test environment may behave differently in production. A certificate may expire on a date nobody wrote down. A SAML assertion may include attributes that were never needed until a new client used them. A JWT may have a clock skew tolerance that is acceptable for one system and fatal for another.

I keep a simple list for every integration involving identity: who authenticates, what credential is used, when it expires, who owns rotation, and what happens when it fails. That list has caught more problems than most architecture documents.

Write the failure list first#

The habit that has helped most is unglamorous. Before building an integration, I write a list of ways it can fail:

• The other system is unavailable.
• The other system is slow but healthy.
• The response is valid but the data is wrong.
• The request succeeds twice.
• The credential expires.
• The volume is ten times larger than the sample.
• The file is empty, truncated, or has a different encoding than the specification claims.

Then I decide, explicitly, what the system does in each case. Not because every case needs handling — some can be accepted — but because an unexamined failure becomes a production incident with an audience.

The point#

Integration work is often described as plumbing. That undersells it. The actual skill is understanding two systems well enough to know where their assumptions disagree, then designing a boundary where the disagreement is visible and controlled.

The demo is the easy part. The edges are the job.