Skip to content

Testing Distributed Systems Across Delivery Teams ​

Executive summary ​

Large software systems are often delivered by multiple teams that own separate applications, services, data pipelines and platforms. Every team can report green tests while a critical customer journey fails across team boundaries. This is not necessarily a lack of tests: it is often a failure of system-level ownership, interface assurance, integration readiness and shared feedback.

Recommendation: use a federated quality engineering operating model. Delivery teams own quality and verification of their changes; a small central or cross-cutting QE capability establishes minimum controls, makes systemic risk visible, enables shared tooling and coordinates assurance for critical cross-team flows. A named system or value-stream owner remains accountable for the end-to-end outcome.

This is a proposed operating model, not a formally standardised industry framework or an empirically proven universal optimum. The fit depends on service coupling, team maturity, release dependencies, regulatory constraints and organisational authority.

The intended end state is not a large central QA phase. It is a system in which teams can verify and deploy independently for routine changes, while genuinely cross-service risks receive explicit ownership and evidence-based assessment.

The failure mode ​

Consider six delivery teams responsible for a frontend, identity service, business APIs, ETL, third-party integrations and cloud platform. Each can demonstrate local component readiness. Yet an end-to-end transaction might still fail because:

  • providers and consumers disagree about field meaning or backward compatibility;
  • asynchronously delivered events are duplicated or reordered;
  • authorisation context is not carried across boundaries;
  • a shared environment has incompatible versions, inaccessible dependencies or unsuitable data;
  • failure recovery, observability and rollback were never validated as a whole.

A programme that waits until a late SIT window to find these problems has converted integration testing into its first effective integration control.

Root causes, not just symptoms ​

LevelTypical causeConsequence
TechnicalContracts, message semantics, versions and failure modes are weakly specified or verifiedLocally correct services fail together
Delivery processTeams integrate late; SIT entry is assumed from deployment successSlow defect discovery and expensive rework
OwnershipNobody owns an entire customer journey or cross-team dependencyRisks fall between delivery backlogs
OrganisationTeams are rewarded for local completion dates rather than usable system outcomesOptimisation of local green status
FeedbackTest results, traces, incident data and dependency status are fragmentedSystem-level readiness cannot be demonstrated

DORA's guidance on loosely coupled teams explicitly links effective delivery to teams being able to test and deploy on demand, without requiring extensive coordination or a common integrated environment. This is both an architecture problem and an operating-model problem.

Compare the operating models ​

ModelStrengthFailure modeAppropriate context
Centralised QAConsistent oversight and specialist capacityHandoffs, late feedback, QA becomes delivery gatekeeperTemporary migration or highly constrained legacy delivery
Fully embedded QEFast feedback and close engineering collaborationDuplication and cross-team assurance gapsSmall, loosely coupled services with clear owners
Federated QETeam ownership plus system-wide assurance and enabling capabilityRequires clear decision rights; central layer can become bureaucraticMultiple teams sharing critical business journeys or platforms

A federated model is not a reporting hierarchy in which one Principal QE approves all tests. It is a distributed ownership arrangement with targeted coordination for cross-boundary risks. Team Topologies is a useful conceptual basis for separating stream-aligned delivery, platform services and enabling capabilities; it does not itself prescribe the QE model in this article. See Team Topologies — Key Concepts.

Quality activityAccountable ownerCentral / cross-cutting QE contribution
Unit, component and local service testsOwning engineering teamGuidance, minimum expectations, coaching
API and event contract compatibilityProvider and consumer teams, with named interface ownerCommon tooling and visibility of incompatible versions
Cross-service integration scenariosNamed flow owner and participating teamsRisk mapping, coordination and evidence synthesis
Critical end-to-end journeysBusiness/system owner with participating teamsRisk-based suite strategy and reporting
Performance, security and resilienceSystem/service owner with relevant specialistsCross-cutting strategy and assurance
Environment and test data readinessPlatform/environment owner; data ownersDefine testability requirements and expose blockers
Production quality and reliabilityService owner and operations/SREConnect incidents and reliability signals to engineering controls
Release decision and accepted residual riskAuthorised delivery/product/release decision-makerIndependent assessment of evidence and quality risk

Decision-right distinction: QE assesses test coverage, readiness and residual quality risk; delivery and product leadership decide priorities, release timing and risk acceptance within organisational policy. A QA lead should not become accountable for software that other teams have not completed.

A useful rule is: every critical cross-team journey must have one directly responsible individual (DRI) for coordinating its verification, with named provider and consumer owners for each boundary. Coordination accountability does not eliminate each team's technical responsibility.

Verification architecture: test at the cheapest trustworthy boundary ​

The UK Home Office's Quality assurance and testing guidance, updated July 2025, recommends weighting component and API integration tests more heavily than UI-driven end-to-end tests where architecture allows, embedding resilience, accessibility and security, and using only targeted E2E coverage.

LayerMain questionTypical execution
Unit / domainIs the business logic correct?Fast checks on each change
Component / local integrationDoes the service behave with its database, queues or adapters?Team-owned CI, reproducible dependencies
Contract / compatibilityCan released provider and consumer versions coexist?Both sides' pipelines; version-aware compatibility checks
Cross-service integrationDo real boundaries, messages, data and failure recovery work?Continuous, risk-selected integrated verification
Critical business E2ECan a user complete a valuable business outcome across the system?Small, resilient, production-like journey suite
Non-functional / operationalDoes the system meet reliability, performance, security and recovery objectives?Component and whole-system checks, including production feedback

Do not infer a fixed numerical test pyramid ratio. Choose verification at the lowest layer that can faithfully expose the risk.

For example, a consumer-driven contract can detect a removed API field before deployment. It does not prove the business semantics of an ETL transformation, distributed transaction recovery or the correctness of an entire customer journey. These need additional targeted tests.

Pact's can-i-deploy illustrates version-aware deployment compatibility checks. It is a product-specific implementation example, not evidence that every architecture needs Pact.

For asynchronous services, prioritise idempotency, retries, ordering assumptions, dead-letter handling, eventual consistency and reconciliation. For shared auth and data flows, prioritise propagation of identity, permissions, data lineage and negative cases.

Three lightweight cross-team controls ​

1. Dependency and interface ownership map ​

For each critical or changing integration, record provider, consumer, interface owner, contract, deployed versions, data semantics, known dependencies, validation evidence and escalation route.

Use the map to identify unowned edges and high-risk changes; avoid creating a complete inventory solely for governance.

2. Evidence-based integration readiness ​

An integrated environment being available does not mean a candidate is ready for formal cross-system testing.

Minimum expectations might include:

  • required component scope and version identifiers are known;
  • local business-rule, component and contract checks pass;
  • dependent endpoints and necessary data are available;
  • deployment and smoke checks pass;
  • logs and traces permit failure diagnosis;
  • exceptions and missing functionality have named owners.

Entry states: ready; conditional entry with a documented non-critical exception and authorised risk acceptance; or not ready when a critical dependency is missing. Automate these checks where practical.

For a fuller treatment of this specific failure mode, see Starting SIT Is Not the Same as Being Ready for SIT.

3. Shared system-quality evidence ​

Aggregate existing CI/CD and operational signals rather than asking each team to write another status report. A useful view shows:

  • contract and integration compatibility by dependency;
  • critical journey verification and last successful run;
  • environment/test-data blockers and their age;
  • unresolved cross-team defects, owners and impact;
  • release exceptions and explicit acceptance decisions;
  • production SLOs, incidents and regression signals.

Every red condition needs an owner and decision path. An unowned “blocked by another team” ticket is a governance failure, not a valid long-term status.

Release assurance without a universal QA sign-off ​

Adopt risk-based system readiness assessment:

  1. Independent low-risk change: service-local tests and contractual checks run automatically; a team deploys within established guardrails.
  2. Material interface or data change: provider/consumer compatibility and selected integration flows provide additional evidence.
  3. Coordinated high-risk release: a named release/system owner assesses critical journeys, operational readiness, security/reliability risks, unresolved defects and rollback options.
  4. Exception: the authorised decision-maker accepts or rejects a clearly described residual risk. QE documents its assessment without silently owning the decision.

A deployment should be observable and recoverable, not merely test-green. Google's SRE workbook on implementing SLOs provides a basis for service-level reliability goals and error-budget decisions. Trace propagation across services improves diagnosis; see OpenTelemetry Context Propagation.

Measurements that reveal systemic risk ​

Avoid treating test counts, pass percentages or pre-project “expected defect rates” as measures of overall quality. Prioritise actionable signals:

SignalWhat it can revealCaution
Cross-team integration failure frequencyUnstable service boundariesNormalise by change volume and severity
Time from change to compatibility feedbackHow early contracts detect failuresSeparate flaky checks from real incompatibility
Percentage of changes independently testable/deployableArchitectural coupling and coordination costNot every regulated system can deploy independently
SIT entry failures / blocked timeWeak readiness controls and environment dependenciesAn initial rise may mean improved detection
Escaped incidents affecting critical journeysMissed system-level risksLow volumes are statistically noisy
Change failure and recovery timeProduction impact and resilienceInterpret within change type and service criticality
SLO attainment / error-budget burnActual user-facing reliabilityRequires sound service-level objectives

DORA provides a loosely coupled teams capability and continuous integration guidance that can inform assessment, but none of these metrics alone proves a particular QE operating model caused an outcome.

Practical implementation ​

A sensible pilot involves one critical business journey, not an organisation-wide QA reorganisation.

First two weeks — map accountability. Identify journey owner, participating services, contracts, missing controls and current failure-detection points. Document who can accept risks and unblock dependencies.

Next three weeks — shift checks left. Implement version compatibility checks for the highest-risk interface changes, agree minimum readiness evidence and make CI results available to relevant teams.

Next three weeks — close system-level gaps. Automate a small number of critical end-to-end checks and cross-service negative scenarios; link failure diagnostics and create a shared exception view.

Continuous — adapt the model. Review incidents, coordination delays, testing bottlenecks and the cost of shared environments. Move checks into team-owned pipelines when they can reliably replace late-stage verification.

This is an illustrative implementation sequence, not an evidence-backed universal eight-week delivery estimate.

Risks and trade-offs ​

  • A central QE bottleneck may reappear. Avoid mandatory approvals for ordinary changes; reserve human assessment for defined risk thresholds.
  • Teams may optimise local green status. Assign explicit owners to critical journeys and their production outcomes.
  • Contract tests can create false confidence. They establish selected compatibility properties, not complete semantic or business correctness.
  • Shared integrated environments are costly and flaky. Reduce dependence through isolated tests and version-aware contracts; retain realistic system checks where needed.
  • Strict consistency can suppress autonomy. Standardise evidence, critical controls and ownership, not every team's test framework or implementation details.
  • Metrics can be gamed or misread. Use trends and incident learning for improvement, not simple inter-team league tables.
  • Legacy coupling and regulatory rules may limit autonomy. In those cases, use stronger coordinated release controls while reducing architectural coupling over time.

Key takeaways ​

  1. Quality belongs to delivery teams; system-level risk must also have a named owner.
  2. The most important gap is often between teams, not within a test suite.
  3. Automated boundary checks and early integration reduce dependence on late SIT.
  4. Central QE should enable consistency, diagnose systemic risk and challenge evidence—not execute or sign off everything.
  5. Release readiness is a risk decision informed by test and operational evidence.
  6. Judge improvement by independent testability, earlier feedback and fewer user-facing failures, not by the number of tests.

Evidence and references ​

The following are primary engineering guidance or practitioner frameworks; they support specific practices, not a controlled comparative claim that federated QE is universally superior.

  1. UK Home Office — Quality assurance and testing (updated 25 July 2025) — layered testing, integrated quality, risk and measurable outcomes.
  2. DORA — Loosely coupled teams (updated 20 October 2025) — independent testing/deployment, architecture, contracts and team coordination.
  3. DORA — Continuous integration — rapid feedback and automated builds.
  4. Team Topologies — Key Concepts — stream-aligned, platform and enabling team concepts.
  5. Microsoft — Architecture strategies for testing — test architecture and entry/exit considerations.
  6. Pact — Can I Deploy — implementation of version compatibility decisions.
  7. Google SRE Workbook — Implementing SLOs — service objectives and reliability-informed decisions.
  8. OpenTelemetry — Context propagation — correlation of distributed requests.

Further research needed: comparative case studies of governance structures, central QE staffing and decision authority; contractual semantics for event-driven systems; cross-team test data and environment ownership; and evidence of effectiveness across enterprise and government delivery environments.

A personal quality engineering knowledge base.