Quality Engineering

Appearance

Quality Engineering Research Digest — 1 August 2026 ​

A focused review of current API-testing practice with Zod, with emphasis on OpenAPI as the source of truth, generated runtime schemas, Playwright and Vitest integration, diagnostic failures and the limits of schema validation.

Executive summary ​

Zod is a strong runtime validation library for TypeScript API tests, but it is not a complete API contract-testing solution. The current best practice is to keep OpenAPI as the shared contract, derive Zod schemas from it where practical, and use those schemas inside API tests to validate real response bodies.

The five main conclusions are:

  1. OpenAPI should remain the source of truth; Zod should be a generated or deliberately scoped runtime projection.
  2. Schema validation and functional assertions solve different problems and should remain separate.
  3. Response validation should avoid coercion, defaults and transforms that can hide provider defects.
  4. Playwright and Vitest are both suitable runners; Zod supplies validation, not transport or test orchestration.
  5. OpenAPI-driven property-based and stateful testing should complement example-based Zod tests.

Selected sources and developments ​

1. Zod 4 and native JSON Schema support ​

Zod 4 is now the current major version. It adds native JSON Schema conversion, metadata registries and improved error formatting. It also favours z.strictObject() and z.looseObject() over the older .strict() and .passthrough() pattern.

These changes make Zod easier to use in API tooling, but native JSON Schema conversion is primarily Zod-to-JSON-Schema. It does not remove the source-of-truth decision. If an organisation already governs its API through OpenAPI, manually maintained Zod schemas still create a second contract unless they are generated or intentionally limited to consumer expectations.

Quality engineering implications ​

  • Standardise new work on Zod 4 rather than creating more Zod 3 patterns.
  • Generate schemas from OpenAPI where tool support is adequate.
  • Treat manually written Zod schemas as test adapters or consumer expectations, not the canonical API definition.
  • Review generated output when upgrading either Zod or the generator.

2. OpenAPI 3.1 and 3.2 align API schemas with modern JSON Schema ​

OpenAPI 3.1 moved its Schema Object onto JSON Schema Draft 2020-12 foundations, and OpenAPI 3.2 continues the modern specification line. This matters because schema-generation and validation tools can work from a more standards-aligned contract than was possible with OpenAPI 3.0's restricted schema dialect.

The practical architecture is:

text
OpenAPI contract
      ↓
Generated TypeScript types and Zod schemas
      ↓
Playwright or Vitest API tests
      ↓
Runtime response validation plus business assertions

Quality engineering implications ​

  • Validate the OpenAPI document itself in CI.
  • Generate test-side types and runtime schemas from the same reviewed specification.
  • Make schema generation deterministic and fail CI when generated files drift.
  • Test documented status codes, content types, headers and error responses; body validation alone is insufficient.

3. OpenAPI-to-Zod generation is becoming the maintainable path ​

Orval can generate Zod schemas from an OpenAPI document and combine them with generated TypeScript clients. This direction matches an OpenAPI-first operating model: the specification remains authoritative, while application and test code consume generated artefacts.

This is preferable to manually duplicating every API model in Zod. However, generation is not automatically correct. OpenAPI dialects, formats, nullable fields, unions, discriminators, recursive schemas and additionalProperties can expose differences between tools.

Quality engineering assessment ​

Adopt generation only with a small compatibility suite. Select representative schemas from the real API and prove that the generator handles them correctly before rolling it across the full specification. Pin generator versions and review generated diffs during upgrades.

Avoid selecting a generator only because it produces concise TypeScript. The important question is whether its runtime behaviour preserves the OpenAPI semantics used by the service.

4. Playwright API testing provides the execution layer ​

  • Source: Microsoft Playwright
  • Current documentation: Accessed 1 August 2026
  • Trend: API tests integrated with browser workflows and fixtures
  • Reference: Playwright API testing

Playwright's APIRequestContext can test APIs directly, prepare server state before browser tests and verify server-side postconditions after UI actions. Zod fits into this layer by validating the untrusted JSON body returned by the service.

A useful pattern is:

ts
import { expect, test } from '@playwright/test'
import * as z from 'zod'

const PetResponse = z.strictObject({
  id: z.string(),
  name: z.string(),
  status: z.enum(['available', 'pending', 'sold'])
})

function validate<T extends z.ZodType>(schema: T, body: unknown): z.output<T> {
  const result = schema.safeParse(body)

  if (!result.success) {
    throw new Error(
      `Response schema validation failed:\n${z.prettifyError(result.error)}`
    )
  }

  return result.data
}

test('creates a pet', async ({ request }) => {
  const response = await request.post('/pets', {
    data: { name: 'Milo', status: 'available' }
  })

  expect(response.status()).toBe(201)
  expect(response.headers()['content-type']).toContain('application/json')

  const pet = validate(PetResponse, await response.json())

  expect(pet.name).toBe('Milo')
  expect(pet.status).toBe('available')
})

The important separation is visible:

  • Playwright validates HTTP behaviour and runs the workflow.
  • Zod validates the response structure and types.
  • Explicit assertions validate the requested outcome and business rules.

Vitest can use the same schema and helper pattern. The runner choice does not change Zod's responsibility.

5. Schema-driven property and stateful testing covers a different risk class ​

Zod tests normally validate examples chosen by the test author. Schemathesis instead generates inputs from OpenAPI, checks response conformance and server errors, minimizes failures and can chain operations using real values from earlier responses.

The latest reviewed release added dynamic OAuth 2.0 and OpenID Connect authentication support, including token refresh and request replay. This makes schema-driven testing more practical for protected enterprise APIs.

Quality engineering implications ​

  • Keep deterministic Playwright or Vitest tests for known business examples and release gates.
  • Add schema-driven testing for boundary values, malformed inputs and undocumented server failures.
  • Add stateful testing for create-read-update-delete flows and dependent operations.
  • Do not attempt to recreate broad fuzzing coverage by writing hundreds of Zod examples manually.
LayerPrimary artefactMain questionSuitable tooling
Contract definitionOpenAPIWhat has the API promised?OpenAPI linting and review
Runtime body validationGenerated or scoped Zod schemaDoes this real payload have the expected structure and types?Zod 4
Protocol validationTest assertions or OpenAPI-aware validatorAre status, headers, media type and documented responses correct?Playwright, Vitest, OpenAPI validators
Functional API testingDeterministic test scenarioDid the operation produce the correct business result?Playwright or Vitest
Workflow testingMulti-call scenarioDo state transitions and dependent operations work?Playwright, Vitest, Arazzo-aware tooling
Generative testingOpenAPI-derived propertiesWhat edge cases did humans fail to enumerate?Schemathesis or equivalent

Practices that prevent false confidence ​

Validate raw responses without repairing them ​

Avoid coercion, defaults and transforms in provider-conformance schemas:

ts
// Can hide an API defect by converting "42" to 42.
const WeakResponse = z.object({ count: z.coerce.number() })

// Fails when the provider returns the wrong type.
const StrongResponse = z.object({ count: z.number() })

Coercion can be appropriate at an application boundary where normalization is intentional. It is usually wrong in a test whose purpose is to detect a provider contract violation.

Choose unknown-key behaviour deliberately ​

z.object() strips unknown keys by default. That can hide unexpected fields if the test assumes it is validating the complete payload. z.strictObject() rejects unknown keys, but using it everywhere can also make harmless additive API changes fail consumer tests.

Use two distinct goals:

  • Provider conformance: validate the complete response according to OpenAPI and its additionalProperties rules.
  • Consumer expectation: validate only the fields and invariants the consumer actually depends on.

Do not mix the two and call both “contract testing.”

Keep dynamic expectations outside reusable schemas ​

Schemas should normally describe stable structure and invariant rules. Values that depend on the request, fixture or test state belong in assertions:

ts
const pet = validate(PetResponse, await response.json())

expect(pet.id).toBe(createdPetId)
expect(pet.name).toBe(requestBody.name)

This keeps schemas reusable and failures easier to diagnose.

Make validation failures readable ​

Use safeParse() and Zod 4's z.treeifyError() or z.prettifyError() to produce useful CI output. A raw ZodError without request, endpoint and response context is difficult to investigate.

At minimum, failure evidence should include:

  • HTTP method and sanitized URL.
  • Expected response/status variant.
  • Zod issue path and message.
  • Correlation or trace ID when available.
  • A sanitized response excerpt or attached test artefact.

Trend synthesis ​

TrendChange to quality engineering
OpenAPI-first validationZod schemas are derived test artefacts rather than a competing contract.
Runtime validationTypeScript types are supplemented by checks against real, untrusted payloads.
Layered API assertionsSchema, protocol, functional and workflow failures are reported separately.
Generated test assetsClients, types, schemas and mocks increasingly come from one reviewed API definition.
Generative API testingDeterministic examples are complemented by property-based and stateful exploration.
  1. OpenAPI 3.2.0
  2. Zod 4 release notes
  3. Orval — Generate Zod schemas from OpenAPI
  4. Playwright — API testing
  5. Schemathesis documentation

For a team already using Playwright, Vitest and OpenAPI:

  1. Keep OpenAPI as the single source of truth.
  2. Validate and lint the OpenAPI document in CI.
  3. Trial OpenAPI-to-Zod generation on a representative subset of endpoints.
  4. Create one shared Zod validation helper with readable failure output.
  5. Validate success and error response bodies after asserting status and media type.
  6. Keep request-dependent values and business rules in normal test assertions.
  7. Separate provider-conformance tests from frontend consumer-expectation tests.
  8. Add schema-driven property testing as a separate CI job, starting with high-risk APIs.
  9. Measure escaped contract defects, validation failures, execution time and maintenance cost before scaling.

Final assessment ​

Zod is a good fit for TypeScript API testing. It is lightweight, readable and integrates cleanly with Playwright and Vitest. The mistake would be to elevate a set of hand-written Zod schemas into a second API contract or to treat successful body parsing as proof that the API works.

The strongest design is:

OpenAPI defines the promise. Zod checks real payloads at runtime. Test assertions verify behaviour. Generative testing searches beyond the examples humans wrote.

Source note ​

This document is an independent study summary based primarily on official specifications, project documentation and release records. Tool capabilities and compatibility should be verified against a representative subset of the organisation's real OpenAPI definitions before adoption.

A personal quality engineering knowledge base.

Content maintained by Tim Yao