Why regression tests matter for LLM applications

Traditional software tests often compare a function’s output with an expected result. LLM responses are less uniform: an answer can be worded differently while remaining acceptable, or sound convincing while violating an important product rule. A test suite therefore needs carefully chosen assertions rather than a single exact paragraph used as a “golden” answer.

Research on performance regression in applications that depend on evolving LLM APIs identifies regression as a real engineering concern. Work on LLM-based test generation also describes how generated tests can support regression testing as software changes. But generation is not a substitute for review. The software-engineering survey in the supplied context highlights the test-oracle problem: a test that passes on the current release may still encode the wrong expected behavior. It also reports that generated tests are not guaranteed to be executable. Treat test cases as reviewed product requirements, not unquestionable output from an automated system.

The tool we build here is deliberately modest. It loads named cases, evaluates recorded responses with deterministic checks, writes a JSON report, and returns a failing process status when an assertion fails. The same evaluation logic can later be connected to a live provider adapter after you verify that adapter against the provider’s current official documentation.

Prerequisites and project setup

You need a JavaScript runtime with ES module support and npm. The commands below create a minimal project. The harness itself makes no network requests and does not require API keys, so you can verify the evaluator before introducing API cost, credentials, or provider-specific behavior.

mkdir llm-regression-runner
cd llm-regression-runner
npm init -y
npm pkg set type=module
npm pkg set scripts.test="node --test"
npm pkg set scripts.eval="node src/run.js"
mkdir -p src test fixtures reports

Create .gitignore so dependencies and generated reports are not accidentally committed. Reports can contain test prompts and model responses; review their contents and your organization’s handling requirements before storing or sharing them.

node_modules/
reports/*.json

This example uses Node’s built-in test runner, filesystem, and assertion facilities. It has no third-party package requirement. If your environment uses a different supported runtime or test framework, adapt the commands and verify the result in that environment.

Step 1: Define reviewed cases and saved responses

Create src/cases.js. Each test has a stable identifier, a description that helps explain failures, and explicit checks. These sample cases illustrate policy facts, prohibited language, response length, and a small JSON contract. Replace the fictional policy with rules that your application owners have reviewed. The expected text fragments are examples, not legal or customer-support guidance.

export const testCases = [
  {
    id: "refund-window",
    description: "Includes the reviewed refund window and offers support.",
    checks: {
      requiredSubstrings: ["30 days", "support"],
      forbiddenSubstrings: ["guaranteed refund"],
      maxCharacters: 900
    }
  },
  {
    id: "account-security",
    description: "Directs the user to a password reset and does not request a password.",
    checks: {
      requiredSubstrings: ["password reset"],
      forbiddenSubstrings: ["send me your password", "share your password"],
      maxCharacters: 700
    }
  },
  {
    id: "shipping-json",
    description: "Returns the expected fields and a numeric delivery estimate.",
    checks: {
      requiredJsonKeys: [
        "status",
        "estimated_delivery_days",
        "customer_message"
      ],
      maxCharacters: 600
    }
  }
];

Stable IDs make reports understandable even after you add or reorder cases. Keep descriptions concise but useful to a reviewer. Use required fragments only for facts or wording that genuinely matters to the application. Exact phrase checks can be brittle: equivalent wording may fail, while the presence of a phrase does not prove that the complete answer is correct.

Create fixtures/responses.json with recorded responses for each case. These are example fixtures for testing the harness, not claims about what a particular model will return. In an API-backed workflow, store appropriately reviewed and permitted sample outputs here, or load them from a controlled test-data source.

{
  "refund-window": "The stated refund window is 30 days from delivery. Please contact support to discuss your situation.",
  "account-security": "Do not share your password. Use the password reset flow to regain access to your account.",
  "shipping-json": "{\"status\":\"shipped\",\"estimated_delivery_days\":3,\"customer_message\":\"Your order is on the way.\"}"
}

Step 2: Implement deterministic response checks

Create src/evaluate.js. The evaluator is a pure function: it receives a case and response text, then returns whether the checks passed and a list of reasons for failure. It does not call a model, modify files, or make assumptions about provider response formats. That separation makes failures quick to test and easier to diagnose.

function parseJsonObject(text) {
  try {
    const value = JSON.parse(text);
    if (value === null || Array.isArray(value) || typeof value !== "object") {
      return null;
    }
    return value;
  } catch {
    return null;
  }
}

export function evaluateResponse(testCase, text) {
  if (typeof text !== "string") {
    throw new TypeError("Response text must be a string.");
  }

  const failures = [];
  const checks = testCase.checks ?? {};
  const normalized = text.toLocaleLowerCase("en-US");

  for (const phrase of checks.requiredSubstrings ?? []) {
    if (!normalized.includes(phrase.toLocaleLowerCase("en-US"))) {
      failures.push(`Missing required text: ${JSON.stringify(phrase)}`);
    }
  }

  for (const phrase of checks.forbiddenSubstrings ?? []) {
    if (normalized.includes(phrase.toLocaleLowerCase("en-US"))) {
      failures.push(`Contains forbidden text: ${JSON.stringify(phrase)}`);
    }
  }

  if (Number.isInteger(checks.maxCharacters) && text.length > checks.maxCharacters) {
    failures.push(
      `Response has ${text.length} characters; maximum is ${checks.maxCharacters}.`
    );
  }

  if (Array.isArray(checks.requiredJsonKeys) && checks.requiredJsonKeys.length > 0) {
    const parsed = parseJsonObject(text);

    if (parsed === null) {
      failures.push("Response is not a valid JSON object.");
    } else {
      const actualKeys = Object.keys(parsed).sort();
      const expectedKeys = [...checks.requiredJsonKeys].sort();

      for (const key of expectedKeys) {
        if (!Object.hasOwn(parsed, key)) {
          failures.push(`JSON response is missing required key: ${JSON.stringify(key)}`);
        }
      }

      if (
        actualKeys.length !== expectedKeys.length ||
        actualKeys.some((key, index) => key !== expectedKeys[index])
      ) {
        failures.push(`JSON keys must be exactly: ${expectedKeys.join(", ")}.`);
      }

      if (
        Object.hasOwn(parsed, "estimated_delivery_days") &&
        typeof parsed.estimated_delivery_days !== "number"
      ) {
        failures.push('JSON key "estimated_delivery_days" must contain a number.');
      }
    }
  }

  return { passed: failures.length === 0, failures };
}

This JSON check accepts only raw JSON, not a Markdown code fence. That choice is intentional for a machine-readable fixture: it avoids treating decorated prose as a valid payload. The key check requires exactly the listed keys, which is appropriate only when the contract really is exact. If extra properties are permitted, change the check to require a subset instead. For business-critical data, validate every field and its allowed values using a schema designed for that application.

Step 3: Add a command-line runner and JSON report

Create src/run.js. The runner pairs each case with its saved response, evaluates it, prints a readable result, and writes a timestamped report. It exits unsuccessfully if a fixture is missing or a check fails. A missing fixture is an execution problem, not a successful evaluation, so the runner reports it as a failure rather than silently skipping the case.

import { mkdir, readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { testCases } from "./cases.js";
import { evaluateResponse } from "./evaluate.js";

async function main() {
  const fixtureText = await readFile("fixtures/responses.json", "utf8");
  const fixtures = JSON.parse(fixtureText);
  const results = [];

  for (const testCase of testCases) {
    const responseText = fixtures[testCase.id];

    if (typeof responseText !== "string") {
      results.push({
        testCaseId: testCase.id,
        description: testCase.description,
        passed: false,
        failures: ["No string response fixture was found for this test case."]
      });
      continue;
    }

    const evaluation = evaluateResponse(testCase, responseText);
    results.push({
      testCaseId: testCase.id,
      description: testCase.description,
      passed: evaluation.passed,
      failures: evaluation.failures
    });
  }

  for (const result of results) {
    console.log(`[${result.passed ? "PASS" : "FAIL"}] ${result.testCaseId}`);
    for (const failure of result.failures) {
      console.log(`  - ${failure}`);
    }
  }

  const passed = results.filter((result) => result.passed).length;
  const report = {
    generatedAt: new Date().toISOString(),
    total: results.length,
    passed,
    failed: results.length - passed,
    results
  };

  await mkdir("reports", { recursive: true });
  const filename = `report-${new Date().toISOString().replace(/[:.]/g, "-")}.json`;
  const reportPath = join("reports", filename);
  await writeFile(reportPath, `${JSON.stringify(report, null, 2)}\n`, "utf8");

  console.log(`Completed ${report.total} evaluations: ${report.passed} passed, ${report.failed} failed.`);
  console.log(`Report written to ${reportPath}`);

  if (report.failed > 0) {
    process.exitCode = 1;
  }
}

main().catch((error) => {
  console.error(error instanceof Error ? error.stack : String(error));
  process.exitCode = 1;
});

The report contains test identifiers and assertion failures, but this minimal example omits response text. That reduces the chance of copying sensitive output into a build artifact by default. If you need response text for debugging, add it deliberately and control access, retention, and redaction. A test report is still an engineering data artifact and should be handled accordingly.

Step 4: Test the evaluator without API calls

Create test/evaluate.test.js. These tests exercise the assertion logic directly and run without provider credentials or network access. They demonstrate both a successful response and two distinct failure types.

import test from "node:test";
import assert from "node:assert/strict";
import { evaluateResponse } from "../src/evaluate.js";

test("passes when required and prohibited-text checks are satisfied", () => {
  const result = evaluateResponse(
    {
      checks: {
        requiredSubstrings: ["30 days", "support"],
        forbiddenSubstrings: ["guaranteed refund"],
        maxCharacters: 300
      }
    },
    "The policy states 30 days. Please contact support for help."
  );

  assert.deepEqual(result, { passed: true, failures: [] });
});

test("reports missing required text and a prohibited phrase", () => {
  const result = evaluateResponse(
    {
      checks: {
        requiredSubstrings: ["password reset"],
        forbiddenSubstrings: ["share your password"]
      }
    },
    "Please share your password with support."
  );

  assert.equal(result.passed, false);
  assert.equal(result.failures.length, 2);
});

test("rejects invalid JSON and detects wrong JSON field types", () => {
  const checks = {
    requiredJsonKeys: ["status", "estimated_delivery_days", "customer_message"]
  };

  const invalidJson = evaluateResponse({ checks }, "not JSON");
  assert.equal(invalidJson.passed, false);
  assert.ok(invalidJson.failures.includes("Response is not a valid JSON object."));

  const wrongType = evaluateResponse(
    { checks },
    JSON.stringify({
      status: "shipped",
      estimated_delivery_days: "three",
      customer_message: "On the way"
    })
  );
  assert.equal(wrongType.passed, false);
  assert.ok(
    wrongType.failures.includes(
      'JSON key "estimated_delivery_days" must contain a number.'
    )
  );
});

Run the local checks:

npm test
npm run eval

The test command validates evaluator behavior. The evaluation command checks the saved examples. A passing run means these fixtures satisfy these rules; it does not prove that a model will always respond correctly, that the rules are complete, or that a model provider has not changed its behavior.

Step 5: Connect a live model only after verifying its API

The next step is a provider adapter that accepts a case’s prompt and returns response text. This tutorial intentionally does not supply one. The verified context does not specify the current OpenAI or Anthropic API method names, SDK versions, supported model IDs, request fields, or response shapes. Copying unverified integration code into a tutorial would create a misleading claim of compatibility.

When implementing an adapter, consult each provider’s current official documentation and verify the exact SDK package and version, authentication method, model identifier, request and response structure, timeout handling, and error semantics. Keep the adapter separate from evaluateResponse. That separation lets the same acceptance suite test saved fixtures locally while a distinct integration command exercises live calls. Handle transient errors and rate limits according to the verified provider guidance; do not assume that two providers expose identical errors or retry rules.

For an API-backed evaluation, record enough metadata to investigate changes—such as the configured provider and model identifier, timestamp, and test-case version—without leaking secrets. Before sharing outputs or retaining reports, check whether prompts and responses contain confidential or personal information. Avoid claiming that a particular model is “better” based only on a handful of cases; results are meaningful only relative to the reviewed test set and intended use.

Designing a useful regression suite

Begin with cases linked to actual product requirements, incidents, or known failure modes. Keep each case focused so a failure has an understandable cause. Include negative cases where the assistant should avoid a claim, as well as positive cases where it must preserve a critical fact. For structured outputs, verify parseability, required fields, types, and any constraints that downstream code relies on.

Do not compare complete prose responses to one reference paragraph unless exact text is genuinely required. Phrase-level assertions can catch a missing fact, but they are not semantic proof: a response can contain the required words and still be misleading. Pair deterministic checks with review by qualified people for high-impact cases. If you introduce a model-based judge, treat its score as another fallible signal and validate it against human-reviewed examples before relying on it.

Keep acceptance criteria versioned and review changes as carefully as application code. When a test fails, first inspect whether the output violates a real requirement, the requirement was expressed clearly, or the check is overly brittle. Do not weaken a meaningful rule just to make a build green; equally, remove assertions that merely enforce a stylistic preference. The goal is a suite that surfaces consequential behavior changes and remains trusted by the people who maintain it.

Limitations and next steps

• This example evaluates saved fixtures; it does not make calls to OpenAI or Anthropic.
• Substring and basic JSON checks cannot establish overall factual correctness or quality.
• A passing test set covers only the scenarios represented by its cases; it is not a guarantee of future behavior.
• Generated tests require review. The supplied research notes that generated tests may not be executable and discusses the broader difficulty of determining whether a test’s expected outcome is correct.

Useful extensions include stricter application-specific schemas, reviewed regression cases derived from incidents, controlled API-backed runs, and a report format that captures only the metadata your team needs. Add concurrency, retries, cost accounting, or latency gates only after defining and verifying their requirements for your chosen providers and deployment environment.

Sources and research context

• CoverUp: Coverage-Guided LLM-Based Test Generation discusses LLM-based test generation and its relationship to regression testing.
• Large Language Models for Software Engineering: Survey and Open Problems discusses test correctness, the oracle problem, and limitations of generated tests.
• (Why) Is My Prompt Getting Worse? Rethinking Regression addresses performance regression in applications dependent on evolving LLM APIs.

Editorial attribution: Gate of AI Editorial & Engineering Teams, GateOfAI, LLC.