Why an Evaluation Gate Belongs Before Fine-Tuning

Fine-tuning changes model behavior using task-specific examples. Research on large-language-model adaptation distinguishes fine-tuning from prompt engineering: prompting guides a model at inference time, while fine-tuning adapts behavior from a training corpus. A broad review of fine-tuning practice describes a lifecycle that spans data preparation, model initialization, optimization, evaluation, and deployment. That lifecycle is important because a completed training run is not itself evidence that a model should be released.

A release gate turns that lifecycle into an engineering control. Before any training submission, validate that examples follow the expected schema, that target responses are present, and that the holdout set is separate. After a provider returns a candidate model, run the same holdout prompts against the baseline and candidate, calculate task-specific metrics, preserve the evidence, and approve promotion only when the defined requirements pass.

This pattern is especially useful for stable and measurable tasks such as classification, extraction, controlled formatting, routing, and code-review conventions. It is less suitable as a way to inject rapidly changing facts. When a workflow needs current policy, inventory, account, or incident information, obtain that information from approved retrieval or internal systems at runtime rather than assuming a fine-tuned dataset remains current.

Prerequisites

• Python 3.10 or newer.
• A labeled training dataset and a separately curated holdout dataset.
• A baseline model and a candidate model that can both be invoked through your organization’s approved inference path.
• An approved process for reviewing data rights, sensitive information, and provider data-processing requirements before remote submission.
• Basic familiarity with JSON, JSON Lines, command-line tools, and Python virtual environments.

The code below uses only the Python standard library. It is therefore useful before choosing a provider and does not make unverified assumptions about a particular SDK. Its input is JSONL data and saved model predictions. Your approved provider integration can generate those predictions later.

Step 1: Create Separate Training and Holdout Files

Use JSONL, where each non-empty line is one JSON object. For this tutorial, the task is support routing. The training file includes an assistant target. The holdout file includes an expected label used only by the evaluator. Do not submit the expected metadata as part of a provider training file unless its documented format explicitly permits it.

Create data/train.jsonl:

{"messages":[{"role":"system","content":"Return JSON with queue and priority."},{"role":"user","content":"My invoice shows two annual subscription charges."},{"role":"assistant","content":"{\"queue\":\"billing\",\"priority\":\"high\"}"}]}
{"messages":[{"role":"system","content":"Return JSON with queue and priority."},{"role":"user","content":"The dashboard fails when I export usage data."},{"role":"assistant","content":"{\"queue\":\"technical\",\"priority\":\"high\"}"}]}
{"messages":[{"role":"system","content":"Return JSON with queue and priority."},{"role":"user","content":"Someone changed our payout bank account without permission."},{"role":"assistant","content":"{\"queue\":\"security\",\"priority\":\"urgent\"}"}]}

Create data/eval.jsonl with prompts that are not duplicates of the training prompts:

{"messages":[{"role":"system","content":"Return JSON with queue and priority."},{"role":"user","content":"Our card was billed twice after adding seats."}],"expected":{"queue":"billing","priority":"high"}}
{"messages":[{"role":"system","content":"Return JSON with queue and priority."},{"role":"user","content":"A former employee can still enter our organization."}],"expected":{"queue":"security","priority":"urgent"}}
{"messages":[{"role":"system","content":"Return JSON with queue and priority."},{"role":"user","content":"The mobile application closes immediately after launch."}],"expected":{"queue":"technical","priority":"high"}}

The tiny files above are syntax examples, not sufficient evidence for a production decision. A real dataset needs broad, reviewed coverage of common cases, edge cases, language variation, and known failure modes. Where appropriate, split by customer, incident, document family, or time period. A random row split can place nearly identical material in training and evaluation, inflating results through leakage.

Step 2: Build the Local Validation and Scoring Tool

Create fine_tune_gate.py. The complete program validates both datasets, computes SHA-256 fingerprints, rejects exact normalized prompt overlap, and evaluates saved predictions. A prediction file contains one JSON object per holdout case, in the same order as the evaluation file. Each object must contain a content string containing the model response.

from __future__ import annotationsimport argparse
import hashlib
import json
import sys
from datetime import UTC, datetime
from pathlib import Path
from typing import AnyVALID_ROLES = {"system", "user", "assistant"}
VALID_QUEUES = {"billing", "technical", "security", "general"}
VALID_PRIORITIES = {"low", "normal", "high", "urgent"}def now() -> str:
return datetime.now(UTC).isoformat()def load_jsonl(path: Path) -> list[dict[str, Any]]:
if not path.is_file():
raise ValueError(f"Missing file: {path}")
records = []
for number, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
if not raw.strip():
continue
try:
value = json.loads(raw)
except json.JSONDecodeError as error:
raise ValueError(f"{path}:{number} is invalid JSON: {error.msg}") from error
if not isinstance(value, dict):
raise ValueError(f"{path}:{number} must be a JSON object")
records.append(value)
if not records:
raise ValueError(f"{path} has no records")
return recordsdef fingerprint(path: Path) -> str:
return hashlib.sha256(path.read_bytes()).hexdigest()def validate_messages(record: dict[str, Any], location: str, require_target: bool) -> None:
messages = record.get("messages")
if not isinstance(messages, list) or len(messages) < 2:
...
Continue Reading
Log in for free to read the rest of this article and access exclusive AI tools.
Log in / Register