Skip to main content
Training guide chapters All documentation

Evaluate progress and decide whether to accept

Use task-level criteria and language-quality evaluation alongside training diagnostics.

Training guide chapters

Commands and code stay in English in every language. Replace uppercase placeholders with values from your own account and paid job; do not paste real keys into shared notebooks or logs.

From loss to task acceptance

Loss improvement is one signal; subject competence requires separate task checks.
  1. Baseline Untouched student performance
  2. Trained model Same evaluation conditions
  3. Task results Quality, correctness and errors
  4. Decision Accept or report unmet requirements

Measure baseline and final criteria

TrainingProject evaluates the untouched student and final model against local validation or final-test rows. Every rubric criterion names a registered deterministic metric and a minimum_score threshold. Built-ins are exact_match, macro_f1, text_similarity and chrf; custom scorers can be registered with an explicit version. Baseline and final held-out evaluation stay local.

  • Description text explains a criterion but is not itself semantic grading.
  • Compare scores per criterion and review retained capabilities, not an average alone.
  • Keep the rubric, data fingerprints and baseline/final evaluation reports together.
Example · python 
# Continue with a TrainingProject whose run has completed.
result = project.evaluate(split="test")
for criterion in result["criteria"]:
    print(criterion["id"], criterion["score"], criterion["accepted"])
print("Accepted:", result["accepted"], "Report:", result["report_path"])
# Scores measure the registered criteria; they do not guarantee broader capability.
Example · python 
import os
from fusionpointai_training import TrainingClient

client = TrainingClient(api_key=os.environ["FUSIONPOINTAI_API_KEY"])
session = client.get_session(os.environ["FUSIONPOINTAI_SESSION_ID"])
server_report = session.get("report")
runtime_report = server_report.get("runtime") if isinstance(server_report, dict) else None
if (
    session.get("status") == "completed"
    and isinstance(runtime_report, dict)
    and runtime_report.get("artifact_sha256") == os.environ["LOCAL_ARTIFACT_SHA256"]
    and server_report.get("delivery") == "runtime_local"
    and server_report.get("verification") == "runtime_reported"
):
    print("Server report matches the local artifact digest.")
else:
    print("Completion is not confirmed for this local artifact.")
# Reconcile by reading state; do not repeat training or invent a result.

German capability evaluation

For a German objective, evaluate comprehension, grammar, writing, translation and reasoning separately where they are in scope. Use hidden tasks, appropriate human-readable rubrics and meaningful error examples. Measure retention of existing capabilities too. Teacher grading can support evaluation, but avoid using the same generated examples as both lessons and final proof of quality.

  • Classification: per-class precision, recall and confusion patterns.
  • Translation: meaning, terminology, numbers, names and omissions.
  • Writing: grammatical accuracy, clarity, tone and task completion.

Run bounded improvement cycles

TrainingProject evaluates curriculum topics, prepares approved lessons when sharing permits, and runs bounded improvement cycles. Stop conditions include acceptance, exhausted budgets, no progress, cancellation or regression. Reports retain unmet objectives and measured criteria; a passed task does not certify broader model capability.

  • Keep final acceptance data untouched throughout improvement.
  • Pin custom evaluator versions for reproducible comparisons.
  • Review each criterion and regression rather than averaging away a failure.