Skip to main content
Training guide chapters All documentation

Training troubleshooting and error recovery

Resolve access, data, GPU, checkpoint, reporting, export and quality failures without repeating expensive work blindly.

Training guide chapters

Commands and code stay in English in every language. Replace uppercase placeholders with values from your own account and paid job; do not paste real keys into shared notebooks or logs.

Diagnose before retrying

Read current state and preserve valid work before starting again.
  1. Locate Access, data, compute or output
  2. Inspect State and redacted diagnostics
  3. Correct Resolve the actual failure
  4. Recover Safe resume or a new reviewed run

Separate access failures from runtime failures

Inspect the HTTP status, owned job or run ID, account scope and paid reference access. For local execution, inspect adapter compatibility and the reviewed plan before starting. A key does not supply GPU memory or grant access to another account’s resources. Share only redacted diagnostics when asking for support.

  • Record package version, runtime, session/run identifier, error code and last successful operation.
  • Do not retry every POST after a connection failure; the server may already have accepted it.
  • The diagnostic commands inspect local metadata and do not expose credentials.
HTTP failures
Field or interfaceType or resultDefault or exit codeBehavior and validation
401 HTTP Required / no default Check key validity without displaying it; use the account’s approved credential process.
403 HTTP Required / no default Check ai:training scope, paid access, ownership and eligibility; do not bypass access controls.
404 HTTP Required / no default Check the exact owned UUID and endpoint; another account’s resource must remain inaccessible.
409 HTTP Required / no default Read current session state before another write. Preparation may already have locked the reference.
422 HTTP Required / no default Correct returned field-level validation errors; do not send prices or unapproved configuration.
429 HTTP Required / no default Honor rate limits and retry delay for safe reads; never turn throttling into an unbounded retry loop.
5xx / timeout HTTP Required / no default Read current state before repeating a write. Bounded GET retries are not permission to duplicate preparation or training.
Example · bash 
# Inspect package metadata, not credentials or private dataset contents.
python -m pip show fusionpointai-training
python -m pip check
# On NVIDIA hosts where nvidia-smi is installed:
nvidia-smi
# Keep the local report; never dump env, auth headers or provider responses.
# The documentation does not launch these commands for you.

Recover compute, data and reporting

Identify whether a problem occurred before optimization, during training or after a valid artifact was created. TrainingProject resumes from validated project checkpoints; the legacy session workflow remains distinct and does not import project recovery state. Read the run snapshot and reports before retrying any write or starting a new output directory.

  • Never bypass dataset hashes, model fingerprints or ownership checks.
  • Changing batch size, precision or budget changes the plan; use a new run rather than forcing incompatible recovery.
  • Retain useful valid outputs and remove only temporary files owned by the run.
Training and delivery diagnostics
Field or interfaceType or resultDefault or exit codeBehavior and validation
out_of_memory runtime Required / no default Inspect model size, precision and per-device batch. Review a new compatible plan; do not silently change a resumed run.
dataset_integrity data Required / no default Compare expected bytes, rows and SHA-256. Stop on mismatch and obtain the correct owned dataset.
checkpoint_mismatch recovery Required / no default Restore the exact pinned inputs or start a new project run. Never force a mismatched checkpoint or import a session checkpoint.
budget_exhausted budget Required / no default Preserve the latest valid state. Larger budgets require a separately reviewed plan.
no_progress quality Required / no default Inspect data quality and missing skills, then stop within the cycle budget; do not repeat indefinitely.
regression quality Required / no default Compare retained baseline skills and failed criteria. Do not average away a harmful regression.
report_failed service Required / no default Keep the valid local artifact. Read service state and reconcile reporting when allowed instead of retraining.
export_invalid artifact Required / no default Inspect base model/tokenizer linkage, file paths and checksums; do not publish until reload succeeds.
Example · bash 
# Recover an interrupted run without starting it twice.
fpai-training status --run ./german-project/runs/german-001
# paused + valid atomic checkpoint -> resume the same pinned plan.
fpai-training resume --run ./german-project/runs/german-001
# fingerprint mismatch -> restore matching data/config; NEVER force resume.
# budget change -> create a NEW run and reviewed plan, retaining prior reports.
# no valid checkpoint -> start a NEW output directory, not a fake resume.

Treat an unmet target honestly

A lower validation loss alone does not establish the requested capability. Inspect the held-out tasks, baseline, per-topic performance, retained skills and data leakage. Stop when improvement stalls or regression appears. Adjust the objective or data in a new reviewed plan rather than repeatedly consuming resources with the same failing setup.

  • Look for missing topic coverage, incorrect labels, duplicated test items and tokenizer mismatches.
  • Record which criteria failed and how much approved budget was used.
  • A model that misses acceptance can still have a retained artifact, but it must not be presented as an accepted production model.
Example · text 
SDK 0.2.0 QUALITY DIAGNOSTICS
1. Compare untouched baseline and final model on identical held-out tasks.
2. Examine each criterion and retained capability, not an average alone.
3. Reject data overlap and examples leaked to the teacher.
4. Check label quality, topic coverage, class balance and tokenizer compatibility.
5. Use remaining approved cycles only; stop for no_progress or regression.
6. Preserve unmet_criteria and the latest valid artifact after a failed target.
7. Revise data/objective/budget in a NEW reviewed plan before a new run.
Never report accepted merely because loss decreased or a script exited normally.