Skip to main content
Training guide chapters All documentation

Failures, cancellation and recovery

Diagnose failures without duplicate writes, lost artifacts or an endless retry loop.

Training guide chapters

Commands and code stay in English in every language. Replace uppercase placeholders with values from your own account and paid job; do not paste real keys into shared notebooks or logs.

Recover an owned run

Resume only when configuration, model and dataset fingerprints still match.
  1. Run Track data position and budgets
  2. Atomic checkpoint Weights, optimizer and random state
  3. Interruption Keep the last valid checkpoint
  4. Validate Check fingerprints and available space
  5. Resume Continue without duplicating work

Read the failure before retrying

Authentication failures, expired reference access, incompatible models, insufficient device memory, invalid datasets and failed criteria need different remedies. TrainingClient retries only eligible GET requests within a bounded count. Mutating requests are not automatically retried. On a timeout, inspect the owned run or API resource before repeating a write.

  • 401/403: verify key scope, account access and paid reference entitlement.
  • 404: confirm the resource belongs to the authenticated account.
  • 409/422: inspect run state and field-level validation details before changing inputs.
Example · python 
import os
from fusionpointai_training import TrainingClient
from fusionpointai_training.errors import APIError, IntegrityError, TrainingError

client = TrainingClient(api_key=os.environ["FUSIONPOINTAI_API_KEY"])
try:
    state = client.get_session(os.environ["FUSIONPOINTAI_SESSION_ID"])
    print("Current state:", state["status"])
except IntegrityError:
    print("Integrity check failed; preserve the report and investigate.")
except TrainingError:
    print("Training failed; inspect the local report before retrying.")
except APIError:
    print("Request failed; inspect access and session state before another write.")
except ValueError:
    print("Invalid local configuration; correct the input before retrying.")
# Never print the key or authorization headers.

Pause, resume and cancel an owned run

The project API and CLI control workers owned by the run. Pause waits for an atomic checkpoint; resume validates configuration, model, data, evaluator and topology fingerprints; cancel stops this run and is terminal. The legacy TrainingClient.cancel method continues to cancel service session state and does not replace project worker controls.

  • Use status before repeating a control after a timeout.
  • Resume only a valid recoverable run; do not force a mismatched checkpoint.
  • Preserve partial outputs and do not label them accepted artifacts.
Example · python 
# Continue with client/session. Cancels API state; not pause/resume.
cancelled = client.cancel(session["id"])
print(cancelled["status"])
# Stop the separate GPU process using your runtime's process controls.

Recover a project run

TrainingProject stores native model or adapter checkpoints and project progress within the owned run directory. Resume validates model, dataset, configuration, evaluator and topology fingerprints before continuing. Checkpoint retention and storage budgets constrain owned run data; a mismatched checkpoint is rejected.

  • The configured retention keeps at most five recovery checkpoints.
  • Pause and cancellation have different states and effects.
  • Do not delete or modify checkpoint files while a run is active.
Example · text 
Checkpoint contents and resume validation
model/adapters + optimizer + scheduler + random states
dataset version + sampler/data position + curriculum progress
model/config/evaluator fingerprints + distributed topology
atomic completion marker + integrity checks + storage budget

Keep at most 3 valid recovery checkpoints by default.
Refuse mismatched resumes before updating weights.