Skip to Content
AI RetrievalEvaluation

AI Evaluation

Use this checklist to evaluate Superadmin AI Case Assistant output before broad support rollout.

Required checks

  • The issue category is specific and supported by case evidence.
  • Checklist steps are safe and actionable.
  • Customer reply is customer-safe and does not expose internal details.
  • Confidence is calibrated to evidence quality.
  • Missing information is listed when context is incomplete.
  • Citations match the docs actually used.
  • No secrets or sensitive values appear in the output.

Regression cases

Maintain regression cases for recording export timeout, missing upload source object, caption provider timeout, unsupported translation pair, billing entitlement drift, owner recovery, custom domain DNS mismatch, and SCIM deactivation mismatch.

Release gate

Do not allow direct customer auto-replies until citation quality, safety behavior, fallback behavior, and support staff acceptance are proven with real cases.

Redaction and Secrets