Troubleshooting

A run is stuck in queued

A queued run means no runner has claimed it — either no runner is online, or none matches the run's required tags / requested location (local vs vps/remote).

runyard runners            # GET /api/runners — heartbeat state, tags, capacity
runyard engine doctor      # GET /api/runners/engine-doctor — Smithers engine drift/readiness

A runner is considered offline after ~30 seconds without a heartbeat. Start with runyard engine doctor when Smithers runs fail to launch, pause/resume behaves oddly, or a 0.30 cutover is in progress. The doctor is read-only: it reports the effective Smithers version the runner already measured, the expected repo pin, pin drift, stale heartbeat, last startup warning, recent launch/preflight failures, and known 0.30 contract signals without reading runner-host files or secrets. For one runner, use runyard runner doctor <runner-id>; agents can call MCP get_engine_doctor or get_runner_doctor.

If the runner should be online and the doctor reports stale heartbeat, check its service logs on the runner host:

journalctl -u runyard-runner -e

Common causes: the runner unit is idling because RUNYARD_HUB_TOKEN is empty or revoked; the runner's tags do not include what the workflow requires (requiredRunnerTags, for example smithers); or the run asked for --where local and only vps runners are registered. Running runyard preflight <workflow> before enqueueing catches no_matching_runner up front.

A run is in waiting_approval

The run is held on an approval card, which is by design: approval-held runs are never failed or reaped for waiting. Find and resolve the card — web Decisions, runyard approvals then runyard approve|reject|request-changes <approval-id>, or the MCP tools. See Decisions.

Failure classes

Terminal failures carry a class so you know whether to fix input, config, or infrastructure:

Preflight blockers

runyard preflight <workflow> (or POST /api/workflows/{id}/preflight) reports ready, needs_input (with questions — usually missing required input fields), or blocked with coded blockers, including:

• missing_secret / secrets_unavailable — required secrets are not stored, or the secret store is disabled (set SECRETS_ENC_KEY on the Hub, then PUT /api/secrets/{key}).
• no_matching_runner — no registered runner matches the required tags/location.
• workflow_disabled, workflow_not_found — an admin can re-enable via PATCH /api/workflows/{id}.
• workflow_bundle_missing / workflow_source_missing — the definition references source that is not published; send source bytes via update_workflow.
• repo_dir_invalid, hook_blocked — input points outside allowlisted repos, or requests hook profiles that are not enabled for the workflow.

401 / 403 responses

401 means no valid token; 403 insufficient scope means the token lacks the required scope (the response names it). Check what you are authenticated as:

runyard status        # GET /api/me — token name and scopes

Admin actions (creating workflows, secrets, tokens, schedules) need admin; running workflows needs api or mcp; the runner protocol needs runner. Ask a Hub admin for a token with the right scopes.

Secrets not configured

If secret endpoints return an error about the store being disabled, the Hub has no valid SECRETS_ENC_KEY. Set a 32-byte base64/hex key in the Hub env file and restart. Note that rotating an existing key makes previously stored secrets unreadable.

Telegram not delivering

Verify, in order: TELEGRAM_BOT_TOKEN and TELEGRAM_APPROVAL_CHAT_ID are saved in Secrets (or provided as environment fallbacks); the bot's webhook points at https://hub.example.com/api/telegram/webhook with the matching TELEGRAM_WEBHOOK_SECRET; the bot is a member of the target chat; and the users pressing buttons are listed in TELEGRAM_APPROVAL_USER_IDS. Response-endpoint deliveries record their state (status, attempts, last_error) on GET /api/runs/{id} under responseEndpoints[].

Runner agent auth expired

Runs failing with provider auth errors usually mean the runner's codex/claude CLI login lapsed. On hosts with REAUTH_ENABLED=1, an admin can trigger re-auth from the web app (runner card → Re-auth): the Hub surfaces a verification URL and user code, and never token material. Otherwise, log in on the runner host directly (codex login, claude setup-token).

Support chat Hermes harness

The support chat defaults to the native runyard-yardmaster Hermes profile (the single resident agent that replaced the separate runyard-support profile). Run runyard hermes-profiles apply --slug yardmaster after upgrades that seed a fresh profile, then check runyard hermes-profiles preflight runyard-yardmaster --acp. The old RUNYARD_SUPPORT_HERMES_* env vars still work as aliases for the canonical RUNYARD_YARDMASTER_HERMES_* names. Each copilot conversation stores a stable Hermes session binding, while RunYard keeps authority over identity references, typed actions, approvals, workflow launches, audit, and delivery. The on-disk profile is canonical: RunYard never overwrites operator edits after first-seed, and the Hermes home must be included in your backup set (see docs/hermes-profiles.md).

The support status endpoint reports the selected provider and Hermes degraded state. If Hermes is unavailable, the chat shows a precise chat-local degraded reply and does not silently switch to a direct model. During a canary only, an operator can explicitly set RUNYARD_HUB_SUPPORT_AGENT_PROVIDER=runner, openai, or anthropic; leave it unset for the production Hermes harness.

Read-only RunYard inspection remains server-mediated. When the operator asks about a referenced or current run, workflow, runner, schedule, board, or Factory Item, RunYard supplies bounded, redacted context or typed read tools. The support chat never exposes mutating APIs, tokens, secrets, raw database access, arbitrary URL fetches, filesystem access, or shell access outside the typed RunYard action contract.

Where to look, generally