--- name: bot-status-api description: Query and operate the MEV bot fleet through the bot-status backend (fleet state, lifecycle, config/compose, incidents, node checks, config). --- # MEV-X Bot Management API The machine API for **MEV-X Bot Management** β€” the same endpoints the dashboards use. One product, made of two cooperating services behind the same host. Use it to query fleet state, act on bots, and configure notifications/accounting. - **Control plane & fleet** (`bot-status`) β€” liveness, chains, wallets, config/compose, incidents, lifecycle, servers, RPC/disk health. Base **`$BASE`** = the URL this doc was fetched from minus `/agents.md` (e.g. `https://bot-status.mev-x.com/550c1f41-…`). - **Notifications & accounting** (`bot-stats`) β€” per-chain Telegram alerts, hourly stats, profitability/ROI, wallet labels, Prometheus export. Base **`$STATS`** = `https://bot-status.mev-x.com/f77a6831-3421-49fd-b8fe-95eca8500757` (same host; embedded in the bot-status dashboard as the πŸ“Š stats-config iframe). Every path below is under **`$BASE`** unless it is written `$STATS/…`. Responses are JSON unless noted. The two share a Postgres DB and the same Telegram chat ids, so e.g. a chain's notification chat set via `$STATS/cfg/chain/{chain}` is also what `bot-status` posts crash/ recovery alerts to. **Identity model:** a bot is identified by `server` + `name`, where `name` is the bot's `name_of_bot` (from its `config.json`), NOT its container name. Discovery is dynamic: any running `*_bot_` container is tracked; telemetry is auto-discovered. > **Agents: re-fetch this doc periodically.** It is the single source of truth and the > fleet ships new endpoints/features regularly β€” don't rely on a cached copy. Re-`GET > /agents.md` at the start of a session (and whenever a call 404s or a response has fields > you don't recognise) so you pick up capabilities added since you last read it. --- ## Fleet Two views of the same fleet, same per-bot data either way. Pick by what you need: if a **per-server** list is required, pull `/bots` (keyed by server); if **per-chain**, pull `/registry` (keyed by chain). **`/registry` is the primary endpoint to use for integrations.** ### Chain names β€” match on the CANONICAL shortName Every chain key we emit (`/registry` keys, `chain` on a bot, `chain` in alerts) is the canonical **shortName**: `POL`, `ETH`, `BNB`, `ARB1`, `BASE`, `AVAX`, `FLR`, `MON`, `PLASMA`, `ROBINHOODCHAIN`, `HYPE`, `BLASTMAINNET`, `INK`, `SONIC`, `OP`. The same chain is spelled differently by every system around us β€” BNB Smart Chain alone is `BNB` (ours), `bsc` (the Vault rpc tree, db-indexer, explorer subdomains), `BSC` (link labels), `binance-smart-chain` (third-party APIs) and "Binance Smart Chain" (humans). **Don't hardcode your own mapping and don't compare raw strings** β€” `POL` vs `polygon` vs `matic` are the same chain, and an exact-match comparison silently drops bots. ### `GET /chains` β€” the chain catalog (join it with `/registry`) Keyed by the **same canonical shortNames as `/registry`**, so the two join directly. It collapses three lookups into one β€” canonical name, a working RPC, and the chain id (previously only obtainable by probing `eth_chainId`): ```json { "POL": { "shortname": "POL", "chain_id": 137, "aliases": ["137","matic","pol","polygon","polygon mainnet","polygon-pos"], "vault_rpc": "polygon", "db_indexer": "polygon", "native_symbol": "POL", "native_decimals": 18, "explorer_url": "https://polygonscan.com", "rpc_active": "https://rpc.newton.mev-x.com/polygon/…", "rpc_reachable": true, "rpc_endpoints": 7, "enabled": true, "known": true } } ``` | field | meaning | |---|---| | `chain_id` | EVM chain id β€” tag settlement jobs without an `eth_chainId` probe | | `vault_rpc` / `db_indexer` | outward segment names. **Not interchangeable**: `AVAX`β†’`avalanche` vs `avax`, `ETH`β†’`ethereum` vs `eth` | | `rpc_active` / `rpc_reachable` | a live-probed **reachable** endpoint (lowest latency) for the chain, or `null`/`false` when none answers β€” skip those chains | | `rpc_endpoints` | how many http endpoints exist for it in the Vault rpc tree | | `enabled` | the fleet currently has at least one **active** bot on this chain | | `known` | `false` = discovered but not yet catalogued here (still usable as a key) | `?enabled=true` returns only the chains the fleet actually runs on. With this a consumer builds `{chain_id, vault_chain: vault_rpc, rpc: rpc_active}` for every `rpc_reachable` chain β€” no hardcoded maps, and new chains appear on their own. Fold any spelling to the canonical name before comparing or keying (`bot_status.chains`): ```python from bot_status import chains chains.canon("Binance Smart Chain") # -> "BNB" (also "bsc", "BSC", "bnb chain", "56") chains.same("bsc", "BNB") # -> True chains.chain_id("polygon") # -> 137 ; chains.from_chain_id(56) -> "BNB" ``` An unrecognised chain comes back uppercased rather than `None`, so a chain discovered before it is catalogued still works as a key. Two OUTWARD segment names exist and are **not interchangeable** β€” `chains.vault_rpc(c)` for the Vault rpc tree (`AVAX`β†’`avalanche`, `ETH`β†’`ethereum`) and `chains.db_indexer(c)` for db-indexer DBs (`AVAX`β†’`avax`, `ETH`β†’`eth`). ### Active and passive bots β€” read this before integrating A bot **exists** when its directory exists on a server (`_bot_` under `/opt/mev-bots`, carrying a compose file and a `.env`). Running is a separate question: | presence | meaning | |---|---| | `active` | a container is running, OR it produced telemetry within the delisting window (**2 days**), OR its container is **present but stopped/crashed** (exited/restarting). A deployed-but-not-running bot β€” or a downed one that produced within 2 days β€” is active and stays visible | | `passive` | **no container at all** (compose-**downed** / removed) **and** silent past the 2-day window. A merely *stopped* container, or a downed bot that produced within 2 days, is NOT passive | | *(gone)* | no directory β€” the row is purged and the bot disappears entirely | **`/registry` and `/bots` return active bots only unless you ask.** Add `?passive=true` to get everything that exists. The default keeps existing integrations seeing the fleet they always saw β€” roughly half the fleet is passive at any time. Which you want depends on the question: - *"which bots exist?"* (inventory, deploy targets, CR work) β†’ **pass `passive=true`** - *"what needs attention now?"* (dashboards, alerting, health) β†’ **leave it off** Getting this wrong under-reports silently, which has bitten this project before. Every bot carries a `presence` field, so you can also filter client-side. Addressing a bot by name is unaffected: `GET /config/{server}/{bot}`, `/act/…`, env, logs, LKG and history all resolve a passive bot with no parameter. Listing hides; addressing resolves. ### `GET /registry` β€” primary (keyed by chain) The whole fleet as `{ "": { "": {…} } }`. The top-level keys **are** the chains (`POL`, `ETH`, `PLASMA`, `BASE`, `ARB1`, `BNB`, `AVAX`, `FLR`, `MON`, `ROBINHOODCHAIN`, …), so to find a chain's bots just index it β€” no walking or per-item filtering: - every Plasma bot: `curl -s $BASE/registry | jq '.PLASMA'` - just their names: `curl -s $BASE/registry | jq '.PLASMA | keys'` - one bot: `curl -s $BASE/registry | jq '.PLASMA.stage_bot_3'` Per-bot fields are the same set as `/bots` (below), and additionally carry the legacy `bot-status-api` field names (`server`, `address`, `status`, `container_status`, `last_tx`, `last_successful_tx`, `up_since`, `wallets` as `{address: balance}`, `config` = latest cached config). Backwards-compatible with the old registry, so legacy consumers move over with just a base-URL swap (`…/bot-status-api//bots` β†’ `…/bot-status//registry`). ### `GET /bots` β€” secondary (keyed by server) The same fleet as `{ "": { "": {…} } }` β€” a server-centric view. To find the bots on a chain, prefer `/registry` (above); with `/bots` you must walk every server and filter each bot by its `chain` field. One object per bot: | field | meaning | |---|---| | `name`, `server` | identity | | `presence` | `active` Β· `passive` β€” does it EXIST vs is it RUNNING (see above). Passive bots are omitted unless you pass `?passive=true` | | `status` | `ok` (fresh tx) Β· `stuck` (container up, no fresh tx) Β· `down` (no container) Β· `no_telemetry`. Health, not existence β€” a passive bot is normally `down` | | `container` | running container name, or null | | `uptime` | docker Status, e.g. `Up 3 hours` | | `image` | running image ref (repo:tag) | | `running_digest` | manifest digest the running container was created from | | `local_digest` | manifest digest that tag resolves to in the server's local docker | | `registry_digest` | latest manifest digest for that tag in the registry | | `image_status` | `uptodate` Β· `update-available` (registry ahead) Β· `restart-pending` (newer pulled locally, not recreated) Β· `pull-and-restart` Β· `unknown` | | `chain` | detected chain shortName (POL/ETH/PLASMA/…) | | `mode` | strategy (TargetSwap/Optimizer) | | `db`, `db_used` | sub-db (e.g. defi-v4) and whether it's used | | `db_block`, `db_behind` | the db-indexer **postgres (amms)** synced block (`variables.block_number`) and blocks behind chain head | | `db_redis_block`, `db_redis_behind` | the db-indexer **redis (routes)** synced block (`block_number_routes`) and blocks behind head. `db_redis_block > db_block` means routes are computed ahead of the amm state (**routes>amms**, inconsistent) | | `address` | telemetry host:port (for the per-bot stats API: `/last`, `/target_tx`, `/bundle_tx`) | | `last_tx` | ISO time of last produced tx | | `vault_ok` | wallet Vault token health (false = stale) | | `wallets`, `balances`, `balances_total` | monitored wallet addresses + native balances | | `autorestart` | docker restart policy set β†’ not auto-remediable | | `needs_restart` | config/compose edited on disk, pending a restart | | `incident` | current open incident summary, or null | | `recovered` | recently resolved incident summary (`{type, resolution, at}`), or null | **Chain-key caveat (registry):** its top-level keys are the *real* chain shortNames now (e.g. robinhood is `ROBINHOODCHAIN`, not the legacy `RH`; arbitrum is `ARB1`). Code that iterates chains and reads each bot's `chain`/`address` is unaffected; only code that hard-keys a specific old chain string (e.g. `data["RH"]`) needs to switch to the new shortName. Grab the shortName from any bot's `chain` field rather than hard-coding it. --- ## Control plane (per bot) - `POST /act/{server}/{bot}/{verb}` β€” lifecycle. `verb` ∈ `restart` (soft: compose restart), `hard` (down+up -d), `start` (up -d), `stop` (down), `pull`. β†’ `{ok, output}`. - **`pull` DOWNLOADS ONLY β€” it never recreates the container.** `docker compose pull`, nothing else. The bot keeps running the old image; the new one just sits on the host. **A `hard` restart is what adopts it.** (This is deliberate: pulling is safe at any time, recreating a trading bot is not. The CR's `pullRequested` flag is the one that does pull-THEN-hard β€” the API verb does not.) An earlier version of this line claimed pull recreates; it never has, and that wording sent an operator away believing a bot was on a new image when it was still on the old one. - **`POST /create` verifies now.** It used to answer `ok: true` for a clone that never happened β€” the clone script exits 3 when the target directory already exists, and non-zero on any `set -e` failure, but ssh_run runs with check=False and returns stdout only, so the failure came back as an ordinary string. The symptom was a bot missing from `/bots` whose config endpoint said "no dir on disk", because create had still written a registry row pointing at a directory that was never made. The script now asserts `test -d "$DST"` and prints a `CLONE_OK` sentinel; without it the call raises with the host's own message, and nothing is registered. - **`{ok: true}` from a lifecycle verb is not proof the bot moved.** The call returns compose's combined output but does **not** check its exit status, so a pull that failed (bad tag, auth) also comes back `ok: true` with the error in `output`. And progress lines in the output do not mean the response arrived mid-download β€” the call waits for completion (900s bound; exceeding it fails loudly). **Confirm with container start time / restart count**, from `/bots` (`uptime`, `up_since`) or `docker ps`, not from the response body. - `GET /logs/{server}/{bot}?tail=N` β€” bot logs (redacted, ANSI). text/plain (one-shot). - `GET /logs/{server}/{bot}/stream?tail=N` β€” **live** logs (`docker logs -f`) as a chunked text/plain stream; `tail` defaults to **1000**. Consume with `curl -N` (or a fetch reader). Closing the connection kills the remote tail immediately β€” no orphaned process on the host. A single stream is capped at `LOG_STREAM_MAX` (default 1h); reconnect to continue. - `GET /logs/{server}/{bot}/grep?pattern=…` β€” **grep at the source** (only matching lines cross the wire), for backend/agents. Grep flags map to query params: `A`/`B`/`C` (after/before/context lines), `i` (ignore case), `E` (extended regex), `F` (fixed string), `v` (invert), `w` (word), `x` (whole line), `n` (line numbers), `m` (max matches). Multiple patterns: repeat `e=` (each a `grep -e`). `tail` = how many recent log lines to search (default 20000); `limit` caps returned lines (default 20000). Redacted, text/plain. e.g. `…/grep?e=panic&e=error&i=1&C=5&tail=50000`. - `GET /config/{server}/{bot}` β€” read config.json β†’ `{ok, output}` (raw file text). - `POST /config/{server}/{bot}` body `{"content": ""}` β€” save a config edit. **Never restarts.** JSON is validated. Works even if the bot is down. On a bot exposing the live-config admin endpoint (below) this **applies the diff live**: only the changed fields are sent to the bot, fields with a live handler take effect immediately, the bot persists all of them, and needs-restart is flagged **only** for the fields the bot says still need one (and cleared when none do). Falls back to writing the whole file + flagging needs-restart when the bot has no live endpoint, or when the edit DELETES a field (a set-patch can't express removal). An unchanged edit is a no-op. - `POST /config-preview/{server}/{bot}` body `{"content": ""}` β€” **classify an edit without applying it**: will this need a restart? β†’ `{live_capable, live[], restart[], removed[], changes{pointer: value}, restart_required?}`, or `{invalid: true, error, path}` when the bot rejects a value. Authoritative when the bot supports `?dry_run=true` (bot-status asks it directly); otherwise classified locally from the advertised live paths. Writes nothing. **Live config patch** (bots on the live-config build only β€” they bind a loopback admin endpoint and write its port to `/assets/config/live_config_admin_port`; bot-status reaches it over SSH): - `GET /live-config-status/{server}/{bot}` β€” cheap capability probe β†’ `{capable: bool}`. - `GET /live-config/{server}/{bot}` β€” the bot's live effective config β†’ `{version, effective: [[path, value], …]}`. `effective` is the set of live-patchable paths (subtrees: a field nested under an entry is live too). - `POST /live-config/{server}/{bot}` body **is** a flat `{"": value, …}` map (one or many fields per call) β†’ `{version, applied_live[], restart_required, restart_fields[]}`. Fields without a live handler still persist but need a restart. A bot rejection (400 malformed/empty, 422 invalid value, 500 disk write failed) comes back as a 400 whose `error` carries the bot's message and path. Prefer `POST /config/{server}/{bot}` for ordinary edits β€” it diffs for you. - **DANGER β€” never live-patch `arbitrage_calc_config.route_profit_search_strategy.statistics_mode`.** It reads like a telemetry toggle; it makes the solver return an EMPTY route vector, i.e. the bot silently stops trading while looking perfectly healthy. The bot side deliberately leaves it restart-only, so it comes back `restart_required` by design, not by omission. (Reported by the bot-side team, 2026-09-01.) - A bot that is WEDGED keeps its port file and its listening socket but never answers. bot-status probes the endpoint (3s) before every patch and, when it doesn't answer, skips the live path and writes the file instead, reporting "its live-config endpoint is not answering (the bot may be wedged)". A hung bot therefore never blocks a config save β€” which is right, since it needs a restart anyway. **Route-search mode** β€” `POST /mode/{server}/{bot}`: - Body `{"mode": "TargetSwap"|"Optimizer"}` sets it; an empty body toggles. `?restart=false` stages without restarting. - The mode is a PAIR: `algo` plus `find_first_profitable_route`, which must be `true` for TargetSwap and `false` for Optimizer (48/48 bots across the fleet, no exceptions). Both move together. - The edit is TEXTUAL β€” a JSON round-trip turns `10000000000000000` into `1e+16`, which the bot rejects. It refuses unless the diff is EXACTLY those two pointers, so it can never touch `statistics_mode` or anything else. - It live-patches only when BOTH pointers are live, else it writes and restarts, so the pair can never be applied half-live to a running bot. Fails closed if it can't tell. - `GET /compose/{server}/{bot}` / `POST /compose/{server}/{bot}` β€” same for docker-compose.yml (secret lines redacted on read; a redacted body is rejected on write). - `GET /dbs/{server}/{bot}` β€” DB variants swappable within the bot's chain, each with how far synced it is: `{chain, head, variants:[{name, block, behind, redis, redis_behind, routes_ahead}]}` where `head` is the public chain head, `block`/`behind` are the variant's **postgres (amms)** synced block and blocks-behind-head, `redis`/`redis_behind` the **redis (routes)** block, and `routes_ahead` flags routes>amms. Read-only (db-reader creds), ~2min cache; unreachable variants report null blocks. - `POST /db/{server}/{bot}/{variant}` β€” swap the bot's DB variant and flag **needs-restart**. The db can live in **two places**, and the swap patches whichever carry it: (1) `.env` `DB_URL` + `REDIS_ROUTES_URL`, and (2) the config's `indexer_config.checkpoint_repository_config.db_loader_config` `db_vault_url` + `redis_routes_vault_url`. **Does NOT restart or start the bot** β€” like a config/compose edit it only stages the change; apply it with a **hard** restart (`down + up -d`) to pick up the new `.env`/config. A stopped bot stays stopped. Errors if neither place holds a db. β†’ `{ok, output}` (output ends "…written to .env (n), config.json (n) (restart to apply)"). The `db` shown on a card comes from the same two places (`.env DB_URL` first, else the config `db_vault_url`). - `POST /db-toggle/{server}/{bot}` β€” flip `load_only_from_files` (enable/disable the DB for this bot) in config.json on disk and flag needs-restart. Does not restart. β†’ `{ok, output}`. - `GET /env/{server}/{bot}` β€” the bot's `.env`, **secrets withheld**: `{ok, output:{editable:[{key,value}], protected:[key,…]}}`. `editable` = non-secret vars with values; `protected` = **names only** for any key matching `*PRIVATE*`, `*TOKEN*` (incl. `VAULT_TOKEN`) or `*PASS*` β€” their **values are never returned** (no read path for the vault token, signing keys, or passwords). No-store. - `POST /env/{server}/{bot}` body `{updates:{K:V,…}, deletes:[K,…]}` β€” set/add or delete **non-protected** env vars. Refuses (400) any protected key. Preserves protected lines, comments, and order; written via stdin so the file (which still holds the token + private keys) never lands in a log. Flags needs-restart (hard restart to apply). β†’ `{ok, output:{updated, deleted}}`. - `POST /env/{server}/{bot}/vault-token` body `{value}` β€” **write-only** VAULT_TOKEN rotation: writes the new token into `.env` and flags needs-restart. Never reads, returns, or logs the value. β†’ `{ok}`. - `POST /create` body `{source_server, source_bot, new_name, target_server?, overwrite?}` β€” create a new bot by **cloning** an existing one on the same server: copies its compose dir, swaps the bot name in config.json + docker-compose.yml, and **strips signing keys** from the copied `.env` (the clone reads wallets from Vault by name β€” create `VAULT_KV_REALM/` first). `new_name` must match `*_bot_`. Writes to disk and leaves the bot **stopped**, registered as a `down` card you then review + start. If the target dir exists it errors unless `overwrite:true` (which moves the old dir aside to `.old.`, never deletes). Holds no keys. β†’ `{ok, server, name, dir, output}`. - `POST /balances/refresh` body `{delay?}` β€” refresh wallet balances now and again after `delay`s (default 15) instead of waiting for the 5-min tick. Call it after a disperse/top-up. β†’ `{ok, refresh_in}`. - `POST /forget/{server}/{bot}` β€” demote a bot to **passive**: drop all caches, close any open accident as manually decommissioned, and take it off the default view. It KEEPS its DB row, its CR and its history. It does not delete: deletion has exactly one meaning now, which is that the directory is gone from the host. A bot whose container was **removed** (compose-downed) comes back as passive next sweep (out of the way, still addressable, still deployable); a running OR merely-stopped one (container still present) is re-discovered **active**. To remove a retired machine's bots outright, mark the SERVER dead (below). - `POST /forget/{server}/{bot}?force=1` β€” **force forget**: run `docker compose down` (stop AND remove the container) first, then forget. Use it for a crashed/stopped bot that keeps coming back: a plain forget leaves the container on disk, and a bot whose container still exists is re-discovered **active** on the next sweep, so the forget never sticks. The bot's files (config/compose/env) stay on the host β€” it remains redeployable. Destructive; the dashboard guards it behind a warning. β†’ `{forgotten, forced}`. - `POST /nodecheck/{server}/{bot}` β€” force a fresh node evaluation β†’ `{reachable, chain_id, syncing, block, ref_block, behind, good, reason}`. Works for down bots too. - `POST /image/{server}/{bot}/refresh` β€” re-read docker-compose.yml, then re-check this bot's image digests now against its `image:` (bypasses the ~10-min cache; so a compose edit / Lock is reflected) β†’ `{ref, running, local, registry, status}`. `running`/`local`/`registry` are the three manifest digests; `running`β‰ `local` means a newer image was pulled but the container wasn't recreated (Restart hard); `local`β‰ `registry` means the registry is ahead (Pull image). - `POST /autodeploy/{server}/{bot}?on=true&branch=dev/master&cadence=weekly` β€” arm/disarm **scheduled redeployment** for ONE bot (`on` ∈ true/false; `branch` required to arm; `cadence` ∈ `nightly`|`weekly`). Deliberately per-bot with no bulk form: this is how CI writes to a production bot on a timer, so opting one in costs one deliberate call. Mirrored to the CR as `spec.autoDeploy` `{enabled, branch, cadence}`, which is what the scheduled job selects on. Read the current state from `/bots` β†’ `auto_deploy`. β†’ `{ok, enabled, branch, cadence}` (400 on a bad combination). - `POST /image/{server}/{bot}/lock` β€” pin the bot's docker-compose `image:` to the digest it's running right now (`repo@sha256:…`), so a later pull/recreate won't drift off it. Compose is edited on disk only; nothing restarts (the running container already IS that image). β†’ `{ok, pinned, output}`. - `POST /breaker/ack/{server}/{bot}` β€” manually acknowledge the bot's open profit-breaker stop. Flips the open `bot_stats.stop_events` row to `acknowledged` and drops the label from the card at once. Does not restart the bot. β†’ `{ok:true, output:{ok:true, acknowledged:"/"}}`; `{ok:false, error}` (400) if there was no open breaker row. **Auto-clear:** the monitor also clears an open stop by itself the moment the bot is restarted β€” a container (re)start after the trip, or (on hosts we can't see docker) a tx after the trip while an *armed* stop had halted the bot. The row goes to `autocleared`, the breaker RE-ARMS (so it trips again if the bot keeps losing), and a loud Telegram notice fires to the chain's chat. This stops a hand-restarted bot from running with the breaker silently disarmed. ### Last known good (per bot) A 24h snapshot of a bot's image + assets, used as the rollback target for the remediation ladder. - `GET /lkg/{server}/{bot}` β€” inspect + **live progress** of the snapshot on the host (read-only, never disturbs an in-flight save) β†’ `{ok:true, dir, state, stage, assets, image, done_at, image_ref, log, output}`. `state` ∈ `in_progress` (a `.lkg.tmp` staging dir exists β€” `stage` is `assets` then `image`; `assets`/`image` are the current growing byte sizes) Β· `complete` (`.done` present; `assets`/`image` are final sizes, `done_at` epoch) Β· `none`. `log` is a tail of the snapshot log; `output` is the raw text. Poll it to watch a snapshot build. `{error}` if the dir can't be resolved. - `POST /lkg/{server}/{bot}` β€” take a fresh snapshot now (bypasses the 24h wait; bot must be **running** β€” needs its live image). β†’ `{ok:true, launched:"/", dir, image, note}` where `note` says the save runs detached (assets tar + `docker save`, ionice'd) and to poll `GET /lkg/{server}/{bot}` for the `.done` marker + sizes. `{error}` if not running / dir unresolved. - `POST /lkg/{server}/{bot}/restore?apply=true` β€” restore the bot to its snapshot: docker-loads the saved image, moves current assets aside (`assets.pre_restore.`, never deleted), extracts the saved assets, and `docker compose up -d --force-recreate`. Disk-guarded (aborts if there's no room to back up current assets). Without `apply=true` it's a **dry-run** β€” returns the plan only, no mutation. β†’ `{ok:true, applied:, dir, output}` (`output` is the host script's stdout, incl. the `RESTORE-DONE …` / `DRY-RUN …` line). `{error}` on failure (e.g. `NO COMPLETE SNAPSHOT`). --- ## Incidents / accidents (auto-remediation) A bot that **panics** (container up, no fresh tx, panic in logs) opens an incident ("accident") with a captured panic-context log; the node is then evaluated. Node good β†’ one restart (dry-run by default). Node bad β†’ flagged, waits for the node to recover. Bots with docker autorestart are not auto-remediated. **Resolution model:** an accident is considered resolved once the bot **runs fine for `incident_resolve_healthy` seconds** (default 1h) of continuous producing β€” with or without our intervention. If the bot is **compose-downed** at any point while an accident is open, it's resolved immediately and marked **human intervention**. Every accident carries an `events` timeline (the resolution log): `opened`, `node_bad`/`node_recovered`, `restart`, `stabilizing`, `relapsed`, `resolved`/`human`. Accidents are pruned oldest-first once the incident-log store passes its 20 GiB cap. - `GET /incidents/{server}/{bot}` β€” a bot's accident history: `[{id, server, name, chain, type, status, remediating, stopped, auto_remediable, attempts, diagnosis, resolution, node_eval, node_bad, events:[{at,kind,detail}], opened_at, last_attempt_at, resolved_at, has_log}]`. `type` is `panic|crash|oom|node|stuck|down` for problems, plus **`lkg-restore`** β€” a last-known-good restoration (manual or auto-remediation) logged as its own already-`resolved` item so restores show as a distinct entry in the list (the outcome is in `resolution`, or `diagnosis` if it failed). - `GET /accidents.json?hours=12` β€” fleet-wide accidents active in the last N hours (same shape). `?server=&bot=` returns one bot's history instead (alias of `/incidents/{server}/{bot}`). - `GET /accidents` β€” fleet accidents HTML page (timeline view). `GET /accidents/{server}/{bot}` is that bot's accident-history subpage. - `GET /incident/{id}/log` β€” the saved panic log (grep -C1000 panic of last 20k lines). text/plain. - `POST /incident/remediate/{server}/{bot}` β€” **manually run the remediation ladder now** on the live bot: one restart if it hasn't been restarted yet for this incident, otherwise a last-known-good restore. Ignores the per-bot auto-remediation toggle **and** `REMEDIATION_DRY_RUN` (an explicit operator action, like the lifecycle buttons). Refuses config faults (a restart can't fix them). β†’ `{ok, step}` where `step` ∈ `restart`/`last-known-good restore`, or `{error}`. - `POST /incident/resolve/{server}/{bot}` β€” **mark the open incident resolved** and clear it from the dashboard. Does not touch the bot β€” for a cause fixed by hand, or a false alarm. β†’ `{ok, output}`. - `GET /r/{token}` β€” one-tap remediate link carried in crash Telegram alerts: runs the ladder for the token's incident and returns a small HTML result page. Tokens are single-use, expiring (`REMEDIATE_LINK_TTL`, 24h), and incident-scoped β€” minted only for remediable (non-fault) incidents. Not for programmatic use; call `POST /incident/remediate/…` directly instead. - `POST /remediation/{server}/{bot}?on=true|false` β€” set the bot's auto-remediation override (omit `on`, or `on=default`/`clear`, to reset to the fleet default). β†’ `{ok:true, enabled:, override:, default:}`. - `POST /remediation/{server}/{bot}/stop` β€” stop remediation for the bot's open incident. --- ## Config / compose history Every bot is scanned every 5 min; a new version is stored on change (100 per bot per kind). - `GET /history/{server}/{bot}/{kind}` β€” versions (`kind` ∈ `config`|`compose`): `[{id, sha, detected_at, bytes}]` (newest first). - `GET /history/version/{id}` β€” that version's raw content. text/plain. - `POST /history/rollback/{id}` β€” write that version to disk (flags needs-restart). - To diff two versions, normalize JSON first (parse + re-stringify) so reformatting isn't noise. --- ## Configuration (servers + tunables) - `GET /cfg` β€” `{settings:[{key,value,type,description,needs_reload}], servers:[{name,ip}], ssh_public_key}`. - `POST /cfg/setting/{key}` body `{"value": …}` β€” set a tunable. Applies live unless `needs_reload`. - `POST /cfg/server` body `{"name","ip"}` β€” add or edit a server (bots+telemetry auto-discovered). - `DELETE /cfg/server/{name}` β€” remove a server. - `GET /cfg/sshkey` β€” `{public_key}`: add to a new server's `~/.ssh/authorized_keys` to onboard it. - `POST /cfg/reload` β€” apply setting/server changes live (rebuilds workers in-process, no restart). --- ## Monitoring β€” RPC health Β· disks Β· low-balance Fleet-wide panels. Reads are cached (periodic background refresh); the `refresh` POSTs force a re-probe on demand. - `GET /rpchealth` β€” every probed Vault RPC endpoint β†’ `{at, rpcs:[{chain, server, visibility, key, url, kind, chain_id_expected, reachable, chain_id, syncing, sync_detail, block, latency_ms, error, at}]}`. Private/LAN endpoints are excluded. A `chain_id` β‰  `chain_id_expected` is a mislabeled node. - `POST /rpchealth/refresh` β€” body `{"url":"…"}` re-probes that one endpoint (returns its row); empty body re-probes the whole tree (returns the full list). - `GET /serverdisks` β€” disks per server β†’ `{at, servers:{:{disks:[{source, fstype, mount, size, used, avail, pct}], reachable, error, ip, dead}}}`. Only disks β‰₯100 GB. `reachable:false` + empty `disks` means the server is inaccessible (not "no disks"); `dead:true` means an operator retired it (see below). - `POST /server/{server}/dead` β€” body `{"dead":true|false}` (default true). Declare a server **retired**, or revive it. β†’ `{ok, server, dead, wiped}`; `400` for an unknown server. Dead means wiped from **our registry**: its bot rows are deleted, its fleet entries dropped, its CRs removed and it is no longer swept. **Nothing on the host is touched** β€” a retired machine is unreachable by definition, so this is a statement about what bot-status tracks, not about what exists on disk. It is deliberately explicit rather than inferred from SSH failure: inferring it would make a transient network blip wipe a server's bots and re-add them a minute later. A server that is merely unreachable keeps its bots (as passive) and keeps being retried. Reviving resumes sweeps; the bots are rediscovered if the machine is back and still has their directories. CRs on a dead server are exempt from the orphan report β€” nothing was deleted there, the machine was retired. - `GET /servermeta` β€” `{server: {country, provider}}` per server (geolocated country code + cloud provider: OVH/Hetzner/AWS/Allnodes/TeraSwitch, else null=unknown). Used by the dashboard's server-grouped view for flags/logos and the providerβ†’regionβ†’name sort. - `POST /serverdisks/refresh` β€” body `{"server":"…"}` re-probes one server β†’ `{ok:true, server, disks, reachable, error, ip}` (its updated entry; `{ok:false, error}` for an unknown server); empty body re-probes all and returns the same shape as `GET /serverdisks`. - `GET /balcfg` β€” low-balance config β†’ `{thresholds:[{chain, min_native, price, usd}], overrides:[{server, name, min}]}` (per-chain native minimum wallet balance + per-bot overrides). - `POST /balcfg/chain/{chain}` body `{"min": |null}` β€” set (or clear) a chain's per-wallet native minimum. `DELETE /balcfg/chain/{chain}` clears it. β†’ `{ok, chain, min_native}`. - `POST /balcfg/bot/{server}/{bot}` body `{"min": |null}` β€” set/clear a bot's override β†’ `{ok:true, override:, effective:}`. - `POST /report/send` β€” build + send the 8-hourly fleet report to the healthcheck chat now β†’ `{ok, text}`. - `GET /links` β€” HTML index of the fleet's dashboards/tools (human page). --- ## Notifications & accounting β€” `bot-stats` (base `$STATS`) The second service. Watches every bot's txs, keeps per-bot profitability/ROI, sends the per-chain Telegram alerts + hourly summaries, and exports Prometheus metrics. All paths here are under **`$STATS`** (`https://bot-status.mev-x.com/f77a6831-3421-49fd-b8fe-95eca8500757`). - `GET $STATS/cfg` β€” the whole config in one call: - `chains: [{chain, enabled, telegram_enabled, telegram_chat_id, attribution, rpc_ref, rpc_active, native_symbol, native_addr, native_decimals, quote_token, explorer_url, stat_explorer_url, min_profit_usd, notify_threshold_usd, sound_threshold_usd, low_balance_usd, tiers, max_loss_usd, auto_stop_mode, coverage_enabled, coverage_top}]` β€” `rpc_active` is the live-resolved RPC url; `tiers` is the win-banner ladder `[{emoji, min_usd}]`; `coverage_enabled`/`coverage_top` drive the 🎯 MEV-coverage report (below). - `coverage_chains: [chain]` β€” the chains with a MEV DB wired up (coverage can be enabled only for these; currently `ROBINHOODCHAIN`, `POL`). - `addresses: [{id, chain, address, name, ours, role, notes}]` β€” the wallet-label table (below). - `tokens`, `settings: {key:value}`, `totals: [{server, name, chain, mined, wins, fees_usd, profit_usd}]`, `rate_fetcher` (price-feed health). - `POST $STATS/cfg/chain/{chain}` β€” **upsert** a chain row (creates it if new); body is any subset of the chain fields above. Applies live (no 5-min wait). Examples: point notifications at a chat `{"telegram_chat_id":"-100123…","telegram_enabled":true}`; set the alert floor `{"notify_threshold_usd":5}`; pin the Vault RPC `{"rpc_ref":"/"}`. β†’ `{ok}` (or `{ok, warn}` if the live reload failed but the row saved). - `GET $STATS/cfg/rpcs/{chain}` β€” Vault RPC candidates for the per-chain RPC dropdown β†’ `{chain, options:[{server, vis, url}]}` (ordered public β†’ cached β†’ private). Empty if the chain isn't in the Vault rpc tree. - `POST $STATS/cfg/test/{chain}` β€” fire a sample Telegram alert to the chain's chat (bypasses the arm gate). β†’ `{ok, chat}` / `{ok:false, error}` (400 if no chat id set for the chain). - `POST $STATS/cfg/coverage-run/{chain}` β€” build the 🎯 MEV-coverage report for a coverage chain **now** and send it to that chain's chat (bypasses the arm gate; also refreshes the coverage gauges). Read-only compute (~20s: Vault MEV DB + telemetry probes). β†’ `{ok, chat}` / `{ok:false, error}` (400 non-coverage chain / no chat, 404 no MEV data this hour). - `POST $STATS/cfg/setting/{key}` body `{"value":…}` β€” set a global setting. Keys: `telegram_enabled` (master arm), `auto_stop_mode` (`dry_run`|…), `max_loss_usd`, `loss_window_hours`. - `POST $STATS/cfg/address` β€” **register a labeled wallet** (a "vault"). Body `{address, chain?, name, ours?, role?}`. The `name` is the label shown in win messages (`0.0005 ETH -> spam vault (~$0.98)`); `chain` null/omitted = all chains; `role` defaults to `vault`. `role` ∈ **`vault`**/**`helper`** (`ours:true` β€” destinations that count toward ROI/PnL) Β· **`external`** (`ours:false` β€” tracked-only external flows, e.g. `blink`: shown in messages but excluded from ROI/PnL). β†’ `{ok}`. - `POST $STATS/cfg/address/del/{id}` β€” remove a labeled wallet (`id` from `$STATS/cfg`). β†’ `{ok}`. - `GET $STATS/metrics` β€” Prometheus exposition. Families (labels `chain`, `bot="/"`): counters `bot_revenue_usd`, `bot_fees_usd`, `bot_wins_total`, `bot_mined_total`; gauges `bot_net_usd_cumulative`, `bot_balance_native`, `bot_balance_usd`, and windowed `bot_roi{window}` / `bot_net_usd{window}` (extra label `window` ∈ `1h`|`12h`|`24h`; ROI = profit/gas, net = revenueβˆ’gas). 🎯 coverage gauges (refreshed each hourly cycle, and on `cfg/coverage-run`): `bot_coverage_seen_ratio`, `bot_coverage_calc_ratio`, `bot_coverage_won_ratio` (MEV-weighted share in [0,1] of the last hour's top-N MEV opps each bot received / valued profitable / landed) + `bot_coverage_mev_window_native{chain,unit}` (total MEV of the scored window). Only present for coverage-enabled chains. - `GET $STATS/health` β€” liveness. `GET $STATS/` β€” the config page (HTML; the dashboard's πŸ“Š stats-config iframe). **Terminology:** REVENUE = gross; GAS = fees; NET = PROFIT = revenue βˆ’ gas; ROI = profit/gas. --- ## Common workflows **Find a bot / read its state:** `GET /registry`, index the chain, match by `name` (e.g. `.PLASMA.stage_bot_3`) β€” or `GET /bots` and walk servers if you want the server-keyed view. Everything (chain, status, wallets, balances, incident) is in that one response β€” no extra calls needed. **Investigate a stuck/panicking bot:** check `incident` on the bot; `GET /incident/{id}/log` for the panic + backtrace; `POST /nodecheck/{s}/{b}` to see if the node is the cause. **Change a bot's config safely:** `GET /config/{s}/{b}` β†’ edit β†’ `POST /config/{s}/{b}` (saves to disk, flags restart, no auto-apply) β†’ `POST /act/{s}/{b}/restart` when ready. History is versioned. **Compare configs across bots:** `GET /config/{s}/{b}` for each bot, JSON-parse, and diff the objects (flatten to dot-paths, report paths whose values differ). The dashboard's βš– compare does this live for any number of bots; agents can do the same for any set. **Onboard a server:** `GET /cfg/sshkey`, add it to the box, `POST /cfg/server {name, ip}`, `POST /cfg/reload`. Bots and telemetry are discovered automatically. **Wire a chain's notifications:** `POST $STATS/cfg/chain/{chain} {"telegram_chat_id":"-100…", "telegram_enabled":true}`, then `POST $STATS/cfg/test/{chain}` to confirm it lands. This same chat also receives `bot-status` crash/recovery alerts for the chain. Enable the master arm once with `POST $STATS/cfg/setting/telegram_enabled {"value":true}`. **Enable the 🎯 MEV coverage report:** for a coverage chain (see `cfg.coverage_chains`), `POST $STATS/cfg/chain/{chain} {"coverage_enabled":true,"coverage_top":50}`. It appends a coverage table (seen/calc/won of the last hour's top-N MEV opps, MEV-weighted) to that chain's hourly message and publishes `bot_coverage_*` gauges. Preview any time with `POST $STATS/cfg/coverage-run/{chain}`. **Register a wallet label (a "vault"):** `POST $STATS/cfg/address {chain, address, name, role}` β€” `role:"vault"` for your own destinations (counted in ROI/PnL), `role:"external"` (`ours:false`) for tracked-only flows like `blink`. The `name` is what shows in win messages. List/remove via `GET $STATS/cfg` (each row has an `id`) + `POST $STATS/cfg/address/del/{id}`. **Per-bot telemetry (stats API):** use the bot's `address` from `/registry` (or `/bots`) β†’ `http://
/last//?profitable_only=true&n=N`, `/target_tx//`, `/bundle_tx//`.