Sandbox Backends
Responsibility
The agent needs to execute commands and run code, but arbitrary commands can't be allowed to hit the host directly. Hermes's terminal tool supports 6 execution backends, chosen by safety/isolation/cost tradeoffs: Local (direct on host), Docker (container isolation), SSH (remote machine), Singularity (HPC container), Modal (cloud sandbox, direct or managed), Daytona (cloud dev environment). Container hardening + namespace isolation; dangerous commands are also intercepted via approval callbacks.
Design Motivation
Why 6 backends instead of 1? The agent's execution scenarios span too wide a range — during dev you just want it fast on the local machine; production batch jobs need container isolation; HPC users must run inside Singularity (Docker isn't an option); unattended cloud runs want a per-second cloud sandbox like Modal. No single backend covers them all. Hermes abstracts "execution" into a uniform interface, plugs backends in by scenario, and shares the dangerous-command approval callback across backends. The Docker backend has to specifically detect whether bind mounts expose host paths — because the safety premise of container isolation is "the container can't see outside," and docker run -v /:/host breaks that. Modal has to distinguish direct (the user's own credentials) from managed (gateway-hosted) — the former bills the user's account, the latter bills the Nous portal quota, and the permission and billing paths are entirely different. These detection logics don't live inside the backends — they're extracted into tool_backend_helpers.py, decoupling backend choice from safety-level judgment.
Key Files
terminal_tool header doc:1-15— lists the six backends local / docker / modal / ssh / singularity / daytonaDocker host access detection:260-290—_docker_volume_uses_host_path/_docker_has_host_access(detects whether bind mounts expose host paths)Singularity / Modal imports:65-90—_get_scratch_dir(from environments/singularity)environments/singularity.py— Singularity scratch directory and SIF cacheModal mode resolution:77-137—coerce_modal_mode/has_direct_modal_credentials/resolve_modal_backend_state(direct vs managed)_DEFAULT_BROWSER_PROVIDER:12— default provider is localsubagent approval callbacks:75-103—_subagent_auto_deny/_subagent_auto_approve(dangerous-command control inside sandboxes)tools/daemon_pool.py— resident thread pool for subagents (carries sandbox execution)tools/ directory— terminal / close_terminal / read_terminal / tool_backend_helpers / environments/
Data Flow
- The agent decides to execute a command → tool_call
terminal→tools/terminal_tool.py. - The backend is chosen by job config (local/docker/modal/ssh/singularity/daytona):
local: direct host execution (fastest, default)docker:tools/terminal_tool.py:260detects whether bind mounts expose host paths and decides permissions accordinglymodal:tools/tool_backend_helpers.py:102resolves direct (user's own Modal credentials) vs managed (gateway-hosted)singularity:tools/environments/singularity.pyprepares scratch and SIFssh/daytona: remote/cloud environments
- Dangerous commands are intercepted by approval callbacks — the main loop uses interactive approval; subagents default to auto-deny (
tools/delegate_tool.py:75), and cron/batch can opt into auto-approve. - Command output is backfilled to the main loop; long tasks can be daemonized (
tools/daemon_pool.py).
The backend list declaration is simple — a single comment at the top of terminal_tool.py: "A terminal tool that executes commands in local, Docker, Modal, SSH, Singularity, and Daytona environments."
The Docker backend's bind-mount detection checks the spec prefix: anything starting with /, ~, ./, ../, or anything with a colon as the second character (like a Windows drive letter C:\) counts as a host path, triggers has_host_access=True, and routes to a stricter permission branch:
def _docker_volume_uses_host_path(volume_spec: str) -> bool:
"""Return True when a docker volume spec bind-mounts a host path."""
if not isinstance(volume_spec, str):
return False
vol = volume_spec.strip()
return bool(vol) and (
vol.startswith(("/", "~", "./", "../")) or
(len(vol) >= 3 and vol[1] == ":" and vol[2] in ("/", "\\"))
)Modal resolution has three tiers: direct (the user's own Modal credentials), managed (Nous portal hosted), auto (default; falls back to direct when managed is unavailable). resolve_modal_backend_state is centralized in one place — if it can't pick a backend, it returns None. has_direct_modal_credentials checks both the MODAL_TOKEN_ID / MODAL_TOKEN_SECRET env vars and the ~/.modal.toml file — both missing means direct is unavailable. managed_nous_tools_enabled queries the Nous Portal account and fails closed on error: a query failure doesn't accidentally open up the managed path.
Singularity is a special case for HPC — users can't install Docker, only Apptainer/Singularity. _find_singularity_executable looks for apptainer first, then singularity; if neither is found, it errors out with install instructions. The scratch directory prefers /scratch (the standard mount on HPC clusters) and falls back to get_sandbox_dir() / "singularity". The container runs with --containall --no-home plus capability dropping — mandatory on a shared HPC cluster, otherwise you'd see into other users' home directories.
Boundaries and Failure
- Docker bind-mount false negative: The detection function treats
/,~,./,../prefixes as host paths, but a named volume (likemydata:/data) passes through. When a named volume backs onto a host path (custom driver),_docker_has_host_accesscan't see it and mis-judges the risk as low. - Modal managed failure fallback: In
automode, managed falls back to direct when unavailable. But if the user only configured managed and has no direct credentials,selected_backendbecomesNone— backend selection fails and only blows up at runtime; terminal creation may not surface a clear error. - Singularity SIF cache bloat: SIF images are often hundreds of MB to several GB; without a rotation policy the scratch directory can fill the disk. HPC nodes usually have a cluster-level cleanup policy on
/scratch; when the two don't align, you get "the cluster wiped the cache but the agent still thinks the SIF is there." - Remote backend network jitter: SSH / Daytona commands run over the network. When the connection drops, the running subprocess's output is lost. Output backfill assumes a single synchronous return — there's no reconnect-and-resume, so long tasks are riskiest on remote backends.
- Timeout and OOM: local / docker have
script_timeout, but remote-backend timeouts rely on the remote process to kill itself. The local scheduler waits for a response that never comes and stalls, or gets falsely declared dead by the lease TTL in_run_job_script_with_claim_heartbeat.
Summary
The sandbox layer is "execution isolation + dangerous-command control": 6 backends cover host through HPC to cloud; Docker detects bind mounts to prevent privilege escalation; Modal distinguishes direct/managed credentials; dangerous commands uniformly go through approval callbacks. Subagents and cron take the non-interactive auto-deny/auto-approve path.