Site Reliability Engineer - Fleet Command & Control

The posting, as Etched wrote it

About Etched

Etched is building hardware for frontier intelligence. We co-design chips, racks, software, and manufacturing to deliver best-in-class throughput and latency across both prefill and decode workloads. Our first products are heavily focused on inference. Backed by hundreds of millions from top-tier investors and staffed by leading engineers, Etched is redefining the infrastructure layer for the fastest growing industry in history.

Job Summary

You'll be one of Etched's first Site Reliability Engineers, building the command-and-control layer for our fleet of frontier inference systems: the alerting, automated recovery and fleet data that keep clusters serving, and the routing that gets each problem to the right expert. Our Supercomputing and Infrastructure teams build the fleet management, rollout and telemetry backbone; but you'll decide which signals define a healthy system, set how we alert and respond, and build agentic hardware health workflows on top of it.

Your job is to design and run a scalable system. Fleet SREs own detection, first response and handoff; domain SREs embedded in each team own the deep product fixes. Over time, most problems should be caught by an alert that measures what users experience, which is then triaged by agents, and sent to an engineer as needed, with the logs, dashboards and recent changes already attached.

Key Responsibilities

• Define what a healthy inference system looks like. In collaboration with software teams building out management and telemetry systems, identify the signals that most clearly show an unhealthy site, rack, node or chip, keep refining them as the fleet changes, and decide which actions (drain, cordon, restart, reroute, roll back) run on their own and which page a person.
• Set a common alerting standard. Define SLOs with each team (latency such as time to first token, error rate, availability), alert on user-facing symptoms, and verify that the metrics behind each alert measure what the alert assumes.
• Cut alert noise. Suppress expected alerts during deploys and maintenance automatically, correlate related alerts into one incident, and retire pages nobody acts on.
• Automate recovery. Turn runbooks into agentic hardware health recovery: service restarts, node drain and traffic shift, spare swaps. Build it with guardrails (scoped permissions, rate limits, blast-radius checks, an audit trail)..
• Route every event to the right engineer. Classify events by domain, page the right domain SRE with context attached, and dispatch on-site technicians for physical work such as tray swaps, reseats and leak response.
• Shape the fleet data. Work with Supercomputing and Infrastructure, who build the telemetry pipeline, so it captures the key signals. Set the tracing standard (OpenTelemetry) so every service can be followed end to end, and report uptime and reliability trends that drive engineering and capacity decisions.
• Run operationally excellent incident ma

[…]

• Clear writing, and the ability to get five or more teams to agree on shared standards.

Strong candidates may also have experience with (Nice-to-have qualifications)

• Operated GPU, accelerator or HPC fleets.
• LLM inference serving: prefill and decode, KV cache, batching, and latency metrics such as time to first token.
• Network debugging from switch to edge: tracing problems across switches, load balancers and Cloudflare, and using edge controls like rate limits to shed or shift traffic during an incident.
• Out-of-band management (BMC, Redfish, IPMI) and fleet-wide firmware rollouts.
• Data center telemetry: power distribution, liquid cooling and CDUs, DCIM.
• PagerDuty or similar incident tooling, including event correlation and automated remediation.
• Working with colocation providers and remote-hands technicians.
• Founding or scaling an SRE function at a startup.

Benefits

• Medical, dental, and vision packages with generous premium coverage
• $500 per month credit for waiving medical benefits
• Housing subsidy of $2,500 per month for those living within walking distance of the office
• Relocation support for those moving to San Jose (Santana Row)
• Various wellness benefits covering fitness, mental health, and more
• Daily lunch and dinner in our office
• Unlimited compute budget subject to ROI justification

How we’re different

Etched believes in the Bitter Lesson http://www.incompleteideas.net/IncIdeas/BitterLesson.html. We are the first inference-focused frontier AI system, betting early on transformer and transformer-like architectures and on increasing model sizes. Our addressable market is the entirety of inference, unlike many of our competitors.

We are a fully in-person team in San Jose (Santana Row), and greatly value engineering skills. We do not have boundaries between engineering and research, and we expect all of our technical staff to contribute to both and work across disciplines as needed.