Site Reliability Engineer

• Own monitoring architecture and signal quality across GPU and host telemetry (DCGM, Prometheus/Grafana) and facility telemetry (BMS, EPMS, DCIM, SCADA, CDU / direct liquid cooling): what we alert on, suppress, and trust. Consume NOC noise-disposition feedback to drive suppression and redesign. Treat alert noise as a design failure, not an operator failure.
• Provide SEV command support: technical incident leadership, bridge coordination with the NOC, paging and escalation (e.g., PagerDuty), and timeline and severity hygiene.
• Run blameless postmortems and drive corrective actions to closed, not filed.
• Gate high-risk change: fleet-scale firmware/BIOS/BMC rollouts (Redfish, canaries, rollback) and SOP/MOP/EOP-governed work on power and cooling systems through a disciplined change management process.
• Lead cross-functional reliability projects spanning compute, network (InfiniBand/RoCE), storage, and facility signal boundaries, including grey failures that cross those boundaries (e.g., NVLink flapping, thermal throttling, power sequencing).
• Partner with Hardware Failure Analysis on systemic failure classes: turn fleet data (MTBF/AFR, repeat-RMA trends) into monitoring, stop-rules, and vendor RCA inputs.
• Build and maintain playbooks, run game days, and keep cross-discipline dependency maps current. Own runbook quality jointly with the NOC (SRE designs; NOC operates and corrects).
• Define error budgets and availability objectives at campus and service boundaries as adopted by the business.
• Participate in on-call rotations and incident response for SEV-class events in the Memphis / Southaven data center campus.

PROFILE SIGNALS:

• Experience in power plants, nuclear power plants, data centers, or facilities like SpaceXSI — current or prior — spanning power, compute, cooling, or related plant systems.
• Proven large-scale incident command experience and calm technical leadership on a bridge. Nuclear control-room, shift supervisor / Senior Reactor Operator (SRO), Shift Technical Advisor, or Navy nuclear experience counts as incident command.
• Demonstrated monitoring and observability design at fleet or campus scale, including alert hygiene, suppression, and signal quality. Treats alert noise as a design failure, not an operator failure.
• Cross-team facilitation and systems engineering depth across software and facility boundaries.

SUCCESS MEASURED BY:

• MTTD / MTTR trend for SEV-class events
• % of SEVs with a blameless postmortem and closed action items
• Alert actionable ratio
• Recurrence rate of incident classes
• Monitoring coverage of critical dependencies across compute, network, storage, power, and cooling
• GPU fleet availability lost to infrastructure events

• Bachelor's degree in Systems Engineering, Computer Science, Electrical Engineering, or a related field (or equivalent experience).
• 5+ years of experience in site reliability, systems engineering, plant operations, or large-scale production operations, preferably in power plants, nuclear power plants, high-performance computing, or data center environments.
• Proven large-scale incident command experience and calm technical leadership on a bridge, whether from an SRE/NOC bridge or a plant control room.
• Demonstrated monitoring and observability design at fleet or campus scale, including alert hygiene, suppression, and signal quality.
• Hands-on experience with at least one of: Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry and controls (SCADA, BMS/BAS, EPMS, DCIM, PLCs, Modbus/BACnet).
• Experience working across at least two of: compute, network, storage, power, and cooling / facilities telemetry.
• Experience writing and operating playbooks, runbooks, or SOP/MOP/EOPs with a 24/7 operations, control room, or NOC partner.
• Proficiency in scripting (Python, Bash) for automation and analysis, plus general experience in at least one systems language (C, C++, Java, Go, Rust, or similar). Not required to be expert in all of them.
• Excellent problem-solving skills with a data-driven approach to reliability engineering.
• Ability to work collaboratively with cross-functional teams, including NOC, data center operations, hardware failure analysis, and infrastructure engineering.

• Experience in power plants, nuclear power plants, or other high-consequence industrial control environments (ICS/OT).
• Current or prior work in data centers or facilities like SpaceXSI spanning power, compute, cooling, and related plant systems, including CDU / direct liquid cooling.
• Experience in AI/ML infrastructure or supercomputing environments: GB200/NVL72 or H100/H200-class GPU fleets, NCCL, InfiniBand/RoCE, Slurm or Kubernetes.
• Fleet firmware and BMC management (Redfish, IPMI), including firmware release validation and staged rollouts.
• Reliability engineering methods: MTBF/AFR analysis, reliability-centered maintenance, grey-failure diagnosis, and vendor RCA/RMA partnership.
• Experience with CMMS (e.g., Maximo) and commissioning / integrated systems testing (IST) and operational turnover.
• Hands-on definition and use of SLOs, SLIs, and error budgets at service or campus boundaries.
• Experience running game days, dependency mapping, and closed-loop corrective action programs.
• Familiarity with data center hardware and plant signals (servers, GPUs, networking, power, cooling) in addition to software telemetry.
• Prior work in a fast-paced startup or tech company like SpaceXSI.

SpaceXSI is an equal opportunity employer. For details on data processing, view our Recruitment Privacy Notice.