Software Engineer, Systems ML Tooling

Job Title

Software Engineer, Systems ML Tooling

Role Summary

Join the MTIA (Meta Training & Inference Accelerator) Software Tooling team to design and build developer tooling for Meta's in-house AI accelerators. The team provides debugging, profiling, memory analysis, and monitoring across the full MTIA stack.

The role focuses on debugging, sanitizer technology, and fault isolation for AI workloads on MTIA hardware and operates at the intersection of compilers, runtime, hardware, and ML frameworks. Candidates may concentrate on debugging/sanitizing technology or simulation infrastructure depending on background and team needs.

Experience Level

Mid-level - the posting expects 4+ years of relevant systems or tooling experience.

Responsibilities

Key responsibilities include end-to-end ownership of tooling components and collaboration with cross-functional partners across the software and hardware stack.

• Design and implement debugging and sanitizer tooling (live debugging, core dumps, memory sanitizers) for accelerator workloads.
• Build multi-layer debugging capabilities: graph-mode, kernel-level diagnostics, and multi-rank fault isolation.
• Contribute to profiling, performance debugging, memory analysis, monitoring, and reliability tooling for training and inference.
• Drive technical design, implementation, production rollout, and long-term maintenance of major tooling components.
• Collaborate with compiler, runtime, kernel, and hardware teams to add instrumentation and hooks required for tooling.
• Apply automation and LLM-guided diagnostics to accelerate root-cause analysis.
• Communicate architecture and debugging methodology through design documents and cross-team reviews.

Requirements

Must-have technical skills and experience; preferred items are noted separately.

• Must-have: 4+ years of systems software engineering, performance engineering, developer tooling, or equivalent practical experience.
• Must-have: Experience building debugging, sanitizer, profiling, simulation, or diagnostic tools for complex software/hardware systems.
• Must-have: Proficiency in C++ and Python, including low-level systems programming and scripting for tool automation.
• Must-have: Cross-stack debugging ability across compiler, runtime, OS/driver, and hardware boundaries; track record delivering tooling to production.
• Preferred: Compiler-based instrumentation and runtime shadow-memory techniques (LLVM passes, interceptors, allocator red zones).
• Preferred: Familiarity with ML framework internals (e.g., PyTorch), AI compiler stacks (MLIR, LLVM, TVM, Triton) and accelerator ecosystems (GPU/CUDA, TPU, custom ASICs).
• Preferred: Experience with Linux debugging and profiling (gdb, perf, eBPF, ftrace, coredump analysis) and with simulator or virtual-platform frameworks (gem5, QEMU, AModel, BModel).
• Preferred: Experience working at device-software boundary (runtime/driver internals, DMA, memory-mapped devices, firmware-assisted error reporting) and integrating AI tools to optimize workflows.

Education Requirements

Bachelor's degree in Computer Science, Computer Engineering, or a related technical field, or equivalent practical experience. Advanced degrees (MS/PhD) in related fields are listed as preferred.

About the Company

Company: Meta Platforms

Headquarters: Menlo Park, California, United States

American technology company that develops social networking products (Facebook, Instagram, WhatsApp) and invests in virtual/augmented reality hardware and software through Reality Labs, focusing on connectivity, advertising, and immersive computing experiences.