Senior Software Engineer - ML Inference & Numerics

Job Title

Senior Software Engineer - ML Inference & Numerics

Role Summary

Develop software to run large AI models efficiently on custom accelerator silicon. Work end-to-end from model reference code to validated hardware performance, with a focus on numerical behavior and automation.

You will own parts of the SDK, build agentic workflows to explore implementations and experiments, and collaborate with teams across models, compilers, runtime, and hardware.

Experience Level

Senior-level engineer. The role expects substantial experience owning complex systems and delivering production-grade software.

Responsibilities

Deliver reliable, measurable model support and numerical tooling for inference on accelerator hardware.

• Design and implement agentic workflows that inspect models, modify implementations and numerical settings, run emulation and hardware experiments, and collect trusted evaluation metrics.
• Port new model architectures from reference code to validated, performant execution on the accelerator; produce reusable SDK support.
• Optimize numerics: evaluate precision, accumulation, and quantization choices and measure impacts on latency, throughput, memory, and model quality.
• Build conversion and emulation tools to compare activations/outputs with reference models, trace accuracy regressions, and validate end-to-end changes.
• Own design, implementation, testing, and delivery of SDK components; produce clear APIs and durable tests to prevent regressions.
• Collaborate with hardware architects to evaluate numerical formats and arithmetic choices and feed findings into hardware and algorithm design.

Requirements

Must-have technical skills and behaviors for the role.

• Deep understanding of LLM inference (prefill, decode, attention, KV caches, MoE, batching, parallelism) and how these map to compute, memory, and communication on accelerators.
• Proven experience owning substantial systems and delivering complex features from design through production; strong software engineering practices.
• Strong Python skills, thoughtful API and interface design, and rigorous approach to abstractions and testing.
• Fluency with agentic tooling and automation; ability to design agentic loops that scale and know when human oversight is required.
• Strong debugging and performance optimization skills across latency, throughput, and memory trade-offs.
• Clear written and verbal communication; ability to form hypotheses, write specs/design docs, and lead cross-functional investigations.

Nice-to-have:

• Experience with ML compilers, accelerator SDKs, or model compression libraries for non-NVIDIA accelerators (e.g., TPU, Trainium/Inferentia, Tenstorrent, Groq, Cerebras).
• Experience with model quantization, sparsity, codebooks, low-rank factorization, or other compression techniques.
• Contributions to distributed inference or serving frameworks such as vLLM or SGLang.
• Prior production experience running inference on custom silicon or low-power accelerators.

Education Requirements

Not specified.

About the Company

Company: Tensordyne

Tensordyne is a startup developing high-performance, low-power AI inference processors and multi-chip systems for generative AI acceleration in data centers. The company focuses on ASIC design, verification, and integration of computational accelerators with third-party SoC IP.