Senior Resiliency Architect, Hardware Diagnostics and Resiliency Modeling

Job Title

Senior Resiliency Architect, Hardware Diagnostics and Resiliency Modeling

Role Summary

Design and implement hardware diagnostics, resiliency models, and automation for datacenter and edge GPUs and SoCs. Work with cross-functional engineering teams to analyze field data, improve fault attribution and recovery, and drive reliability improvements across system-to-datacenter scales.

Experience Level

Senior level. The role requires at least 5+ years of relevant industry experience.

Responsibilities

Primary responsibilities focus on diagnostics, modeling, automation, and cross-team collaboration to improve system reliability and availability.

• Develop and maintain diagnostics suites to stress-test GPUs and SoCs, detect hardware defects and silent data corruption across package/board/rack configurations.
• Analyze coverage gaps from silicon failures and customer workloads; enhance tests for coverage, repeatability, and run-time optimization.
• Model and analyze RAS metrics (FIT, transient/permanent error rates, Availability) from device to rack to datacenter.
• Analyze field interruption and failure data to improve fault attribution, containment, and recovery strategies.
• Design, build, and maintain automation and tooling to orchestrate diagnostic jobs across multiple compute clusters.
• Develop and maintain dashboards that integrate diverse data sources, provide analytics and visualization, and support secure multi-user access.
• Collaborate with engineering teams to ensure telemetry correctly classifies failures as hardware faults, test-script issues, or environment/configuration problems.
• Ensure diagnostics and availability models evolve with changing AI workloads.

Requirements

Must-have technical skills, tool proficiency, and professional capabilities.

• Minimum 5+ years relevant experience in hardware diagnostics, reliability engineering, or related fields.
• Deep knowledge of GPU architecture and programming; experience with hardware diagnostics.
• Understanding of reliability concepts: FIT, AVF estimation, Availability, and ECC/parity/CRC strategies.
• Proficiency in data mining, automation, and scripting (Python, SQL or similar).
• Proficiency in C/C++ and CUDA; experience using AI-assisted coding/debugging tools.
• Strong debugging, statistical judgment, and analytical skills.
• Experience with container-based job execution and cluster orchestration; ability to build automation tooling.
• Good interpersonal skills and ability to collaborate with on-site and remote teams.
• Nice to have: experience developing embedded software, visualization/data-analytics tools, and deep familiarity with resiliency trade-offs in AI datacenters.

Education Requirements

Master's degree in Computer Engineering, Electrical Engineering, Computer Science, or a closely related field; or equivalent practical experience.

About the Company

Company: NVIDIA

Headquarters: Santa Clara, California, USA

NVIDIA is a global leader in accelerated computing, renowned for its innovative solutions in AI and digital twins that transform diverse industries. The company specializes in networking technologies, providing end-to-end InfiniBand and Ethernet solutions for servers and storage that optimize performance and scalability. NVIDIA serves sectors such as high-performance computing, enterprise data centers, and cloud computing, constantly reinventing its products and services to stay ahead in the market.