GPU Performance & Validation Engineer

Key Responsibilities: GPU Performance & Production Validation Design and implement validation frameworks to assess GPU node and cluster readiness, functionality, utilization, and performance. Establish repeatable testing methodologies for AI/ML and HPC workloads across large-scale GPU clusters. Execute industry-standard and workload-specific benchmarks across GPU compute, storage, and networking. Profile workloads and investigate bottlenecks across hardware, operating systems, interconnects, and supporting infrastructure. Establish performance baselines and acceptance criteria for new deployments and infrastructure changes. Partner with engineering teams to resolve issues and verify improvements through repeatable testing. Evaluate throughput, latency, and scalability across InfiniBand and RoCE environments. Benchmark storage performance and assess its impact on GPU workload efficiency. Perform Linux system benchmarking to identify configuration gaps, resource constraints, and performance inconsistencies. Assess GPU and DPU performance using NVIDIA tools and other relevant benchmarking utilities. Compare results across nodes, clusters, and locations to identify deviations and optimization opportunities. Automation & Validation Frameworks Develop automated validation tools and micro-benchmarking frameworks using Python, Go, and Kubernetes. Build repeatable workflows for test execution, results collection, analysis, and reporting. Integrate benchmarking and validation into CI/CD pipelines and infrastructure lifecycle processes. Maintain testing documentation, benchmark configurations, and reproducible evaluation procedures. Expand continuous validation coverage as hardware, software, and workload requirements evolve. Observability & Performance Analysis Develop monitoring and dashboards using Prometheus, Grafana, OpenTelemetry, and ELK. Define telemetry requirements for GPU utilization, cluster health, network and storage performance, and workload behavior. Correlate benchmark results with infrastructure telemetry to support root-cause analysis. Establish observability practices for ongoing performance assessment and troubleshooting. Architecture & Technical Collaboration Translate validation results into recommendations for infrastructure design, configuration, tuning, and capacity planning. Document hardware evaluations, testing methodologies, performance findings, and remediation recommendations. Collaborate with engineering, infrastructure, and research teams to align testing with production workload requirements. Evaluate emerging hardware, benchmarking tools, and architectures to support infrastructure planning. Lead technical investigations and communicate findings to technical and business stakeholders.

Required Qualifications: Experience validating, profiling, and tuning large-scale GPU clusters and accelerator-based infrastructure. Strong knowledge of NVIDIA ClusterKit, Nsight, DCGM, NVIDIA validation tools, and MLPerf, including GPU and DPU performance assessment. Experience benchmarking network and storage performance in InfiniBand and RoCE environments using ClusterKit, iPerf, or comparable tools. Hands-on Linux benchmarking experience using the Phoronix Test Suite or equivalent tools. Strong programming and automation skills using Python and Go, with experience developing tools for Kubernetes environments. Experience working in Ubuntu Linux environments. Experience building repeatable validation frameworks, micro-benchmarks, and automated testing workflows. Strong knowledge of OpenTelemetry, Prometheus, ELK, and Grafana, including defining observability practices for HPC infrastructure. Experience supporting geographically distributed HPC environments and using performance data to guide architectural decisions. Ability to troubleshoot across GPU compute, Linux, networking, storage, and workload layers. Demonstrated ability to lead technical initiatives, document findings, and influence decisions across engineering and research teams. Ability to evaluate emerging technologies and translate findings into practical infrastructure recommendations. Education Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent professional experience.