Software Engineer, SI Workload Scheduling, SI Infra Supercomputing
$140,000 per year
Apply (opens in a new tab)Software Engineer, SI Workload Scheduling, SI Infra Supercomputing
Posted 11 days ago
SENIORITY
Senior
SALARY
$140,000 per year
About the role
- Own and evolve the Slurm scheduler configuration, plugins, and policies that govern job scheduling across our GPU supercomputers
- Design and implement scheduling policies that maximize cluster utilization, minimize job queue times, and enforce fair-share allocation across teams and priorities
- Build tooling and automation around job submission, preemption, backfill, and resource reservation to support the dynamic needs of AI engineering
- Develop monitoring and observability infrastructure to provide real-time visibility into cluster utilization, job throughput, and scheduling efficiency
- Debug and root cause scheduling failures and resource contention issues across thousands of nodes and implement fixes to prevent recurrence
- Coordinate with the operations team managing the training cluster to maintain high availability and job throughput
- Work closely with the ML team to understand workload patterns and evolving resource requirements
- Members of the Supercomputing AI Infrastructure team are expected to be adaptable to the dynamic requirements of AI software engineering and capable of contributing across all parts of the AI and HPC software stack
- Deep hands-on experience with Slurm (configuration, plugins, scheduling algorithms, accounting) or comparable cluster/workload scheduling systems (e.g., PBS Pro/OpenPBS, LSF, Grid Engine, HTCondor, Kubernetes-based batch schedulers)
- Strong knowledge of Python and Linux systems administration
- Experience operating and tuning HPC job schedulers at scale (>=thousands of nodes)
- Understanding of GPU cluster topologies, networking fabrics, and how scheduling decisions impact training performance
- Experience building tooling and automation for cluster operations (job orchestration, resource monitoring, capacity planning)
- Familiarity with containerization and orchestration technologies (Docker, Kubernetes, Pyxis/Enroot)
- Understanding of modern machine learning training workflows and resource requirements
- Experience with parallel programming concepts and distributed systems
- Medical plans > plan options with $0 payroll deduction
- Family-building, fertility, adoption and surrogacy benefits
- Dental (including orthodontic coverage) and vision plans, both have options with a $0 paycheck contribution
- Company Paid (Health Savings Accounts) HSA Contribution when enrolled in the High-Deductible medical plan with HSAHealthcare and Dependent Care Flexible Spending Accounts (FSA)401(k) with employer match, Employee Stock Purchase Plans, and other financial benefits
- Company paid Basic Life, AD&DShort-term and long-term disability insurance (90 day waiting period)
- Employee Assistance Program
- Sick and Vacation time (Flex time for salary positions, Accrued hours for Hourly positions), and Paid Holidays
- Back-up childcare and parenting support resources
- Voluntary benefits to include: critical illness, hospital indemnity, accident insurance, theft & legal services, and pet insurance
- Weight Loss and Tobacco Cessation Programs
- Tesla Babies program
- Commuter benefits
- Employee discounts and perks program
- Expected Compensation $140,000 - $252,000/annual salary + cash and stock awards + benefits
Before you apply
Applying takes about a minute. These four things decide how fast it moves after that.
Your profile is current
It's what we read first. Occupations, seniority and locations matter more than a long history.
Two examples you can talk through
Not a portfolio — just two pieces of work where you can explain the decisions and what you'd change.
A number in mind
What you're on now and what would make you move. We negotiate better when we know both.
Your notice period
Employers plan around it, and it's the question that stalls offers most often.
Once you apply, someone reads it and calls you before anything reaches the employer — usually within two working days.
