Skip to main content

Staff Software Engineer, Inference Infrastructure

Posted today

cohereBrooklyn (NY)

SENIORITY

Lead

Apply (opens in a new tab)

About the role

Overview: As a Member of Technical Staff on the Model Serving team, you help build, deploy, and operate Cohere’s AI platform that delivers large language models through easy API access. You’ll ship scalable, low-latency NLP services and ensure high availability in production. You’ll collaborate with multiple teams and tailor deployments to customer needs. This role combines cutting-edge ML infrastructure with customer-facing impact, enabling widespread AI adoption. You will work in a fast-paced environment to push the boundaries of scalable model serving. Compensation / Benefits open and inclusive culture health and dental benefits with mental health budget 6 weeks of vacation parental leave top-up (up to 6 months) remote-flexible with global offices and stipend co-working stipend
Responsibilities: Develop, deploy and operate the AI platform delivering Cohere's LLMs via API endpoints Design and deploy optimized NLP models to production with low latency, high throughput, and high availability Collaborate with cross-functional teams to support, troubleshoot, and scale mission-critical systems Interface with customers to create customized deployments meeting specific requirements Investigate and resolve complex infra issues in distributed, Linux-based environments Manage compute/storage/network resources and costs across multi-cloud/hybrid settings
Key requirements: 5+ years of engineering experience running production infrastructure at large scale Experience designing large, highly available distributed systems with Kubernetes and GPU workloads Experience with Kubernetes dev and production coding and support Experience with GCP, Azure, AWS, OCI, multi-cloud on-prem / hybrid serving Experience in complex Linux-based computing environments Experience in compute/storage/network resource and cost management Strong collaboration and troubleshooting skills for mission-critical systems Grit and adaptability to solve evolving technical challenges Familiarity with accelerators (GPUs/TPUs) and their impact on latency and throughput Strong understanding or working experience with distributed systems Experience in Golang, C++ or other high-performance server languages collaboration troubleshooting adaptability Kubernetes distributed systems cloud platforms (GCP, Azure, AWS, OCI)

Before you apply

Applying takes about a minute. These four things decide how fast it moves after that.

Your profile is current

It's what we read first. Occupations, seniority and locations matter more than a long history.

Two examples you can talk through

Not a portfolio — just two pieces of work where you can explain the decisions and what you'd change.

A number in mind

What you're on now and what would make you move. We negotiate better when we know both.

Your notice period

Employers plan around it, and it's the question that stalls offers most often.

Once you apply, someone reads it and calls you before anything reaches the employer — usually within two working days.

More like this