Site Reliability Engineer
Site Reliability Engineer
Posted 8 days ago
SENIORITY
Manager
About the role
- Lead the design and implementation of enterprise observability standards across logging, metrics, tracing and alerting
- Own and evolve the observability platform strategy and roadmap, including Datadog and SolarWindsDefine and implement instrumentation standards for applications, APIs and infrastructure to ensure consistent telemetry collection
- Partner with development teams to embed observability into CI/CD pipelines as a first-class deployment requirement
- Design and implement automated monitoring, alerting and self-healing patterns to improve service reliability and availability
- Develop reusable observability frameworks, libraries and deployment templates for engineering teams
- Establish monitoring-as-code, alerting-as-code and dashboard-as-code standards
- Design service-level indicators (SLIs), service-level objectives (SLOs) and error budgets in partnership with product and engineering teams
- Ensure monitoring coverage for cloud, on-prem, containerized and hybrid environments
- Implement distributed tracing and log correlation to enable end-to-end transaction visibility
- Define observability and tracing standards for AI-enabled applications, copilots and autonomous or agentic workflows, including model calls, prompts and responses, retrieval steps, tool invocations, latency, token consumption, cost and failure states
- Implement end-to-end AI tracing that correlates user requests across orchestration layers, models, retrieval-augmented generation (RAG), vector stores, APIs, tools and downstream services
- Establish monitoring and evaluation patterns for AI response quality, hallucination risk, model or prompt drift, guardrail violations, safety events and degraded user outcomes
- Define telemetry governance for AI workloads, including sensitive-data redaction, access controls, retention, sampling and secure handling of prompts, responses and trace data
- Serve as the escalation point for complex monitoring, performance and reliability issues
- Collaborate with enterprise and platform architects to ensure observability requirements are built into reference architectures
- Drive root cause analysis and reliability improvements following major incidents
- Develop platform governance standards for monitoring, alerting thresholds, and on-call best practices
- Ensure observability is tightly integrated with CMDB and Service Mapping for accurate service context and impact analysis
- Partner with DevOps and platform engineering teams to improve deployment reliability and pipeline automation
- Tune alerts continuously to reduce noise and improve signal quality
- Support capacity planning, performance engineering and load testing efforts
- Engage with Datadog, SolarWinds and ServiceNow vendors and user communities to stay ahead of platform capabilities
- Develop training and enablement materials for engineering teams on observability best practices
- Contribute to reliability engineering practices including chaos testing, fault injection and resiliency validation
- Drive continual improvement of incident response, post-incident reviews and operational readiness
- Integrate observability platforms with ServiceNow for incident, event, change and problem automation
- Defines observability policy aligned with CMDB governance, service modeling and configuration management standards
- Ability to demonstrate success implementing monitoring, logging, and tracing at scale across complex environments
- Deep expertise with Datadog and SolarWinds monitoring platforms
- Strong experience designing observability for cloud, hybrid and on-prem environments (Azure, Kubernetes, VMs)
- Expertise in logging platforms, APM, distributed tracing and infrastructure monitoring
- Working knowledge of generative AI and LLM application architectures, including copilots, agents, orchestration frameworks, retrieval-augmented generation (RAG), vector databases, model gateways and tool-based integrations
- Experience with Open
- Telemetry or comparable instrumentation standards and the ability to apply trace context across AI, application, API and infrastructure dependencies
- Understanding of AI observability concepts including prompt and model versioning, token and cost analysis, latency decomposition, quality evaluation, drift detection, guardrails and human feedback signals
- Ability to balance diagnostic trace depth with privacy, security, retention and cost requirements when collecting AI telemetry
- Strong proficiency in automation and scripting (Python, PowerShell)
- Experience implementing observability into CI/CD pipelines (GitHub Actions, Azure DevOps, GitLab)
- Strong understanding of infrastructure-as-code and configuration management (Terraform, Ansible, etc.)Experience with container and orchestration platforms (Docker, Kubernetes)
- Strong understanding of networking, operating systems and application architectures
- Experience with event-driven monitoring, anomaly detection and predictive alerting
- Excellent troubleshooting, root cause analysis and incident management skills
- Strong documentation, architecture diagramming and platform governance capabilities
- Ability to lead cross-functional reliability initiatives and influence engineering standards
Before you apply
Applying takes about a minute. These four things decide how fast it moves after that.
Your profile is current
It's what we read first. Occupations, seniority and locations matter more than a long history.
Two examples you can talk through
Not a portfolio — just two pieces of work where you can explain the decisions and what you'd change.
A number in mind
What you're on now and what would make you move. We negotiate better when we know both.
Your notice period
Employers plan around it, and it's the question that stalls offers most often.
Once you apply, someone reads it and calls you before anything reaches the employer — usually within two working days.
