Senior AIOps ML Engineer

Key Responsibilities: Lakehouse Architecture & Data Engineering: Design and evolve Lakehouse schema (Delta Lake / Apache Iceberg) for multi-domain observability data at petabyte scale. Build and maintain robust ingestion pipelines from OTel Collector through Kafka to the Lakehouse, ensuring exactly-once semantics and strict schema enforcement. Implement dbt transformation models to generate mart-ready, denormalized fact and dimension tables. Define and enforce data quality contracts and SLAs. Optimize query performance utilizing partitioning strategies, Z-ordering, bloom filters, and materialized views. ML Model Development & AIOps: Design, train, and deploy machine learning models for streaming multivariate anomaly detection, root-cause analysis, and incident forecasting. Build low-latency streaming inference pipelines (Flink / Spark Streaming) for real-time anomaly scoring. Develop sophisticated log intelligence models (clustering, NLP classification, error deduplication). Implement unsupervised and semi-supervised methods for User Experience frustration detection and KPI correlation. Own the ML feature store, managing feature engineering, versioning, and backfill pipelines. Instrument model performance tracking, including drift detection, accuracy monitoring, and automated retraining triggers. AIOps Platform & Productionization: Design and operate the end-to-end AIOps workflow, spanning signal ingestion, feature computation, model inference, alert routing, and auto-remediation hooks. Build high-performance model serving infrastructure supporting real-time REST/gRPC endpoints and async batch scoring with strict p 99 latency SLOs. Integrate AIOps insights with incident management platforms (Pager Duty, Opsgenie) and internal runbooks. Define and publish metrics from the Business KPI mart to quantify business impact. Security & Compliance Observability: Partner with the Security team to build the Security mart schema, including threat feed ingestion, UEBA baselines, and CVE correlation pipelines. Train anomalous-access and lateral-movement detection models. Ensure all data handling adheres strictly to data residency requirements, PII masking standards, and audit-log protocols. Collaboration & Engineering Standards: Define telemetry schema contracts with the OTel Instrumentation team. Author ML platform RFCs and contribute actively to observability data model standards. Mentor junior ML and data engineers and conduct rigorous design reviews.

Required Skills: AI Agents Kafka and Streaming technologies (Flink / Spark)Lakehouse architectures (Delta Lake / Apache Iceberg)Machine Learning (Anomaly detection, time-series analysis)Observability tools and concepts (OTel, APM, Logs)MLOps practices (feature store management, drift detection, model retraining)Strong proficiency in SQL and Python

Qualifications: 10+ years of experience in a relevant field.