LLMOps Engineer at ALX

Role Summary

• Project A is ALX’s AI learning platform — a set of LLM products used by learners. Every one generates a stream of LLM data, and every one has hypotheses baked into it about what “working” means. The LLMOps Engineer owns the analyzer function: turning that stream into an honest answer about whether the products work. Take RAG as one example — documents must be stored accurately, fetched accurately, and fetched in the right mixture: three separate failure modes, each needing its own eval. Every product decomposes like that. This is a junior-to-mid role with a deliberate growth path: you start close to the technical lead’s designs and grow into full ownership of the function.
• You will work in collaboration with Anthropic Engineers, a cross functional  team of AI engineers, product managers and data scientists to design world class learning experiences.

Specific Responsibilities

Evaluation Suites

• Build and run eval suites per product, decomposed by failure mode, running on schedule and on every release — regression testing so nothing ships if it broke what worked.
• Keep evals cost-effective as the product line grows.

Reporting, Data & Collaboration

• Own the reporting loop — findings from evals and platform data in front of the team and stakeholders, including surfacing unintended or problematic model behaviour before learners do.
• Steward the core datasets the team depends on, including classified customer-support data.
• Partner with the AI Product Manager on instrumentation — they instrument the product, you build the evals over what is captured. This is a measurement role, not infrastructure — no model hosting or serving.

Skill Requirements - Essential

• Python & data: solid Python and a data inclination,  comfortable shaping and analysing messy LLM-generated data.
• Decomposition: the ability to look at an AI product and decompose it into success and failure metrics.
• Eval landscape: familiarity with Langfuse, RAGAS, DSPy, or similar — depth in one, awareness of the rest. These tools are learnable; we hire the fundamentals underneath them.
• Desirable (not required): experience keeping evals cheap at scale; dashboarding and reporting; classical statistics.

Essential Traits for Success

• You want to own a function, not execute tickets.
• You communicate well and like collaborating,  you will support every builder on the team.
• You can point to any project, even a small one, where you measured an AI system honestly.