Staff Data Scientist, Agent Improvement Loop

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Staff Data Scientist, Agent Improvement Loop based in Brazil.

As a Staff Data Scientist, you will play a key role in improving the performance, reliability, and quality of AI-powered agents in a fast-paced, international environment. You will lead the development of evaluation strategies, analyze production data, and transform insights into measurable improvements. Working closely with Product, Engineering, Delivery, and Customer Success teams, you will help identify performance gaps and drive continuous optimization. You will contribute to the evolution of agent capabilities, conversational experiences, and supporting technical infrastructure. This role combines advanced data science, hands-on engineering, experimentation, and strategic thinking. You will also mentor colleagues and help establish best practices for evaluating and improving AI systems at scale. Your work will directly influence customer outcomes and the development of next-generation conversational AI solutions.

Accountabilities

• Own and develop the evaluation strategy for AI agents, including pre-release behavioral testing, production monitoring, and automated evaluation systems.
• Conduct continuous analysis of agent traces and conversations at scale to identify recurring failure patterns, determine root causes, and prioritize improvement opportunities based on business impact.
• Drive the complete agent improvement cycle by translating analytical findings into actionable fixes, implementing improvements, and measuring their effectiveness through robust before-and-after comparisons and causal analysis.
• Improve agent skills, orchestration frameworks, supporting infrastructure, and developer tools using evidence gathered from production traces, collaborating with relevant engineering teams and contributing directly to their codebases.
• Partner with Delivery and Customer Success teams to improve underperforming customer conversational workflows and establish repeatable processes for maintaining high-quality agent performance.
• Design, calibrate, and validate LLM-as-a-judge evaluation systems against human assessments, measuring consistency, reliability, limitations, and overall effectiveness.
• Develop and maintain production-quality Python and TypeScript code for evaluation and analytical tools, ensuring appropriate testing, documentation, version control, and CI/CD integration.
• Mentor data scientists and engineers in measurement methodologies, experimentation, evaluation practices, and data-driven decision-making.
• Contribute to quarterly and annual strategic planning, helping define priorities and long-term initiatives for improving AI agent quality.

Requirements

• Bachelor's or Master's degree in a technical discipline such as Computer Science, Statistics, Applied Mathematics, Operations Research, Economics, or a related field.
• At least 8 years of experience delivering data science solutions with measurable business impact in cross-functional environments.
• Demonstrated hands-on experience evaluating LLM-based systems or AI agents, including evaluation design, LLM-as-a-judge methodologies, evaluation rubrics, and judge calibration.
• Advanced proficiency in Python and SQL, with working knowledge of TypeScript or the willingness to develop expertise in it.
• Strong software engineering practices, including version control, code reviews, automated testing, CI/CD, and experience building and deploying tools for shared production environments.
• Practical experience working with production GenAI systems, including prompt engineering, tool design, agent frameworks such as LangGraph, and LLM observability platforms such as LangSmith.
• Ability to transform complex and unstructured trace and conversational data into actionable insights and communicate findings effectively to both technical and non-technical stakeholders.
• Strong product and business acumen, with the ability to connect technical quality improvements to customer experience and measurable business outcomes.
• Excellent analytical thinking, problem-solving, communication, and collaboration skills, combined with the ability to work independently and influence cross-functional teams.
• Experience with conversational AI, customer-facing agents, or tool-using AI systems is considered a strong advantage.
• Knowledge of Natural Language Processing (NLP), clustering techniques, and methodologies for discovering and categorizing recurring system failures is a plus.
• Familiarity with causal inference and experimentation methods, including A/B testing, difference-in-differences, and pre/post analysis, as well as the ability to recognize the limitations of available data.
• Exposure to MLOps or LLMOps platforms such as Vertex AI, Amazon SageMaker, MLflow, or similar technologies, as well as big data tools, is desirable.
• Understanding of AI safet