Senior QA AI Engineer 

RESPONSIBILITIES: 

• Define and drive the overall testing strategy for AI agents and LLM-powered solutions
• Design, implement, and maintain evaluation frameworks for non-deterministic AI systems
• Develop testing approaches for response quality, accuracy, consistency, hallucinations, and edge cases
• Create and execute manual and automated test scenarios for AI agents
• Design benchmarks, datasets, and evaluation criteria to measure AI system performance
• Collaborate closely with AI Engineers to improve model behavior and product quality
• Analyze AI outputs and identify quality, reliability, and safety risks
• Perform API, backend, and data validation testing
• Establish QA best practices for AI-powered applications
• Support production releases, monitoring, and troubleshooting activities

REQUIREMENTS: 

• 5+ years of professional QA experience, including experience defining testing strategies and leading quality initiatives
• Proven experience testing AI/LLM-powered applications and non-deterministic systems required
• Strong understanding of AI agents, LLMs, prompt engineering, and AI evaluation methodologies
• Strong Python skills required
• Experience designing automated testing solutions and evaluation frameworks
• Experience validating AI-generated responses, detecting hallucinations, and evaluating model performance
• Strong API testing and backend validation experience
• High SQL knowledge
• Experience working with complex distributed applications and multi-platform architectures
• Experience with Azure DevOps or similar test management tools
• C# knowledge is considered a strong advantage
• English — Upper Intermediate or higher, both written and verbal