
Automation and AI product testing
Job Summary
We are looking for an experienced AI Automation Tester with hands-on experience in testing AI-powered products and applications. The ideal candidate should have strong expertise in Playwright automation, along with practical experience in LLM evaluations (Evals), Golden Datasets (Goldens), and AI Observability.
Key Responsibilities
Design, develop and execute automated test scripts using Playwright.
Test AI-powered applications, LLM-based products and Generative AI solutions.
Work on Evals to assess AI response accuracy, relevance, quality and consistency.
Create, maintain and validate Golden Datasets (Goldens) for benchmarking AI outputs.
Implement and work with AI Observability to monitor model performance, response quality, latency and failures.
Perform functional, regression, API and end-to-end testing.
Identify, report and track defects to ensure product quality and reliability.
Never pay to get work. If a listing asks for a fee, it is a scam. The ten signs →