Principal Software Engineer

Principal Software Engineer

Location

Bangalore India

Department

AI Engineering / Quality Assurance

Experience Level

Mid to Senior

We are looking for a highly motivated Senior AI Test / Automation Engineer to design and scale automated validation frameworks for AI/ML models, LLM-based applications, and agentic systems. This role is critical to ensure that AI solutions meet enterprise standards for quality, reliability, safety, and compliance before and after production deployment.

Key Responsibilities

Build and maintain AI test automation frameworks for pre-qualification and continuous validation of models and agent workflows

Develop comprehensive test suites, including

Unit, integration, and end-to-end (E2E)

Functional, regression, performance, and safety testing

Validate AI system behavior, including

Non-deterministic LLM outputs

Hallucinations and edge cases

Multi-step agent decision-making

Design and manage evaluation systems

Golden datasets

Benchmarking pipelines (accuracy, latency, reliability)

Automate testing within CI/CD pipelines for model updates, prompt changes, and tool integrations

Implement observability and telemetry to enable traceability, monitoring, and audit readiness

Collaborate cross-functionally with ML, MLOps, Product, and Security teams to define quality gates and release criteria

Track and report quality KPIs, including test coverage, defect leakage, and system reliability

Drive root-cause analysis and continuous improvement across the AI testing lifecycle

Required Skills

Core Engineering

Strong programming skills in Python; familiarity with Bash, TypeScript, or Go

Experience with test automation frameworks such as PyTest, Playwright, Selenium, or Cypress

Proficiency in CI/CD tools (GitHub Actions, Jenkins, GitLab CI)

Experience with cloud platforms (AWS, Azure, GCP) and containers (Docker, Kubernetes)

AI / ML & Agentic Systems

Hands-on experience with LLM ecosystems (OpenAI, Anthropic, Bedrock)

Familiarity with

RAG architectures and vector databases (Pinecone, Weaviate)

Agent frameworks (LangChain, LlamaIndex, AutoGen)

AI Testing Techniques

Experience with non-deterministic testing approaches (statistical assertions, tolerance thresholds)

Knowledge of evaluation methods

LLM-as-a-judge

BLEU, ROUGE, semantic similarity scoring

Experience with prompt and agent regression testing

Understanding of AI safety testing, including adversarial testing, bias/fairness validation, and jailbreak detection

Tooling (Preferred)

AI testing & observability tools: LangSmith, TruLens, Arize, Weights & Biases

Evaluation tools

DeepEval, Ragas, PromptFoo, Giskard

Monitoring

Prometheus, Grafana, OpenTelemetry

Soft Skills

Strong analytical and problem-solving skills

Excellent communication and cross-functional collaboration

Data-driven mindset with focus on quality KPIs

Detail-oriented with a strong bias toward automation and scalability

Experience Requirements

7+ years in QA, SDET, or test automation engineering

Proven experience building and scaling automation frameworks

Hands-on experience with AI/ML systems or LLM-based applications

Experience testing RAG pipelines or agentic workflows

Owned end-to-end AI test strategy and architecture

Defined quality metrics and release gates

Delivered scalable validation pipelines for production AI systems

Supported audit and compliance readiness

Preferred

Experience in enterprise or regulated environments (SOC2, ISO 27001, etc.)

Exposure to

Shift-left testing practices

Production observability and monitoring

Chaos or resilience testing

Senior-Level Differentiators

Education

Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related field

Nice-to-have

ISTQB certification

Cloud/ML certifications (AWS, Azure, GCP)

AI testing certifications

What Success Looks Like

AI systems that are accurate, reliable, and safe

Fully automated test pipelines integrated into CI/CD

Measurable improvements in defect leakage and model quality

Strong observability and auditability across AI systems

Scalable validation frameworks supporting rapid AI innovation