AI - Software Development Engineer in Test (AI - SDET)

This role goes beyond traditional QA; the ideal candidate should be capable of validating, evaluating, and improving AI-powered systems, agents, copilots, and LLM-driven workflows while building scalable automation frameworks and intelligent quality solutions. You will work closely with Engineering, AI/ML, Platform, Product, and DevOps teams to ensure reliability, safety, observability, scalability, and quality of modern AI-native applications and Agentic systems.Who Are We.

Newfold Digital is a leading web technology company serving nearly seven million customers globally. Established in 2021 through the combination of leading web services providers Endurance Web Presence and Web.com Group, our portfolio of brands includes: Bluehost, BigRock, ResellerClub, CrazyDomains, HostGator, Network Solutions, Register.com, Web.com and many others.

We help customers of all sizes build a digital presence that delivers results. With our extensive product offerings and personalized support, we take pride in collaborating with our customers to serve their online presence needs.

Key Responsibilities

AI/LLM Quality Engineering

Design, validate, and optimise testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines, and autonomous AI agents.

Develop AI evaluation (Evals) frameworks for:Hallucination detectionGroundednessToxicity & bias validationResponse qualityContext relevancePrompt regression testingDeterminism & consistencySafety and guardrail validation

Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics.

Create synthetic datasets and benchmark datasets for AI testing scenarios.

Test multi-agent orchestration workflows and agent memory/context retention.

Validate MCP (Model Context Protocol) integrations and tool interoperability.

Work on Context Engineering, Prompt Engineering, and Prompt Optimisation strategies.

Evaluate latency, token usage, model cost optimisation, throughput, and scalability.

Automation Engineering

Develop and maintain scalable UI/API automation frameworks using:Python (mandatory)Robot Framework (preferred)Playwright/Selenium/Cypress

Automate testing for

REST APIsGraphQL APIsEvent-driven systemsStreaming systemsWebhooksMicroservicesMFE (Micro Frontend) architectures

Build reusable automation libraries, utilities, and AI-assisted testing workflows.

Integrate automation into CI/CD pipelines and quality gates.

AI Engineering & Platform Validation

Validate RAG architectures including:Chunking strategiesEmbedding qualityRetrieval accuracyRe-rankingVector search relevance

Work with vector databases such as:PineconeWeaviateChromaDBQdrantFAISS

Build or validate AI agents leveraging:LangChainLangGraphCrewAIAutoGenSemantic KernelOpenAI SDKs

Validate AI workflows involving:Tool callingFunction callingAgent memoryAgent planning/executionMulti-modal AI systemsStructured outputs

DevOps, Observability & Reliability

Build and maintain quality workflows integrated with:GitHub ActionsJenkinsGitLab CI/CDAzure DevOps

Work with Docker and Kubernetes environments.

Perform cloud-based testing and deployments across AWS/Azure/OCI/GCP.

Validate distributed systems reliability and resiliency.

Monitor AI system behaviour using observability platforms/logging/tracing tools.

Contribute to shift-left and quality engineering best practices.

Required Skills

Must Have

Strong programming experience in Python

Strong understanding of Software Testing & Quality Engineering principles

Hands-on experience with

AI/LLM testingPrompt EngineeringContext EngineeringAI EvalsRAG systemsAI Agents

Experience with:CI/CD pipelinesDocker & KubernetesCloud platforms (AWS/Azure/OCI/GCP)API testing & automationMicroservices testingUI automation

Experience with SQL and data validation

Strong debugging, analytical, and troubleshooting skills

Experience working in Agile/Scrum environments

Preferred / Good to Have

Experience with Robot Framework

Experience with Playwright

Knowledge of

LangChainLangGraphLlamaIndexCrewAISemantic KernelMCP (Model Context Protocol)

Hands-on experience with

CursorGitHub CopilotClaudeGeminiOpenAI APIs

Experience with vector databases and embeddings

Exposure to MLOps workflows and ML lifecycle testing

Knowledge of performance testing and security testing

Experience validating AI governance/compliance requirements

Experience with contract testing, chaos testing, or resilience engineering

Familiarity with accessibility testing and cross-browser/device testing

Exposure to synthetic monitoring and production observability

What We’re Looking For

Engineers who actively experiment with AI tools and build side projects, prototypes, PoCs, or open-source contributions.

Strong problem solvers who can combine traditional QE practices with modern AI-native engineering approaches.

Passion for innovation, developer productivity, automation acceleration, and intelligent testing.

Ability to rapidly learn emerging AI frameworks, tools, and architectures.

Nice Portfolio Signals

Candidates with one or more of the following will stand out

GitHub repositories showcasing AI/automation projects

AI agents or copilots built personally or professionally

Contributions to open-source automation or AI ecosystems

Blogs, talks, workshops, or technical content on AI/testing

Experience integrating AI into SDLC or QE workflows

Experience with autonomous testing systems or self-healing automation

Education

Bachelor’s/Master’s degree in Computer Science, Engineering, AI/ML, or related field preferred.

Equivalent hands-on experience and demonstrable projects are highly valued.

Bonus Skills (High Impact)

AI Red Teaming

LLM Security Testing (Prompt Injection, Jailbreaks, Data Leakage)

Agent Reliability & Safety Testing

AI Cost Optimisation

Distributed Systems Testing

Test Data Engineering

Feature Flag Testing

AI Observability & Telemetry

Reinforcement Learning feedback