Contents
Every question you asked at our TestGuild webinar, answered in full.
We ran a webinar with the TestGuild community: Powering Up QA Teams in the Age of AI – Testing With an Agentic Validation Layer, presented by our CEO Daniel Mauno Pettersson. We expected questions. We did not expect the chat to fill faster than we could type.
That's a good sign. It means QA engineers are past the "is this hype?" stage and into the "would this survive contact with my actual application?" stage. Those are better questions, and they deserve better answers than what fits in a webinar chat box.
So this post is the follow-up. Every question from the session, answered properly, grouped by theme. If you registered, you've already received the recording; if a colleague sent you this link, the questions will still make sense without it, because most of them are the questions every QA team is asking right now.
The webinar, in short
Development velocity has outrun verification. Coding agents mean more pull requests, more releases, more non-developers shipping changes – and every change still needs someone to confirm it works for a real user. That work didn't get automated; it landed on QA teams already busy keeping brittle scripted tests alive.
Daniel's alternative is an agentic validation layer: QA agents that test by goal ("complete a checkout") instead of by script, re-deriving the journey on every run – so there's no selector to break, and nothing to "heal". The agents never see your code; they use the product like a user and judge outcomes, not implementation. The practical takeaway was a split: delegate the execution – regression runs, role permutations, long-tail coverage – and keep the judgment: strategy, risk, release calls. Teams like Upsales, which replaced 320 hours of manual regression a month, show what that looks like in practice.
What the polls told us
We polled the room before and during the session. Two results frame everything that follows.
Most teams in the room run scripted automation or a mix of approaches for regression – Playwright, Cypress, Selenium territory. And while nearly two thirds are already experimenting with AI somewhere in their testing, not a single team said they run autonomous QA agents in production.
Live poll
How does your team handle regression today?
- Scripted automation (Playwright, Cypress, Selenium)42%
- A mix of these35%
- Mostly manual16%
- AI-assisted scripts – generation or "self-healing"6%
- Autonomous QA agents0%
Source: live audience poll, QA.tech webinar with TestGuild, August 2026
That's the room the questions came from: teams past curiosity, not yet past the practical blockers. So the questions weren't about whether agentic testing is interesting – they were about what breaks between the demo and the pipeline. Determinism. Environments. Login flows. Test data. Audit trails.
Nearly two thirds of the room is already experimenting with AI somewhere in testing – but nobody has autonomous agents running in production yet. That gap is exactly where the questions live:
Live poll
Is your team using AI anywhere in testing today?
- Experimenting63%
- AI-generated or "self-healing" scripts31%
- Not yet6%
- Autonomous QA agents in production0%
Source: live audience poll, QA.tech webinar with TestGuild, August 2026
Questions by category
Everything below is a real attendee question, lightly edited. We grouped them into eight categories, roughly in the order a team hits them when evaluating agentic testing: trust and reproducibility first, then CI/CD and environments, logins and access, test data, regulated processes and reporting, scope and limits, integrations, and the bigger business questions last. Skim for yours, or read straight through.
Trust and reproducibility
"If the agent is non-deterministic, how do I hand a reproducible bug to my developers?"
This was the most upvoted question of the session, and it's the right one to start with, because it separates two things that usually get blurred: how the agent finds a problem and how the problem is recorded.
The path to a failure is non-deterministic. The record of it is not. Every run captures the agent's step-by-step actions, a screenshot at each step, and the errors or unexpected states it hit. When a test fails, you're not told "checkout broke" – you're shown the exact step, the UI state at that moment, and the evidence around it. A developer can trace what happened without re-running anything. If you do want to confirm whether something is a real, repeatable issue or a one-off, you rerun the test, and dependent tests resume from the same prior state for consistency. Confirmed bugs push to Linear or Jira with the whole evidence trail attached.
There's a subtlety worth being honest about: the steps of a test are stored and stay the same between runs, so the test itself is stable – it's the execution of each step that has flexibility in it. That flexibility is the point. It's what lets the agent catch more variations of a user journey than a script that walks one hardcoded path ever would.
"How do you prevent the agent from passing a test it shouldn't pass, or hallucinating a result?"
Two mechanisms. First, execution and assessment are separated: the agent that performs the test is deliberately a bit flexible, and a second assessment step afterwards reviews whether what was tested actually matches the stated goal and expected outcome. The examiner checks the examinee. Second, everything the agent claims is backed by artifacts – screenshots, logs, the action trace – so a wrong verdict is visible rather than buried. False-positive rates are exactly the thing we tell teams to measure during a proof of concept, and it's a fair evaluation criterion for any tool in this category.
"If the agent finds its way around obstacles – pop-ups, error messages, workarounds – does it document them?"
Yes, and this is one of the quiet advantages of an agent over a script. A script that survives an unexpected modal does so silently. The agent records what it encountered and how it got past it, as part of the run. Beyond the pass/fail verdict, runs also surface issues the agent noticed in passing: JavaScript console errors, accessibility violations, visual oddities. A human tester notices things on the way to the goal; that behavior is worth keeping.
One of our pilot customers put it memorably when an agent, faced with a broken UI element, found another way to complete the goal: the agent was "almost too smart for its own good." Which is why strictness is configurable per test – some journeys should adapt, and some should fail loudly the moment anything drifts. You decide which is which, per test, not the tool.
Maintenance is the tax that makes all of this worth solving: two thirds of the room spend at least a tenth of their testing time keeping existing tests alive, and a quarter spend a third of it or more – time that goes to scripts, not to coverage.
Live poll
How much of your testing time goes to maintaining existing tests?
- 10–30%42%
- 30–50%25%
- Less than 10%17%
- Hard to say – we don't track it17%
Source: live audience poll, QA.tech webinar with TestGuild, August 2026
CI/CD, environments and infrastructure
"How does a goal-based approach integrate with a CI/CD pipeline?"
The same way scripted suites do, minus the maintenance. Runs trigger from your pipeline – GitHub Actions, GitLab, Bitrise, or any CI via the REST API and webhooks – on a pull request, on deploy, on a schedule, or manually. Your pipeline kicks off a run in the cloud, gets results back, and can gate a merge on them.
Repeatability, the second half of that question, deserves the honest answer: a goal-based run is not bit-for-bit deterministic, and that's by design (see the reproducibility answer above – stored steps, flexible execution, deterministic evidence). In a pipeline this matters less than people expect, because the failure signal you act on is the assessed outcome plus evidence, not the exact pixel path taken to get there.
There's also a statistical argument Daniel made on the webinar that bears repeating. If you run 100 tests on every change and each has even a 1% chance of environmental flake, you'll see a failing test on essentially every run – regardless of tooling. The fix isn't a more rigid script; it's running the tests that matter for the change. Because the agent maintains a model of your application, it can scope testing to what a diff actually touches instead of brute-forcing the full suite every time.
"Can we run this against ephemeral environments – Docker, preview deploys?"
Yes – and it's the setup we'd wish for every customer. An isolated preview environment per PR, with data as close to production-like as possible, is the ideal target for agentic testing: the agent tests the change in isolation before merge, and nothing it does can collide with anyone else's work. Applications and environments support dynamic preview URLs, and platform gates like Vercel's deployment protection are handled with bypass rules rather than workarounds. If your environment only exists on a laptop or inside a private network, a tunnel exposes it to the agents for the duration of a run.
If you don't have ephemeral environments yet, you're not blocked – plenty of teams run against a shared staging – but it's worth saying that teams who invest in isolated preview environments get disproportionately more from every kind of automated verification, agentic or not.
"What infrastructure do we need to host this? On-prem server, or cloud?"
No servers on your side. QA.tech is a cloud service: the agents and browsers run on our infrastructure and connect to your application from there. For applications that aren't publicly reachable, you open a controlled path instead of installing anything – IP allow-listing against our published runner IPs, or an SSH tunnel into private networks. Your pipeline talks to the platform over the API.
Logins, security gates and access
"How does the agent get past SSO, 2FA, and the security rules on our dev and staging environments?"
It doesn't get past them – it goes through them, like a user. Credentials live in managed configurations, and the agent drives the real login flow. Two-factor flows work because the platform can provision managed email inboxes: the agent receives the verification email, parses the code or magic link, and continues. Signup flows get a unique address per run, so each execution starts clean.
What about the gates that exist specifically to keep bots out? Those are handled by explicit, controlled access rather than evasion: bypass headers for deployment protection, Cloudflare WAF and Turnstile allowances, IP allow-listing, or a tunnel. Deliberately, there is no capability to defeat a security control you haven't chosen to open. One honest caveat from our own sales calls: consumer Google sign-in is extremely good at blocking automation, ours included – test accounts with direct credentials or Microsoft/enterprise SSO are the reliable route.
"And the data setup and teardown we currently do via APIs and scripts?"
Keep it. Your existing seeding and teardown endpoints slot in as API call steps inside a test: call the endpoint that creates the state, run the UI journey, call the endpoint that resets it. Nothing about goal-based testing requires abandoning the API plumbing you've already built – it usually makes that plumbing more valuable, because the agent can use it mid-journey.
Test data
"Is there an agent for preparing test data for each execution?"
Not a separate agent – a configuration system that covers the same need. Configs hold fixed values, uploaded files, and one-time-use items: single-use values like invite codes or IDs that are consumed once per run, so every execution gets fresh data. System-managed inboxes generate a unique email per run for signup and verification flows. For anything that needs to exist in the backend before the journey starts, an API step seeds it. And dependent tests can chain, so one test creates the data the next one uses – a multi-user scenario where an admin invites a user and the user accepts is one chain, not two scripts and a prayer.
There's one more layer worth knowing about: the agent can invent plausible filler on its own. If a form needs a name or an address and you haven't pinned a value, it generates one, the way a human tester types "Test Testsson" without asking permission. The rule of thumb: use configurations when the data must be exact, reusable, or valid against your backend; let the agent make it up when any reasonable value will do.
"Same path, different data, different expected outcome – how does the agent handle data-driven flows?"
By making the variability live in the data and the stated outcome, not in duplicated scripts. Each test is a goal plus an expected result in plain language; a different input with a different expected result is a different test, sharing the same journey. The agent reads the live UI at runtime, so whatever the page renders back for a given input – a validation message, a computed total, a different next screen – is what gets assessed. And because assertions are on meaning ("the total reflects the discount") rather than on a static value, dynamic attributes don't break anything. A human tester doesn't need six scripts to check six discount codes; neither does the agent.
"On dynamic testing – is this similar to producing fake data? Do you have tools to generate scrubbed data?"
Two different needs, worth separating – and "dynamic testing" is neither of them, strictly speaking. Dynamic here means the agent reads the live UI at runtime and reacts to what the page actually renders. It's about responding to real application state, not fabricating data.
The data questions split cleanly from there. Per-run synthetic input – emails, one-time values, files – is generated and managed by the platform, as above. Scrubbing or anonymizing your production database into a staging dataset is not something QA.tech does; that stays in your data pipeline, where it belongs, because your team knows what's sensitive.
And the recommendation Daniel repeated on the webinar, and repeats to every customer: don't point agents (or any automated testing) at real production data at all. Give them an environment with representative seeded data instead.
Regulated teams, V&V and reporting
"We have a strict Verification & Validation process. Specified steps, recorded expected and actual results. How does this fit?"
Better than people expect, because goal-based doesn't mean vague. You are not forced into open-ended goals: a test case can specify exact, ordered steps with an expected outcome, and the agent follows the defined procedure. What the agent adds is the evidence discipline that V&V actually demands: every run records expected versus actual per test, with step-by-step actions and screenshots, tied to a specific environment and build, rerunnable, and exportable. Test cases carry revision history, so the procedure you validated is versioned, and runs push to your tracker for the audit trail.
The honest framing: you keep the rigor – defined steps, recorded results, traceability – and delegate the execution and the paperwork. In regulated industries the screenshots-and-steps documentation is half the job, and it comes free with every run.
"How do you report results – by goal?"
Yes, outcome-first. Each test is assessed as pass or fail against its stated goal, backed by the step trace, screenshots, and any errors or unexpected states. Runs group the tests you executed together, so you get the overall picture plus the per-test breakdown, and failures pinpoint the step and UI state that broke. Real bugs go to Linear or Jira with evidence attached.
"Where do the test results reside?"
In the QA.tech cloud platform – each run and its evidence (steps, screenshots, expected versus actual, errors) is stored there and viewed in the QA.tech UI. From there the data flows out on your terms: pull results via the API to build your own reports, or push failures into Linear or Jira. The system of record is QA.tech; the access points are the UI, the API, and your tracker.
"Can I export a report as a flat file – high-level results plus detailed steps and expected results?"
Yes, via the REST API: run-level and test-level results, including the step-by-step trace, come back as structured data you can format into CSV, JSON, or whatever your reporting stack wants. Test definitions themselves export too, including steps, which gives regulated teams a readable, versionable record of what each test does. What doesn't exist today is a one-click branded PDF – the data is all there, the formatting is yours.
"What governance and guardrail challenges are enterprise teams actually running into?"
A question from the chat panel that deserves more air than it got. Five themes recur across enterprise evaluations.
Access and network control comes first: keeping test traffic inside approved boundaries with IP allow-listing, tunnels and deployment-protection rules, rather than opening anything wide. Then data handling – synthetic inputs and dedicated test credentials instead of real PII and production accounts. Test data does pass through the underlying AI models during execution, which is exactly why dedicated test accounts are a standing rule, not a suggestion.
The remaining three: scope guardrails (constraining the agent with defined steps, project rules and environment boundaries, so flexible execution stays inside the lines you drew), auditability (traceable evidence on every run, pushed to the tracker for a defensible record), and credential management (logins and secrets living in managed configurations, never hardcoded into test text).
None of these are exotic. They're the same governance questions every automation tool faces – the difference is that an agent's flexibility makes teams ask them earlier, which is healthy.
Scope: what it tests, and what it doesn't
We spent real airtime in the webinar on what agents can't test, because that honesty is what makes the delegate-versus-keep split workable. Here's where each boundary question landed.
"Can this be applied to automated API testing?"
As part of end-to-end journeys, yes; as a standalone API testing tool, no. An API call step can fire an HTTP request during a test – setup, teardown, or checking a backend response alongside the UI flow – and you can assert on the network requests the browser makes during a journey: URL, method, status code, payload. So "verify the right API call fired with the right data when the user clicked submit" is a natural test. "Fuzz our public API against its OpenAPI spec" is a different product category, and we'd rather say so than stretch.
"How does it help desktop applications where firmware and software are integrated?"
The agents test what runs in a browser or on a mobile device. Thick desktop clients – native Windows applications included, since that came up separately in the chat – and embedded firmware sit outside that reach, and hardware was named explicitly in the webinar as something that stays with your team. Where it helps an integrated product is the web-facing layer – the dashboard, admin console, or config portal that monitors the device – and API steps can trigger or confirm device-side outcomes mid-journey if they're reachable over an API. One of our customers, Pricer, runs electronic shelf labels in 28,000 stores; agents cover their nine web and mobile applications while the QA team focuses on strategy and the actual hardware.
"Are the mobile devices real devices or emulators?"
Cloud-hosted emulators and simulators today – iOS Simulators and Android Emulators, with the model, OS version and orientation configurable per test. Physical-device testing is on the roadmap but not available yet, and we'd rather tell you that plainly than let a demo imply otherwise. For mobile web, device presets simulate viewports, locales and user agents in the browser; for native apps, builds upload via API or CI and results track the build version they ran against.
"Are you covering performance QA?"
Not as a primary function, and we won't pretend otherwise. Load testing wants predictable, high-volume, cheap requests; an agent reasoning through a UI is the opposite of that profile. Parallel agent runs will incidentally put some pressure on your system, and slow pages do surface in results, but if load testing is the problem you're solving this week, use a load testing tool – we wrote a comparison of ten of them. The validation layer's job is functional correctness from the user's side.
"QAs increasingly need to test RAG models, chatbots, LLM features. Can we do that with this?"
Partially, and it's an area where goal-based testing has a structural advantage. You can't script an assertion for generative output, because the output is different every time – but you can state what a valid output must satisfy, and have an agent judge the result against that expectation: the answer cites a source, the generated note follows the template, the chatbot's reply actually addresses the question. That's meaning-level assessment, which is what the assessment step already does for every test. What this doesn't replace is model evaluation proper – benchmarks, red-teaming, regression suites over prompts. Testing the feature around the model, in the product, as a user experiences it: yes, that's squarely in scope.
"Running all these tests across viewports sounds expensive – monetarily and environmentally. How do you mitigate the AI overhead?"
By not brute-forcing. Tests run when you trigger them, not continuously. Viewports and devices are opt-in per test plan, so you multiply across the combinations that matter rather than every combination by default. The agent reuses what it knows – the knowledge graph and agent cache mean it isn't re-exploring your product from scratch on every run, and dependent tests resume from saved state instead of repeating setup. And the diff-scoped approach from the CI/CD answer applies here too: the cheapest and greenest test run is the one you didn't need.
"How do you handle model allocation to keep costs down without compromising the outcome?"
A sharper version of the cost question, also from the chat panel. Work is routed to the right-sized model per step rather than one heavy model doing everything: lighter models handle simple perception and navigation, and the more capable ones are reserved for hard reasoning and the pass/fail judgment. That last part matters – the assessment is the one place you never want to economize, and it's also the one step that is never served from cache; it always looks at what's actually on screen. Combined with reused app memory and dependent tests resuming from saved state, the principle is simple: match model power to task difficulty, and spend the intelligence where the verdict gets made.
Plugging into your stack
"Any integrations with test case management systems like Xray? Does it interact with MCP servers?"
Both, and the MCP answer is worth slowing down for. On test management: TestRail and Xray connect under project settings, and you can pull existing cases in and turn them into agent-executed tests – a migration path rather than a rewrite. One current limit, stated plainly: results don't push back to TestRail or Xray yet; the link to the original case is tracked, reporting back is the part still to come.
On MCP, it runs in both directions. QA.tech has an MCP server, so Claude Code, Cursor or Codex can list tests, start runs, and read results – your coding agent can ask the validation layer whether its change actually works. And custom MCP integrations let you register your own MCP servers so the QA.tech chat agent can call your internal tools, with per-tool permissions.
"Can it read stories, tasks, enhancements for context?"
Yes, three ways: it connects to Jira or Linear and reads issues to ground tests in what's being built; during a PR review it reads the diff, so tests target the actual change; and you can add specs, requirements or docs to the project knowledge base, which the agents use when creating and running tests. One of our customers, 4C Strategies, fed their dense user manuals in and generated tests for highly customized workflows – documentation they already had became coverage.
"Do you offer knowledge layers or plugins for the code-producing engineers, to bake context in sooner?"
This is the same MCP answer wearing different clothes, and it's a sharp instinct. The QA.tech MCP server is that layer: connected to a coding assistant, the engineer's agent can check existing coverage, run the relevant tests against a change, and read failures before a PR ever goes up. The CLI goes a step further and can generate Claude Code subagent and skill files, so the validation context lives in the repo where the code gets written. Verification moving earlier, without QA losing the independent layer – that's the shape we think this whole category settles into.
"How do you synthesize truth going into the knowledge graph, and keep it fresh?"
The knowledge graph is built from observation: crawling sessions explore your application and record pages, actions and features, and every test run adds what the agent saw. Freshness comes from the same two loops – connect your repository and merged changes update the graph as they ship; otherwise recurring crawls pick up drift. Staleness, when it happens, is self-correcting in a way scripts aren't: an agent that encounters a UI that doesn't match its memory observes the new reality, completes the goal against what's actually there, and the graph updates. A script in the same situation just fails.
Importantly, the graph is memory, not gospel: the assessment of each run judges what's on screen now, never what the graph remembers being there.
The bigger questions
"How does pricing work?"
Usage-based: what counts is test executions, not seats, with plans by team size and a fair-usage policy so a busy month doesn't produce a surprise bill. We don't publish flat prices because the honest number depends on how much you run – which you control, per the cost answer above. Every engagement starts with a free proof of concept on two or three of your most critical journeys, which also means you see the cost profile on your own application before committing to anything.
"Can we use self-hosted models, or are we required to use the public models?"
Today the agents run on frontier foundation models operated through our cloud – there's no bring-your-own-model or self-hosted option. We'd rather be plain about what that implies: test traffic and screenshots are processed by those model providers, which is one more reason we insist on testing against non-production environments and data. QA.tech holds SOC 2 Type 2 for Security, Availability and Confidentiality; if your compliance requirements are stricter than that (data residency, on-prem inference), talk to us about where the roadmap is rather than assuming – this area moves quickly.
"What is the future of manual testing?"
Daniel's answer on the webinar is the one we'd stand behind in writing: manual testing was never the problem. In many ways manual testing is the best testing – a human sees what users see, notices what's off, adapts to anything. Its only flaws were that it's slow and expensive, so we rationed it, and then built scripts to imitate a fraction of it. If you'd always had unlimited testers available around the clock, you'd never have written a selector in your life.
Agentic testing is closer to manual testing at automated pace than it is to automation in the traditional sense. Which means the manual tester's skills – decomposing a product into what needs checking, writing a test case in plain language, judging whether a result actually matters – are precisely the skills that direct QA agents. The repetitive execution goes away. The craft doesn't; it gets a larger lever. The QA teams we work with end up owning test strategy, risk analysis and the hard cases, while agents grind through the regression runs – including the long-tail coverage that topped the room's list of what it struggles to test:
Live poll
Which of these does your team struggle to test today?
- Long-tail features nobody has time for47%
- Multi-role permutations13%
- Login, 2FA and email flows13%
- Cross-device journeys13%
- Third-party systems like Salesforce or HubSpot13%
Source: live audience poll, QA.tech webinar with TestGuild, August 2026
Nobody in that poll answered "less QA," and that matches what we see: the time doesn't disappear, it moves up the stack.
Keep going
Three places to go from here, depending on what you're solving:
If you're making the case internally, the Past the Bottleneck ebook is the argument in a shareable form, and the first webinar's recording covers the verification-gap story from the engineering side – worth sending to your engineering lead, because this conversation goes better with both sides in the room.
If you're evaluating, the case studies show the shapes this takes on real teams – including Crystal Intelligence, whose canvas-based UI had no selectors to script against at all, which made it "untestable" right up until the tests stopped needing selectors.
And if you'd rather just see agents run against your own application – the thing 86% of the room said "not yet" to, entirely reasonably, on the webinar – that's what a free POC is for. We start with two or three of your most critical journeys and you judge the results on your own product, not ours.
Thanks again to Joe and the TestGuild community for the sharpest question set we've had in a webinar chat. Keep them coming.