198 / How To Build AI Agents We Can Trust, with Alexis Zheng

Alexis Zheng is Chief Product Officer at G2 and brings 20 years of experience in building AI systems. In this episode of Product Momentum – recorded in person at Mind The Product – Chicago – Alexis explains how product teams can build trustworthy AI agents without sacrificing their potential. Her message is clear: if we expect our agents to safely meet user needs and business objectives, we need to make deliberate decisions around defining success, establishing boundaries, and monitoring results.

These responsibilities extend beyond engineering. Product managers and UX designers need to join their Developer counterparts to evaluate performance, draw a balance between autonomy and control, and deeply embed human judgment in the overarching product strategy.

Here’s more of what we learned:

Trust but Verify: Measuring AI Performance with Continuous Evaluation

Trustworthy AI agents require more than promising outputs, Alexis says. Product teams need measurable criteria against which we can determine whether an agent achieves its intended objectives. Alexis itemizes three essential capabilities: evaluation metrics, guardrails, and observability.

She explains: metrics establish what success looks like; guardrails prevent inappropriate actions; and observability helps teams monitor performance after deployment. “You have to design metrics so you know if the system is doing the right thing,” she continues.

“Building the Harness”: Designing Agents for Flexibility and Control

Evaluating AI performance is only one part of the challenge, Alexis adds; controlling agent behavior is another. Unlike traditional software, AI agents can produce different outcomes from similar inputs. Restricting them too tightly limits their capabilities, while allowing too much freedom often introduces unintended consequences. Alexis describes the infrastructure and practices needed to manage this tension as building the harness.

“You have to find a way to let the model or let the agent be free, but at the same time also manage the unintended outcomes,” she says. “For those of us on product teams, this means making deliberate design decisions about where agents can operate independently and where constraints become necessary.”

Apply Human Judgment to Enhance AI’s ‘Average Intelligence’

Even well-designed, carefully evaluated AI agents can’t replace the human judgment that shapes successful products. Teams using similar AI models often arrive at alarmingly similar solutions – what Alexis refers to as average intelligence – so product managers need to inject their own creativity, customer understanding, and strategic perspective to the development process. Rather than blindly allowing AI to dictate product decisions, PdMs should use it to advance a distinct vision.

“To understand how to leverage AI, we need to put ourselves in a position that we are doing the steering, that we are using AI to accomplish our vision.”

• [06:30] How do you know that your AI agents are doing the right thing? Or that they’re delivering the desired outcome? That is what the Eval is trying to do, is to really understand what the agents are doing in production and to make sure they’re achieving the outcome as humans who actually want them to achieve; otherwise you may not end up with what you asked them to do.
• [12:30] Verifying the verifier. And that is really hard. That’s why today you have expert in the loop, which isn’t very common if you talk about it like RLHF (reinforcement learning from human feedback), to have a human expert to look at the result.
• [18:12] AI stands for artificial intelligence, but it also stands for average intelligence. The products we build today seem to look like our competitors’ products. We all seem to be regressing to the mean. The human element is what distinguishes us from all the rest.
• [20:46] Be intentional in how you deploy AI tools; don’t simply defer to the tool’s solutions. Agents are doing the work, but we humans still need to do the steering; otherwise, you don’t know what the agents are going to end up with.
• [22:19] Deterministic roadmaps vs. Probabilistic roadmaps. As we move through more probabilistic roadmaps, we begin to narrow the field of possible selections – this is known as converging. The key to AI is this: how fast we are converging?

The post 198 / How To Build AI Agents We Can Trust, with Alexis Zheng appeared first on ITX Corp..