Building a generative AI chat prototype has become straightforward. With a large language model (LLM), a vector database, a frontend interface, and an AI coding assistant, a developer can assemble a working demo in days. However, deploying such applications at institutions like the World Bank Group—where data accuracy, policy context, and citation verifiability are essential—requires far more than a working proof of concept.Â
How do you guarantee an LLM assistant remains reliable across hundreds of user requests?
How do you ensure it does not hallucinate indicators, skip citations, or lose context over multi-turn dialogues?Â
How do you measure response quality for questions users have not yet asked?
To address these challenges, we built an automated, persona-driven evaluation framework using Confident AI’s open-source DeepEval library as our scoring backbone. While DeepEval provides core metric primitives, our framework introduces custom innovations: dynamic persona composition, intelligent pre-filtering, and a replay mechanism for rapid rubric iteration. Originally created for the Data360 Chat Assistant, this framework is now packaged as a reusable blueprint for building trustworthy AI applications across the institution (GitHub Repository).
The unpredictable path of multi-turn chat
Traditional software testing is linear: input A yields output B. For conversational AI, this approach fails because user interactions are dynamic, branching dialogues. A user might greet the assistant, ask for GDP trends, request data formatted in a table, compare neighboring countries, and finally request a chart.
Evaluating isolated responses misses compounding multi-turn failures:
- Context Drift: The model forgets constraints (such as country or year range) specified in earlier turns.
- Orchestration Failures: The router correctly classifies intent on Turn 1 but strays onto an incorrect path on Turn 3.
- State and Transport Breaks: Persistent Server-Sent Events (SSE) data streams can cut off mid-transit, or database session state can fail to reload, forcing the model to restart.
To catch these failures, our framework executes tests directly against the live chatbot API—requesting guest auth tokens, persisting PostgreSQL session IDs, and parsing streamed responses to simulate real user sessions.
