Key Findings From The AVA Study
Here are the key findings from the five month AVA study : Participation & coverage dynamics During May 12 to Aug 31, 2025 , 2,679 unique individuals signed up for AVA.[:cite[1]{ln=1}] Abstention (reasoned “no respon...
Here are the key findings from the five month AVA study : Participation & coverage dynamics During May 12 to Aug 31, 2025 , 2,679 unique individuals signed up for AVA.[:cite[1]{ln=1}] Abstention (reasoned “no response”) was high early when the knowledge base had 50 reports (ranged 40–70% ), then dropped sharply after the corpus expanded to over 4,000 World Bank reports on July 9, 2025 , and stayed below 10% for the rest of the study.[:cite[1]{ln=6}], [:cite[1]{ln=7}] Across the full deployment, AVA generated responses with citations .[:cite[1]{ln=8}] User engagement and session behavior Of users, 73.4% engaged in only one session , while 26.6% returned for multiple sessions .[:cite[2]{ln=2}] Users who used AVA in multiple sessions reported more time saved than those who used it in a single session (difference in differences: b = 1.84 , and upper bound b = 3.27 , both with p < .10 ).[:cite[3]{ln=3}] Task types users asked AVA for Users’ questions often centered on policy diagnostics, design, and evaluation , and the study reports that most queries ask AVA to complete diagnostic tasks .[:cite[4]{ln=3}], [:cite[4]{ln=4}] The study also found that two policy themes — Human Capital and Fiscal Policy/Private Sector —dominated, making up two thirds of queries .[:cite[5]{ln=2}] AVA responses were typically grounded in multi document evidence , averaging five documents per query (range 1–23 ).[:cite[6]{ln=1}] Satisfaction and perceived usefulness Based on endline perception data from 118 AVA users, the tool was “ viewed very positively ,” with high endorsement for efficiency, accuracy, and overall value .[:cite[7]{ln=1}], [:cite[7]{ln=2}] From pop up feedback (n=100 users): 68% found content relevant, 65% found citations relevant, 72% were satisfied, and 72% would recommend the tool.[:cite[7]{ln=4}] Effect on efficiency / productivity (mixed impact results) In the intention to treat (ITT) analysis, there were no significant differences in self reported time saved between treatment and control (lower bound b = 0.56 , upper bound b = 0.78 ; both p .05 ).[:cite[8]{ln=3}] The study also reports no impact when respondents estimated output counts or whether AI helped them produce more outputs and/or save time for those outputs (e.g., outputs: b = 0.08 , p .05 ; time saved: b = 0.78 , p .05 ).[:cite[9]{ln=5}] Why AVA was trusted in practice (mechanism level qualitative insight) Citation verification was described as a major value driver, working through: Faster source finding by narrowing reading scope to relevant sections.[:cite[10]{ln=3}] Easier scanning via highlighted excerpts and page level context.[:cite[10]{ln=4}] Streamlined fact checking by clicking citations to confirm evidence grounding.[:cite[10]{ln=5}] Participants framed this as turning verification into a “lightweight, reliable shortcut,” improving trust and saving time (with an example of going from days to minutes ).[:cite[11]{ln=1}], [:cite[10]{ln=7}] If you want, I can also extract the main qualitative themes (e.g., how users positioned AVA in their workflows, and how “reasoned abstention” shaped trust calibration), but the list above covers the core “what the study found” points.