Navigating the AI Landscape
With over 5,000 AI tools available, choosing the right one can be overwhelming. We suggest a pragmatic approach: focus on the few large language models (LLMs) that power most tools, such as Open AI, Anthropic, or Llama, etc. We recommend using one of the major, off-the-shelf, turn-key options for daily use, considering factors like existing software ecosystems, cost, core functions, and unique value propositions.
AI Throughout the Evaluation Life Cycle
At the session we outlined numerous ways AI can support evaluators throughout the entire evaluation process:
- Design Phase: AI can assist in learning about topics, summarizing RFPs, outlining proposals, drafting needs assessments, supporting logic model development, performing literature reviews, drafting evaluation questions, and preparing project materials.
- Implementation Phase: Use cases include developing data collection toolkits, creating scripts for focus groups or interviews, planning outreach, suggesting quality checks, assessing project status, and scoring assessments.
- Use Phase: AI can partner to help with data cleaning and review, recoding data, providing language assistance (such as R, excel, or python), developing visualizations, suggesting key insights for different audiences, and tailoring communications.
Best Practices for Using AI in Evaluation
Some valuable tips we’ve found for effectively integrating AI into evaluation work include:
- Building prompts using the POP framework: Persona, Objective, Parameters.
- Letting the AI do the heavy lifting, but breaking complex tasks into manageable steps.
- Providing feedback and requesting revisions as needed.
- Tailoring the final output to your specific needs.
- Don’t get discouraged – there’s a learning curve, but AI is designed to help you along the way.
- Be mindful of privacy concerns, especially when handling sensitive data.
Looking Ahead
We encourage evaluators to dive in and start using AI tools purposefully in their daily work. We believe that while AI is already powerful, it will continue to improve with time. The key is to start the learning process now and stay involved in the evolving field.
We concluded our session by sharing a few studies highlighting the benefits for speed and quality in work for those leveraging AI tools and emphasized how it (AI chatbots) can complement human intelligence but can not replace it.
As the field of evaluation continues to evolve, it’s clear that AI will play an increasingly important role. By embracing these tools responsibly and intentionally, evaluators can enhance their practice, increase efficiency, and ultimately deliver more value to their clients and stakeholders.
Participant Questions
Following the 35 minute presentation, participants shared their questions and insights. A few key ones included:
Question: How and when do you talk to clients about using AI as a tool in your daily work?
Response: We shared that we do disclose use of AI, but clients don’t normally ask which specific tasks are being assisted by AI tools. We do ensure data integrity and privacy is being maintained as described in any contracts and we focus on the quality of deliverables that clients expect from us. Several participants mentioned how different tasks, settings, and contracts demand unique considerations for disclosing use. As an example, we wouldn’t likely disclose re-wording of a sentence, but qualitative coding assistance may warrant disclosure.
Question: How do you balance the time saving AI can provide with the time to review the output for quality and accuracy?
Response: We feel that it depends on the task. For example, drafting a rubric vs completing a needs assessment draft are very different. The rubric can be developed by AI very quickly and saves considerable time, and the human review is quick, a clear time savings. Drafting of a needs assessment is a quick task for AI as well, but given the complexity, amount of data being pulled, and nuance, it will require significantly more time to review. The net time savings may still be there, but probably not as clearly with the previous task. Assessing your use cases for AI tools is important, but, as you are learning and navigating where it fits well, the presenters encouraged experimentation.
We closed the session with a reminder that AI tools can be beneficial even for evaluators without technical expertise and that the media hype shouldn’t distract us from finding use cases that support our work.