Overview
The Extension Foundation is seeking Integrated Pest Management (IPM) subject matter experts from the Cooperative Extension System, USDA, and related agricultural research organizations to assist with the evaluation of a benchmark question set used to assess an AI-based agricultural advisory system.
This work supports the development and evaluation of ExtensionBot, an AI-powered conversational assistant designed to provide research-based pest management guidance grounded in Extension publications and USDA resources.
The goal of this project is to ensure that the system accurately reflects real-world agricultural questions and decision contexts faced by farmers, crop advisors, and Extension educators.
Your expertise will help validate whether the questions used to test the system are realistic, relevant, and representative of common IPM challenges.
Project Background
ExtensionBot is a conversational AI system designed to help farmers and agricultural professionals access research-based pest management guidance drawn from Extension and USDA publications.
Unlike general-purpose AI tools, ExtensionBot uses a retrieval-based architecture, meaning its responses are grounded in trusted Extension and research content.
To ensure the system performs well in real-world agricultural scenarios, we are building a benchmark evaluation dataset consisting of approximately 100 IPM-related questions.
These questions are used to test how well different AI systems answer common pest management inquiries.
Before using this dataset for formal evaluation, we are asking IPM experts to review the questions themselves to confirm that they represent authentic, reasonable, and meaningful agricultural decision scenarios.
Role of the Reviewer
Subject matter experts will review a set of approximately 100 IPM-related questions and provide feedback on their quality and appropriateness.
Reviewers will not be evaluating AI-generated answers.
Instead, reviewers will assess whether the questions themselves represent realistic agricultural inquiries.
For each question, reviewers will be asked to consider:
- Is this a reasonable question that a farmer, crop consultant, or Extension client might ask?
- Does the question represent a valid IPM scenario or diagnostic situation?
- Is the wording clear and understandable for an agricultural audience?
- Does the question reflect a meaningful pest management decision or concern?
- Are there any questions that appear misleading, unrealistic, or poorly framed?
Reviewers may also flag questions that:
- need clarification
- should be revised
- should be removed from the evaluation dataset
Why This Work Matters
Artificial intelligence systems are increasingly being used to provide agricultural information and decision support.
However, evaluating these systems requires carefully designed test questions that reflect real-world agricultural practice.
By validating this question set, IPM experts help ensure that the evaluation process measures what truly matters:
- realistic pest identification challenges
- appropriate treatment decision contexts
- integrated pest management principles
- regulatory and pesticide safety considerations
This expert review helps ensure that the evaluation framework reflects the realities of agricultural production and Extension practice.
Scope of Work
Reviewers will:
- Review approximately 100 IPM-related questions
- Provide simple ratings or feedback on each question
- Flag questions that are unrealistic, confusing, or inappropriate
- Suggest edits or improvements when helpful
- 2-3 Zoom meetings
Estimated time commitment:
8-12 hours
Deliverables
Reviewers will submit:
- A completed evaluation form based on a formal rubric
- Optional written comments or suggested revisions for specific questions
Desired Expertise
We are seeking reviewers with experience in areas such as:
- Integrated Pest Management
- Crop protection
- Plant pathology
- Entomology
- Weed science
- Agronomy
- Extension crop advisory services
- Experience working directly with farmers, crop consultants, or agricultural producers is particularly valuable.
Compensation
Participants will be compensated with a $1000 honorarium.
About the Extension Foundation
The Extension Foundation is a nonprofit organization that supports the U.S. Cooperative Extension System by developing technology, tools, and programs that help Extension professionals better serve agricultural communities.
This project builds on Extension Foundation’s work to develop ExtensionBot, a conversational AI system designed to provide accessible, research-based agricultural guidance grounded in Extension expertise.