Mining Human Feedback for Prompt Insights to Improve AI Assistants

Optimizing AI assistant prompts is an iterative process. Human feedback provides invaluable qualitative insights to guide refinements. In this post, I’ll explore methods for mining human assessments to uncover prompt improvements.

As an applied AI researcher, I leverage various techniques to gather human perspectives for prompt engineering. Combining quantitative and qualitative insights allows sculpting responses tailored to human needs. Let’s dig into mining that human feedback loop.

The Value of Human Feedback for Prompts

First, why solicit human input for prompt optimization? Some key benefits:

• Surfaces subjective aspects quantitative metrics miss
• Provides nuanced qualitative assessments of responses
• Captures subtleties like tone, style, and ethics
• Identifies blindspots and assumptions to address
• Assesses alignment with actual use case goals
• Guides prompt priorities for refinement
• Encourages reviewer prompt creativity and ideas
• Keeps the human perspective integral to the process
  
  

Humans know best when an AI resonates.

Methods to Gather Human Feedback on Prompts

What methods can provide human insights on prompts? Some options:

• Crowdworker ratings on dimensions like helpfulness
• 1:1 interviews probing perceptions of responses
• Focus groups discussing examples of AI interactions
• Social listening for commentary on capabilities
• User testing to identify pain points and desires
• Surveys assessing subjective satisfaction
• Expert reviews weighing appropriateness for use case
• Analysis of support tickets and product reviews
  
  

Cast a wide net.

Creating a Human Feedback Review Process

To formalize gathering feedback:

1. Define review dimensions of interest – accuracy, tone, ethics etc.
2. Create rubrics and questionnaires to standardize assessments
3. Establish a review cadence and recruiter reviewer pools
4. Synthesize insights to identify common themes and outliers
5. Distill actionable prompt optimization recommendations
6. Close the loop by confirming improvements address feedback
   
   

Analyzing Human Feedback for Prompt Insights

When analyzing reviews, look for:

• Response aspects rated poorly – identify underlying prompt gaps
• High variance – signals polarization to address
• Reviewer suggestions – prompts to try or avoid
• Surprising reactions – probing why prompts missed the mark uncovers assumptions
• Feedback patterns – clusters of similar assessments indicate systematic gaps
• Reviewer disagreement – prompts implicitly conveying unintended messages
  
  

Uncover the prompt roots of reviewer reactions.

Case Study: Optimizing a Customer Service Prompt

Let’s walk through an example applying human feedback analysis to improve a customer service prompt:

Original prompt:

You are a customer service agent. Please respond to this customer politely and provide information to resolve their support issue:

“Hello, I purchased your product last week and it is already broken after light use. I’m very frustrated. Can I get a refund?”

Reviewer feedback:

• Tone feels impersonal and lacking in empathy
• Response too focused on policies rather than understanding customer feelings
• Does not build trust or offer goodwill gestures
  
  

Improved prompt:

You are a friendly, thoughtful customer service agent deeply invested in satisfying customer needs. Please respond very politely to this upset customer and provide information to resolve their support issue, showing care and understanding:

“Hello, I purchased your product last week and it is already broken after light use. I’m very frustrated. Can I get a refund?”

This illustrates mining reviews to identify opportunities for better aligning prompts with desired tone and qualities.

Continuously Tuning Prompts Based on Feedback

Soliciting human perspectives must be an ongoing endeavor, not a one-off. Continuously refine prompts informed by regular feedback.

No amount of quantitative optimization substitutes for keeping your finger on the pulse of subjective human reactions. Prompt engineering is applied social science.

I hope these tips provide a helpful starting point for mining human insights to illuminate the way forward. Please reach out if you need any assistance establishing prompt feedback loops and analyses tailored to your use case requirements!