VOI System Documentation

The Value of Information (VOI) system is EIAS's core intelligence layer that optimizes question selection during expert interviews. This document explains the conceptual foundations, implementation, and usage of the VOI system.

Overview

Traditional interviews follow fixed scripts. EIAS uses VOI-based question selection to: - Adapt questions based on what the expert has already shared - Maximize information gain per question asked - Know when to stop when additional questions have diminishing returns - Balance exploration of new topics vs. deepening existing understanding

Core Concepts

Value of Information (VOI)

VOI quantifies the expected benefit of asking a specific question:

VOI(q, e | K, R) = E[U(K' | R)] - U(K | R)

Where: - q = candidate question - e = expert profile (competence, domains) - K = current knowledge state - K' = expected knowledge state after receiving answer - R = research context (goals, uncertainty areas) - U = utility function

In plain terms: VOI measures how much asking this question would improve our knowledge state, given what the expert might say.

Knowledge State (K)

The knowledge state tracks everything we know (and don't know) about the research topic:

interface KnowledgeState {
  subQuestions: SubQuestionState[];  // Atomic research questions
  findings: FindingState[];           // Extracted information
  contradictions: ContradictionState[]; // Conflicting findings
  expertCompetence?: ExpertCompetenceState; // Calibration data
}

Sub-Questions are the atomic components of the research question. Each has: - status: uncovered | partial | answered - confidence: 0.0-1.0, how certain we are - priority: high | medium | low

Findings are individual pieces of information extracted from expert responses, linked to the sub-questions they address.

Contradictions occur when findings conflict, requiring resolution.

Utility Function U(K|R)

Utility measures the quality of our knowledge state:

U(K|R) = 0.35×Coverage + 0.30×Confidence + 0.20×Coherence + 0.15×Actionability

Component	Weight	Description
Coverage	35%	Percentage of sub-questions addressed (weighted by priority)
Confidence	30%	Average confidence across findings
Coherence	20%	1 - (contradictions / findings)
Actionability	15%	High-priority questions that are fully answered

Diminishing Returns

The utility function applies diminishing returns to prevent over-optimization:

// Values above 70% get progressively smaller marginal gains
if (value > 0.7) {
  value = 0.7 + (1 - 0.7) * (1 - e^(-0.5 * (value - 0.7)))
}

This means: - Going from 0% to 70% coverage provides full value - Going from 70% to 100% provides diminishing marginal utility - Encourages breadth over excessive depth on any single topic

VOI Calculation

Hybrid Approach

EIAS uses a hybrid approach for speed and quality:

1. Fast closed-form calculation (<100ms) for VOI estimation
2. LLM-based generation (~2s) for candidate questions

Step-by-Step Process

1. Calculate current utility U(K)
   └─▶ Coverage, Confidence, Coherence, Actionability

2. For each candidate question:
   a. Estimate information gain
      └─▶ Which sub-questions would it address?
      └─▶ What confidence boost is expected?

   b. Estimate expected utility U(K')
      └─▶ Apply gains with diminishing returns

   c. Calculate VOI = U(K') - U(K)

3. Rank questions by VOI
   └─▶ Highest VOI = most valuable question

4. Apply threshold checks
   └─▶ VOI < 0.05? Stop interviewing
   └─▶ VOI > 0.30? High-value question

Thresholds

const VOI_THRESHOLDS = {
  STOP_THRESHOLD: 0.05,      // Stop when VOI falls below this
  FOLLOW_UP_THRESHOLD: 0.15, // Prefer follow-up over topic switch
  HIGH_VALUE_THRESHOLD: 0.30 // Considered high-value question
};

Expert Calibration

Cooke's Classical Model

EIAS uses Cooke's Classical Model to weight expert responses based on their demonstrated competence:

Competence = Statistical Accuracy × Informativeness

Metric	Description	Measurement
Statistical Accuracy	Does stated confidence match actual accuracy?	Golden question calibration
Informativeness	Does the expert provide decisive judgments?	Entropy of confidence distribution

Golden Questions

Calibration uses "golden questions" with known answers:

1. Expert answers questions with confidence levels
2. System compares stated confidence vs. actual accuracy
3. Competence score is calculated and used to weight VOI

Example calibration flow:

Golden Question: "What is the typical insulin absorption rate?"
Expert Answer: "4-6 units/hour" (confidence: 80%)
Known Answer: "4-6 units/hour"
Result: Correct at 80% confidence → contributes to accuracy score

Competence Weighting

Expert competence affects VOI calculation: - High-competence experts (>0.7) reduce uncertainty in VOI estimates - Low-competence experts widen confidence intervals - Competence is used to weight findings in aggregation

Integration with Interview Graph

Interview Flow

┌────────────────────────────────────────────────────────────┐
│                    Interview Session                        │
├────────────────────────────────────────────────────────────┤
│                                                            │
│  START ─▶ GREETING ─▶ [Expert responds]                   │
│                              │                             │
│                              ▼                             │
│                        SYNTHESIZE                          │
│                    ┌─────────────────┐                     │
│                    │ Extract findings │                    │
│                    │ Update K state   │                    │
│                    │ Calculate utility│                    │
│                    └────────┬────────┘                     │
│                              │                             │
│                              ▼                             │
│                    VOI Question Selection                  │
│                    ┌─────────────────┐                     │
│                    │ Generate candidates│                  │
│                    │ Calculate VOI each│                   │
│                    │ Select highest VOI│                   │
│                    └────────┬────────┘                     │
│                              │                             │
│              ┌───────────────┼───────────────┐            │
│              │               │               │            │
│         VOI < 0.05     VOI >= 0.05     VOI >= 0.30        │
│              │               │               │            │
│              ▼               ▼               ▼            │
│          WRAP_UP        INTERVIEW       INTERVIEW         │
│        (low value)    (continue)     (high value)        │
│                                                            │
└────────────────────────────────────────────────────────────┘

State Annotations

The interview state includes VOI-related fields:

interface InterviewState {
  // ... other fields

  knowledgeState: KnowledgeState;        // Core VOI data structure
  utilityScores: UtilityScores | null;   // Current utility breakdown
  candidateQuestions: VOIResult[];       // Questions ranked by VOI
  selectedQuestion: VOIResult | null;    // Chosen question for turn
}

API Usage

VOI Service

The VOI service provides high-level API access:

import { voiService } from '@/lib/services/voi.service';

// Calculate current project utility
const utility = await voiService.calculateProjectUtility(projectId);

// Get VOI for a specific question
const voi = await voiService.estimateQuestionVOI(
  projectId,
  invitationId,
  "What challenges have you encountered?"
);

// Generate and rank candidate questions
const rankedQuestions = await voiService.generateRankedQuestions(
  projectId,
  invitationId,
  5 // number of candidates
);

// Select the best next question
const bestQuestion = await voiService.selectNextQuestion(
  projectId,
  invitationId
);

// Get full VOI analysis with recommendations
const analysis = await voiService.getVOIAnalysis(projectId);

Calibration Service

import { calibrationService } from '@/lib/services/calibration.service';

// Create a golden question
await calibrationService.createGoldenQuestion({
  projectId: 'optional-project-id',
  question: 'What is the typical dosage range?',
  knownAnswer: '10-20mg daily',
  domain: 'pharmacology',
  difficulty: 'medium'
});

// Get calibration questions for an expert
const questions = await calibrationService.getCalibrationQuestions({
  projectId: 'project-id',
  domains: ['pharmacology'],
  count: 5,
  excludeAnsweredBy: 'expert-id'
});

// Record a calibration response
await calibrationService.recordCalibrationResponse({
  goldenQuestionId: 'question-id',
  expertId: 'expert-id',
  response: '10-20mg daily',
  confidenceLevel: 0.8
});

// Get expert competence
const competence = await calibrationService.getExpertCompetence('expert-id');

REST API Endpoints

VOI Endpoints

Endpoint	Method	Description
/api/projects/[id]/voi	GET	Get VOI analysis for project
/api/projects/[id]/voi/questions	POST	Generate ranked questions
/api/projects/[id]/voi/estimate	POST	Estimate VOI for question

Calibration Endpoints

Endpoint	Method	Description
/api/calibration/questions	GET	Get calibration questions
/api/calibration/questions	POST	Create golden question
/api/calibration/responses	POST	Record calibration response
/api/calibration/experts/[id]	GET	Get expert competence

See API Reference for full documentation.

Best Practices

Designing Sub-Questions

Good sub-questions are: - Atomic: Address one specific aspect - Prioritized: Ranked by importance to research goals - Observable: Can be addressed through expert knowledge

Example for a diabetes medication study:

High Priority:
- What are the primary efficacy endpoints?
- What safety concerns are most critical?

Medium Priority:
- What patient populations respond best?
- How does this compare to existing treatments?

Low Priority:
- What is the optimal dosing schedule?
- Are there notable drug interactions?

Calibration Strategy

1. Include domain-relevant golden questions
2. Mix difficulty levels (easy, medium, hard)
3. Calibrate before critical interviews
4. Re-calibrate periodically for long-running projects

Interpreting VOI Scores

VOI Score	Interpretation	Action
> 0.30	High value	Ask this question
0.15 - 0.30	Moderate value	Good follow-up candidate
0.05 - 0.15	Low value	Consider if time permits
< 0.05	Minimal value	Consider wrapping up

Troubleshooting

VOI Always Low

Symptoms: VOI scores consistently below threshold

Possible causes: 1. Sub-questions already well-covered 2. Questions not matching sub-question topics 3. Expert competence very low

Solutions: - Review sub-question coverage in knowledge state - Regenerate candidate questions - Check expert calibration data

VOI Not Decreasing

Symptoms: VOI stays high despite many turns

Possible causes: 1. Sub-questions too broad 2. Diminishing returns not configured 3. New findings creating new sub-questions

Solutions: - Break down sub-questions into atomic parts - Verify useDiminishingReturns is enabled - Review synthesis extraction rules

References

• Cooke, R. M. (1991). Experts in Uncertainty: Opinion and Subjective Probability in Science
• Howard, R. A. (1966). "Information Value Theory"
• Aspinall, W. P. (2010). "A route to more tractable expert advice" Nature, 463(7279)

Related Documentation

• Architecture - System overview
• API Reference - Full API documentation
• Database Schema - Data model for VOI entities