Testing Guide

This document describes the testing strategy, patterns, and practices used in EIAS.

Overview

EIAS uses Vitest as its testing framework, providing: - Fast test execution with native ESM support - Jest-compatible API - TypeScript support out of the box - Built-in mocking utilities

Test Structure

tests/
├── setup.ts                    # Global test configuration and mocks
├── helpers/
│   ├── fixtures.ts            # Reusable test fixtures
│   └── api.ts                 # API testing utilities
├── unit/
│   └── voi/
│       ├── utility.test.ts    # VOI utility function tests
│       └── anduryl.test.ts    # Expert calibration tests
├── integration/
│   ├── calibration.test.ts    # Calibration service tests
│   ├── voi-interview.test.ts  # VOI integration tests
│   └── synthesis/
│       ├── synthesis.test.ts  # Synthesis service tests
│       ├── stats.test.ts      # Statistics tests
│       └── contradictions.test.ts
└── e2e/
    └── adaptive-interview.test.ts  # Full interview flow tests

Running Tests

# Run all tests
npm test

# Run tests once (CI mode)
npm run test:run

# Run with coverage
npm run test:coverage

# Run specific test file
npm test -- tests/unit/voi/utility.test.ts

# Run tests matching pattern
npm test -- -t "calculateUtility"

# Watch mode (default)
npm test

Test Categories

Unit Tests (tests/unit/)

Unit tests verify individual functions in isolation. They are fast, deterministic, and don't require external services.

Example: VOI Utility Tests

// tests/unit/voi/utility.test.ts
import { describe, it, expect } from 'vitest';
import { calculateCoverage, calculateUtility } from '@/lib/voi/utility';

describe('calculateCoverage', () => {
  it('returns 0 for empty sub-questions', () => {
    expect(calculateCoverage([])).toBe(0);
  });

  it('returns 1 for all answered', () => {
    const subQuestions = [
      createSubQuestion('1', 'answered'),
      createSubQuestion('2', 'answered'),
    ];
    const coverage = calculateCoverage(subQuestions);
    expect(coverage).toBeCloseTo(1.0);
  });
});

Best Practices: - Test one function/behavior per test - Use descriptive test names - Use toBeCloseTo() for floating-point comparisons - Create helper functions for test fixtures

Integration Tests (tests/integration/)

Integration tests verify that multiple components work together correctly. They may use mocked external services.

Example: Calibration Service Tests

// tests/integration/calibration.test.ts
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { calibrationService } from '@/lib/services/calibration.service';
import prisma from '@/lib/prisma';

describe('calibrationService', () => {
  beforeEach(() => {
    vi.clearAllMocks();
  });

  it('creates golden question', async () => {
    vi.mocked(prisma.goldenQuestion.create).mockResolvedValue({
      id: 'gq-1',
      question: 'Test question',
      knownAnswer: 'Test answer',
      // ... other fields
    });

    const result = await calibrationService.createGoldenQuestion({
      question: 'Test question',
      knownAnswer: 'Test answer',
    });

    expect(result.id).toBe('gq-1');
    expect(prisma.goldenQuestion.create).toHaveBeenCalled();
  });
});

Best Practices: - Mock external dependencies (database, APIs) - Test service-level business logic - Verify correct interaction with dependencies - Clear mocks between tests

E2E Tests (tests/e2e/)

End-to-end tests verify complete user flows through the system. They test the integration of all components.

Example: Adaptive Interview Flow

// tests/e2e/adaptive-interview.test.ts
describe('E2E Adaptive Interview', () => {
  it('completes full interview flow with VOI termination', async () => {
    // Initialize knowledge state
    const knowledgeState = createInitialKnowledgeState();

    // Simulate interview turns
    for (let turn = 0; turn < 10; turn++) {
      const voi = await calculateBestVOI(knowledgeState);

      if (voi < VOI_THRESHOLDS.STOP_THRESHOLD) {
        // VOI-based termination
        break;
      }

      // Simulate expert response
      await simulateExpertResponse(knowledgeState);
    }

    // Verify interview completed successfully
    expect(knowledgeState.findings.length).toBeGreaterThan(0);
  });
});

Best Practices: - Test complete user journeys - Use realistic test data - Verify final state, not just function calls - Test error scenarios and edge cases

Test Setup

Global Setup (tests/setup.ts)

The setup file configures global mocks for external dependencies:

// tests/setup.ts
import { vi, beforeAll, afterAll, afterEach } from 'vitest';

// Mock Prisma client
vi.mock('@/lib/prisma', () => ({
  default: {
    project: {
      findFirst: vi.fn(),
      findUnique: vi.fn(),
      findMany: vi.fn(),
      create: vi.fn(),
      update: vi.fn(),
      delete: vi.fn(),
    },
    // ... other models
  },
}));

// Mock Anthropic SDK
vi.mock('@anthropic-ai/sdk', () => ({
  default: vi.fn().mockImplementation(() => ({
    messages: {
      create: vi.fn().mockResolvedValue({
        content: [{ type: 'text', text: 'Mocked response' }],
      }),
    },
  })),
}));

// Reset mocks after each test
afterEach(() => {
  vi.clearAllMocks();
});

Vitest Configuration

// vitest.config.ts
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    environment: 'node',
    setupFiles: ['./tests/setup.ts'],
    include: ['tests/**/*.test.ts'],
    coverage: {
      provider: 'v8',
      reporter: ['text', 'html'],
      include: ['src/**/*.ts'],
      exclude: ['src/**/*.d.ts', 'src/app/**'],
    },
  },
  resolve: {
    alias: {
      '@': '/src',
    },
  },
});

Test Fixtures

Creating Test Data

Use helper functions to create consistent test data:

// tests/helpers/fixtures.ts
import type { SubQuestionState, FindingState } from '@/lib/voi/types';

export function createSubQuestion(
  id: string,
  status: 'uncovered' | 'partial' | 'answered',
  priority: 'high' | 'medium' | 'low' = 'medium'
): SubQuestionState {
  return {
    id,
    title: `SubQuestion ${id}`,
    status,
    confidence: status === 'answered' ? 0.9 : status === 'partial' ? 0.5 : 0,
    dependencies: [],
    findingIds: [],
    priority,
  };
}

export function createFinding(
  id: string,
  subQuestionIds: string[],
  confidence: number = 0.8
): FindingState {
  return {
    id,
    content: `Finding ${id}`,
    summary: `Summary ${id}`,
    confidence,
    expertId: 'test-expert',
    turnNumber: 1,
    subQuestionIds,
  };
}

export function createKnowledgeState(
  subQuestions: SubQuestionState[],
  findings: FindingState[] = []
): KnowledgeState {
  return {
    subQuestions,
    findings,
    contradictions: [],
  };
}

Using Fixtures

import { createSubQuestion, createKnowledgeState } from '../helpers/fixtures';

describe('VOI calculation', () => {
  it('calculates higher VOI for uncovered high-priority questions', () => {
    const state = createKnowledgeState([
      createSubQuestion('1', 'uncovered', 'high'),
      createSubQuestion('2', 'answered', 'low'),
    ]);

    const voi = calculateVOI('Question about topic 1?', state);
    expect(voi).toBeGreaterThan(0.2);
  });
});

Mocking Patterns

Mocking Prisma

import { vi } from 'vitest';
import prisma from '@/lib/prisma';

// Mock a single method
vi.mocked(prisma.project.findUnique).mockResolvedValue({
  id: 'project-1',
  name: 'Test Project',
  // ... other fields
});

// Mock to throw an error
vi.mocked(prisma.project.create).mockRejectedValue(
  new Error('Database error')
);

// Verify mock was called
expect(prisma.project.findUnique).toHaveBeenCalledWith({
  where: { id: 'project-1' },
});

Mocking Claude API

import { vi } from 'vitest';

// Mock in setup.ts or test file
vi.mock('@anthropic-ai/sdk', () => ({
  default: vi.fn().mockImplementation(() => ({
    messages: {
      create: vi.fn().mockResolvedValue({
        content: [{ type: 'text', text: 'Mocked response' }],
      }),
    },
  })),
}));

// Override for specific test
import Anthropic from '@anthropic-ai/sdk';

it('handles API errors gracefully', async () => {
  vi.mocked(Anthropic).mockImplementationOnce(() => ({
    messages: {
      create: vi.fn().mockRejectedValue(new Error('API Error')),
    },
  }));

  // Test error handling...
});

Mocking LangChain

vi.mock('@langchain/anthropic', () => ({
  ChatAnthropic: vi.fn().mockImplementation(() => ({
    invoke: vi.fn().mockResolvedValue({
      content: 'This is a mock response.',
    }),
  })),
}));

Testing Best Practices

1. Arrange-Act-Assert Pattern

it('calculates utility correctly', () => {
  // Arrange
  const state = createKnowledgeState([
    createSubQuestion('1', 'answered', 'high'),
  ]);

  // Act
  const utility = calculateUtility(state);

  // Assert
  expect(utility.coverage).toBeCloseTo(1.0);
  expect(utility.total).toBeGreaterThan(0);
});

2. Test Edge Cases

describe('calculateCoverage', () => {
  it('returns 0 for empty array', () => {
    expect(calculateCoverage([])).toBe(0);
  });

  it('handles single item', () => {
    const result = calculateCoverage([createSubQuestion('1', 'answered')]);
    expect(result).toBeCloseTo(1.0);
  });

  it('handles all uncovered', () => {
    const result = calculateCoverage([
      createSubQuestion('1', 'uncovered'),
      createSubQuestion('2', 'uncovered'),
    ]);
    expect(result).toBe(0);
  });
});

3. Test Error Handling

it('throws error for invalid project', async () => {
  vi.mocked(prisma.project.findUnique).mockResolvedValue(null);

  await expect(
    voiService.calculateProjectUtility('invalid-id')
  ).rejects.toThrow('Project not found');
});

4. Use Type-Safe Mocks

import { mockDeep } from 'vitest-mock-extended';
import type { PrismaClient } from '@prisma/client';

const prismaMock = mockDeep<PrismaClient>();

Coverage Targets

Category	Target	Current
Statements	80%	-
Branches	70%	-
Functions	80%	-
Lines	80%	-

Viewing Coverage

# Generate coverage report
npm run test:coverage

# View HTML report
open coverage/index.html

CI Integration

Tests run automatically on pull requests via GitHub Actions:

# .github/workflows/test.yml
name: Tests
on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '18'
      - run: npm ci
      - run: npm run test:run
      - run: npm run test:coverage

Troubleshooting

Common Issues

"Cannot find module '@/lib/...' "

Ensure the test uses the correct path alias. Check vitest.config.ts for alias configuration.

"Mock not being called"

1. Verify the mock is defined before the test imports the module
2. Check that vi.mock() path matches the import path
3. Ensure vi.clearAllMocks() runs in afterEach

"Async test timeout"

Increase the timeout for slow tests:

it('handles slow operation', async () => {
  // test code
}, 10000); // 10 second timeout

"Type error in mock"

Use type assertions or vi.mocked() helper:

vi.mocked(prisma.project.findUnique).mockResolvedValue({...});

Related Documentation

• Architecture - System overview
• API Reference - API documentation
• Contributing - Contribution guidelines