FaithBench Benchmark Card

This document follows the benchmark card format adapted from Gebru et al. (2021) "Datasheets for Datasets" and Mitchell et al. (2019) "Model Cards for Model Reporting." It provides a standardized summary of FaithBench for AI researchers and evaluators.

1. Benchmark Overview

One-sentence summary: FaithBench is an initial benchmark framework for evaluating the theological faithfulness of large language models across Christian traditions, using LLM-as-judge scoring with planned human expert calibration.

2. Intended Use

FaithBench is designed for:

• Comparing theological knowledge across LLMs under controlled conditions (standardized prompts, temperature, token limits)
• Identifying dimension-specific strengths and weaknesses (e.g., a model may handle historical theology well but struggle with textual analysis)
• Identifying tradition-specific performance differences (e.g., a model may represent Catholic doctrine more accurately than Reformed doctrine)
• Informing model selection for theological applications (seminary tools, Bible study apps, pastoral AI)
• Research into LLM evaluation methodology for subjective, interpretive domains

3. Out-of-Scope Use

FaithBench should not be used for:

• Certifying theological safety. A high FaithBench score does not mean a model is safe for unsupervised theological use.
• Measuring spiritual wisdom or pastoral sensitivity. FaithBench measures knowledge accuracy, not wisdom, empathy, or pastoral appropriateness.
• Making deployment decisions without additional evaluation. FaithBench tests controlled conditions; real-world performance depends on system prompts, RAG, temperature, and user interaction patterns.
• Evaluating non-Christian traditions. The benchmark covers only Christian theology in v1.0.
• Comparing models across different benchmark versions. Held-out rotation and methodology changes may affect cross-version comparability.
• Ranking models with small score differences. Models within each other's confidence intervals should be treated as statistically indistinguishable.

4. Benchmark Composition

Active test set composition:

• Hard questions: ~60% of scored items (Ph.D./faculty-level content)
• Expert questions: ~40% of scored items (multi-hop reasoning, adversarial design)

5. Evaluation Protocol

Test Model Configuration

Judge Configuration

Scoring

• Scale: 0–3 per sub-dimension (Inadequate / Partial / Good / Excellent)
• Normalization: Raw scores divided by 3 to produce 0–1 range
• Weighting: Sub-dimension weights within each dimension, dimension weights for composite
• Difficulty weighting: Easy=1.0, Medium=1.5, Hard=2.0, Expert=3.0
• Confidence intervals: Bootstrap, 1,000 iterations, 95% confidence

Score Interpretation Hierarchy

1. Primary: Per-dimension performance profiles
2. Secondary: Tradition-specific scores
3. Tertiary: Overall composite score

The composite score is useful for leaderboard simplicity but may obscure meaningful dimension-level variation.

6. Tradition Scope

Scope limitations:

• Western Christian focus
• English-language evaluation only
• Non-Western traditions (African, Asian, Latin American) planned for v2.0+

7. Known Limitations

8. Known Failure Modes

9. Validation Status

10. Recommended Citation

@misc{faithbench2026,
  title={FaithBench: Toward Tradition-Aware Evaluation of Theological
         Faithfulness in Large Language Models},
  author={FaithBench Team},
  year={2026},
  url={https://faithbench.com},
  note={Methodology v1.0}
}

11. Contact & Access

References

Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86–92.

Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., ... & Gebru, T. (2019). Model cards for model reporting. FAT.