First release.
aiEvalR provides a statistical and psychometric framework for
evaluating large language models and other AI systems, organized around
seven modules.
ai_generalizability()
fits a crossed G-study (case x prompt x model x run) via lme4;
ai_dstudy() projects the generalizability coefficient and
dependability index (Phi) for any measurement design without
refitting. Also ai_conditional_sem() and
ai_decision_consistency().ai_test_retest() (ICC), ai_internal_consistency()
(Cronbach's alpha), ai_bootstrap_reliability(), and the
ai_reliability() wrapper.stress_test(), prompt_sensitivity(),
adversarial_stability(), and the ai_robustness() wrapper.ai_group_disparity() (demographic parity, equalized
odds), ai_dif_mh() (Mantel-Haenszel screening), and ai_fairness(),
which integrates with the companion aiDIF package for item-level
IRT-based AI-scoring bias when item MLEs are supplied.ece(), brier_score(), calibration_curve(),
empirical_output_interval(), and the ai_calibration() wrapper,
with input validation throughout.hallucination_rate() (aggregation of adjudicated
verdicts), lexical_overlap() (a lexical baseline, deliberately named
to avoid implying factual-consistency detection), and
citation_accuracy().ai_dashboard() combines rescaled module scores into a
descriptive overall profile.Comprehensive test suite (~93% line coverage), verifying statistical behavior against known-value cases rather than merely checking for non-null output.
getting-started: a module-by-module tour with runnable examples.psychometric-core: a deeper treatment of the generalizability-theory
workflow (G-studies and D-studies).