Package: aiEvalR 0.1.0

aiEvalR: Statistical and Psychometric Evaluation of AI Systems

Evaluates artificial intelligence (AI) systems as measurement instruments using psychometric methods. Provides multi-facet generalizability theory (G-study and D-study) via 'lme4', reliability via the intraclass correlation coefficient (ICC), calibration via the expected calibration error (ECE) and Brier score, robustness stress testing, and group disparity diagnostics. Item-level differential item functioning (DIF) based on item response theory (IRT) is delegated to the 'aiDIF' package. Methods follow Cronbach, Gleser, Nanda and Rajaratnam (1972, <ISBN:9780471188506>) and Brennan (2001) <doi:10.1007/978-1-4757-3456-0>.

Authors:Subir Hait [aut, cre]

aiEvalR_0.1.0.tar.gz
aiEvalR_0.1.0.zip(r-4.7-any)aiEvalR_0.1.0.zip(r-4.6-any)aiEvalR_0.1.0.zip(r-4.5-any)
aiEvalR_0.1.0.tgz(r-4.6-any)aiEvalR_0.1.0.tgz(r-4.5-any)
aiEvalR_0.1.0.tar.gz(r-4.7-any)aiEvalR_0.1.0.tar.gz(r-4.6-any)
aiEvalR_0.1.0.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
aiEvalR/json (API)

# Install 'aiEvalR' in R:
install.packages('aiEvalR', repos = c('https://subirhait.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/causalfragility-lab/aievalr/issues

On CRAN:

Conda:

4.30 score 265 downloads 25 exports 0 dependencies

Last updated from:79c93e72ed. Checks:9 OK. Indexed: yes.

TargetResultTimeFilesSyslog
linux-develOK149
source / vignettesOK187
linux-releaseOK150
macos-releaseOK152
macos-oldrelOK136
windows-develOK84
windows-releaseOK79
windows-oldrelOK77
wasm-releaseOK112

Exports:adversarial_stabilityai_bootstrap_reliabilityai_calibrationai_conditional_semai_dashboardai_decision_consistencyai_dif_mhai_dstudyai_fairnessai_generalizabilityai_group_disparityai_internal_consistencyai_reliabilityai_robustnessai_test_retestbrier_scorecalibration_curvecitation_accuracyeceempirical_output_intervalfairness_dashboardhallucination_ratelexical_overlapprompt_sensitivitystress_test

Dependencies:

Getting Started with aiEvalR
Why evaluate an AI system like a test? | Reliability: is the system consistent? | Robustness: does the answer survive small changes? | Fairness: do outcomes differ by group? | Calibration: are stated confidences trustworthy? | Hallucination: aggregating adjudicated labels | An integrated dashboard | Where to next
Rendered fromgetting-started.Rmdusingknitr::rmarkdown

Last update: 2026-07-27
Started: 2026-07-27

The Generalizability-Theory Core
From one reliability number to a variance decomposition | A G-study | A D-study: how many conditions do you need? | Conditional error and decision consistency | References
Rendered frompsychometric-core.Rmdusingknitr::rmarkdown

Last update: 2026-07-27
Started: 2026-07-27

Readme and manuals

Help Manual

Help pageTopics
Adversarial stabilityadversarial_stability
Bootstrap confidence interval for AI reliability estimatesai_bootstrap_reliability
Overall calibration summary for AI system outputsai_calibration
Conditional standard error of measurement for AI outputsai_conditional_sem
Integrated AI Evaluation Dashboardai_dashboard
Decision consistency for AI classification/scoring decisionsai_decision_consistency
Mantel-Haenszel differential functioning test for AI-scored binary itemsai_dif_mh
D-study: project generalizability/dependability for given facet sizesai_dstudy
Overall fairness summary for AI system outputsai_fairness
Multi-facet generalizability (G-)study for AI outputsai_generalizability
Group-level disparity in AI outcomesai_group_disparity
Internal consistency of AI responses across parallel itemsai_internal_consistency
Overall reliability summary for AI system outputsai_reliability
Overall robustness summary for AI system outputsai_robustness
Test-retest reliability of AI responsesai_test_retest
Brier scorebrier_score
Calibration curve (reliability diagram data)calibration_curve
Citation accuracycitation_accuracy
Expected Calibration Errorece
Empirical output interval for AI numeric outputsempirical_output_interval
Fairness dashboard summary (text form)fairness_dashboard
Hallucination rate across AI outputshallucination_rate
Lexical overlap between an AI output and a reference sourcelexical_overlap
Prompt sensitivity analysisprompt_sensitivity
Stress test AI responses under a set of perturbationsstress_test