huggingface-community-evals

# Overview This skill is for **running evaluations against models on the Hugging Face Hub on local hardware**. It covers: - `inspect-ai` with local inference - `lighteval` with local inference - choosing between `vllm`, Hugging Face Transformers, and `accelerate` - smoke tests, task selection, and backend fallback strategy It does **not** cover: - Hugging Face Jobs orchestration - model-card or `model-index` edits - README table extraction - Artificial Analysis imports - `.eval_results` generation or publishing - PR creation or community-evals automation If the user wants to **run the same eval remotely on Hugging Face Jobs**, hand off to the `hugging-face-jobs` skill and pass it one of the local scripts in this skill. If the user wants to **publish results into the community evals workflow**, stop after generating the evaluation run and hand off that publishing step to `~/code/community-evals`. > All paths below are relative to the directory containing this `SKILL.md`. # When To Use Which Script | Use case | Script | |---|---| | Local `inspect-ai` eval on a Hub model via inference providers | `scripts/inspect_eval_uv.py` | | Local GPU eval with `inspect-ai` using `vllm` or Transformers | `scripts/inspect_vllm_uv.py` | | Local GPU eval with `lighteval` using `vllm` or `accelerate` | `scripts/lighteval_vllm_uv.py` | | Extra command patterns | `examples/USAGE_EXAMPLES.md` | # Prerequisites - Prefer `uv run` for local execution. - Set `HF_TOKEN` for gated/private models. - For local GPU runs, verify GPU access before starting: ```bash uv --version printenv HF_TOKEN >/dev/null nvidia-smi ``` If `nvidia-smi` is unavailable, either: - use `scripts/inspect_eval_uv.py` for lighter provider-backed evaluation, or - hand off to the `hugging-face-jobs` skill if the user wants remote compute. # Core Workflow 1. Choose the evaluation framework. - Use `inspect-ai` when you want explicit task control and inspect-native flows. - Use `lighteval` when the benchmark is naturally expressed as a lighteval task string, especially leaderboard-style tasks. 2. Choose the inference backend. - Prefer `vllm` for throughput on supported architectures. - Use Hugging Face Transformers (`--backend hf`) or `accelerate` as compatibility fallbacks. 3. Start with a smoke test. - `inspect-ai`: add `--limit 10` or similar. - `lighteval`: add `--max-samples 10`. 4. Scale up only after the smoke test passes. 5. If the user wants remote execution, hand off to `hugging-face-jobs` with the same script + args. # Quick Start ## Option A: inspect-ai with local inference providers path Best when the model is already supported by Hugging Face Inference Providers and you want the lowest local setup overhead. ```bash uv run scripts/inspect_eval_uv.py \ --model meta-llama/Llama-3.2-1B \ --task mmlu \ --limit 20 ``` Use this path when: - you want a quick local smoke test - you do not need direct GPU control - the task already exists in `inspect-evals` ##