Open source LLM evaluation for RAG: accuracy + cost + hallucination

[vignesh2027]

Hey LlamaIndex community!

For teams building RAG applications with LlamaIndex, choosing the right LLM backbone requires systematic evaluation. I built an open source framework specifically useful for this.

Why it matters for LlamaIndex RAG:

• Hallucination Rate: the most critical RAG failure mode is when the LLM ignores retrieved context and fabricates. This metric catches that.
• Cost per 1K tokens: RAG prompts are long (context + query). Cost compounds fast. Know your per-model cost before scaling.
• Latency p95: RAG adds retrieval latency. LLM latency is the other half. Tail latency = user experience.
• Accuracy: ground truth comparison on QA benchmarks
• Reasoning Quality: CoT depth (important for multi-hop RAG reasoning)

pip install llm-evaluation-framework
llm-eval compare --models gpt-4o-mini --models claude-3-haiku-20240307 --benchmark mmlu --samples 100

Live demo (no API key): https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework

Open source, free forever. Built with LiteLLM so all LlamaIndex-supported models work. Feedback very welcome!