What GraphRAG evaluation metrics actually measure
GraphRAG evaluation metrics measure the quality of a system that combines a knowledge graph with retrieval-augmented generation. Microsoft Research introduced the GraphRAG term and described GraphRAG as an extension of retrieval-augmented generation that uses a knowledge graph to support language-model answers. The system may retrieve entities, relationships, communities, documents, or generated summaries, so a single accuracy score is rarely enough. As of 25 September 2026, the most useful evaluation separates retrieval, graph construction, answer quality, operating performance, and user outcomes. A system can find relevant facts and still produce an unsupported conclusion, or it can produce a polished answer at an unacceptable cost per query.
Also worth reading: How Can Enterprises Reduce LLM Token Costs Without Lowering AI Quality in 2026? · How Do Modern Enterprises Measure and Optimize Learning Return on Investment Using an Enterprise Learning Metrics Platform? · How do enterprises measure the ROI of corporate AI training programs in 2026?
Evaluation should begin by defining the question types the system is expected to handle. Local fact questions, global synthesis questions, relationship questions, causal questions, and temporal questions place different demands on the index. For a direct fact, a traditional vector retriever may be enough. For a question such as “Which recurring mechanisms appear across the approved pharmaceutical research documents?”, the evaluator needs evidence that relevant documents and entities were selected, connected correctly, and combined into a coherent answer. The correct unit of evaluation is therefore not only the final paragraph; it is the full chain from corpus and query to evidence and answer.
A defensible evaluation program records a dataset, a query set, expected evidence, and scoring rules. Microsoft’s public GraphRAG implementation and documentation are useful technical references, but no public benchmark establishes a universal pass percentage for every enterprise GraphRAG deployment. Teams should treat published benchmark scores as baselines or methodological examples rather than promises about their own results.
The core GraphRAG quality metrics
The first metric group measures retrieval. Recall at k, or Recall@k, reports how many required evidence items appear in the first k retrieved objects. If an answer depends on five source documents and four are returned, evidence recall is 80% for that query. Precision@k reports how many returned objects are genuinely relevant, while normalized discounted cumulative gain, or nDCG@k, rewards relevant items appearing near the top. Mean reciprocal rank rewards a system that places the first acceptable answer near the beginning. These measures should be calculated separately for documents, entities, relationships, and community summaries when those objects are part of the retrieval design.
The second group measures graph quality. Entity linking accuracy asks whether mentions referring to the same real-world concept were merged correctly. Relation precision asks whether extracted relationships are supported by the source text, and relation recall asks whether important supported relationships were captured. Community or community-summary quality should be checked against a human-defined taxonomy, especially when the graph uses local-global or hierarchical retrieval. Path validity measures whether a relationship chain exists in the graph, but it does not prove that the chain answers the user’s question. A graph can be technically valid and still be incomplete, stale, or biased toward frequently repeated entities.
The third group measures generation. Groundedness or faithfulness measures whether every factual claim can be traced to retrieved evidence. Answer correctness compares the response with an expert answer or annotated reference. Completeness checks whether the answer covers all required elements, such as dates, populations, conditions, and exceptions. Citation precision measures whether each citation actually supports the nearby statement, while citation recall measures whether important claims have citations. Human reviewers should use a 1–5 rubric, but scores need written criteria and adjudication. An evaluator that calls every plausible sentence “correct” is measuring writing style, not factual reliability.
| Metric group | Example measure | What it tells you | Common failure it exposes |
|---|---|---|---|
| Retrieval | Recall@10 | Whether needed evidence appears in the results | Missing relevant documents or entities |
| Ranking | nDCG@10 or MRR | Whether the best evidence is placed first | Correct material buried below distractors |
| Graph | Relation precision | Whether extracted links are defensible | Hallucinated or over-broad relationships |
| Generation | Citation precision | Whether citations support generated claims | Fluent unsupported statements |
| Outcomes | Expert acceptance rate | Whether users accept the answer for work | Technically accurate but unusable responses |
| Operations | Cost per accepted answer | What useful answers cost after retries | Expensive retries and excessive context |
Start with a frozen sample of real business questions rather than a collection of easy prompts. A typical enterprise pilot might contain 100–300 questions, divided into 20% simple factual lookups, 40% multi-hop or relationship questions, 20% aggregation questions, and 20% adversarial or unanswerable questions. Those percentages are a design starting point, not an industry standard. The sample should reflect the languages, document formats, departments, and risk levels found in production. Pharmaceutical research, for example, may require careful separation between a study result, an author’s hypothesis, and a company press claim.
Each test item should contain a question, relevant evidence, acceptable answer criteria, and a policy for unanswerable cases. Annotators should identify required facts, optional context, acceptable source documents, and claims that would make an answer incorrect. For questions that require several documents, mark relationships between evidence pieces. This lets the team distinguish a retrieval failure from a reasoning failure. If a required document is absent, blaming the language model hides an indexing problem; if all evidence is retrieved but the conclusion changes the relationship, the problem lies in graph construction or generation.
A baseline is essential. Run the same questions through a strong vector RAG system, a hybrid lexical-vector system, a long-context model where feasible, and the proposed GraphRAG system. Keep the language model, prompt format, context budget, and answer temperature as similar as practical. In controlled comparisons, change one component at a time. A result that appears better only because it used a larger model or more tokens is not evidence for the graph itself. The Microsoft GraphRAG paper, “From Local to Global: A Graph RAG Approach to Query-Focused Summarization,” is useful for understanding this distinction between local and global retrieval behavior, but its reported results should not be transplanted to a different corpus.
Practical scoring and acceptance thresholds
There is no universal GraphRAG pass mark. A sensible initial policy is to require at least 90% evidence recall@10 on high-priority factual queries, at least 95% citation precision for claims marked as verified, and at least 85% expert acceptance for low-risk internal questions. Those figures are proposed operating thresholds, not standards established by Microsoft or AWS. Risk should change the threshold. A customer-support answer may tolerate a lower score than a regulated clinical, legal, or safety statement. Teams should also set maximum thresholds for unsupported claims, usually zero for questions in which unsupported advice could cause material harm.
Measure every query at least five times when outputs are nondeterministic, or use repeated runs sufficient to estimate variability. Report the mean, median, and worst decile instead of only the average. A system with 95% average citation precision may be unacceptable if its worst 10% of answers contain invented drug interactions. Confidence intervals are useful when the evaluation set is smaller than 300 questions. For a pilot, 100 questions can reveal major failure patterns, but it cannot support precise claims about rare errors. Confidence in the final result depends on sample size, annotator agreement, and the percentage of queries representing the highest-risk use cases.
Use both automatic and human evaluation. Exact-match and reference-based measures work for short factual answers, while rubric-based expert review is better for synthesis, explanation, and policy interpretation. Automatic embedding similarity is generally weak for determining whether a claim is supported, so it should not be the sole correctness measure. Record latency separately for retrieval, graph traversal, model generation, reranking, and citation verification. A median response under 10 seconds may be acceptable for internal research, while an interactive learning assistant may need a first useful response under 4 seconds even if a complete answer takes longer.
Comparing GraphRAG with alternative retrieval designs
Vector RAG is usually simpler and cheaper for direct semantic lookup. Hybrid retrieval adds lexical and dense signals, which often improves exact-name, terminology, and regulatory-code searches. Long-context generation can avoid retrieval misses when the entire document set fits the context window, but it consumes tokens and can still overlook material buried in a long prompt. Agentic RAG can route questions through multiple tools, but it introduces orchestration latency and makes failures harder to reproduce. GraphRAG is most defensible when relationships, aggregation, provenance, or cross-document synthesis are central to the task.
| Design choice | GraphRAG | Vector RAG | Hybrid or long-context retrieval | Agentic RAG |
|---|---|---|---|---|
| Best fit | Relationship and global synthesis questions | Similarity-based factual search | Exact terms or small document sets | Multi-step tool workflows |
| Typical complexity | High | Low to medium | Medium | High |
| Main advantage | Structured connections and community evidence | Simple implementation and predictable cost | Better coverage across search styles | Flexible task routing |
| Main weakness | Indexing and maintenance overhead | Weak on explicit multi-hop reasoning | Token cost or context limits | Harder debugging and evaluation |
| Cost pattern | Graph build plus query and storage costs | Usually lowest per query | Depends on context and reranking | Multiple model and tool calls |
| Evaluation focus | Relation quality, coverage, groundedness | Recall, ranking, faithfulness | Coverage and context efficiency | Tool success, latency, total cost |
Cost, latency, and pricing considerations in 2026
The Microsoft GraphRAG software is open source, so software licensing does not require a GraphRAG license fee. That does not make GraphRAG free. Costs include document parsing, entity extraction, relationship resolution, community summarization, embedding generation, storage, retrieval, reranking, model inference, observability, and human evaluation. A small proof of concept may be built with existing cloud credits, but a production system needs a budget for refreshes and error analysis. If the source corpus changes daily, graph updates may need to be incremental; if it changes hourly, the operational design is different.
A useful cost formula is total cost per accepted answer. Divide monthly platform cost by the number of answers that pass the correctness and citation rubric, rather than by total requests. Include retries, failed queries, evaluation labor, and support time where appropriate. Track token use by stage. A system that uses 12,000 tokens for retrieval context but increases expert acceptance from 70% to 90% may still be economical, while a system that uses 25,000 tokens and improves acceptance by two percentage points may not. As a planning rule, set an engineering budget ceiling before launch, such as 2,000 model input tokens per ordinary query and 500 for a fast local-fact route, then revise the limits using measured quality. These are internal targets, not GraphRAG standards.
Published vendor case studies should be read carefully. The supplied research context includes an AWS item claiming that GraphRAG deployment cut drug research cycles by 87%. That number describes a particular reported deployment, not a general expected result for GraphRAG. Before using such a figure in a business case, ask for the baseline cycle length, task scope, number of documents, model configuration, human review requirements, and whether the comparison included indexing time. GraphRAG evaluation is strongest when the claimed business effect is connected to a measured system metric, such as time to verified evidence or expert acceptance per hour.
Common mistakes that distort evaluation
A frequent mistake is evaluating only polished final answers. If the evaluator cannot inspect retrieved evidence, entity links, community summaries, and citations, it cannot tell whether the graph helped. Another mistake is comparing GraphRAG with a weak vector baseline. A fair comparison uses a tuned baseline, comparable model settings, and the same evaluation set. Teams also confuse more retrieved context with better answers. Larger context windows can increase cost and distract the model without improving relation reasoning.
Metric gaming is another risk. A system can maximize recall by returning every document in the corpus, or maximize precision by returning one easy passage. Graph quality can be inflated by counting duplicate mentions as separate entities. A model can appear well grounded when citations point to broad documents rather than the exact supporting passage. A fair report should include a refusal or abstention test, because a system that answers every unanswerable question is usually less trustworthy than one that identifies insufficient evidence.
Finally, avoid treating an LLM judge as an unquestionable authority. Use human review for high-risk items, blind reviewers where possible, and a second judge to audit disagreements. If annotators disagree, preserve the disagreement instead of averaging it away. Documentation should state the model, prompt, graph schema, index date, embedding model, retrieval parameters, and evaluation date. Without that record, a later improvement may be impossible to reproduce.
When enterprises should adopt, pause, or expand GraphRAG
Adopt GraphRAG when the task requires relationships across documents, the corpus contains reliable entities and concepts, and errors are costly enough to justify graph maintenance. Good early candidates include research intelligence, policy mapping, knowledge discovery, technical support dependencies, and enterprise learning paths. Expand only after a controlled pilot shows a measurable advantage over hybrid RAG. A reasonable pilot is 4–8 weeks, with 100–300 annotated questions, weekly error review, and a final comparison against at least one simpler baseline. The pilot should include users who can identify domain-specific false claims.
Pause when the corpus is too small, too unstable, or too weakly governed for reliable graph construction. If most questions are direct lookups, improve metadata, lexical search, and citations first. Also pause if the graph is being used to hide weak source quality. A sophisticated graph cannot make contradictory or outdated documents trustworthy. For an enterprise knowledge-port product, it is often better to expose source dates, ownership, and unresolved conflicts than to present one synthesized answer without provenance.
The decision to expand should be based on three gates: quality, economics, and user behavior. Quality requires better evidence coverage or fewer unsupported claims than the baseline. Economics requires acceptable cost per accepted answer and predictable latency. User behavior requires that people actually use the graph-derived results, inspect citations, and report fewer downstream errors. If the graph is technically active but users still export raw search results, the feature may be a demonstration rather than a product capability. Mentorship and learning teams can use the same measures to test whether a recommended concept path is accurate, whether a learner can explain the reasoning, and whether the system reduces time spent searching for prerequisite material.
The final recommendation for 2026 is to evaluate GraphRAG as a system, not as a model. Use Recall@k, nDCG or MRR, entity-linking and relation measures, citation precision, groundedness, completeness, expert acceptance, latency, and cost per accepted answer. Publish a baseline, a frozen test set, a reproducible configuration, and an error taxonomy. Choose GraphRAG for relationship-heavy work, use hybrid retrieval for simple lookups, and retain human review where the cost of a wrong answer is high. This approach does not assume that graph retrieval is always better; it establishes whether the additional graph layer earns its place in the enterprise workflow.