An overview of how we think about context graphs at
@harvey.
Context graphs are a promising direction for improving agent quality and efficiency. We ran a series of experiments on Legal Agent Bench to evaluate how structured context improves online agent performance.
For each LAB task we evaluated, an agent first reviews each document in its client matter and builds a graph of relevant entities and their relationships – e.g. parties, dates, draft vs. executed status, which versions belong to the same contract.
To evaluate the impact of the graph, we compared the performance of an agent with and without the context graph, using LAB’s standard grading setup. Overall, the agent with the context graph improved rubric pass rate by up to 30% while reducing task cost by up to ~50%.
In some cases the graph led to massive improvements. For example, in one firm-knowledge task, the agent was asked to find the firm’s most recent secured financing and retrieve the executed credit and security agreements as precedents for a new facility.
An agent working from raw files attempted to search through several files sequentially, and looked at many different matters, but ultimately selected the wrong precedent (0% of rubric criteria). With the context graph, the same agent found the relevant matter and its executed agreements immediately (75% of rubric criteria).
We’ll have more to share soon on how we’re building this into Harvey, along with more experiments like this one!