OpenEvals Simplifies LLM Evaluation Process for Developers

Zach Anderson
Feb 26, 2025 12:07

LangChain introduces OpenEvals and AgentEvals to streamline evaluation processes for large language models, offering pre-built tools and frameworks for developers.

LangChain, a prominent player in the field of artificial intelligence, has launched two new packages, OpenEvals and AgentEvals, aimed at simplifying the evaluation process for large language models (LLMs). These packages provide developers with a robust framework and a set of evaluators to streamline the assessment of LLM-powered applications and agents, according to LangChain.

Understanding the Role of Evaluations

Evaluations, often referred to as evals, are crucial in determining the quality of LLM outputs. They involve two primary components: the data being evaluated and the metrics used for evaluation. The quality of the data significantly impacts the evaluation’s ability to reflect real-world usage. LangChain emphasizes the importance of curating a high-quality dataset tailored to specific use cases.

The metrics for evaluation are typically customized based on the application’s goals. To address common evaluation needs, LangChain developed OpenEvals and AgentEvals, sharing pre-built solutions that highlight prevalent evaluation trends and best practices.

Common Evaluation Types and Best Practices

OpenEvals and AgentEvals focus on two main approaches to evaluations:

1. Customizable Evaluators: The LLM-as-a-judge evaluations, which are widely applicable, allow developers to adapt pre-built examples to their specific needs.
2. Specific Use Case Evaluators: These are designed for particular applications, such as extracting structured content from documents or managing tool calls and agent trajectories. LangChain plans to expand these libraries to include more targeted evaluation techniques.

LLM-as-a-Judge Evaluations

LLM-as-a-judge evaluations are prevalent due to their utility in assessing natural language outputs. These evaluations can be reference-free, enabling objective assessment without needing ground truth answers. OpenEvals aids this process by providing customizable starter prompts, incorporating few-shot examples, and generating reasoning comments for transparency.

Structured Data Evaluations

For applications that require structured output, OpenEvals offers tools to ensure the model’s output adheres to a predefined format. This is crucial for tasks such as extracting structured information from documents or validating parameters for tool calls. OpenEvals supports exact match configuration or LLM-as-a-judge validation for structured outputs.

Agent Evaluations: Trajectory Evaluations

Agent evaluations focus on the sequence of actions an agent takes to accomplish a task. This involves assessing tool selection and the trajectory of applications. AgentEvals provides mechanisms to evaluate and ensure agents are using the correct tools and following the appropriate sequence.

Tracking and Future Developments

LangChain recommends using LangSmith for tracking evaluations over time. LangSmith offers tools for tracing, evaluation, and experimentation, supporting the development of production-grade LLM applications. Notable companies like Elastic and Klarna utilize LangSmith to evaluate their GenAI applications.

LangChain’s initiative to codify best practices continues, with plans to introduce more specific evaluators for common use cases. Developers are encouraged to contribute their own evaluators or suggest improvements via GitHub.

Image source: Shutterstock