Meeseeks – Meituan’s open-source benchmark for evaluating model instruction-following capabilities

What is Meeseeks？

Meeseeks is an open-source large model benchmark released by Meituan’s M17 team, designed to evaluate a model’s instruction-following capabilities. Meeseeks uses a three-tier evaluation framework to comprehensively assess whether models generate responses strictly according to user instructions, without evaluating the factual correctness of the content. It incorporates a multi-turn correction mode, allowing models to revise their outputs based on feedback, testing their self-correction ability. By using objective evaluation criteria and eliminating vague instructions, Meeseeks ensures consistency and accuracy in results. Its challenging dataset design effectively differentiates model performance and provides developers with clear directions for optimization.

Main Features of Meeseeks

• Instruction-Following Evaluation:

  • Tier 1: Evaluates whether the model correctly understands the core task intent, whether the overall structure of the response meets the instructions, and whether each individual unit adheres to instruction details.

  • Tier 2: Focuses on the execution of specific constraints, such as content constraints (topic, style, language, word count, etc.) and format constraints (template compliance, number of units, etc.).

  • Tier 3: Assesses adherence to fine-grained rules, such as rhyming, keyword avoidance, prohibition of repetition, and symbol usage.

• Multi-Turn Correction Mode: If the model’s first response does not fully satisfy all instructions, the evaluation framework generates clear feedback, indicating which instruction items were unmet and requiring the model to revise the answer accordingly.

• Objective Evaluation Standards: Eliminates vague instructions; all evaluation items are objectively determinable, ensuring consistency and accuracy of results.

• Challenging Dataset Design: Test cases are designed to be difficult, effectively differentiating models and providing developers with clear directions for improvement.

Technical Principles of Meeseeks

• Three-Tier Evaluation Framework:

  • Tier 1: Uses natural language processing (NLP) techniques to parse user instructions and extract core task intent and structural requirements. For example, intent recognition algorithms determine whether the model understands a task such as “generate nicknames.”

  • Tier 2: Checks the model’s response against content and format constraints. For example, text analysis algorithms verify whether a generated review meets word count limits or follows a specified style.

  • Tier 3: Performs fine-grained rule checks on the model’s response. For example, regular expressions verify whether a review contains prohibited words or complies with specific writing techniques.

Project Links

• GitHub Repository: https://github.com/ADoublLEN/Meeseeks

• HuggingFace Dataset: https://huggingface.co/datasets/meituan/Meeseeks

Application Scenarios for Meeseeks

• Model Evaluation and Optimization: Provides standardized evaluation of instruction-following capabilities, helping developers identify and improve weaknesses in understanding and executing instructions.

• Model Training and Fine-Tuning: Meeseeks datasets and multi-turn correction feedback can supplement training and guide fine-tuning, improving real-world performance.

• Model Deployment and Applications: Evaluates whether models can strictly follow user instructions in content generation, customer service, education, and other scenarios, producing high-quality, compliant outputs.

• Model Research and Analysis: Serves as a standardized benchmark to support academic research and industry analysis, enabling in-depth study of model performance differences and improvement methods.

• Model Safety and Compliance: Assesses compliance of generated content, helping ensure outputs meet legal, ethical, and privacy standards.