HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attribution¶
HAGRID：用于具有归因的生成信息搜索的人类LLM协作数据集
The rise of large language models (LLMs) had a transformative impact on search, ushering in a new era of search engines that are capable of generating search results in natural language text, imbued with citations for supporting sources. Building generative information-seeking models demands openly accessible datasets, which currently remain lacking. In this paper, we introduce a new dataset, HAGRID (Human-in-the-loop Attributable Generative Retrieval for Information-seeking Dataset) for building end-to-end generative information-seeking models that are capable of retrieving candidate quotes and generating attributed explanations. Unlike recent efforts that focus on human evaluation of black-box proprietary search engines, we built our dataset atop the English subset of MIRACL, a publicly available information retrieval dataset. HAGRID is constructed based on human and LLM collaboration. We first automatically collect attributed explanations that follow an in-context citation style using an LLM, i.e. GPT-3.5. Next, we ask human annotators to evaluate the LLM explanations based on two criteria: informativeness and attributability. HAGRID serves as a catalyst for the development of information-seeking models with better attribution capabilities.
大型语言模型 （LLMs） 的兴起对搜索产生了变革性的影响，开创了搜索引擎的新时代，这些搜索引擎能够以自然语言文本生成搜索结果，并充满对支持来源的引用。构建生成式信息搜索模型需要可公开访问的数据集，而目前仍然缺乏这些数据集。在本文中，我们引入了一种新的数据集HAGRID（Human-in-the-loop Attributable Generative Retrieval for Information-seeking Dataset），用于构建能够检索候选引语和生成归因解释的端到端生成信息搜索模型。与最近专注于对黑匣子专有搜索引擎进行人工评估的工作不同，我们在MIRACL的英文子集之上构建了我们的数据集，MIRACL是一个公开的信息检索数据集。HAGRID是建立在人与LLM协作的基础上的。我们首先使用 LLM，即 GPT-3.5 自动收集遵循上下文引用样式的归因解释。接下来，我们要求人类注释者根据两个标准评估解释LLM：信息性和可归因性。HAGRID是开发具有更好归因能力的信息搜索模型的催化剂。