人工智能环境下图书编目的挑战与进路
图书馆学理论重构与AI伦理框架
这组文献探讨了AI技术与图书馆学基本理论(如阮冈纳赞五律)的融合,并从宏观角度分析了生成式AI带来的机遇、伦理挑战及未来发展趋势。
- AI and Ranganathan: Modernizing "Save the Time of the Reader"(S. Kundu, 2025, International Journal For Multidisciplinary Research)
- Artificial intelligence and the five laws: a new vision for library science(Dattatraya Kalbande, D. Hemke, N. Motewar, 2025, Library Hi Tech News)
- GENERATIVE AI IN LIBRARIES: OPPORTUNITIES, CHALLENGES, AND FUTURE DIRECTIONS(Hardik Vanik, Dr. Rajeshkumar M. Gamit, 2025, Towards Excellence)
- Navigating the AI Landscape in Libraries: A PRISMA-Based Systematic Analysis of AI Applications in Libraries(K. Vrindha, S. C., 2025, Journal of Web Librarianship)
自动化编目流程与多模态元数据提取
该组论文侧重于AI在编目实务中的具体应用,包括自动分类、罗马拼音转写、科学实验元数据提取、纸质档案数字化及Web文档推荐等技术实现。
- Enhancing Cataloging with Generative AI: Converting Wade-Giles to Pinyin(Li Sun, 2025, Cataloging & Classification Quarterly)
- AI-supported cataloger: a deep dive into intelligent document classification(Yi-Shuai Xu, A. Y. Yanti Idaya, M.S.S. Kassim, 2025, Library Hi Tech News)
- SMART METADATA MANAGEMENT FOR PRINT ARCHIVES(Rohit Chandwaskar, Sathyabalaji Kannan, Srikanta Kumar Sahoo, Rashmi Manhas, Takveer Singh, Ms. Usha Kiran Barla, 2025, ShodhKosh: Journal of Visual and Performing Arts)
- Metadata for Scientific Experiment Reporting: A Case Study in Metal-Organic Frameworks(Xintong Zhao, Kyle Langlois, Jacob Furst, Scott McClellan, Xiaohua Hu, Yuan An, Diego A. Gómez-Gualdrón, Fernando J. Uribe-Romo, Jane Greenberg, 2023, ArXiv Preprint)
- Enhancing Web Document Recommendations with Generative AI and Differential Metadata Selection(Krutharth R Kumar, Neha Honniganur, Shroff Gaurav, M. Ramesh, Numa Rahamath S A, A. Nayak, 2025, 2025 IEEE 17th International Conference on Computational Intelligence and Communication Networks (CICN))
- 基于词项聚类的文本语义标签抽取研究 (Word Clustering Based Text Semantic Tagging Extraction Method)(Xiong Li, Zhiming Ding, Xing Su, Limin Guo, 2018, 计算机科学)
- Linguistic foundation for NLP(Guodong Zhou, 2012, No journal)
元数据建模、关联数据与人机协作模式
这些研究关注如何利用LLM和本体驱动的方法改进元数据建模,探讨从MARC向BIBFRAME/LOD的范式转变,并强调“人机协作”(Human-in-the-loop)在词表开发和模型解构中的重要性。
- A Generative AI-driven Metadata Modelling Approach(Mayukh Bagchi, 2024, ArXiv)
- Toward Generative AI–Driven Metadata Modeling: A Human–Large Language Model Collaborative Approach(Mayukh Bagchi, 2025, Library Trends)
- Human-in-the-Loop and AI: Crowdsourcing Metadata Vocabulary for Materials Science(Jane Greenberg, Scott McClellan, Addy Ireland, Robert Sammarco, Colton Gerber, Christopher B. Rauch, Mat Kelly, John Kunze, Yuan An, Eric Toberer, 2025, ArXiv Preprint)
- Cataloguing, Metadata, and Generative AI. Early Experiences and Future Perspectives(Gino Roncaglia, 2026, JLIS.it)
- Better Recommendations: Validating AI-generated Subject Terms Through LOC Linked Data Service(K. Tang, Yi Jiang, 2025, ArXiv)
大语言模型在编目场景下的性能优化与技术挑战
这组文献深入探讨了LLM在处理编目任务时的技术瓶颈,如幻觉问题、逻辑不一致性、受限生成下的推理能力下降,并提出了RAG(检索增强生成)等优化方案。
- Prompt-Reverse Inconsistency: LLM Self-Inconsistency Beyond Generative Randomness and Prompt Paraphrasing(Jihyun Janice Ahn, Wenpeng Yin, 2025, ArXiv)
- CRANE: Reasoning with constrained LLM generation(Debangshu Banerjee, Tarun Suresh, Shubham Ugare, Sasa Misailovic, Gagandeep Singh, 2025, ArXiv)
- RETA-LLM: A Retrieval-Augmented Large Language Model Toolkit(Jiongnan Liu, Jiajie Jin, Zihan Wang, Jiehan Cheng, Zhicheng Dou, Ji-Rong Wen, 2023, ArXiv Preprint)
编目人员角色转型与行业应用感知
该组文献通过案例研究和调查,探讨了编目专业人员对AI工具的接受度、职业认知的变化,以及AI在不同规模图书馆(如社区图书馆、学术图书馆)中的应用实践。
- AI in Technical Services(Brinna Michael, 2025, TCB: Technical Services in Religion & Theology)
- AI for Cataloging and Metadata Creation: Perspectives and Future Opportunities from Cataloging and Metadata Professionals(Suzhen Chen, Mingyan Li, 2024, Technical Services Quarterly)
- Integration of Ai-driven Tools in Academic Library Services(A. Hasan, 2025, International Journal For Multidisciplinary Research)
- Bridging The Digital Divide: A Framework For AIPowered Cataloging And Patron Engagement In Small Community Libraries(Shaunit Bajoria, 2025, IOSR Journal of Computer Engineering)
- Bringing DeepMind Technology to the Table: Envisioning Library Services Using DeepMind Visualization AI(Noor Abutayeh, 2024, Public Library Quarterly)
AI生成资源的编目规范与质量控制
这组论文探讨了如何对AI生成的资源(如AI图像)进行描述和编目,涉及作者身份界定、编目标准(如PCC指南)的演进以及元数据质量评估准则。
- Cataloging Computer-Generated Resources: Past, Present, and Possible Future(Sarah Hovde, 2025, Cataloging & Classification Quarterly)
- Assessing and optimising metadata quality for AI-generated images: A comparative analysis of generative AI and image recognition approaches(Akara Thammastitkul, 2025, Journal of Information Science)
本组文献全面覆盖了人工智能环境下图书编目领域的多个维度:从阮冈纳赞五律的理论焕新到自动化分类、转写等具体业务流程的优化;从元数据模型的本体重构到人机协作的词表构建;从解决LLM幻觉与不一致性的技术进路到AI生成资源编目规范的建立。同时,文献还深入调研了馆员的职业感知与角色转型,为构建智能化、规范化且符合伦理的现代图书馆编目体系提供了理论支撑与实践指南。
总计26篇相关文献
Abstract Generative AI tools are transforming library cataloging by improving efficiency and expanding access to multilingual resources. This study examines the application of AI in automating the transliteration of Wade-Giles romanized titles into Pinyin in MARC records. Three generative AI tools—ChatGPT, Copilot, and Gemini—were evaluated for their efficiency, and they exhibited distinct strengths and limitations. The study highlights persistent challenges, including inconsistent outputs, fluctuating accuracy levels, and the need for data pre-processing. These findings paint a promising future for AI integration in cataloging workflows but also highlight the need for continuous refinement to achieve scalable performance optimization.
ABSTRACT This study examined the role of artificial intelligence (AI) in the field of cataloging and metadata creation through a case study approach. It explored the perceptions of cataloging and metadata professionals regarding the application and effectiveness of AI in their job duties, as well as the main challenges they encountered when utilizing AI for cataloging purposes. Quantitative data was collected through surveys to understand the target audience’s perspectives on AI in their work. Qualitative data from open-ended survey questions provided deeper insights. Both data types were analyzed to assess AI’s impact on cataloging and metadata workflows.
Abstract The article discusses how catalogers describe resources generated by artificial intelligence and how these resources could be represented in bibliographic records. It briefly reviews the history and current state of computer-generated resources, as well as the concept of authorship in cataloging standards. It then provides an overview of the history of cataloging approaches to non-human authors, including the PCC’s recent FAQ on “Cataloging of Resources Generated Using Artificial Intelligence (AI) Software.” The article finishes by evaluating the PCC guidance and suggesting a range of alternative and/or additional practices that catalogers could adopt.
The small community libraries, though an important repository of local knowledge and local culture, tend to be severely strained in resources thereby not afforded the ability to apply modern digital technologies. This disparity is creating a digital divide between such institutions and larger, better endowed institutions and influencing the efficiency and involvement of patronage. This issue will be discussed in the paper by creating an elaborate conceptual model of an AI-based system, with a specific focus on the needs of small libraries. The proposed Community-Centric AI Library System (CAILS) integrates Natural Language Processing (NLP) to automatically label the metadata and categorization of subjects and a hybrid recommender system that provides personalized and community-focused recommendations of what to read. The paper adopts the conceptual-analytical method of deconstructing the architecture of CAILS, its key modules, an NLP-based cataloging engine, a privacyconcerned patron profile and recommendation engine, and a user-friendly librarian-in-the-loop interface. We comparatively case study this AI-based solution with the old and traditional way of doing things in the library and how this solution can assist in greatly reducing the workload cue of the librarian both in the cataloging and the reader advisory services. Moreover, we also comment on how the system will enhance the degree of involvement with patrons due to the metamorphosis of the library catalog as a unilateral search engine to an active discovery engine. It is a critical analysis of the colossal implications of such a technology, including professional development of the librarians, becoming more democratic in accessing information, and maintaining the library as a communally-oriented organization. It also covers though, the inherent limitations and ethical concerns, such as algorithmic bias, data privacy and hindrances to practical implementation. The paper ends by saying that being considerate about implementing AI may not only allow the small libraries to overcome the limitations of operational inefficiencies but it will also help them to interact with their communities better and, as such, the inefficiencies in their processes will not be applicable to them but instead they will be as useful and active in the digital world.
Abstract Artificial Intelligence (AI) has become an essential part of modern life, and its impact on libraries is increasingly evident. This study identifies a significant rise in research exploring the intersection of AI and library services in recent years. However, the actual implementation of AI in libraries remains in its early stages at many institutions. The research systematically analyzes scholarly articles on AI in libraries using the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) method. Data were collected from the Scopus and Web of Science databases, with all publications assessed using the SPIDER tool. The quality of the selected articles was evaluated using the Critical Appraisal Skills Programme (CASP) checklist. The findings reveal a notable surge in research activity exploring the relationship between AI and libraries. Specifically, ChatGPT shows potential to enhance library services in areas such as reference services, classification, and cataloging. While AI offers advantages like improved search accuracy, automated cataloging, and content creation, concerns about biased results, privacy, and security persist. Although AI could play a crucial role in future library services, it is not a substitute for human librarians. Key challenges include funding constraints and fears of job loss among library staff. The study also highlights a gap in the development of AI technologies specifically designed for libraries, emphasizing that while AI is valuable, it cannot yet replace the nuanced judgment of a librarian.
This study aims to explore the development of automatic document classification, analyzes its related applications in library contexts, identifies key challenges and emerging opportunities and proposes future research directions. The study examines historical and modern document classification methods. It synthesizes technical progress with insights from real-world cases. Furthermore, it investigates the implementation challenges libraries face and the opportunities offered by emerging technologies and proposes potential directions for future research. Automatic cataloging and document classification have evolved significantly. AI-driven approaches have brought notable improvements in efficiency and consistency. Tools such as Annif, JEX and AutoMSC have demonstrated practical utility in real-world settings. However, several challenges remain, including scarcity of data and annotated resources, insufficient model interpretability, standard incompatibility and ethical concerns. Meanwhile, opportunities are emerging through human-AI collaboration, data-sharing initiatives, open-access models and advancements in multimodal and multilingual technologies. This paper brings together recent developments and real-world examples to provide a comprehensive view of how automatic document classification and cataloging are evolving in libraries today. It provides insights for researchers, library professionals and technologists working to build more intelligent, equitable and inclusive library services.
study examined the role of artificial intelligence (AI) in the field of cataloging and meta-data creation through a case study approach. It explored the perceptions of cataloging and metadata professionals regarding the application and effectiveness of AI in their job duties, as well as the main challenges they encountered when utilizing AI for
Generative artificial intelligence (AI) heralds a transformative era for libraries, redefining their role as knowledge hubs through innovative content creation. This conceptual paper explores how generative AI can revolutionize library services by enhancing personalization, efficiency, and innovation. Focusing on theoretical insights without empirical data, it examines opportunities, challenges, and future trends, drawing from library science and AI ethics literature. Key opportunities include AI-driven personalized recommendations, enabling tailored reading lists, and virtual storytelling, fostering inclusive user engagement (Cox 425). Operational efficiencies, such as automated metadata generation, streamline cataloging, enhancing resource accessibility (Hadi et al. 5). Challenges encompass ethical dilemmas like algorithmic bias, which risks marginalizing voices, and privacy concerns tied to data-intensive models (Bender et al. 612). Implementation barriers, including integration costs and staff training needs, further complicate adoption (Massaro 45). Future directions propose multimodal AI and federated learning to create privacy-conscious, immersive services, alongside policy frameworks for ethical integration (UNESCO 12). The paper advocates for cautious adoption, emphasizing transparency and equity to align AI with libraries’ democratic mission. Results highlight generative AI’s potential to transform libraries into dynamic, co-creative ecosystems while underscoring the need for ethical oversight to mitigate societal risks like misinformation. This framework offers actionable insights for librarians, policymakers, and researchers, positioning libraries as leaders in the AI-driven information landscape.
This article explores the integration of AI-generated subject terms into library cataloging, focusing on validation through the Library of Congress Linked Data Service. It examines the challenges of traditional subject cataloging under the Library of Congress Subject Headings system, including inefficiencies and cataloging backlogs. While generative AI shows promise in expediting cataloging workflows, studies reveal significant limitations in the accuracy of AI-assigned subject headings. The article proposes a hybrid approach combining AI technology with human validation through LOC Linked Data Service, aiming to enhance the precision, efficiency, and overall quality of metadata creation in library cataloging practices.
Artificial Intelligence (AI) in academic libraries is transforming the landscape of information management and service delivery. This research article explores the opportunities and challenges presented by AI-driven tools in enhancing academic library services. It examines different useful AI applications, such as chatbots, recommendation systems, automated cataloging, and predictive analytics, and their impact on improving user experience, streamlining workflows, and optimizing collection management. The study also addresses the ethical considerations and potential challenges associated with AI integration, including data privacy, and the changing roles of librarians. The findings of this research add to the expanding body of knowledge on the strategic implementation of AI technologies in academic library services to satisfy the evolving needs of library users in the digital era.
Abstract: The emergence of Artificial Intelligence (AI) has transformed the landscape of information management, access, and retrieval in libraries. This paper explores how AI-driven technologies modernize Dr. S. R. Ranganathan’s timeless Fourth Law of Library Science — “Save the Time of the Reader.” By integrating machine learning, natural language processing, and recommender systems, libraries today can anticipate user needs, automate cataloging, personalize information services, and enhance search precision. The study highlights how AI applications such as chatbots, semantic search tools, and intelligent metadata management not only expedite user access but also redefine the librarian’s role in the digital age. Through this synergy between Ranganathan’s principles and AI innovation, libraries can continue to uphold their foundational mission of facilitating efficient and meaningful access to knowledge
ABSTRACT Traditional knowledge sources are undergoing digital transformation, prompting libraries to adapt to new technological requirements. This article focuses on the potential of DeepMind Visualization AI to revolutionize library services by exploring its impact on information access, user experience, and operational efficiency. Specifically, the article examines how this advanced AI technology is set to enhance library cataloging, user support systems, and community engagement initiatives. Additionally, ethical considerations and challenges in integrating DeepMind Visualization AI into library operations are addressed. Through these discussions, it becomes evident that DeepMind’s visualization AI promises to redefine library services by enhancing efficiency, usability, and fostering stronger community connections.
This conceptual paper reinterprets S.R. Ranganathan’s five laws of library science in the context of artificial intelligence (AI), examining their continued relevance and adaptability in the digital age. By aligning AI capabilities with these foundational principles, this paper aims to explore how AI can enhance information access, optimize resource management and personalize library services while maintaining the ethical and philosophical core of Library and Information Science (LIS). This study uses a conceptual analysis approach to critically examine AI applications in LIS, including automated cataloging, AI-driven search systems, personalized recommendations and intelligent chatbots. It also addresses ethical considerations such as algorithmic bias, data privacy and equitable access. This paper proposes an AI-enhanced reinterpretation of Ranganathan’s laws, offering a guiding framework for responsible AI adoption in libraries. This study highlights the transformative potential of AI in libraries, demonstrating its ability to improve operational efficiency, user engagement and accessibility. However, it also emphasizes the necessity of aligning AI implementation with ethical principles to prevent biases and ensure inclusivity. By conceptualizing an AI-driven adaptation of Ranganathan’s laws, this paper provides a roadmap for integrating AI into library services without compromising their core values. This research offers a novel perspective by reconceptualizing Ranganathan’s five laws in the era of AI, providing LIS professionals with a theoretical framework to guide AI integration. It contributes to the discourse on ethical and sustainable AI adoption in libraries, ensuring that technological advancements support rather than undermine traditional LIS principles.
Metacognitive metadata management of print archives is one of the essential steps that must be taken to integrate the old archival methodology with the new digital intelligence. With the shift of the cultural institution and libraries towards dynamic digital environments, replacing the static cataloging systems with the dynamic digital ecosystems, the needs of making the metadata efficient, accurate, and information-rich increases manifold. This paper will describe an intelligent model of metadata management, which uses Artificial Intelligence (AI), Optical Character Recognition (OCR), Natural Language Processing (NLP), and linked data frameworks to perform intelligent work on archives. The framework suggested combines automated metadata extraction, real-time validation, semantic enrichment, and entity recognition based on machine learning pipelines that are compatible with existing metadata standards (dublin core, MARC, METS and PREMIS). A prototype implementation presents the ways AI-based workflows can be applied to facilitate metadata completeness, interoperability, and retrieval accuracy in digitized print repositories. Scanned archival data has been evaluated through experimental evaluations where there is a significant difference in accuracy and efficiency compared to traditional manual cataloging systems. Performance insights, system architecture, scalability and semantic alignment issues and long-term preservation implications are also addressed in the study. Finally, the study will add a model of the future that unites automation with semantic intelligence, improves discoverability, sustainability, and accessibility of cultural heritage collections with smart metadata governance and intelligent interoperability of archival.
No abstract available
This paper presents a novel framework for web document recommendation that integrates generative AI methods with differential metadata selection. The approach combines advanced natural language processing techniques including transformer-based large language models, TF-IDF, and decision tree classifiers to enhance the semantic understanding and relevance of recommended documents. The system constructs hierarchical ontologies and applies metaheuristic optimization to produce personalized and diverse recommendations in large-scale scientific archives. Experimental evaluations demonstrate superior precision, recall, and overall effectiveness compared to existing recommendation models. The framework supports continual updating and adaptation to evolving user preferences and document collections, offering a robust solution for academic and research document discovery.
The quality of metadata is critical for the effective organisation, retrieval and management of increasingly large repositories of AI-generated images. However, the absence of inherent contextual information in such images presents significant challenges in ensuring accurate and relevant metadata. This study evaluates the effectiveness of generative AI and image recognition approaches in improving metadata quality, based on 10 comprehensive criteria: semantic accuracy and relevance; thematic consistency and contextual adaptability; user-centric language and accessibility; consistency across data sets; emotional and aesthetic alignment; inclusivity and bias control; error detection and quality control; retrievability and search optimization; interpretive depth and granularity; and statistical performance measures. Results reveal that generative AI is particularly effective in generating contextually rich, adaptive keywords, while image recognition demonstrates superior object identification precision. Based on these insights, we propose a hybrid approach that integrates the contextual strengths of generative AI with the accuracy of image recognition. This hybrid method achieves significant improvements across all evaluations, including higher precision, recall, F1-score, cosine similarity and Jaccard index. By optimising metadata quality, the proposed approach facilitates more accurate and accessible retrieval of AI-generated images, significantly enhancing their usability and integration across diverse digital platforms.
Abstract:For decades, the modeling of metadata has been core to the functioning of any academic library. Metadata’s importance has only increased with the pervasiveness of generative artificial intelligence–driven information activities and services. However, several challenges impact a library metadata model’s reusability, crosswalk, and interoperability with other metadata models. This paper posits that these problems stem from an underlying assumption that there should be only a few core metadata models that would be sufficient for any information service using them, irrespective of the heterogeneity of intradomain or interdomain settings. To that end, this paper advances a contrary view and substantiates its argument in three key steps. First, the paper introduces a novel way of thinking about a library metadata model as an ontology-driven composition of five functionally interlinked representation levels from perception to definition via properties. Second, the paper introduces the representational manifoldness implicit in each of the five levels, which cumulatively contributes to a conceptually entangled library metadata model. Finally, and most importantly, the paper proposes a generative AI–driven, human–large language model collaboration-based metadata modeling approach to disentangle the entanglement inherent in each representation level, which would lead to a conceptually disentangled metadata model. Throughout the paper, the author provides motivating scenarios and examples from libraries handling cancer information.
Since decades, the modelling of metadata has been core to the functioning of any academic library. Its importance has only enhanced with the increasing pervasiveness of Generative Artificial Intelligence (AI)-driven information activities and services which constitute a library's outreach. However, with the rising importance of metadata, there arose several outstanding problems with the process of designing a library metadata model impacting its reusability, crosswalk and interoperability with other metadata models. This paper posits that the above problems stem from an underlying thesis that there should only be a few core metadata models which would be necessary and sufficient for any information service using them, irrespective of the heterogeneity of intra-domain or inter-domain settings. To that end, this paper advances a contrary view of the above thesis and substantiates its argument in three key steps. First, it introduces a novel way of thinking about a library metadata model as an ontology-driven composition of five functionally interlinked representation levels from perception to its intensional definition via properties. Second, it introduces the representational manifoldness implicit in each of the five levels which cumulatively contributes to a conceptually entangled library metadata model. Finally, and most importantly, it proposes a Generative AI-driven Human-Large Language Model (LLM) collaboration based metadata modelling approach to disentangle the entanglement inherent in each representation level leading to the generation of a conceptually disentangled metadata model. Throughout the paper, the arguments are exemplified by motivating scenarios and examples from representative libraries handling cancer information.
The article deals with the intersection of generative artificial intelligence (AI) and bibliographic/metadata practices, assessing how large language models (LLMs) can support cataloguing and metadata creation while navigating the constraints of formal knowledge architectures. In the first section, the article discusses the evolution of cataloguing paradigms from MARC to Linked Open Data (LOD), emphasizing the shift from rigid records to semantic, entity-based models like FRBR, RDA, and BIBFRAME. The second section deals with the epistemological clash between deterministic, rule-based metadata standards (the "architect") and probabilistic, generative AI systems (the "oracle").Three strategies are proposed for integrating AI into bibliographic workflows:1) Specialized AI systems trained exclusively on controlled, high-quality datasets.2) Retrieval-Augmented Generation (RAG), blending LLMs with authoritative knowledge bases.3) Next-generation LLMs enhanced via reasoning models, multimodal inputs, expanded context windows, and small/medium-scale local models to align generative outputs with metadata standards.Key challenges include hallucinations, data sparsity in bibliographic corpora, and the obsolescence of MARC-centric experiments. The article argues for caution against retrofitting AI onto outdated data models, urging alignment with LOD and IFLA’s Library Reference Model (LRM). Ethical considerations (bias, transparency, AI literacy) and the potential of local SLMs/MSLMs for privacy-sensitive applications are highlighted.
Code generation, symbolic math reasoning, and other tasks require LLMs to produce outputs that are both syntactically and semantically correct. Constrained LLM generation is a promising direction to enforce adherence to formal grammar, but prior works have empirically observed that strict enforcement of formal constraints often diminishes the reasoning capabilities of LLMs. In this work, we first provide a theoretical explanation for why constraining LLM outputs to very restrictive grammars that only allow syntactically valid final answers reduces the reasoning capabilities of the model. Second, we demonstrate that by augmenting the output grammar with carefully designed additional rules, it is always possible to preserve the reasoning capabilities of the LLM while ensuring syntactic and semantic correctness in its outputs. Building on these theoretical insights, we propose a reasoning-augmented constrained decoding algorithm, CRANE, which effectively balances the correctness of constrained generation with the flexibility of unconstrained generation. Experiments on multiple open-source LLMs and benchmarks show that CRANE significantly outperforms both state-of-the-art constrained decoding strategies and standard unconstrained decoding, showing up to 10% points accuracy improvement over baselines on challenging symbolic reasoning benchmarks GSM-symbolic and FOLIO.
While the inconsistency of LLMs is not a novel topic, prior research has predominantly addressed two types of generative inconsistencies: i) Randomness Inconsistency: running the same LLM multiple trials, yielding varying responses; ii) Paraphrase Inconsistency: paraphrased prompts result in different responses from the same LLM. Randomness Inconsistency arises from the inherent randomness due to stochastic sampling in generative models, while Paraphrase Inconsistency is a consequence of the language modeling objectives, where paraphrased prompts alter the distribution of vocabulary logits. This research discovers Prompt-Reverse Inconsistency (PRIN), a new form of LLM self-inconsistency: given a question and a couple of LLM-generated answer candidates, the LLM often has conflicting responses when prompted"Which are correct answers?"and"Which are incorrect answers?". PRIN poses a big concern as it undermines the credibility of LLM-as-a-judge, and suggests a challenge for LLMs to adhere to basic logical rules. We conduct a series of experiments to investigate PRIN, examining the extent of PRIN across different LLMs, methods to mitigate it, potential applications, and its relationship with Randomness Inconsistency and Paraphrase Inconsistency. As the first study to explore PRIN, our findings offer valuable insights into the inner workings of LLMs and contribute to advancing trustworthy AI.
Metadata vocabularies are essential for advancing FAIR and FARR data principles, but their development constrained by limited human resources and inconsistent standardization practices. This paper introduces MatSci-YAMZ, a platform that integrates artificial intelligence (AI) and human-in-the-loop (HILT), including crowdsourcing, to support metadata vocabulary development. The paper reports on a proof-of-concept use case evaluating the AI-HILT model in materials science, a highly interdisciplinary domain Six (6) participants affiliated with the NSF Institute for Data-Driven Dynamical Design (ID4) engaged with the MatSci-YAMZ plaform over several weeks, contributing term definitions and providing examples to prompt the AI-definitions refinement. Nineteen (19) AI-generated definitions were successfully created, with iterative feedback loops demonstrating the feasibility of AI-HILT refinement. Findings confirm the feasibility AI-HILT model highlighting 1) a successful proof of concept, 2) alignment with FAIR and open-science principles, 3) a research protocol to guide future studies, and 4) the potential for scalability across domains. Overall, MatSci-YAMZ's underlying model has the capacity to enhance semantic transparency and reduce time required for consensus building and metadata vocabulary development.
Research methods and procedures are core aspects of the research process. Metadata focused on these components is critical to supporting the FAIR principles, particularly reproducibility. The research reported on in this paper presents a methodological framework for metadata documentation supporting the reproducibility of research producing Metal Organic Frameworks (MOFs). The MOF case study involved natural language processing to extract key synthesis experiment information from a corpus of research literature. Following, a classification activity was performed by domain experts to identify entity-relation pairs. Results include: 1) a research framework for metadata design, 2) a metadata schema that includes nine entities and two relationships for reporting MOF synthesis experiments, and 3) a growing database of MOF synthesis reports structured by our metadata scheme. The metadata schema is intended to support discovery and reproducibility of metal-organic framework research and the FAIR principles. The paper provides background information, identifies the research goals and objectives, research design, results, a discussion, and the conclusion.
Although Large Language Models (LLMs) have demonstrated extraordinary capabilities in many domains, they still have a tendency to hallucinate and generate fictitious responses to user requests. This problem can be alleviated by augmenting LLMs with information retrieval (IR) systems (also known as retrieval-augmented LLMs). Applying this strategy, LLMs can generate more factual texts in response to user input according to the relevant content retrieved by IR systems from external corpora as references. In addition, by incorporating external knowledge, retrieval-augmented LLMs can answer in-domain questions that cannot be answered by solely relying on the world knowledge stored in parameters. To support research in this area and facilitate the development of retrieval-augmented LLM systems, we develop RETA-LLM, a {RET}reival-{A}ugmented LLM toolkit. In RETA-LLM, we create a complete pipeline to help researchers and users build their customized in-domain LLM-based systems. Compared with previous retrieval-augmented LLM systems, RETA-LLM provides more plug-and-play modules to support better interaction between IR systems and LLMs, including {request rewriting, document retrieval, passage extraction, answer generation, and fact checking} modules. Our toolkit is publicly available at https://github.com/RUC-GSAI/YuLan-IR/tree/main/RETA-LLM.
本组文献全面覆盖了人工智能环境下图书编目领域的多个维度:从阮冈纳赞五律的理论焕新到自动化分类、转写等具体业务流程的优化;从元数据模型的本体重构到人机协作的词表构建;从解决LLM幻觉与不一致性的技术进路到AI生成资源编目规范的建立。同时,文献还深入调研了馆员的职业感知与角色转型,为构建智能化、规范化且符合伦理的现代图书馆编目体系提供了理论支撑与实践指南。