Unfamiliar Finetuning Examples Control How Language Models Hallucinate

★

☆

SHARE

Overview

Paper Summary

Conflicts of Interest

Identified Weaknesses

Rating Explanation

Good to know

Topic Hierarchy

File Information

Paper Summary

Paperzilla title

LLM Hallucinations Mimic Unfamiliar Training Data

This paper finds that unfamiliar examples in an LLM's finetuning data significantly influence its hallucinations, with the model's predictions mirroring responses associated with these examples. This suggests that manipulating the finetuning data could steer the model towards more desirable responses, like expressing uncertainty when it doesn't know.

Possible Conflicts of Interest

None identified

Identified Weaknesses

Focus on Question Answering Tasks

The study primarily uses question-answering tasks as testbeds, which may not fully represent the complexity of long-form generation where hallucinations are more prevalent.

Limited Scope of Unfamiliarity

The paper defines unfamiliar inputs as those outside the pretrained model's knowledge but within the finetuning data distribution. Real-world queries often fall into a spectrum of partial familiarity, which isn't fully addressed.

Scalability of Conservative Reward Models

While conservative reward models show promise, their reliance on ground-truth rewards for labeling during training poses scalability challenges, especially for large datasets.

Rating Explanation

This paper presents a novel perspective on how LLMs hallucinate and offers a potential solution through conservative reward models. While the focus on QA tasks and the limited scope of unfamiliarity are limitations, the core findings and the proposed approach are valuable contributions to the field.

Good to know

This is our free standard analysis. Paperzilla Pro fact-checks every citation, researches author backgrounds and funding sources, and uses advanced AI reasoning for more thorough insights.

Explore Pro →