<ul data-eligibleForWebStory="true"><li>Recent copyright agreements highlight the need for controlling language models' reproduction of copyrighted text.</li><li>Existing methods sacrifice model utility or fail to adequately prevent verbatim leakage.</li><li>A new method called Obliviate is introduced to selectively suppress exact reproduction of specified sequences while maintaining semantic understanding.</li><li>Obliviate identifies memorized passages and adjusts the model's output distribution to reduce the probability of exact reproduction using a Kullback-Leibler divergence penalty.</li><li>Consistency loss is enforced on non-target tokens to preserve fluency and task performance.</li><li>Obliviate is evaluated on various models using synthetic memorization benchmarks and copyrighted excerpts like Moby Dick and Alice in Wonderland.</li><li>It significantly reduces verbatim recall while minimally affecting downstream accuracy on different benchmarks.</li><li>The method is compared against other unlearning and copyright techniques and proves effective in ensuring copyright compliance in language models.</li></ul>

Obliviate: Efficient Unmemorization for Protecting Intellectual Property in Large Language Models

Discover more