A naukri.com initiative
Arxiv
1d
188
Image Credit: Arxiv
Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2B
- In-Context Learning (ICL) is an intriguing ability of large language models (LLMs).
- Research finds that Gemma-2 2B uses a two-step strategy, contextualize-then-aggregate, for task information assembly.
- In the lower layers, the model builds up representations of individual fewshot examples, contextualized by preceding examples.
- In the higher layers, these representations are aggregated to identify the task and prepare predictions.
Read Full Article
11 Likes
For uninterrupted reading, download the app