A naukri.com initiative
Medium
1w
204
Image Credit: Medium
Entropy, Parrots and Probability
- The language model ChatGPT gathers information from almost everything published online, with the aim of predicting the next word in a sentence using probability scores. However, the task of understanding spoken language is incredibly challenging, given the endless possible word combinations and the vagaries of linguistic meaning.
- To make sense of human language patterns, ChatGPT relies on one of computer science's greatest contributions, Claude Shannon's information theory, which is built on four key elements: stochastic data, probability, surprise and entropy.
- Stochastic data is random and unpredictable, meaning that probabilities are a key element in understanding how likely a particular outcome might be in any given context.
- Using probabilities which are the backbone of stochastic data, ChatGPT can help form an understanding of how likely a particular outcome might be. For instance, the probability of rain, or with computational models the possible outcomes of a stock price might be.
- Surprise is closely linked to the concept of information content; outcomes that are less likely to occur carry more 'information' when they do are not expected.
- Entropy refers to the extent of randomness and is a crucial tool for language models as it allows them to predict and categorize unexpected and unseen data. Information theory offers a framework for modeling complex data sets and finding constructive ways to solve them.
- Given that no two individuals use language in exactly the same way, the sheer scale of the task faced by ChatGPT is enormous. Nonetheless, using the tools offered by information theory, progress is being made in tackling the challenge of deciphering human language.
- It is hoped that the greater the model's accuracy becomes, the more it will enhance computational intelligence and support new paradigms for building smarter and more nuanced agents.
- AI developers use information theory for chunking and organizing data sets to develop an understanding in RAG training solutions, for instance, and to identify where to split data and tackle the challenges in smaller increments.
- To improve probabilities scores of data, developers must find ways to quantify the information within and around the machine's challenge. This is where statistical tools like Shannon's information theory can immensely contribute.
- To sum it up, ChatGPT's ability to understand human language is probabilistic, and improvements are being made with the aid of theories like information theory and its crucial elements: probability, surprise, stochastic data, and entropy.
Read Full Article
12 Likes
For uninterrupted reading, download the app