Tokenization

tokenization

Here is a live Tokenizer called Tiktokenizer

Tokens are the fundemantal unit, the “atom” of Large Language Models (LLMs). Tokenization is the process of translating strings (i.e. text) and converting them into sequences of tokens and vice versa.

text = "The scaling up of AI models has two major consequences. First, AI models are becoming more powerful and capable of more tasks, enabling more applications. More people and teams leverage AI to increase productivity, create economic value, and improve quality of life. Second, training large language models (LLMs) requires data, compute resources, and specialized talent that only a few organizations can afford. This has led to the emergence of model as a service: models developed by these few organizations are made available for others to use as a service. Anyone who wishes to leverage AI to build applications can now use these models to do so without having to invest up front in building a model. In short, the demand for AI applications has increased while the barrier to entry for building AI applications has decreased. This has turned AI engineering—the process of building applications on top of readily available models—into one of the fastest-growing engineering disciplines. Building applications on top of machine learning (ML) models isn’t new. Long before LLMs became prominent, AI was already powering many applications, including product recommendations, fraud detection, and churn prediction. While many principles of productionizing AI applications remain the same, the new generation of large-scale, readily available models brings about new possibilities and new challenges, which are the focus of this book. This chapter begins with an overview of foundation models, the key catalyst behind the explosion of AI engineering. I’ll then discuss a range of successful AI use cases, each illustrating what AI is good and not yet good at. As AI’s capabilities expand daily, predicting its future possibilities becomes increasingly challenging. However, existing application patterns can help uncover opportunities today and offer clues about how AI may continue to be used in the future. To close out the chapter, I’ll provide an overview of the new AI stack, including what has changed with foundation models, what remains the same, and how the role of an AI engineer today differs from that of a traditional ML engineer."
tokens = text.encode('utf-8') # raw bytes
tokens = list(map(int, tokens)) # convert to a list of integers 0-255

print("---")
print(text)
print("length of text:", len(text))
print("---")
print(tokens)
print("length of tokens:", len(tokens))

---
The scaling up of AI models has two major consequences. First, AI models are becoming more powerful and capable of more tasks, enabling more applications. More people and teams leverage AI to increase productivity, create economic value, and improve quality of life. Second, training large language models (LLMs) requires data, compute resources, and specialized talent that only a few organizations can afford. This has led to the emergence of model as a service: models developed by these few organizations are made available for others to use as a service. Anyone who wishes to leverage AI to build applications can now use these models to do so without having to invest up front in building a model. In short, the demand for AI applications has increased while the barrier to entry for building AI applications has decreased. This has turned AI engineering—the process of building applications on top of readily available models—into one of the fastest-growing engineering disciplines. Building applications on top of machine learning (ML) models isn’t new. Long before LLMs became prominent, AI was already powering many applications, including product recommendations, fraud detection, and churn prediction. While many principles of productionizing AI applications remain the same, the new generation of large-scale, readily available models brings about new possibilities and new challenges, which are the focus of this book. This chapter begins with an overview of foundation models, the key catalyst behind the explosion of AI engineering. I’ll then discuss a range of successful AI use cases, each illustrating what AI is good and not yet good at. As AI’s capabilities expand daily, predicting its future possibilities becomes increasingly challenging. However, existing application patterns can help uncover opportunities today and offer clues about how AI may continue to be used in the future. To close out the chapter, I’ll provide an overview of the new AI stack, including what has changed with foundation models, what remains the same, and how the role of an AI engineer today differs from that of a traditional ML engineer.
length of text: 2141
---
[84, 104, 101, 32, 115, 99, 97, 108, 105, 110, 103, 32, 117, 112, 32, 111, 102, 32, 65, 73, 32, 109, 111, 100, 101, 108, 115, 32, 104, 97, 115, 32, 116, 119, 111, 32, 109, 97, 106, 111, 114, 32, 99, 111, 110, 115, 101, 113, 117, 101, 110, 99, 101, 115, 46, 32, 70, 105, 114, 115, 116, 44, 32, 65, 73, 32, 109, 111, 100, 101, 108, 115, 32, 97, 114, 101, 32, 98, 101, 99, 111, 109, 105, 110, 103, 32, 109, 111, 114, 101, 32, 112, 111, 119, 101, 114, 102, 117, 108, 32, 97, 110, 100, 32, 99, 97, 112, 97, 98, 108, 101, 32, 111, 102, 32, 109, 111, 114, 101, 32, 116, 97, 115, 107, 115, 44, 32, 101, 110, 97, 98, 108, 105, 110, 103, 32, 109, 111, 114, 101, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 46, 32, 77, 111, 114, 101, 32, 112, 101, 111, 112, 108, 101, 32, 97, 110, 100, 32, 116, 101, 97, 109, 115, 32, 108, 101, 118, 101, 114, 97, 103, 101, 32, 65, 73, 32, 116, 111, 32, 105, 110, 99, 114, 101, 97, 115, 101, 32, 112, 114, 111, 100, 117, 99, 116, 105, 118, 105, 116, 121, 44, 32, 99, 114, 101, 97, 116, 101, 32, 101, 99, 111, 110, 111, 109, 105, 99, 32, 118, 97, 108, 117, 101, 44, 32, 97, 110, 100, 32, 105, 109, 112, 114, 111, 118, 101, 32, 113, 117, 97, 108, 105, 116, 121, 32, 111, 102, 32, 108, 105, 102, 101, 46, 32, 83, 101, 99, 111, 110, 100, 44, 32, 116, 114, 97, 105, 110, 105, 110, 103, 32, 108, 97, 114, 103, 101, 32, 108, 97, 110, 103, 117, 97, 103, 101, 32, 109, 111, 100, 101, 108, 115, 32, 40, 76, 76, 77, 115, 41, 32, 114, 101, 113, 117, 105, 114, 101, 115, 32, 100, 97, 116, 97, 44, 32, 99, 111, 109, 112, 117, 116, 101, 32, 114, 101, 115, 111, 117, 114, 99, 101, 115, 44, 32, 97, 110, 100, 32, 115, 112, 101, 99, 105, 97, 108, 105, 122, 101, 100, 32, 116, 97, 108, 101, 110, 116, 32, 116, 104, 97, 116, 32, 111, 110, 108, 121, 32, 97, 32, 102, 101, 119, 32, 111, 114, 103, 97, 110, 105, 122, 97, 116, 105, 111, 110, 115, 32, 99, 97, 110, 32, 97, 102, 102, 111, 114, 100, 46, 32, 84, 104, 105, 115, 32, 104, 97, 115, 32, 108, 101, 100, 32, 116, 111, 32, 116, 104, 101, 32, 101, 109, 101, 114, 103, 101, 110, 99, 101, 32, 111, 102, 32, 109, 111, 100, 101, 108, 32, 97, 115, 32, 97, 32, 115, 101, 114, 118, 105, 99, 101, 58, 32, 109, 111, 100, 101, 108, 115, 32, 100, 101, 118, 101, 108, 111, 112, 101, 100, 32, 98, 121, 32, 116, 104, 101, 115, 101, 32, 102, 101, 119, 32, 111, 114, 103, 97, 110, 105, 122, 97, 116, 105, 111, 110, 115, 32, 97, 114, 101, 32, 109, 97, 100, 101, 32, 97, 118, 97, 105, 108, 97, 98, 108, 101, 32, 102, 111, 114, 32, 111, 116, 104, 101, 114, 115, 32, 116, 111, 32, 117, 115, 101, 32, 97, 115, 32, 97, 32, 115, 101, 114, 118, 105, 99, 101, 46, 32, 65, 110, 121, 111, 110, 101, 32, 119, 104, 111, 32, 119, 105, 115, 104, 101, 115, 32, 116, 111, 32, 108, 101, 118, 101, 114, 97, 103, 101, 32, 65, 73, 32, 116, 111, 32, 98, 117, 105, 108, 100, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 32, 99, 97, 110, 32, 110, 111, 119, 32, 117, 115, 101, 32, 116, 104, 101, 115, 101, 32, 109, 111, 100, 101, 108, 115, 32, 116, 111, 32, 100, 111, 32, 115, 111, 32, 119, 105, 116, 104, 111, 117, 116, 32, 104, 97, 118, 105, 110, 103, 32, 116, 111, 32, 105, 110, 118, 101, 115, 116, 32, 117, 112, 32, 102, 114, 111, 110, 116, 32, 105, 110, 32, 98, 117, 105, 108, 100, 105, 110, 103, 32, 97, 32, 109, 111, 100, 101, 108, 46, 32, 73, 110, 32, 115, 104, 111, 114, 116, 44, 32, 116, 104, 101, 32, 100, 101, 109, 97, 110, 100, 32, 102, 111, 114, 32, 65, 73, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 32, 104, 97, 115, 32, 105, 110, 99, 114, 101, 97, 115, 101, 100, 32, 119, 104, 105, 108, 101, 32, 116, 104, 101, 32, 98, 97, 114, 114, 105, 101, 114, 32, 116, 111, 32, 101, 110, 116, 114, 121, 32, 102, 111, 114, 32, 98, 117, 105, 108, 100, 105, 110, 103, 32, 65, 73, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 32, 104, 97, 115, 32, 100, 101, 99, 114, 101, 97, 115, 101, 100, 46, 32, 84, 104, 105, 115, 32, 104, 97, 115, 32, 116, 117, 114, 110, 101, 100, 32, 65, 73, 32, 101, 110, 103, 105, 110, 101, 101, 114, 105, 110, 103, 226, 128, 148, 116, 104, 101, 32, 112, 114, 111, 99, 101, 115, 115, 32, 111, 102, 32, 98, 117, 105, 108, 100, 105, 110, 103, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 32, 111, 110, 32, 116, 111, 112, 32, 111, 102, 32, 114, 101, 97, 100, 105, 108, 121, 32, 97, 118, 97, 105, 108, 97, 98, 108, 101, 32, 109, 111, 100, 101, 108, 115, 226, 128, 148, 105, 110, 116, 111, 32, 111, 110, 101, 32, 111, 102, 32, 116, 104, 101, 32, 102, 97, 115, 116, 101, 115, 116, 45, 103, 114, 111, 119, 105, 110, 103, 32, 101, 110, 103, 105, 110, 101, 101, 114, 105, 110, 103, 32, 100, 105, 115, 99, 105, 112, 108, 105, 110, 101, 115, 46, 32, 66, 117, 105, 108, 100, 105, 110, 103, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 32, 111, 110, 32, 116, 111, 112, 32, 111, 102, 32, 109, 97, 99, 104, 105, 110, 101, 32, 108, 101, 97, 114, 110, 105, 110, 103, 32, 40, 77, 76, 41, 32, 109, 111, 100, 101, 108, 115, 32, 105, 115, 110, 226, 128, 153, 116, 32, 110, 101, 119, 46, 32, 76, 111, 110, 103, 32, 98, 101, 102, 111, 114, 101, 32, 76, 76, 77, 115, 32, 98, 101, 99, 97, 109, 101, 32, 112, 114, 111, 109, 105, 110, 101, 110, 116, 44, 32, 65, 73, 32, 119, 97, 115, 32, 97, 108, 114, 101, 97, 100, 121, 32, 112, 111, 119, 101, 114, 105, 110, 103, 32, 109, 97, 110, 121, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 44, 32, 105, 110, 99, 108, 117, 100, 105, 110, 103, 32, 112, 114, 111, 100, 117, 99, 116, 32, 114, 101, 99, 111, 109, 109, 101, 110, 100, 97, 116, 105, 111, 110, 115, 44, 32, 102, 114, 97, 117, 100, 32, 100, 101, 116, 101, 99, 116, 105, 111, 110, 44, 32, 97, 110, 100, 32, 99, 104, 117, 114, 110, 32, 112, 114, 101, 100, 105, 99, 116, 105, 111, 110, 46, 32, 87, 104, 105, 108, 101, 32, 109, 97, 110, 121, 32, 112, 114, 105, 110, 99, 105, 112, 108, 101, 115, 32, 111, 102, 32, 112, 114, 111, 100, 117, 99, 116, 105, 111, 110, 105, 122, 105, 110, 103, 32, 65, 73, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 32, 114, 101, 109, 97, 105, 110, 32, 116, 104, 101, 32, 115, 97, 109, 101, 44, 32, 116, 104, 101, 32, 110, 101, 119, 32, 103, 101, 110, 101, 114, 97, 116, 105, 111, 110, 32, 111, 102, 32, 108, 97, 114, 103, 101, 45, 115, 99, 97, 108, 101, 44, 32, 114, 101, 97, 100, 105, 108, 121, 32, 97, 118, 97, 105, 108, 97, 98, 108, 101, 32, 109, 111, 100, 101, 108, 115, 32, 98, 114, 105, 110, 103, 115, 32, 97, 98, 111, 117, 116, 32, 110, 101, 119, 32, 112, 111, 115, 115, 105, 98, 105, 108, 105, 116, 105, 101, 115, 32, 97, 110, 100, 32, 110, 101, 119, 32, 99, 104, 97, 108, 108, 101, 110, 103, 101, 115, 44, 32, 119, 104, 105, 99, 104, 32, 97, 114, 101, 32, 116, 104, 101, 32, 102, 111, 99, 117, 115, 32, 111, 102, 32, 116, 104, 105, 115, 32, 98, 111, 111, 107, 46, 32, 84, 104, 105, 115, 32, 99, 104, 97, 112, 116, 101, 114, 32, 98, 101, 103, 105, 110, 115, 32, 119, 105, 116, 104, 32, 97, 110, 32, 111, 118, 101, 114, 118, 105, 101, 119, 32, 111, 102, 32, 102, 111, 117, 110, 100, 97, 116, 105, 111, 110, 32, 109, 111, 100, 101, 108, 115, 44, 32, 116, 104, 101, 32, 107, 101, 121, 32, 99, 97, 116, 97, 108, 121, 115, 116, 32, 98, 101, 104, 105, 110, 100, 32, 116, 104, 101, 32, 101, 120, 112, 108, 111, 115, 105, 111, 110, 32, 111, 102, 32, 65, 73, 32, 101, 110, 103, 105, 110, 101, 101, 114, 105, 110, 103, 46, 32, 73, 226, 128, 153, 108, 108, 32, 116, 104, 101, 110, 32, 100, 105, 115, 99, 117, 115, 115, 32, 97, 32, 114, 97, 110, 103, 101, 32, 111, 102, 32, 115, 117, 99, 99, 101, 115, 115, 102, 117, 108, 32, 65, 73, 32, 117, 115, 101, 32, 99, 97, 115, 101, 115, 44, 32, 101, 97, 99, 104, 32, 105, 108, 108, 117, 115, 116, 114, 97, 116, 105, 110, 103, 32, 119, 104, 97, 116, 32, 65, 73, 32, 105, 115, 32, 103, 111, 111, 100, 32, 97, 110, 100, 32, 110, 111, 116, 32, 121, 101, 116, 32, 103, 111, 111, 100, 32, 97, 116, 46, 32, 65, 115, 32, 65, 73, 226, 128, 153, 115, 32, 99, 97, 112, 97, 98, 105, 108, 105, 116, 105, 101, 115, 32, 101, 120, 112, 97, 110, 100, 32, 100, 97, 105, 108, 121, 44, 32, 112, 114, 101, 100, 105, 99, 116, 105, 110, 103, 32, 105, 116, 115, 32, 102, 117, 116, 117, 114, 101, 32, 112, 111, 115, 115, 105, 98, 105, 108, 105, 116, 105, 101, 115, 32, 98, 101, 99, 111, 109, 101, 115, 32, 105, 110, 99, 114, 101, 97, 115, 105, 110, 103, 108, 121, 32, 99, 104, 97, 108, 108, 101, 110, 103, 105, 110, 103, 46, 32, 72, 111, 119, 101, 118, 101, 114, 44, 32, 101, 120, 105, 115, 116, 105, 110, 103, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 32, 112, 97, 116, 116, 101, 114, 110, 115, 32, 99, 97, 110, 32, 104, 101, 108, 112, 32, 117, 110, 99, 111, 118, 101, 114, 32, 111, 112, 112, 111, 114, 116, 117, 110, 105, 116, 105, 101, 115, 32, 116, 111, 100, 97, 121, 32, 97, 110, 100, 32, 111, 102, 102, 101, 114, 32, 99, 108, 117, 101, 115, 32, 97, 98, 111, 117, 116, 32, 104, 111, 119, 32, 65, 73, 32, 109, 97, 121, 32, 99, 111, 110, 116, 105, 110, 117, 101, 32, 116, 111, 32, 98, 101, 32, 117, 115, 101, 100, 32, 105, 110, 32, 116, 104, 101, 32, 102, 117, 116, 117, 114, 101, 46, 32, 84, 111, 32, 99, 108, 111, 115, 101, 32, 111, 117, 116, 32, 116, 104, 101, 32, 99, 104, 97, 112, 116, 101, 114, 44, 32, 73, 226, 128, 153, 108, 108, 32, 112, 114, 111, 118, 105, 100, 101, 32, 97, 110, 32, 111, 118, 101, 114, 118, 105, 101, 119, 32, 111, 102, 32, 116, 104, 101, 32, 110, 101, 119, 32, 65, 73, 32, 115, 116, 97, 99, 107, 44, 32, 105, 110, 99, 108, 117, 100, 105, 110, 103, 32, 119, 104, 97, 116, 32, 104, 97, 115, 32, 99, 104, 97, 110, 103, 101, 100, 32, 119, 105, 116, 104, 32, 102, 111, 117, 110, 100, 97, 116, 105, 111, 110, 32, 109, 111, 100, 101, 108, 115, 44, 32, 119, 104, 97, 116, 32, 114, 101, 109, 97, 105, 110, 115, 32, 116, 104, 101, 32, 115, 97, 109, 101, 44, 32, 97, 110, 100, 32, 104, 111, 119, 32, 116, 104, 101, 32, 114, 111, 108, 101, 32, 111, 102, 32, 97, 110, 32, 65, 73, 32, 101, 110, 103, 105, 110, 101, 101, 114, 32, 116, 111, 100, 97, 121, 32, 100, 105, 102, 102, 101, 114, 115, 32, 102, 114, 111, 109, 32, 116, 104, 97, 116, 32, 111, 102, 32, 97, 32, 116, 114, 97, 100, 105, 116, 105, 111, 110, 97, 108, 32, 77, 76, 32, 101, 110, 103, 105, 110, 101, 101, 114, 46]
length of tokens: 2153

Byte-Pair Encoding Algorithm

find the pair of bytes that occur most frequently
mint a new token representation of them

def get_most_frequent_pair(ids):
    # count the frequency of each pair of bytes
    pair_counts = {}
    for pair in zip(ids, ids[1:]): # zip() pairs each byte with the next one
        pair_counts[pair] = pair_counts.get(pair, 0) + 1 # get() returns the value of the pair, or 0 if it doesn't exist
    return pair_counts

most_frequent_pair = get_most_frequent_pair(tokens)
# print(most_frequent_pair)
print(sorted(((value, key) for key, value in most_frequent_pair.items()), reverse=True))

[(57, (101, 32)), (52, (115, 32)), (48, (105, 110)), (40, (32, 116)), (40, (32, 97)), (35, (110, 103)), (31, (111, 110)), (29, (32, 111)), (28, (116, 105)), (26, (114, 101)), (26, (97, 116)), (25, (116, 104)), (24, (101, 114)), (24, (97, 110)), (23, (100, 32)), (23, (44, 32)), (22, (104, 101)), (21, (101, 115)), (21, (32, 109)), (20, (105, 111)), (20, (103, 32)), (19, (111, 100)), (19, (110, 32)), (19, (99, 97)), (19, (32, 99)), (18, (111, 102)), (18, (108, 105)), (18, (100, 101)), (18, (32, 65)), (17, (116, 32)), (17, (110, 101)), (17, (108, 101)), (17, (105, 108)), (17, (104, 97)), (17, (102, 32)), (16, (110, 100)), (16, (101, 110)), (16, (97, 115)), (16, (65, 73)), (15, (111, 32)), (15, (110, 115)), (15, (109, 111)), (15, (105, 99)), (15, (73, 32)), (15, (46, 32)), (15, (32, 112)), (15, (32, 98)), (14, (121, 32)), (14, (116, 111)), (14, (115, 101)), (14, (111, 114)), (14, (101, 108)), (14, (100, 105)), (14, (32, 102)), (13, (112, 108)), (13, (97, 112)), (13, (32, 105)), (13, (32, 101)), (11, (119, 32)), (11, (114, 111)), (11, (105, 116)), (11, (101, 97)), (11, (97, 108)), (11, (32, 119)), (10, (112, 114)), (10, (112, 112)), (10, (108, 115)), (10, (105, 115)), (10, (103, 101)), (10, (101, 100)), (10, (32, 115)), (10, (32, 104)), (10, (32, 100)), (9, (118, 101)), (9, (114, 32)), (9, (110, 99)), (9, (109, 97)), (9, (104, 105)), (9, (101, 119)), (9, (101, 99)), (9, (99, 111)), (9, (99, 104)), (9, (32, 114)), (8, (116, 101)), (8, (115, 116)), (8, (115, 44)), (8, (114, 97)), (8, (102, 111)), (8, (97, 98)), (8, (32, 108)), (7, (118, 105)), (7, (117, 116)), (7, (117, 115)), (7, (114, 105)), (7, (111, 119)), (7, (111, 117)), (7, (111, 109)), (7, (105, 101)), (7, (103, 105)), (7, (100, 97)), (7, (99, 101)), (7, (98, 101)), (7, (97, 114)), (7, (97, 105)), (7, (32, 117)), (7, (32, 110)), (6, (226, 128)), (6, (119, 104)), (6, (117, 105)), (6, (110, 116)), (6, (110, 105)), (6, (109, 101)), (6, (108, 121)), (6, (108, 97)), (6, (108, 32)), (6, (99, 116)), (6, (97, 32)), (5, (119, 105)), (5, (117, 114)), (5, (116, 97)), (5, (115, 115)), (5, (114, 103)), (5, (112, 111)), (5, (112, 32)), (5, (111, 118)), (5, (111, 112)), (5, (108, 117)), (5, (108, 108)), (5, (108, 100)), (5, (104, 111)), (5, (102, 101)), (5, (101, 101)), (5, (99, 114)), (5, (98, 108)), (5, (97, 100)), (4, (128, 153)), (4, (118, 97)), (4, (117, 110)), (4, (117, 101)), (4, (117, 99)), (4, (116, 117)), (4, (116, 114)), (4, (115, 105)), (4, (115, 99)), (4, (114, 118)), (4, (114, 110)), (4, (112, 97)), (4, (111, 115)), (4, (105, 122)), (4, (104, 32)), (4, (102, 117)), (4, (101, 118)), (4, (101, 109)), (4, (101, 44)), (4, (99, 108)), (4, (98, 117)), (4, (97, 118)), (4, (97, 109)), (4, (84, 104)), (4, (32, 84)), (3, (119, 101)), (3, (117, 100)), (3, (116, 44)), (3, (115, 46)), (3, (114, 115)), (3, (113, 117)), (3, (112, 101)), (3, (111, 111)), (3, (110, 121)), (3, (110, 111)), (3, (109, 105)), (3, (108, 111)), (3, (102, 114)), (3, (102, 102)), (3, (101, 120)), (3, (101, 46)), (3, (100, 117)), (3, (99, 105)), (3, (98, 111)), (3, (98, 105)), (3, (97, 121)), (3, (97, 103)), (3, (97, 99)), (3, (73, 226)), (3, (32, 103)), (3, (32, 73)), (2, (153, 108)), (2, (128, 148)), (2, (122, 97)), (2, (121, 44)), (2, (120, 112)), (2, (117, 112)), (2, (117, 108)), (2, (117, 97)), (2, (116, 121)), (2, (115, 111)), (2, (115, 104)), (2, (115, 97)), (2, (114, 116)), (2, (114, 44)), (2, (112, 116)), (2, (111, 116)), (2, (111, 99)), (2, (110, 97)), (2, (109, 112)), (2, (105, 114)), (2, (105, 112)), (2, (105, 102)), (2, (105, 98)), (2, (103, 111)), (2, (103, 97)), (2, (103, 46)), (2, (101, 116)), (2, (101, 113)), (2, (100, 46)), (2, (99, 117)), (2, (77, 115)), (2, (77, 76)), (2, (76, 77)), (2, (76, 76)), (2, (41, 32)), (2, (32, 77)), (2, (32, 76)), (2, (32, 40)), (1, (153, 116)), (1, (153, 115)), (1, (148, 116)), (1, (148, 105)), (1, (122, 105)), (1, (122, 101)), (1, (121, 115)), (1, (121, 111)), (1, (121, 101)), (1, (120, 105)), (1, (119, 111)), (1, (119, 97)), (1, (119, 46)), (1, (116, 119)), (1, (116, 116)), (1, (116, 115)), (1, (116, 46)), (1, (116, 45)), (1, (115, 226)), (1, (115, 117)), (1, (115, 112)), (1, (115, 110)), (1, (115, 107)), (1, (115, 102)), (1, (115, 41)), (1, (114, 121)), (1, (114, 114)), (1, (114, 102)), (1, (114, 100)), (1, (114, 99)), (1, (114, 46)), (1, (112, 117)), (1, (111, 108)), (1, (111, 107)), (1, (110, 226)), (1, (110, 118)), (1, (110, 117)), (1, (110, 108)), (1, (110, 46)), (1, (110, 44)), (1, (109, 115)), (1, (109, 109)), (1, (109, 32)), (1, (108, 114)), (1, (108, 112)), (1, (108, 46)), (1, (107, 115)), (1, (107, 101)), (1, (107, 46)), (1, (107, 44)), (1, (106, 111)), (1, (105, 118)), (1, (105, 109)), (1, (105, 100)), (1, (105, 97)), (1, (104, 117)), (1, (103, 226)), (1, (103, 117)), (1, (103, 115)), (1, (103, 114)), (1, (103, 108)), (1, (102, 97)), (1, (101, 121)), (1, (101, 111)), (1, (101, 104)), (1, (101, 103)), (1, (101, 102)), (1, (101, 58)), (1, (101, 45)), (1, (100, 121)), (1, (100, 111)), (1, (100, 44)), (1, (99, 107)), (1, (99, 99)), (1, (99, 32)), (1, (98, 121)), (1, (98, 114)), (1, (98, 97)), (1, (97, 117)), (1, (97, 106)), (1, (97, 102)), (1, (97, 44)), (1, (87, 104)), (1, (84, 111)), (1, (83, 101)), (1, (77, 111)), (1, (76, 111)), (1, (76, 41)), (1, (76, 32)), (1, (73, 110)), (1, (72, 111)), (1, (70, 105)), (1, (66, 117)), (1, (65, 115)), (1, (65, 110)), (1, (58, 32)), (1, (45, 115)), (1, (45, 103)), (1, (40, 77)), (1, (40, 76)), (1, (32, 121)), (1, (32, 118)), (1, (32, 113)), (1, (32, 107)), (1, (32, 87)), (1, (32, 83)), (1, (32, 72)), (1, (32, 70)), (1, (32, 66))]

chr(101), chr(32) # most common tokens changed back into characters

('e', ' ')

top_ranking_pair = max(most_frequent_pair, key=most_frequent_pair.get) # key=most_frequent_pair.get ranks the keys by the value of the pair
top_ranking_pair

(101, 32)

def merge_tokens(ids, pair, idx):
    # in the list of ints (ids), replace all consecutive occurrences of the pair with a new token
    newids = []
    i = 0
    while i < len(ids):
        # if we are not at the very last position AND the pair matches, replace it
        if i < len(ids) - 1 and ids[i] == pair[0] and ids[i+1] == pair[1]:
            newids.append(idx)
            i += 2 # skip the next two bytes
        else:
            newids.append(ids[i])
            i += 1
    return newids

print("length of tokens before merge:", len(tokens))
print("---Merging tokens---")
tokens_after_merge = merge_tokens(tokens, top_ranking_pair, 255)
print("---")
print(tokens_after_merge)
print("length of tokens after merge:", len(tokens_after_merge))

length of tokens before merge: 2153
---Merging tokens---
---
[84, 104, 255, 115, 99, 97, 108, 105, 110, 103, 32, 117, 112, 32, 111, 102, 32, 65, 73, 32, 109, 111, 100, 101, 108, 115, 32, 104, 97, 115, 32, 116, 119, 111, 32, 109, 97, 106, 111, 114, 32, 99, 111, 110, 115, 101, 113, 117, 101, 110, 99, 101, 115, 46, 32, 70, 105, 114, 115, 116, 44, 32, 65, 73, 32, 109, 111, 100, 101, 108, 115, 32, 97, 114, 255, 98, 101, 99, 111, 109, 105, 110, 103, 32, 109, 111, 114, 255, 112, 111, 119, 101, 114, 102, 117, 108, 32, 97, 110, 100, 32, 99, 97, 112, 97, 98, 108, 255, 111, 102, 32, 109, 111, 114, 255, 116, 97, 115, 107, 115, 44, 32, 101, 110, 97, 98, 108, 105, 110, 103, 32, 109, 111, 114, 255, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 46, 32, 77, 111, 114, 255, 112, 101, 111, 112, 108, 255, 97, 110, 100, 32, 116, 101, 97, 109, 115, 32, 108, 101, 118, 101, 114, 97, 103, 255, 65, 73, 32, 116, 111, 32, 105, 110, 99, 114, 101, 97, 115, 255, 112, 114, 111, 100, 117, 99, 116, 105, 118, 105, 116, 121, 44, 32, 99, 114, 101, 97, 116, 255, 101, 99, 111, 110, 111, 109, 105, 99, 32, 118, 97, 108, 117, 101, 44, 32, 97, 110, 100, 32, 105, 109, 112, 114, 111, 118, 255, 113, 117, 97, 108, 105, 116, 121, 32, 111, 102, 32, 108, 105, 102, 101, 46, 32, 83, 101, 99, 111, 110, 100, 44, 32, 116, 114, 97, 105, 110, 105, 110, 103, 32, 108, 97, 114, 103, 255, 108, 97, 110, 103, 117, 97, 103, 255, 109, 111, 100, 101, 108, 115, 32, 40, 76, 76, 77, 115, 41, 32, 114, 101, 113, 117, 105, 114, 101, 115, 32, 100, 97, 116, 97, 44, 32, 99, 111, 109, 112, 117, 116, 255, 114, 101, 115, 111, 117, 114, 99, 101, 115, 44, 32, 97, 110, 100, 32, 115, 112, 101, 99, 105, 97, 108, 105, 122, 101, 100, 32, 116, 97, 108, 101, 110, 116, 32, 116, 104, 97, 116, 32, 111, 110, 108, 121, 32, 97, 32, 102, 101, 119, 32, 111, 114, 103, 97, 110, 105, 122, 97, 116, 105, 111, 110, 115, 32, 99, 97, 110, 32, 97, 102, 102, 111, 114, 100, 46, 32, 84, 104, 105, 115, 32, 104, 97, 115, 32, 108, 101, 100, 32, 116, 111, 32, 116, 104, 255, 101, 109, 101, 114, 103, 101, 110, 99, 255, 111, 102, 32, 109, 111, 100, 101, 108, 32, 97, 115, 32, 97, 32, 115, 101, 114, 118, 105, 99, 101, 58, 32, 109, 111, 100, 101, 108, 115, 32, 100, 101, 118, 101, 108, 111, 112, 101, 100, 32, 98, 121, 32, 116, 104, 101, 115, 255, 102, 101, 119, 32, 111, 114, 103, 97, 110, 105, 122, 97, 116, 105, 111, 110, 115, 32, 97, 114, 255, 109, 97, 100, 255, 97, 118, 97, 105, 108, 97, 98, 108, 255, 102, 111, 114, 32, 111, 116, 104, 101, 114, 115, 32, 116, 111, 32, 117, 115, 255, 97, 115, 32, 97, 32, 115, 101, 114, 118, 105, 99, 101, 46, 32, 65, 110, 121, 111, 110, 255, 119, 104, 111, 32, 119, 105, 115, 104, 101, 115, 32, 116, 111, 32, 108, 101, 118, 101, 114, 97, 103, 255, 65, 73, 32, 116, 111, 32, 98, 117, 105, 108, 100, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 32, 99, 97, 110, 32, 110, 111, 119, 32, 117, 115, 255, 116, 104, 101, 115, 255, 109, 111, 100, 101, 108, 115, 32, 116, 111, 32, 100, 111, 32, 115, 111, 32, 119, 105, 116, 104, 111, 117, 116, 32, 104, 97, 118, 105, 110, 103, 32, 116, 111, 32, 105, 110, 118, 101, 115, 116, 32, 117, 112, 32, 102, 114, 111, 110, 116, 32, 105, 110, 32, 98, 117, 105, 108, 100, 105, 110, 103, 32, 97, 32, 109, 111, 100, 101, 108, 46, 32, 73, 110, 32, 115, 104, 111, 114, 116, 44, 32, 116, 104, 255, 100, 101, 109, 97, 110, 100, 32, 102, 111, 114, 32, 65, 73, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 32, 104, 97, 115, 32, 105, 110, 99, 114, 101, 97, 115, 101, 100, 32, 119, 104, 105, 108, 255, 116, 104, 255, 98, 97, 114, 114, 105, 101, 114, 32, 116, 111, 32, 101, 110, 116, 114, 121, 32, 102, 111, 114, 32, 98, 117, 105, 108, 100, 105, 110, 103, 32, 65, 73, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 32, 104, 97, 115, 32, 100, 101, 99, 114, 101, 97, 115, 101, 100, 46, 32, 84, 104, 105, 115, 32, 104, 97, 115, 32, 116, 117, 114, 110, 101, 100, 32, 65, 73, 32, 101, 110, 103, 105, 110, 101, 101, 114, 105, 110, 103, 226, 128, 148, 116, 104, 255, 112, 114, 111, 99, 101, 115, 115, 32, 111, 102, 32, 98, 117, 105, 108, 100, 105, 110, 103, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 32, 111, 110, 32, 116, 111, 112, 32, 111, 102, 32, 114, 101, 97, 100, 105, 108, 121, 32, 97, 118, 97, 105, 108, 97, 98, 108, 255, 109, 111, 100, 101, 108, 115, 226, 128, 148, 105, 110, 116, 111, 32, 111, 110, 255, 111, 102, 32, 116, 104, 255, 102, 97, 115, 116, 101, 115, 116, 45, 103, 114, 111, 119, 105, 110, 103, 32, 101, 110, 103, 105, 110, 101, 101, 114, 105, 110, 103, 32, 100, 105, 115, 99, 105, 112, 108, 105, 110, 101, 115, 46, 32, 66, 117, 105, 108, 100, 105, 110, 103, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 32, 111, 110, 32, 116, 111, 112, 32, 111, 102, 32, 109, 97, 99, 104, 105, 110, 255, 108, 101, 97, 114, 110, 105, 110, 103, 32, 40, 77, 76, 41, 32, 109, 111, 100, 101, 108, 115, 32, 105, 115, 110, 226, 128, 153, 116, 32, 110, 101, 119, 46, 32, 76, 111, 110, 103, 32, 98, 101, 102, 111, 114, 255, 76, 76, 77, 115, 32, 98, 101, 99, 97, 109, 255, 112, 114, 111, 109, 105, 110, 101, 110, 116, 44, 32, 65, 73, 32, 119, 97, 115, 32, 97, 108, 114, 101, 97, 100, 121, 32, 112, 111, 119, 101, 114, 105, 110, 103, 32, 109, 97, 110, 121, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 44, 32, 105, 110, 99, 108, 117, 100, 105, 110, 103, 32, 112, 114, 111, 100, 117, 99, 116, 32, 114, 101, 99, 111, 109, 109, 101, 110, 100, 97, 116, 105, 111, 110, 115, 44, 32, 102, 114, 97, 117, 100, 32, 100, 101, 116, 101, 99, 116, 105, 111, 110, 44, 32, 97, 110, 100, 32, 99, 104, 117, 114, 110, 32, 112, 114, 101, 100, 105, 99, 116, 105, 111, 110, 46, 32, 87, 104, 105, 108, 255, 109, 97, 110, 121, 32, 112, 114, 105, 110, 99, 105, 112, 108, 101, 115, 32, 111, 102, 32, 112, 114, 111, 100, 117, 99, 116, 105, 111, 110, 105, 122, 105, 110, 103, 32, 65, 73, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 115, 32, 114, 101, 109, 97, 105, 110, 32, 116, 104, 255, 115, 97, 109, 101, 44, 32, 116, 104, 255, 110, 101, 119, 32, 103, 101, 110, 101, 114, 97, 116, 105, 111, 110, 32, 111, 102, 32, 108, 97, 114, 103, 101, 45, 115, 99, 97, 108, 101, 44, 32, 114, 101, 97, 100, 105, 108, 121, 32, 97, 118, 97, 105, 108, 97, 98, 108, 255, 109, 111, 100, 101, 108, 115, 32, 98, 114, 105, 110, 103, 115, 32, 97, 98, 111, 117, 116, 32, 110, 101, 119, 32, 112, 111, 115, 115, 105, 98, 105, 108, 105, 116, 105, 101, 115, 32, 97, 110, 100, 32, 110, 101, 119, 32, 99, 104, 97, 108, 108, 101, 110, 103, 101, 115, 44, 32, 119, 104, 105, 99, 104, 32, 97, 114, 255, 116, 104, 255, 102, 111, 99, 117, 115, 32, 111, 102, 32, 116, 104, 105, 115, 32, 98, 111, 111, 107, 46, 32, 84, 104, 105, 115, 32, 99, 104, 97, 112, 116, 101, 114, 32, 98, 101, 103, 105, 110, 115, 32, 119, 105, 116, 104, 32, 97, 110, 32, 111, 118, 101, 114, 118, 105, 101, 119, 32, 111, 102, 32, 102, 111, 117, 110, 100, 97, 116, 105, 111, 110, 32, 109, 111, 100, 101, 108, 115, 44, 32, 116, 104, 255, 107, 101, 121, 32, 99, 97, 116, 97, 108, 121, 115, 116, 32, 98, 101, 104, 105, 110, 100, 32, 116, 104, 255, 101, 120, 112, 108, 111, 115, 105, 111, 110, 32, 111, 102, 32, 65, 73, 32, 101, 110, 103, 105, 110, 101, 101, 114, 105, 110, 103, 46, 32, 73, 226, 128, 153, 108, 108, 32, 116, 104, 101, 110, 32, 100, 105, 115, 99, 117, 115, 115, 32, 97, 32, 114, 97, 110, 103, 255, 111, 102, 32, 115, 117, 99, 99, 101, 115, 115, 102, 117, 108, 32, 65, 73, 32, 117, 115, 255, 99, 97, 115, 101, 115, 44, 32, 101, 97, 99, 104, 32, 105, 108, 108, 117, 115, 116, 114, 97, 116, 105, 110, 103, 32, 119, 104, 97, 116, 32, 65, 73, 32, 105, 115, 32, 103, 111, 111, 100, 32, 97, 110, 100, 32, 110, 111, 116, 32, 121, 101, 116, 32, 103, 111, 111, 100, 32, 97, 116, 46, 32, 65, 115, 32, 65, 73, 226, 128, 153, 115, 32, 99, 97, 112, 97, 98, 105, 108, 105, 116, 105, 101, 115, 32, 101, 120, 112, 97, 110, 100, 32, 100, 97, 105, 108, 121, 44, 32, 112, 114, 101, 100, 105, 99, 116, 105, 110, 103, 32, 105, 116, 115, 32, 102, 117, 116, 117, 114, 255, 112, 111, 115, 115, 105, 98, 105, 108, 105, 116, 105, 101, 115, 32, 98, 101, 99, 111, 109, 101, 115, 32, 105, 110, 99, 114, 101, 97, 115, 105, 110, 103, 108, 121, 32, 99, 104, 97, 108, 108, 101, 110, 103, 105, 110, 103, 46, 32, 72, 111, 119, 101, 118, 101, 114, 44, 32, 101, 120, 105, 115, 116, 105, 110, 103, 32, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 32, 112, 97, 116, 116, 101, 114, 110, 115, 32, 99, 97, 110, 32, 104, 101, 108, 112, 32, 117, 110, 99, 111, 118, 101, 114, 32, 111, 112, 112, 111, 114, 116, 117, 110, 105, 116, 105, 101, 115, 32, 116, 111, 100, 97, 121, 32, 97, 110, 100, 32, 111, 102, 102, 101, 114, 32, 99, 108, 117, 101, 115, 32, 97, 98, 111, 117, 116, 32, 104, 111, 119, 32, 65, 73, 32, 109, 97, 121, 32, 99, 111, 110, 116, 105, 110, 117, 255, 116, 111, 32, 98, 255, 117, 115, 101, 100, 32, 105, 110, 32, 116, 104, 255, 102, 117, 116, 117, 114, 101, 46, 32, 84, 111, 32, 99, 108, 111, 115, 255, 111, 117, 116, 32, 116, 104, 255, 99, 104, 97, 112, 116, 101, 114, 44, 32, 73, 226, 128, 153, 108, 108, 32, 112, 114, 111, 118, 105, 100, 255, 97, 110, 32, 111, 118, 101, 114, 118, 105, 101, 119, 32, 111, 102, 32, 116, 104, 255, 110, 101, 119, 32, 65, 73, 32, 115, 116, 97, 99, 107, 44, 32, 105, 110, 99, 108, 117, 100, 105, 110, 103, 32, 119, 104, 97, 116, 32, 104, 97, 115, 32, 99, 104, 97, 110, 103, 101, 100, 32, 119, 105, 116, 104, 32, 102, 111, 117, 110, 100, 97, 116, 105, 111, 110, 32, 109, 111, 100, 101, 108, 115, 44, 32, 119, 104, 97, 116, 32, 114, 101, 109, 97, 105, 110, 115, 32, 116, 104, 255, 115, 97, 109, 101, 44, 32, 97, 110, 100, 32, 104, 111, 119, 32, 116, 104, 255, 114, 111, 108, 255, 111, 102, 32, 97, 110, 32, 65, 73, 32, 101, 110, 103, 105, 110, 101, 101, 114, 32, 116, 111, 100, 97, 121, 32, 100, 105, 102, 102, 101, 114, 115, 32, 102, 114, 111, 109, 32, 116, 104, 97, 116, 32, 111, 102, 32, 97, 32, 116, 114, 97, 100, 105, 116, 105, 111, 110, 97, 108, 32, 77, 76, 32, 101, 110, 103, 105, 110, 101, 101, 114, 46]
length of tokens after merge: 2096

Keep iterating over the next largest pair to mint

vocab_size = 276 # desired final volcabulary size 
number_of_merges = vocab_size - 256 # number of merges to perform
ids = list(tokens) # copy so we don't destory original list

# building a BINARY TREE of merges (starting not from root but from the leaves)
merges = {} # (child1, child2) -> mapping to new token
for i in range(number_of_merges):
  most_frequent_pair = get_most_frequent_pair(ids) # find most commonly occuring pair
  pair = max(most_frequent_pair, key=most_frequent_pair.get)
  idx = 256 + i
  print(f"merging {pair} into new token {idx}")
  ids = merge_tokens(ids, pair, idx)
  merges[pair] = idx

merging (101, 32) into new token 256
merging (115, 32) into new token 257
merging (105, 110) into new token 258
merging (111, 110) into new token 259
merging (32, 116) into new token 260
merging (32, 97) into new token 261
merging (97, 116) into new token 262
merging (258, 103) into new token 263
merging (101, 114) into new token 264
merging (105, 259) into new token 265
merging (111, 100) into new token 266
merging (111, 102) into new token 267
merging (114, 101) into new token 268
merging (105, 108) into new token 269
merging (104, 256) into new token 270
merging (65, 73) into new token 271
merging (101, 110) into new token 272
merging (100, 32) into new token 273
merging (32, 109) into new token 274
merging (46, 32) into new token 275

Note that some prior minted tokens are also eligible for merging

print(len(tokens))
print(len(ids))
print(f"compression ratio: {len(tokens) / len(ids):.2f}X")

2153
1653
compression ratio: 1.30X

The Technical User’s Introduction to LLM Tokenization - Christopher Samiullah

The Tokenizer is completely seperate, independent module to the LLM. It has its own training dataset of text, on which on train the vocabulary using the Byte-Pair Encoding Algorithm.

The Tokenizer then translates back and forth between raw text and sequences of tokens. The LLM later only ever sees the tokens and never deals with any text

Decoding

Given the integers, what is the text?

# dictionary mapping token ids to bytes
vocab = {idx: bytes([idx]) for idx in range(256)}
# in order of all the merges, populate the vocab list
# run in order of insertion of merges into the dictionary
for (p0, p1), idx in merges.items():
    # concatenate the bytes
    vocab[idx] = vocab[p0] + vocab[p1]
    print(vocab[idx])
def decode(ids):
    # given ids, return the text
    # get the tokens
    # concat all bytes together
    tokens = b"".join(vocab[idx] for idx in ids)
    text = tokens.decode('utf-8', errors='replace') # utf-8 needs the replace erroring to fix the issue of some tokens not being valid utf-8
    return text
print("----")
print(decode(tokens))
print("----")
print(decode([257]))

b'e '
b's '
b'in'
b'on'
b' t'
b' a'
b'at'
b'ing'
b'er'
b'ion'
b'od'
b'of'
b're'
b'il'
b'he '
b'AI'
b'en'
b'd '
b' m'
b'. '
----
The scaling up of AI models has two major consequences. First, AI models are becoming more powerful and capable of more tasks, enabling more applications. More people and teams leverage AI to increase productivity, create economic value, and improve quality of life. Second, training large language models (LLMs) requires data, compute resources, and specialized talent that only a few organizations can afford. This has led to the emergence of model as a service: models developed by these few organizations are made available for others to use as a service. Anyone who wishes to leverage AI to build applications can now use these models to do so without having to invest up front in building a model. In short, the demand for AI applications has increased while the barrier to entry for building AI applications has decreased. This has turned AI engineering—the process of building applications on top of readily available models—into one of the fastest-growing engineering disciplines. Building applications on top of machine learning (ML) models isn’t new. Long before LLMs became prominent, AI was already powering many applications, including product recommendations, fraud detection, and churn prediction. While many principles of productionizing AI applications remain the same, the new generation of large-scale, readily available models brings about new possibilities and new challenges, which are the focus of this book. This chapter begins with an overview of foundation models, the key catalyst behind the explosion of AI engineering. I’ll then discuss a range of successful AI use cases, each illustrating what AI is good and not yet good at. As AI’s capabilities expand daily, predicting its future possibilities becomes increasingly challenging. However, existing application patterns can help uncover opportunities today and offer clues about how AI may continue to be used in the future. To close out the chapter, I’ll provide an overview of the new AI stack, including what has changed with foundation models, what remains the same, and how the role of an AI engineer today differs from that of a traditional ML engineer.
----
s

merges

{(101, 32): 256,
 (115, 32): 257,
 (105, 110): 258,
 (111, 110): 259,
 (32, 116): 260,
 (32, 97): 261,
 (97, 116): 262,
 (258, 103): 263,
 (101, 114): 264,
 (105, 259): 265,
 (111, 100): 266,
 (111, 102): 267,
 (114, 101): 268,
 (105, 108): 269,
 (104, 256): 270,
 (65, 73): 271,
 (101, 110): 272,
 (100, 32): 273,
 (32, 109): 274,
 (46, 32): 275}

most_frequent_pair

{(84, 270): 1,
 (270, 115): 3,
 (115, 99): 4,
 (99, 97): 9,
 (97, 108): 11,
 (108, 263): 2,
 (263, 32): 11,
 (32, 117): 6,
 (117, 112): 2,
 (112, 32): 5,
 (32, 267): 9,
 (267, 32): 9,
 (32, 271): 12,
 (271, 274): 3,
 (274, 266): 8,
 (266, 101): 12,
 (101, 108): 14,
 (108, 257): 7,
 (257, 104): 5,
 (104, 97): 12,
 (97, 257): 8,
 (257, 116): 7,
 (116, 119): 1,
 (119, 111): 1,
 (111, 274): 1,
 (274, 97): 4,
 (97, 106): 1,
 (106, 111): 1,
 (111, 114): 14,
 (114, 32): 4,
 (32, 99): 9,
 (99, 259): 4,
 (259, 115): 1,
 (115, 101): 5,
 (101, 113): 1,
 (113, 117): 3,
 (117, 272): 1,
 (272, 99): 2,
 (99, 101): 6,
 (101, 115): 11,
 (115, 46): 3,
 (46, 32): 15,
 (32, 70): 1,
 (70, 105): 1,
 (105, 114): 1,
 (114, 115): 1,
 (115, 116): 8,
 (116, 44): 3,
 (44, 32): 15,
 (257, 97): 9,
 (97, 114): 6,
 (114, 256): 9,
 (256, 98): 1,
 (98, 101): 6,
 (101, 99): 8,
 (99, 111): 5,
 (111, 109): 7,
 (109, 263): 1,
 (263, 274): 3,
 (274, 111): 3,
 (256, 112): 5,
 (112, 111): 5,
 (111, 119): 7,
 (119, 264): 2,
 (264, 102): 1,
 (102, 117): 4,
 (117, 108): 2,
 (108, 261): 2,
 (261, 110): 9,
 (110, 273): 10,
 (273, 99): 2,
 (97, 112): 5,
 (112, 97): 3,
 (97, 98): 8,
 (98, 108): 5,
 (108, 256): 6,
 (256, 267): 5,
 (267, 274): 3,
 (256, 116): 5,
 (116, 97): 2,
 (97, 115): 7,
 (115, 107): 1,
 (107, 115): 1,
 (115, 44): 8,
 (32, 272): 7,
 (272, 97): 1,
 (256, 97): 5,
 (112, 112): 10,
 (112, 108): 13,
 (108, 105): 12,
 (105, 99): 15,
 (99, 262): 10,
 (262, 265): 15,
 (265, 115): 3,
 (32, 77): 2,
 (77, 111): 1,
 (112, 101): 3,
 (101, 111): 1,
 (111, 112): 5,
 (97, 110): 15,
 (110, 100): 3,
 (100, 260): 4,
 (260, 101): 1,
 (101, 97): 3,
 (97, 109): 4,
 (109, 257): 1,
 (257, 108): 2,
 (108, 101): 6,
 (101, 118): 4,
 (118, 264): 6,
 (264, 97): 2,
 (97, 103): 3,
 (103, 256): 5,
 (256, 271): 2,
 (271, 260): 2,
 (260, 111): 7,
 (111, 32): 13,
 (32, 258): 5,
 (258, 99): 6,
 (99, 268): 5,
 (268, 97): 7,
 (115, 256): 7,
 (112, 114): 8,
 (114, 266): 3,
 (266, 117): 3,
 (117, 99): 4,
 (99, 116): 6,
 (116, 105): 5,
 (105, 118): 1,
 (118, 105): 6,
 (105, 116): 11,
 (116, 121): 2,
 (121, 44): 2,
 (268, 262): 1,
 (262, 256): 1,
 (256, 101): 1,
 (259, 111): 1,
 (109, 105): 1,
 (99, 32): 1,
 (32, 118): 1,
 (118, 97): 4,
 (108, 117): 5,
 (117, 101): 2,
 (101, 44): 4,
 (44, 261): 4,
 (273, 105): 1,
 (105, 109): 1,
 (109, 112): 2,
 (114, 111): 7,
 (111, 118): 5,
 (118, 256): 1,
 (256, 113): 1,
 (117, 97): 2,
 (121, 32): 8,
 (32, 108): 4,
 (105, 102): 2,
 (102, 101): 3,
 (101, 46): 2,
 (32, 83): 1,
 (83, 101): 1,
 (259, 100): 1,
 (100, 44): 1,
 (44, 260): 4,
 (260, 114): 2,
 (114, 97): 4,
 (97, 258): 3,
 (258, 263): 1,
 (108, 97): 3,
 (114, 103): 4,
 (256, 108): 2,
 (110, 103): 3,
 (103, 117): 1,
 (256, 109): 6,
 (109, 266): 4,
 (257, 40): 1,
 (40, 76): 1,
 (76, 76): 2,
 (76, 77): 2,
 (77, 115): 1,
 (115, 41): 1,
 (41, 32): 1,
 (32, 268): 5,
 (268, 113): 1,
 (117, 105): 1,
 (105, 268): 1,
 (268, 257): 1,
 (257, 100): 3,
 (100, 262): 4,
 (262, 97): 2,
 (97, 44): 1,
 (112, 117): 1,
 (117, 116): 7,
 (116, 256): 1,
 (256, 268): 1,
 (268, 115): 1,
 (115, 111): 2,
 (111, 117): 7,
 (117, 114): 4,
 (114, 99): 1,
 (273, 115): 1,
 (115, 112): 1,
 (99, 105): 3,
 (105, 97): 1,
 (105, 122): 4,
 (122, 101): 1,
 (101, 100): 3,
 (260, 97): 1,
 (108, 272): 3,
 (272, 116): 3,
 (116, 260): 2,
 (260, 104): 5,
 (104, 262): 5,
 (262, 32): 5,
 (32, 259): 2,
 (259, 108): 1,
 (108, 121): 3,
 (121, 261): 5,
 (261, 32): 1,
 (32, 102): 6,
 (101, 119): 9,
 (119, 32): 10,
 (32, 111): 6,
 (103, 97): 2,
 (110, 105): 3,
 (122, 262): 2,
 (265, 257): 8,
 (257, 99): 6,
 (110, 261): 1,
 (261, 102): 1,
 (102, 102): 2,
 (102, 111): 8,
 (114, 100): 1,
 (100, 46): 2,
 (32, 84): 4,
 (84, 104): 3,
 (104, 105): 5,
 (105, 257): 5,
 (111, 260): 1,
 (260, 270): 11,
 (270, 101): 2,
 (101, 109): 2,
 (109, 264): 1,
 (264, 103): 1,
 (103, 272): 2,
 (99, 256): 1,
 (261, 257): 1,
 (97, 32): 3,
 (32, 115): 6,
 (115, 264): 2,
 (264, 118): 4,
 (101, 58): 1,
 (58, 274): 1,
 (100, 101): 4,
 (118, 101): 2,
 (108, 111): 3,
 (101, 273): 5,
 (273, 98): 1,
 (98, 121): 1,
 (121, 260): 1,
 (104, 101): 4,
 (256, 102): 2,
 (109, 97): 5,
 (97, 100): 5,
 (100, 256): 2,
 (97, 118): 2,
 (97, 269): 4,
 (269, 97): 3,
 (111, 116): 2,
 (116, 104): 5,
 (104, 264): 1,
 (264, 257): 2,
 (116, 111): 5,
 (117, 115): 6,
 (32, 65): 2,
 (65, 110): 1,
 (110, 121): 3,
 (121, 259): 1,
 (259, 256): 2,
 (256, 119): 1,
 (119, 104): 6,
 (104, 111): 5,
 (32, 119): 7,
 (119, 105): 4,
 (105, 115): 5,
 (115, 104): 2,
 (101, 257): 8,
 (32, 98): 8,
 (98, 117): 4,
 (117, 269): 5,
 (269, 100): 5,
 (100, 261): 1,
 (261, 112): 8,
 (110, 32): 7,
 (32, 110): 3,
 (110, 111): 2,
 (32, 100): 4,
 (100, 111): 1,
 (116, 32): 10,
 (32, 104): 4,
 (118, 263): 1,
 (263, 260): 1,
 (258, 118): 1,
 (102, 114): 3,
 (114, 259): 1,
 (259, 116): 2,
 (258, 32): 1,
 (100, 263): 6,
 (263, 261): 4,
 (261, 274): 1,
 (108, 46): 1,
 (32, 73): 3,
 (73, 110): 1,
 (114, 116): 2,
 (270, 100): 1,
 (273, 102): 1,
 (271, 261): 3,
 (257, 258): 2,
 (273, 119): 2,
 (104, 269): 2,
 (269, 256): 2,
 (116, 270): 4,
 (270, 98): 1,
 (98, 97): 1,
 (114, 114): 1,
 (114, 105): 1,
 (105, 264): 1,
 (264, 260): 2,
 (116, 114): 2,
 (114, 121): 1,
 (116, 117): 4,
 (114, 110): 3,
 (110, 101): 6,
 (273, 271): 1,
 (271, 32): 7,
 (272, 103): 7,
 (103, 258): 6,
 (258, 101): 6,
 (101, 264): 5,
 (264, 263): 4,
 (263, 226): 1,
 (226, 128): 6,
 (128, 148): 2,
 (148, 116): 1,
 (270, 112): 1,
 (111, 99): 2,
 (115, 257): 2,
 (257, 267): 3,
 (257, 259): 2,
 (259, 260): 2,
 (100, 269): 2,
 (269, 121): 3,
 (261, 118): 2,
 (108, 115): 3,
 (115, 226): 1,
 (148, 258): 1,
 (258, 116): 1,
 (267, 260): 3,
 (270, 102): 3,
 (102, 97): 1,
 (116, 101): 2,
 (116, 45): 1,
 (45, 103): 1,
 (103, 114): 1,
 (119, 263): 1,
 (100, 105): 6,
 (105, 112): 2,
 (108, 258): 1,
 (32, 66): 1,
 (66, 117): 1,
 (97, 99): 3,
 (99, 104): 9,
 (104, 258): 2,
 (258, 256): 1,
 (110, 263): 1,
 (32, 40): 1,
 (40, 77): 1,
 (77, 76): 2,
 (76, 41): 1,
 (41, 274): 1,
 (257, 105): 1,
 (115, 110): 1,
 (110, 226): 1,
 (128, 153): 4,
 (153, 116): 1,
 (119, 46): 1,
 (32, 76): 1,
 (76, 259): 1,
 (259, 103): 1,
 (103, 32): 1,
 (101, 102): 1,
 (256, 76): 1,
 (77, 257): 1,
 (257, 98): 4,
 (109, 256): 1,
 (109, 258): 1,
 (258, 272): 1,
 (119, 97): 1,
 (108, 268): 1,
 (100, 121): 1,
 (32, 112): 9,
 (99, 108): 4,
 (117, 100): 2,
 (268, 99): 1,
 (109, 109): 1,
 (109, 272): 1,
 (272, 100): 1,
 (97, 117): 1,
 (117, 273): 1,
 (273, 100): 2,
 (101, 116): 2,
 (116, 265): 4,
 (265, 44): 1,
 (104, 117): 1,
 (112, 268): 2,
 (268, 100): 2,
 (265, 46): 1,
 (32, 87): 1,
 (87, 104): 1,
 (114, 258): 1,
 (265, 105): 1,
 (122, 263): 1,
 (257, 268): 1,
 (268, 109): 2,
 (258, 260): 2,
 (115, 97): 2,
 (109, 101): 3,
 (270, 110): 2,
 (32, 103): 2,
 (272, 264): 1,
 (264, 262): 1,
 (265, 32): 3,
 (103, 101): 3,
 (101, 45): 1,
 (45, 115): 1,
 (98, 114): 1,
 (114, 263): 1,
 (263, 257): 1,
 (98, 111): 3,
 (111, 115): 4,
 (115, 115): 3,
 (115, 105): 2,
 (105, 98): 2,
 (98, 269): 3,
 (269, 105): 3,
 (105, 101): 6,
 (273, 110): 2,
 (108, 108): 4,
 (104, 261): 2,
 (261, 114): 1,
 (99, 117): 2,
 (117, 257): 1,
 (111, 111): 1,
 (111, 107): 1,
 (107, 46): 1,
 (112, 116): 2,
 (116, 264): 3,
 (264, 32): 3,
 (101, 103): 1,
 (258, 257): 2,
 (257, 119): 1,
 (117, 110): 4,
 (265, 274): 2,
 (270, 107): 1,
 (107, 101): 1,
 (101, 121): 1,
 (121, 115): 1,
 (101, 104): 1,
 (258, 100): 1,
 (101, 120): 3,
 (120, 112): 2,
 (115, 265): 1,
 (263, 46): 2,
 (73, 226): 2,
 (153, 108): 2,
 (108, 260): 1,
 (104, 272): 1,
 (272, 32): 1,
 (32, 114): 1,
 (115, 117): 1,
 (99, 99): 1,
 (115, 102): 1,
 (108, 32): 3,
 (256, 99): 1,
 (32, 101): 2,
 (104, 32): 2,
 (32, 269): 1,
 (269, 108): 1,
 (114, 262): 1,
 (262, 263): 1,
 (32, 105): 2,
 (257, 103): 1,
 (103, 111): 2,
 (111, 266): 2,
 (266, 261): 2,
 (32, 121): 1,
 (121, 101): 1,
 (261, 116): 1,
 (116, 46): 1,
 (65, 257): 1,
 (257, 271): 1,
 (271, 226): 1,
 (153, 257): 1,
 (257, 101): 1,
 (100, 97): 1,
 (116, 263): 2,
 (116, 257): 1,
 (257, 102): 2,
 (115, 263): 1,
 (263, 108): 1,
 (103, 263): 1,
 (32, 72): 1,
 (72, 111): 1,
 (119, 101): 1,
 (264, 44): 2,
 (120, 105): 1,
 (112, 262): 1,
 (262, 116): 1,
 (264, 110): 1,
 (110, 257): 1,
 (108, 112): 1,
 (110, 99): 1,
 (116, 266): 1,
 (266, 97): 2,
 (97, 121): 3,
 (273, 267): 1,
 (267, 102): 1,
 (102, 264): 2,
 (116, 258): 1,
 (258, 117): 1,
 (117, 256): 1,
 (98, 256): 1,
 (256, 117): 1,
 (273, 258): 1,
 (117, 268): 1,
 (268, 46): 1,
 (84, 111): 1,
 (256, 111): 1,
 (270, 99): 1,
 (105, 100): 1,
 (99, 107): 1,
 (107, 44): 1,
 (273, 104): 1,
 (119, 260): 1,
 (270, 114): 1,
 (111, 108): 1,
 (267, 261): 2,
 (260, 266): 1,
 (109, 260): 1,
 (261, 260): 1,
 (265, 97): 1,
 (76, 32): 1,
 (264, 46): 1}

Encoding

Given a string, what are the tokens?

def encode(text):
    tokens = list(text.encode('utf-8'))
    while True:
        # this time we dont care for frequency of pairs
        raw_pairs_in_sequence = get_most_frequent_pair(tokens)
        # get the lowest index pair (earliest merges first
        # all pairs inside raw_pairs_in_sequence, get index, get pair with min number index
        # float inf is fallback for non merging pairs
        # pair returns the most eligible merging pair from merged list
        pair = min(raw_pairs_in_sequence, key=lambda p: merges.get(p, float('inf')))
        if pair not in merges:
            break  # nothing else can be merged
        idx = merges[pair]
        tokens = merge_tokens(tokens, pair, idx)
    return tokens
print(encode("hello, world!"))

[104, 101, 108, 108, 111, 44, 32, 119, 111, 114, 108, 100, 33]