What Is a Token in AI? The Jargon of AI Explained for Investors
- Noel Ong
- 10 hours ago
- 17 min read
I use AI in my own work, and I can see how far it extends what one person can do. But when this industry talks about itself, the words get in the way. This Insight takes one real presentation by a leading AI company, quotes the jargon exactly as spoken, and translates it piece by piece, so that by the end you can read any AI announcement and know what is actually being said.
1.00 — WHY I AM WRITING THIS
The language got between me and the industry - The AI Token Mystery
I am writing this Insight for a simple reason: the language of AI still confuses me. I use AI, and I can see how far it extends what one person can do. But when this industry talks about itself, the words get in the way.
To understand any industry, you have to understand what its people are talking about. Nobody in resources would expect a newcomer to know what JORC means, yet AI announcements assume the whole world speaks their language. What settled it for me was a recent presentation by Kimi, one of the Chinese AI companies now competing with the biggest American labs. I watched it wanting to learn, and the words were mind-numbing: token efficiency, optimisers, context length, agent swarms. If I was going to understand what these companies are building, and what the market is paying for, I had to decode the language first. This Insight is that decoding, for me as much as for you.
The presentation was given by Yang Zhilin, the founder of Moonshot AI, the Beijing company behind the Kimi models, at NVIDIA's GTC conference in San Jose on 19 March 2026. The session was called "How We Scaled Kimi K2.5". Here is a taste of it, quoted as spoken:
If that reads like another language, this piece is for you. By the end, you will be able to re-read that paragraph and follow every word. And because one company from Beijing is not the whole industry, each section pairs the Kimi talk with a second voice, OpenAI, Anthropic, Google, Microsoft, NVIDIA, Meta or xAI, saying the same kind of thing in their own words. The jargon is not one company's habit. It is the working language of a global industry that now carries trillion-dollar price tags.
2.00 — THE UMBRELLA WORDS: AI, MACHINE LEARNING AND WHAT AN LLM ACTUALLY IS
Five nested words, and the engine behind the app
Most confusion starts above the jargon, with words that sound interchangeable but are not. They nest inside each other like Russian dolls, and FIG. 01 draws the nesting.
Artificial intelligence (AI) is the whole field: any attempt to make computers do things that normally need human intelligence. Machine learning is the approach that now dominates the field: instead of programmers writing rules, the computer works out its own rules from examples. Deep learning is machine learning done with a neural network, a program loosely inspired by the brain, built from many layers of simple connected units. Generative AI is the branch that produces new material, text, images, video and computer code, rather than just sorting or scoring things. And a large language model (LLM) is the kind of generative AI behind the chat tools: a very large neural network trained on enormous amounts of text to predict what comes next. The letters in GPT describe exactly that: Generative, Pre-trained, Transformer, where the transformer is the 2017 model design that made today's AI work at scale. You do not need to know how a transformer works, only that when you read the word, it means the standard engine design the whole industry builds on.
The other distinction worth having is between the engine, the product and the company. The model is the engine. The chatbot or app is the car built around it. The company is the maker. OpenAI makes the GPT models that power ChatGPT. Anthropic makes Claude. Google makes Gemini. xAI makes Grok. Meta built Llama. Moonshot AI makes Kimi. When a headline says "GPT-5.6 beats Gemini", it is comparing engines, not apps, and certainly not businesses.
3.00 — THE TOKEN: THE UNIT EVERYTHING ELSE IS PRICED IN
A small piece of a word, and the industry's whole accounting system
Now the word in this article's title. A token is a small chunk of text, roughly three-quarters of an English word. Before an AI model reads anything, the text is chopped into these chunks, a step called tokenisation, and everything the model reads and writes is counted in them. OpenAI's own documentation gives the rule of thumb: 1,000 tokens is about 750 words. FIG. 02 shows a sentence cut into tokens; FIG. 03 puts sizes in scale.
Why does one small word carry so much weight? Because the token plays three roles at once, and each role turns up in a different kind of announcement.
The price role first. FIG. 04 compares what the major labs charged per million tokens in late July 2026. Two things stand out. Output tokens cost more than input tokens everywhere, because reading is done in one parallel pass while writing happens one token at a time and takes more computing effort. And the spread is enormous: from about US$0.28 per million output tokens at the cheapest Chinese model to US$90 at OpenAI's dearest professional tier, a gap of roughly 320 times. Price is a choice about quality, speed and brand, not a fixed cost of the technology.
Then the usage role, which produced the most quoted number of 2026. Google's chief executive Sundar Pichai told the company's I/O conference in May that Google's products went from processing 9.7 trillion tokens a month two years ago to about 480 trillion a year ago, to "over 3.2 quadrillion per month" now, with its developer services handling "roughly 19 billion tokens per minute". FIG. 05 draws that curve. NVIDIA's founder Jensen Huang put the same idea in one sentence at GTC, the same conference where the Kimi talk was given: "Tokens are the building blocks of AI... your data center is now a factory to generate tokens."
4.00 — TRAINING AND INFERENCE: BUILDING THE PLANT AND RUNNING IT
The two lives of a model, and the words that live inside each
Every AI model has two lives. Training is the first: the model is shown trillions of tokens of text and slowly adjusts itself until it can predict language well. It happens once, takes months, and costs an enormous amount. Inference is the second: the finished model answers questions, every day, for everyone. It is the recurring cost of every token sold. FIG. 06 draws the split, in a shape any resources investor will recognise.
Inside the model sit its parameters, also called weights: the billions of numbers the model adjusted during training, which hold everything it learned. Model sizes are quoted in parameters the way deposits are quoted in tonnes. Kimi K2, the model in our presentation, has one trillion of them. During training, progress is tracked by a single error score called the loss: the lower the loss, the better the model predicts. The arithmetic runs on GPUs, graphics processors from NVIDIA and others that turned out to be ideal for AI, or on TPUs, the equivalent chips Google designs for itself.
And then there is the word that made me stop the Kimi video: the optimiser. The optimiser is the recipe the model follows while it learns, the set of rules deciding how much to adjust after each mistake. Almost the whole industry uses a 2014-era recipe called Adam. Moonshot's claim at GTC was that its new optimiser, called Muon (and MuonClip once stabilised), gets roughly twice the learning out of the same data. That is why Yang Zhilin said the line quoted in Section 1.00: with a two-times token efficiency, "50 trillion tokens" of data behaves like 100 trillion. Not magic; a better recipe. The claim is the vendor's own, from its own paper, and is best read that way.
5.00 — SCALING LAWS, TOKEN EFFICIENCY AND THE DATA WALL
Why "bigger" has rules, and why the data is running out
The AI boom rests on an empirical observation called the scaling laws: make the model bigger, feed it more data and give it more computing power, all together, and its results improve along a curve you can predict in advance. The landmark papers behind this were published by OpenAI researchers in 2020 and DeepMind researchers in 2022, and the industry has bet hundreds of billions of dollars that the curve keeps holding. OpenAI's Sam Altman stated it plainly in a 2025 essay: "The intelligence of an AI model roughly equals the log of the resources used to train and run it."
But one of those three inputs is not like the others. Chips can be bought and models can be enlarged, yet the supply of good training text is finite. The industry calls this the data wall: the highest-quality human-written text, roughly speaking the useful part of the internet, has largely already been used. Anthropic's chief executive Dario Amodei put the underlying puzzle well in a February 2026 interview: "Humans don't see trillions of words. There is an actual sample efficiency difference here", meaning people learn language from far less data than models need. Two responses dominate. One is synthetic data: text generated by AI models to train other AI models. The other is token efficiency: squeezing more learning out of every token you already have, which is exactly what Moonshot's optimiser claim is about, and why a dry-sounding piece of mathematics was the headline of a major conference talk.
6.00 — CONTEXT WINDOWS: HOW MUCH THE MODEL CAN HOLD IN ITS HEAD
The working memory, measured in tokens
The context window is the model's working memory: the maximum number of tokens it can consider at once, covering both what you give it and what it writes back. Once a conversation or document exceeds the window, the earliest material falls out of view. Window sizes have become a headline specification, and FIG. 07 compares the current flagships.
Why the arms race for bigger windows? Because the window sets what kind of work the model can take on. A model that holds a paragraph can answer a question. A model that holds a data room can audit it. In the Kimi talk, this is the second of the three things Moonshot says it scaled: their Kimi Linear design mixes a cheaper form of attention (the mechanism the model uses to weigh which earlier tokens matter) with the standard, expensive kind, in a three-to-one ratio, precisely to make very long working memories affordable. The stated goal: an AI that can run "for days or even weeks" on a single task without losing the thread. Anthropic's marketing makes the same promise from the other side of the Pacific: its Claude Sonnet 4.5 launch boasted that the model "maintains focus for more than 30 hours on complex, multi-step tasks".
7.00 — GETTING ANSWERS OUT: PROMPTS, REASONING, MULTIMODAL AND LOOKING THINGS UP
The words you meet when you actually use the tools
The vocabulary so far describes how models are built. This section covers the words that describe using them, the ones an investor meets in product launches rather than research papers.
A prompt is simply the instruction you type, and prompt engineering is the craft of writing instructions that get better results. A reasoning model is the industry's newest product category: a model that can spend extra time working through a hard problem step by step before answering, producing an internal chain of thought. The extra thinking happens at answer time, so the industry calls it test-time compute, and the crucial commercial detail is that the thinking is billed in tokens like everything else. NVIDIA's Jensen Huang explained it at CES in January 2026 in one line: "We also have test-time scaling, which is another way of saying thinking. You think in real time." And in another: "Instead of a one-shot answer, inference is now a thinking process."
Multimodal means one model handling more than one kind of material: text, images, audio and video together. The Kimi presentation is a worked example: Moonshot says K2.5 was the first open model trained on vision and text jointly "from day one" rather than bolting image skills onto a finished text model, and reported a result worth noticing: training the model on vision tasks alone improved its text results too, and a strong text model lifted its vision results. The claim is the vendor's own, but the direction, single models that see and read, is industry-wide.
Then there is the machinery for making a model useful on your information. RAG, retrieval-augmented generation, lets a model look answers up in a nominated set of documents instead of relying on what it absorbed in training, which sharply reduces invention. The look-up usually runs on a vector database, a special store that searches by meaning rather than by exact words. Samso has covered vector databases before, in the AI & Capital piece "How to buy the AI build-out on the ASX", where our research found there is no pure-play way to buy one on the ASX. Guardrails are the rules wrapped around a model to stop harmful or off-limits outputs.
Which brings us to the industry's most famous word for failure. A hallucination is a confident, fluent, wrong answer. It happens because a language model is a prediction machine: it produces the most plausible next tokens, and plausible is not the same as true. The rate has fallen with each generation, but it has not reached zero, and no vendor claims it has.
8.00 — AGENTS AND AGENT SWARMS: FROM ANSWERING TO DOING
The industry's favourite word of 2026
An agent is an AI model given tools and permission to take steps on its own: search the web, run computer code, file the result, then decide what to do next. The adjective agentic now appears in nearly every AI announcement, and it marks the industry's core shift: from software that answers questions to software that completes tasks.
The Kimi presentation's third big idea takes this one step further. In an agent swarm, an orchestrator (a lead agent) breaks a large job into pieces and hands them to a team of sub-agents working in parallel, then collects and assembles the results. Yang Zhilin's own analogy was a company: a chief executive decomposing work across departments, with researchers, developers and fact-checkers each doing their part. FIG. 08 redraws the structure he described. Moonshot's stated ambition is swarms of hundreds or a thousand agents finishing tasks fast enough "to produce real economical value".
Every major lab is saying a version of the same thing. Google's Sundar Pichai told I/O 2026 "we're firmly in our agentic Gemini era". Microsoft's Satya Nadella told Build 2026 that "agents effectively are a new execution environment... a new paradigm", and described a security system where "over 100 specialized agents are working together to discover, debate and prove exploitable vulnerabilities end to end". Anthropic's Claude launches lead on "long-running agents" that keep checking their own work. When you read "agentic" in an announcement, this is the whole of what is meant: the software acts, in steps, with tools, increasingly in teams.
9.00 — THE WORDS OF THE RACE: OPEN WEIGHTS, FRONTIER, AGI AND THE REST
The vocabulary of ambition, and what it is worth
The last group of words describes not the machinery but the race itself, and these are the words that move markets.
Open weights versus closed is the industry's deepest divide. An open-weight model's parameters can be downloaded and run by anyone, free: this is Kimi, DeepSeek and Alibaba's Qwen, and historically Meta's Llama. A closed or proprietary model can only be used through the maker's paid service: OpenAI's, Anthropic's and Google's flagships. Note what "open" does not include: usually not the training data or the full recipe, so open weights is not quite "open source" in the traditional software sense. The Kimi talk's framing, that open models are "closing the gap" with proprietary ones, is now conventional wisdom even at NVIDIA, whose chief executive told Axios in July 2026: "These Chinese models are excellent... Free AI should be great for chips."
That the divide is strategy rather than ideology was proven in April 2026, when Meta, the company that built its AI reputation on giving Llama away, launched the first model from its Superintelligence Labs, Muse Spark, as a closed product, saying only that "we hope to open-source future versions of the model". The open champion went closed the moment it believed it had a lead; the challengers stay open because free distribution is how a challenger wins users from an incumbent.
Then the ambition words. A frontier model is simply one of the handful of most capable models in the world at a given moment. AGI, artificial general intelligence, means an AI able to match humans across essentially all thinking work, and superintelligence means exceeding us. These are the words the money is raised on, and the industry's leaders now use them as near-term forecasts, on the record: Mark Zuckerberg in July 2025: "Developing superintelligence is now in sight." Sam Altman in June 2025: "Humanity is close to building digital superintelligence." Google DeepMind's Demis Hassabis on AGI, in 2026: "2030 is when I expect it to arrive, either plus or minus a year." Elon Musk went furthest in a January 2026 podcast, saying he expected AGI "next year". Alignment and safety name the work of keeping such systems doing what we intend; Anthropic in particular builds its brand on it.
Three more words complete the set, all live in 2026. Distillation is training a cheap model on an expensive model's answers, the student copying the master's homework; in July 2026, Washington accused Moonshot of doing exactly this to American models, an accusation that is reported and denied, not proven. Sovereign AI is a country deciding it needs its own models, data and data centres rather than renting someone else's, the phrase behind national AI programmes worldwide, including the partnership between OpenAI and Australia's NextDC. And a wrapper is investor slang for a thin product built on someone else's model: worth remembering, because a company whose "AI product" is a wrapper owns a user interface, not the technology.
10.00 — FIVE QUESTIONS TO ASK OF ANY AI ANNOUNCEMENT
The vocabulary, put to work
Learning the words is only useful if it changes how you read. Here are five questions the vocabulary now lets you ask of any AI announcement, product launch or "AI strategy" slide.
What is the unit? Tokens, parameters, users, gigawatts or dollars? Each unit flatters a different thing. A company quoting tokens is bragging about volume; parameters, about size; neither is revenue.
Is that a usage number or a money number? "3.2 quadrillion tokens a month" measures work done, not dollars earned, and says nothing at all about profit.
Who measured it? Vendor benchmarks are chosen by the vendor, from a rotating menu, after the fact. "Best on X" means best on the test they picked. Independent verification is rare and valuable.
Which life of the model is the money for? Training (the build) or inference (the running)? Capital raised "for compute" can mean either, and they have completely different payback profiles.
Who pays whom? Agents consume tokens; token sellers promote agents. Chipmakers invest in labs that buy chips. Follow the direction of the enthusiasm and check whether it matches the direction of the money.
And now, the test. Here is the passage from Section 1.00 again: "Suppose you have 50 trillion high-quality tokens, and then you apply this new optimizer... all of a sudden you have a two times token efficiency... We introduce this new learning paradigm of agent swarms... at the end of the day, we're going to have a swarm of agents that each of them have a super long context." Read it once more. A company with a fixed supply of training text says its new learning recipe doubles what that text teaches its model; and its software now works as coordinated teams, each member able to hold enormous amounts of material in working memory. That is all it ever said. The ideas are ambitious; the language was the only barrier.
The condensed decoder below covers the terms this piece taught. The full glossary, more than 45 terms in plain English, lives permanently on the Samso website, with the downloadable Samso AI Decoder PDF, and every future Samso piece on AI links back to it.
References & sources
This piece is built on one primary transcript (supplied) and the public statements, documents and pricing pages listed below. Quotes are reproduced verbatim from official transcripts or company pages except where noted as reported through a third-party transcript. Market-sensitive and dated figures (prices, context windows, token volumes) are quoted as at the dates shown and must be refreshed on publication day. No locator map appears because the subject is not geographic (house rule 4.2, waived). All figures are original Samso illustrations.
Yang Zhilin (founder, Moonshot AI) — "How We Scaled Kimi K2.5", NVIDIA GTC, San Jose, 19 March 2026. Transcript supplied by Samso; session corroborated by Constellation Research, China Biz Insider and 36Kr coverage. Source of: the opening quotation; MuonClip optimiser and the two-times token efficiency claim; Kimi K2's one trillion parameters; Kimi Linear and the three-to-one attention mix; the agent swarm design and company analogy; K2.5's joint vision-and-text training claims. All capability claims are the company's own account.
Jensen Huang (NVIDIA) — GTC 2026 keynote, 16 March 2026, official NVIDIA transcript ("Tokens are the building blocks of AI"; "your data center... is now a factory to generate tokens"; "three scaling laws"); CES 2026 keynote, January 2026, Rev.com transcript ("test-time scaling... another way of saying thinking"; "inference is now a thinking process"); interview with Axios, 22 July 2026 ("These Chinese models are excellent"; "Free AI should be great for chips").
Sundar Pichai (Google) — I/O 2026 opening keynote, 19 May 2026, official written version at blog.google. Source of: 9.7 trillion / ~480 trillion / ~3.2 quadrillion monthly tokens; "roughly 19 billion tokens per minute"; "we're firmly in our agentic Gemini era".
Satya Nadella (Microsoft) — Build 2026 opening keynote, 2 June 2026, official Microsoft transcript. Source of: "tokens per dollar per watt"; "agents effectively are a new execution environment"; the 100-plus-agent security example.
Sam Altman (OpenAI) — essays at blog.samaltman.com: "Three Observations" (February 2025; the log-of-resources scaling line), "The Gentle Singularity" (June 2025; "Humanity is close to building digital superintelligence").
Dario Amodei (Anthropic) — interview on the Dwarkesh Podcast, February 2026, quoted from the published transcript ("Humans don't see trillions of words..."); Anthropic, "Claude Sonnet 4.5" launch post, 29 September 2025 ("maintains focus for more than 30 hours").
Mark Zuckerberg / Meta — "Personal Superintelligence" letter, meta.com, 30 July 2025 ("Developing superintelligence is now in sight"); "Introducing Muse Spark", about.fb.com, 8 April 2026 ("we hope to open-source future versions of the model").
Demis Hassabis (Google DeepMind) — interview with Fast Company, 22 May 2026 (AGI "2030... plus or minus a year"); Yann LeCun — Davos panel remarks as reported by Fortune, 23 January 2026 ("completely LLM-pilled"; "digging the same trench").
Elon Musk (xAI) — post on X, 17 January 2026 (Colossus 2 "First Gigawatt training cluster in the world"); Moonshots podcast, January 2026, quoted from the published transcript (AGI "next year"). Podcast-transcript quotes to be re-verified against audio before publication.
Pricing and specifications — official pricing/documentation pages of OpenAI, Anthropic, Google, xAI, Moonshot AI and DeepSeek, all fetched 26 July 2026. Known conflicts noted in the FIG. 04 and FIG. 07 captions (Claude context-window tiers; OpenAI's newest window unconfirmed on an official page; DeepSeek's launch-versus-current prices). Refresh all on publication day.
OpenAI Help Center — "What are tokens and how to count them" (the 1,000-tokens-per-750-words rule).
Scaling-law literature — Kaplan et al., "Scaling Laws for Neural Language Models" (2020); Hoffmann et al. ("Chinchilla"), "Training Compute-Optimal Large Language Models" (2022).
Moonshot distillation accusation — reported by US press in July 2026 following the Kimi K3 release; an accusation, denied and unproven, and described as such in Section 9.00.
Photographs (PLATE 1 to PLATE 5) — Unsplash, used under the Unsplash licence: Maximus Beaumont (taxi meter); Taylor Vick (data centre); Vishnu Mohanan (circuit board); Susan Q Yin (library); Frames For Your Heart (construction cranes). All retrieved 27 July 2026.
Samso archive — "Tokens Are the Bill, Not the Value" (AI & Capital, Part 1, published 25 July 2026; source of the volume-versus-value framing revisited in Sections 3.00 and 10.00, and first publication of the Google token-volume figures redrawn here as FIG. 05) and "How to buy the AI build-out on the ASX" (AI & Capital, Part 2; source of the ASX vector-database finding cited in Section 7.00; link to be added on publication).
The Samso Way – Seek the Research
Here at Samso, we pride ourselves on delivering content for investors that is independent and informed by over three decades of experience in the industry. Our content is well-researched and is only created if I see merit in discussing the company's story.
Our mission is simple: cut through the noise and spotlight what matters—genuine stories, grounded insights, and real opportunity.
Our content is well-researched and is only created if the team sees merit in discussing the company or concept. Investors can explore our three core platforms:
There may be numerous paths to success in investing, but the common thread among successful individuals is that they remain committed to making informed decisions. Equip yourself with the right knowledge and tools, and you will be well on your way to achieving your financial goals.
Most importantly, investors need to be absolutely diligent in understanding their own risk-reward tolerance and capabilities. Never bite off more than you can chew. As they say, Rome wasn’t built in a day, and the Great Wall stood because it took centuries to complete.
The Samso Philosophy:
Stay curious. Stay sharp. And remember—digging deeper always uncovers the real value.
In Life, there is no such thing as a Free Lunch.
Never bite off more than you can chew is my parting comment.
Happy Investing, and the only four-letter word you need to know is DYOR.
To support our independent work, please head over to our Support Page and give us a helping hand in any of the ways listed. This is a new initiative for the Samso Platform, and it was always the concept of Samso when we started this journey in 2018.
Disclaimer
The information or opinions provided herein do not constitute investment advice, an offer, or solicitation to subscribe for, purchase, or sell the investment product(s) mentioned herein. It does not take into consideration, nor have any regard to your specific investment objectives, financial situation, risk profile, tax position and particular, or unique needs and constraints.
Samso is a trusted platform that equips dedicated investors with up-to-date industry knowledge and insights from top CEOs and thought leaders. By staying informed on business advancements and market trends, investors can enhance their financial decisions through a combination of expert guidance and their own research.
Samso Insights | www.samso.com.au | An Investor Lens on ASX-Listed Companies