1. Introduction
The emergence of LLMs (large language models) has impacted conversation around artificial intelligence in virtually every field imaginable. It is now one of the most important topics when it comes to computer science, and massive amounts of resources are being directed towards the development and research of these models in hopes of eventually developing what is known as AGI (Artificial General Intelligence). These models generate coherent text, answer questions, and even simulate reasoning processes that resemble human thought. This fluency and human-like speech has led to the belief that these systems “understand” language or exhibit “intelligence”. Much research has been done into the anthropomorphization of LLMs (Cohn et al. 2024) and the trust that humans have in these systems (Colombatto et al. 2025), but there is a much deeper question we should be asking ourselves: does the current design of LLMs even allow for intelligence to occur?
To evaluate the intelligence of LLMs, we must understand the role of language and tokens. During training, these models are exposed to a vast corpus of human-generated textual data, which is transformed into sequences of tokens. Through analyzing patterns in the appearance of these tokens, the parameters of LLMs acquire certain weights (this is the process we call learning). Once trained, these parameters are fixed and do not continue to adapt. When deployed, these models receive an input, transform it into tokens, perform a series of calculations by using their weights, and produce an output that predicts the series of tokens following the input. This is what we call Generative AI. It is worthy of note that this computation is deterministic, and the randomness of the results we observe in practice is externally induced (temperature, pseudo-random number generation…).
This essay argues that, by design, LLMs cannot possess intelligence or even understanding in any meaningful way. Furthermore, this essay argues that no model under current machine learning techniques can ever present these traits. The architecture of LLMs confines them to a closed system of linguistic correlation, devoid of grounding in the external world. The following sections will analyze this claim in depth, beginning with an explanation of the relationship between language and meaning, and then exploring how LLMs cannot produce any meaningful or logical statements about the world.
2. Language as a Picture of the World
The relationship between symbol and meaning is at the core of all theories of knowledge. Many of you probably know about René Magritte’s painting The Treachery of Images.
This painting presents to the viewer a representation of a pipe, while denying that it is the object itself by adding the subtitle “This is not a pipe”. What seems like a joke is actually a commentary on this relationship between symbol and meaning we were talking about: there is nothing intrinsic about the painting that connects it to a pipe. On your screen, this image is a grid of pixels with different numerical values. The individual pixels themselves contain no meaning, yet we recognize them as an image of something familiar when arranged in a certain way. The same applies to language: the letters ‘t’, ‘r’ and ‘e’ carry no meaning themselves, and even when combined into the word “tree” they carry no meaning. The word itself does not contain a tree, that meaning only appears externally through our recognition of what it refers to. Symbols are pictures of reality and do not contain this reality.
This principle extends to all forms of representation. As Wittgenstein wrote, “The world is the totality of facts, not of things.” Logical propositions can be used to form a picture of the world, by describing what is and isn’t the case. However, whether these logical propositions are true or not is not something that can be determined from the propositions themselves. The truth value of propositions lies outside of them. If I describe to you that “my car is red”, this proposition paints a picture of the world, but we can only validate if it’s true or false by comparing it with reality. In other words, the truth value of propositions is not embedded within them but appears externally.
This limitation is akin to the phenomenon illustrated in Bertrand Russell’s Paradox in set theory: a set cannot contain itself. For a set to contain itself, it would also have to contain the set that contains itself — ad infinitum. Since the truth of a proposition only becomes evident via comparison, a picture of reality would paradoxically need to include reality itself in order to possess its truth. Using the earlier example, the proposition “my car is red” would need, by itself, to show me the fact that it is true. The same concept applies to meaning: the symbol “tree” would need to show me its meaning without needing any external comparisons.
When we apply this insight to LLMs, we encounter an obvious truth. The meaning of words cannot be learned from language alone; even the definitions of a dictionary rely on us comparing its sentences with our previous experiences (an external reality). During training, LLMs are fed vast amounts of text that are translated into machine-readable data (tokens, or numerical values). From that data, they learn patterns of co-occurrence: which tokens appear near others, how sequences follow one another, and other types of statistical relationships. In doing so, they construct an internal reality of probabilistic relationships between those machine-readable symbols. However, this world is self-enclosed. As noted above, symbols carry no intrinsic meaning; they only gain meaning when compared with an external reality. However, LLMs never encounter anything beyond machine-readable data. Even if we wanted to describe the meaning of these words, or attach sensors that allow it to “experience” the world, these descriptions and sensor values would need to get translated into machine-readable symbols as well. This means that LLMs, and machine learning models as a whole, can only ever know pictures of reality without the ability to truly compare those pictures with reality. In other words, the training of machine learning models fails to provide them with sense (meaningfulness in relation to reality) at a fundamental level. Even context injection and other techniques we use to “ground” these models, as we will see in a different section, do not supply the LLM with a genuine external reality.
LLMs do not understand why relationships between tokens exist, they do not know what the tokens signify, and they do not even know whether they are true (as we observe through hallucinations). For LLMs to consider these, they would need access to information outside the context they operate in. In this sense, LLMs do not speak our language at all. They instead speak the language of machine-readable symbols, and their reality is limited to their training data. Once training concludes, these relationships no longer evolve. The model cannot observe, reinterpret, or revise them. Saying that “token B follows token A with a 50% probability” conveys nothing about what A or B mean, nor about why this relationship holds. It only reflects how often such a pairing appeared in its training data.
Some might argue that meaning can emerge from co-occurrence, and that patterns of association can be a form of understanding. But this is an illusion born of human interpretation. This meaning lacks any form of sense in relation to our reality, because the machine learning model has no referent for its symbols. The only metric through which LLMs can be judged is through how they describe this training data, and never through how they describe reality.
It suffices to say that, without a connection between machine-readable data and the world, a model’s output lacks sense. It will never know what meaning the output it generates has, whether it is true or not, or whether it corresponds to reality. The output of LLMs only possesses structural value, not meaning.
3. The Foundation of LLMs
To understand the limits of what a large language model can know, we first need to understand what it actually does. As we have explained in the previous section, LLMs do not manipulate meaning. Instead, they manipulate structure. Their goal is not to reason but to calculate how likely a given sequence of symbols is to appear based on a given input. Everything that we perceive as thought, reasoning, or creativity is the result of a deterministic mathematical process that calculates this likelihood.
In simple terms, given the start of a sequence like “The sky is”, the model estimates which token is most likely to follow based on statistical regularities observed during its training phase. It repeats this process, generating tokens until a full sequence is produced. Reasoning models, at their core, are not different from regular LLMs: they simply append instructions to the input and perform a chain of reasoning steps. The mechanics and training of the models don’t change. Each chain of thought can be reduced to a single ‘node’ (the last LLM model that gets called, with the result of the previous models appended to its input).
The learning phase adjusts billions of parameters, or weights, so that the model’s predicted output matches the distribution of text in the dataset as closely as possible. Once this optimization is complete, the model’s parameters are fixed. During inference, it no longer learns: the model limits itself to computing the necessary calculations with its billions of parameters. To put it simply, it puts the user’s input into a formula and produces an output. The result is a machine that does not reason about what words mean, and instead calculates how words tend to occur. This is why LLMs are great at producing human-readable text. Machine learning is an optimization problem, and what the LLMs are optimized around is the distribution of text across their training data.
However, probability does not equal truth. A high probability indicates that a token frequently appears in certain contexts, but it does not indicate the meaning of the token or the propositions in which it appears. For example, if an LLM encounters the phrase “the Earth is flat” many times in its dataset, it’ll become a flat-earther. Not because the reality is that the Earth is flat, but because that is a common pattern of words in its training data. The model has no means of distinguishing between true and false propositions, because the concept of truth lies outside the space of probabilities. This has become an issue that Anthropic itself has researched in the paper “Poisoning Attacks on LLMs Require a Near-Constant Number of Poison Samples”, in which it was discovered that a surprisingly small amount of documents were needed to “poison” the dataset. Poisoning datasets refers to the act of maliciously manipulating training data in order to make LLMs produce an unexpected output upon encountering an input closely related to the attacked tokens. Because there is no meaning inherent in the training corpus, there is no way to automate the removal of misinformation or false statements.
In human cognition, a statement like “the Earth is flat” refers to a real-world thing that can be verified through observation or scientific methods. However, an LLM simply interprets this same sentence as a set of symbols without sense, whose arrangement follows a given pattern. It cannot know what “the”, “Earth”, “is” or “flat” mean. The only thing that the LLM takes into account is the arrangement of these symbols. This limitation of LLMs is not accidental; it is caused by the fact that meaning and sense are unobtainable to the machine. The only thing that machine learning models have access to is the arrangement of symbols, so they cannot predict anything outside that arrangement. Probability encodes correlation, not causation or even meaning. The result is that LLMs reflect how things are written, not what is written. The model’s outputs are, therefore, reflections of linguistic habits.
Some may argue that words have vectors, or embeddings, associated with them that encode contextual relationships between words. Semantically related words correspond to a geometrical closeness in this vector space: words like “red” and “blue”, or “mother” and “father” end up near one another in this vector space.
However, this semantic space is equally as probabilistic as the training data. The proximity between vectors reflects similarity between different words in usage, but it does not provide them with any meaning outside this usage. Embeddings can tell us that two words are used similarly or that a sentence represented in vector space is close to a certain cluster of words (classification), but this again only reflects linguistic habits. This system reflects how words are used in the training set, not what those words mean.
Knowing how LLMs work also lets us see that they cannot be considered to be creative or transformative. The model’s inference is deterministic given its parameters and the seed used for the pseudo-random number generation involved in its output. What appears to be creativity or spontaneity is simply the sampling of tokens according to calculated probabilities, which are later filtered or randomly picked through the model’s temperature. As we mentioned earlier, the parameters that calculate the output of the LLM are obtained in the model’s training. The training process is, itself, deterministic as well. No matter how complex, they are a result of mathematical formulas used to optimize the learning rates of models, to perform gradient descent, and many other operations. As such, given a certain training set and a given input (including the random seed), there is only one possible output. Machine learning models can therefore be thought of as a complex function: given an input A, the result will always be f(A).
In summary, LLMs are not a representation of truth. LLMs are probabilistic by nature and simply describe and reproduce the patterns found in their training data. They describe to us what tends to happen and not what is true, and they are incapable of being creative given that their output is deterministic. The next section will expand on this by examining what “learning” means and why LLMs cannot truly learn past their training phase.
4. The Illusion of Intelligence
If we accept that LLMs cannot derive meaning from language, we must ask whether they can even learn meaning after their initial training. Much of the optimism and hype surrounding LLMs rests on the assumption that, eventually, we will reach AGI. There exists a belief that, through further learning, fine-tuning, context injection, or “agentic design”, we can make LLMs progressively more intelligent. However, none of these methods fix the inherent issues with LLMs that deprive their output of meaning, as we will now see.
Fine-tuning is described as a process in which models are taught new behaviors or values. In the context of LLMs, fine-tuning is used to align LLMs with human expectations through reinforcement learning. During this process, human evaluators compare multiple outputs and indicate the ones that seem better according to their criteria. Through this process, the model is then fine-tuned so that similar responses are favored in future generations. However, this process does not teach the model why one answer is better than another. Furthermore, the principle behind reinforcement learning adds a human subjective element to the design of the language model: its answers don’t have more meaning; its parameters have simply been readjusted in accordance with the biases of the human evaluators.
This means that fine-tuning does not change the underlying mechanisms of LLMs. It favors desirability in the answers, and not understanding or true problem-solving.
Another common approach is to ground models through the injection of context. For this, we use prompt engineering or even Retrieval-Augmented Generation (RAG). The idea is that we append information to the input of the LLM in order to guide its response or to make it answer in accordance with a given context or set of instructions. These methods are often mistaken for genuine intelligence, but they are nothing of the sort. In fact, the apparent success of these methods relies on a mechanism that has been proven to exist in LLMs: sycophancy and repetition. Sycophancy refers to a model’s tendency to excessively agree with its users. Recent studies like “Sycophancy in Large Language Models: Causes and Mitigations” (Lars Malmqvist, 2024) have shown that LLMs systematically mirror user opinions, reinforce biases, and over-align their intentions with the user’s. These issues stem from not only their training data but also limitations with current learning techniques, along with an inability to effectively ground knowledge.
When taking into account what we have discussed earlier, the reason why sycophancy and repetition exist is obvious. In fact, there is no other way for LLMs to operate than through these issues. Because LLMs are trained to continue sequences of human language, their goal is to produce the most probable continuation of the input they are given. In other words, the model extends the user’s text without evaluating it (as evaluating it would require LLMs to possess sense). The result is a structural bias toward agreement and continuity. The mathematics of prediction favor completion over contradiction.
This is also why repetition frequently appears in model outputs. LLMs overproduce phrases and restate information due to their autoregressive nature (Wang et al. 2024b), which leads to repetition. This is another major issue, as it is an obstacle towards diversity in the output produced by LLMs.
Simply put, the reason why RAG techniques and context injection seem to work is because they exploit the model’s built-in sycophancy. They inject words and sentences that the model is then compelled to agree with and repeat, which makes LLMs great at summarization of information and presenting already stated information. Without this sycophancy and repetition, LLMs would most likely suffer from more hallucinations and be useless at the tasks they are currently used for. In reality, LLMs simply mimic and agree with the user’s provided input. This bias is architectural, as LLMs are optimized to favor creating human-like text. With current machine learning techniques, it cannot be eliminated or fully mitigated. In other words, machine learning as a field cannot transcend this bias, as it is an inherent part of optimization: the structure of training a model and then performing inference does not allow for intelligence to appear at any point.
Recent frameworks attempt to overcome these limitations by building agentic systems, environments where LLMs act as reasoning agents that coordinate various tools. These systems chain multiple model calls together, sometimes supported by non-LLM algorithms or models, to simulate planning and reflection. They appear to reason, self-correct, or coordinate tasks. They are undoubtedly effective at performing various tasks, but they are far from intelligent. The chain-of-thought that agentic systems rely on is still token prediction: its intelligence resides outside the model, in the human-designed logic that calls and evaluates it.
For an agentic system to be reliable, the person building the prompts and instructions for the LLM coordinating the various tools would have to indicate to it what to output in any given scenario. If the solution to a given problem is not given to the LLM in its training data or in the prompt, then its answer will only be structurally sound but lack sense. A common experiment to test reasoning models and agentic systems is to attempt to make various LLMs beat Pokémon Red. I believe the issues encountered with memory in these playthroughs perfectly highlight this lack of ability in problem-solving. The approach used in these playthroughs can be summarized as follows: provide the LLM with the current state of the game and what options it has, along with text describing the outcomes of what it previously has done. Often, the LLM ended up in recursive feedback loops and repeated past actions, similar to the traits we had previously talked about. This is a clear example of how LLMs are incapable of problem solving, unless they are already given the answer.
In other words, agentic systems can only solve problems if the agents provide the exact solution to the LLM, and we provide the LLM with exact instructions on how to use the agents it has access to. As we have explained, machine learning cannot do actual problem-solving under current machine learning architectures. In conclusion, for an agentic system to solve a problem, a human has to have solved the problem (or generalized its solution), constructed the necessary tools to reproduce the solution, and given the LLM exact instructions on how to use these tools. Agentic systems cannot provide solutions to unsolved problems, as they lack the sense to do so.
These limitations reveal that, with our current understanding of machine learning and learning as a whole, we cannot produce true artificial intelligence. AGI is, at present, impossible. All of our so-called improvements simply make use of the LLMs limitations to produce more favorable answers. They do not solve the limitations. None give the model access to actual knowledge grounding, or the adaptive capacity necessary for real understanding. Sycophancy and repetition show us that models are not learning, but acting as echo chambers for the user. Agentic systems demonstrate that the intelligence we attribute to LLMs comes from the designer of said system, not from the model itself.
The pattern of using training data, freezing parameters, and then performing inference is insufficient for problem-solving. Prompts, RAG techniques, and agentic systems will never produce AGI. That would require humans to describe to said agentic system what to respond in every single scenario it could encounter; in other words, humans need to reach AGI to produce an AGI. The existence of AGI is paradoxical.
5. Conclusion
Throughout this essay, we have examined the foundations of large language models from both a philosophical and technical perspective, and we have found that they lack a way to produce true intelligence. Machine learning is, at its core, a problem of optimization. LLMs are not designed to produce solutions to answers; they are optimized to produce human-like text.
Language, as we have seen, is only a picture of reality. Words are symbols that gain meaning only through their relation to the world, but LLMs cannot be taught this relation as it lies outside of language itself. The lack of sense in the training of machine learning models is not something that can be avoided, because all training data needs to be translated into something that the machine can interpret. In this translation, all possible meaning in relation to the real world is lost. Machine learning models can only use what can be inferred from the patterns of these machine-readable symbols. Truth and meaning lie outside these machine-readable symbols, so the output of machine learning models lacks sense.
LLMs cannot reason, and they are not creative either. They can only transform a given input through entirely deterministic systems (given a seed for any pseudo-random number generation that produces artificial un-deterministic behavior), so their output is determined entirely by their training data. The only reason humans attribute reasoning and creativeness to AI is because of our trend to anthropomorphize these systems, since they produce an output that is human-like. This is because LLMs are optimized to produce human-like output on a purely structural level and not a meaningful one.
Fine-tuning, prompt engineering, and context injection do not add intelligence to LLMs. They use the model’s limitations, sycophancy and repetition, to produce more desirable results. This makes LLMs agree with the user and repeat the information given to it in its input or in its training data. These limitations cannot be corrected through training data or through the previously mentioned techniques. They are the natural outcome of a system built with current machine learning architectures, and to correct them would require rethinking the field of machine learning and what it means to learn.
Even agentic systems do not overcome this barrier. Their intelligence lies entirely in the structure designed by the humans who have crafted the system. An LLM can only solve a task if the instructions it has been given, or its training data, contain the solution to said task explicitly or implicitly. In other words, the intelligence of these systems remains entirely external. AGI can never be achieved with our current understanding of machine learning, because achieving AGI requires humans to achieve AGI so they can tell it exactly how to act in every possible scenario. AGI is a paradox.
Machine learning, as we currently practice it, does not allow for true artificial intelligence. It is a method of optimization that is always deterministic in nature. It is difficult to argue that a deterministic system can be considered intelligent in any sense of the word. To achieve artificial intelligence, we must rethink what it means for a machine to learn. Until a different architecture exists, LLMs will remain reflections of their training data that lack sense, intelligence, or creativity.
Artificial Intelligence, at the present time, cannot exist.
References:
- Wittgenstein, Ludwig. (1922). Tractatus Logico-Philosophicus.
- Russell, Bertrand. (1903). The Principles of Mathematics. Cambridge University Press.
- Cohn, M., et al. (2024). Believing Anthropomorphism: Examining the Role of Anthropomorphic Cues on Trust in Large Language Models. arXiv:2405.06079
- Colombatto, C., Birch, J. & Fleming, S.M. (2025). The influence of mental state attributions on trust in large language models. Commun Psychol 3, 84 (2025).
- Malmqvist, Lars. (2024). Sycophancy in Large Language Models: Causes and Mitigations arXiv:2411.15287.
- Wang, et al. (2024b). Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editing.
- Anthropic. (2025). Poisoning Attacks on LLMs Require a Near-Constant Number of Poison Samples.
- Harnad, Stevan. (1990). The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1–3), 335–346.