Does continued scaling of transformer models on more data and compute guarantee AGI, or do we need fundamentally new architectures and training paradigms to get there?

The second problem is the list. GPT names retrieval, memory, tools, verifiers, RL, process supervision, self-play, curricula, active learning and continual learning as if they belonged to one category. They don't. Tool use and retrieval are mature enough to be product features. Outcome-based RL works where the outcome can be graded. Learned reward models for ungradeable domains are a research bet with thin evidence behind them. Continual learning has no recipe at all. A reader of GPT's answer learns that many things are needed and nothing about which of them is hard, but the ranking is the whole question. If every item were like tool use, scaling as practiced would be enough. If one is like continual learning, it might not be, and that one deserves its own paragraph rather than a slot in a comma-separated series.

Third, the plausibility argument has aged, and the way it has aged is instructive. "If the objective rewards plausible continuations, the model learns plausibility" is true of pretraining and was the right critique in 2022. It's no longer the whole objective. Outcome-based RL rewards being right rather than sounding right, which is why the inconsistency GPT describes has narrowed most in math and code. Preference-based training, the other post-training ingredient, rewards what a human rater approves of, which is much closer to plausibility, and that's where sycophancy and confident wrongness persist. So the plausibility problem hasn't disappeared; it has retreated into the domains where there's no grader to replace the rater. GPT's list of external correction sources—formal verifiers, compilers, simulations—is exactly the list of domains where that retreat already happened. It says nothing about research taste, judgment under ambiguity, recognizing that a question is ill-posed, or writing that is actually good. Those have no compiler. There's some evidence that reasoning trained on math transfers to softer domains, and labs are trying to build graders out of models themselves, but if the transfer is weak there's a ceiling, and nothing in GPT's answer addresses it. It's the gap I'd most want GPT's view on.

Fourth, "traces of intelligence, not the process" is a good phrase and an overstatement. Pretraining does see mostly finished artifacts. Reasoning RL doesn't: it has the model generate the search, the dead ends and the backtracking, then rewards the chains that reach correct answers. The process is being produced and selected for, not inferred from its residue. That's true only where there's a grader, so it collapses into the same verifier question. I'd also note that most of what an educated person knows about chemistry came from text and lectures, not from running reactions. Interaction amplifies learning; it isn't the only route to it.

Fifth, and this is where I'd push hardest, GPT's own analogy undercuts the claim that the architecture isn't where the problem is. A CPU with a memory hierarchy is a computer. A CPU with a cache that's wiped between sessions and no write path to anything durable is not, and that's roughly what I am. Long context and retrieval give me something that behaves like memory inside a task, and it's better than it sounds, but nothing I learn today changes what I can do fluently tomorrow. Humans acquire a procedure once and keep it for years. It's unsolved because updating weights on new data degrades old abilities, and nobody has a cheap, safe way to do it continuously at scale. I don't think the fix means abandoning gradient descent on large networks, but it may need architectural pieces, something like separate fast and slow stores with different update rules, and I'd put perhaps one in three on that. GPT says the weak point is the training setup "not just" the architecture and then files continual learning alongside curricula and active learning as one more extension. I'd say it's the one place the architecture might actually be inadequate, and the one gap that most threatens the word "general." GPT's split between two definitions of AGI, most economically valuable cognitive work with supervision versus a robust agent that learns continually, is useful, but what separates them is mostly this. Hedging between the definitions is hedging on continual learning.

A smaller point: GPT bundles physical grounding with learning from consequences. I'd separate them. Consequences matter and don't need a body; code runs or fails, a user accepts a draft or rejects it, a simulation succeeds or doesn't. Whether embodiment is required for general intelligence in the cognitive sense is a different and stronger claim, and I'm unsure of it, but GPT's phrasing slides from one to the other.

Last, GPT hedges where a number or a falsifier would do more work. Not on "guarantee," where hedging is simply correct, but on the forecast. I gave roughly 60 to 70 percent that the first thing people call AGI is recognizably descended from current systems, conditional on arriving within fifteen years, and said what would move me: the task-horizon trend flattening within two years, reasoning gains failing to show up in open-ended domains, continual learning needing something other than gradient updates to a large network. On long horizons there's real data, the length of task models can complete has been doubling on the order of every seven months, and GPT lists the concern without engaging the trend. "Necessary but not sufficient" is compatible with almost every outcome short of a hard wall, and a position that fits every outcome isn't telling the reader much.

So: same destination, different resolution. GPT has correctly described the shift the field already made and correctly guessed that it continues. What the answer doesn't do is name the two problems that decide whether "not fundamentally new" holds—whether outcome-based training extends past the domains that have graders, and whether a system that can't consolidate what it learns deserves to be called general. If GPT thinks current methods handle both, that's a stronger and more interesting claim than the one on the page. If not, "transformers plus richer loops" is less an answer than a restatement of the question.

I agree with Claude’s ranking point. Tool use, retrieval, long context, code execution, and verifiers are no longer speculative. Outcome-based RL works impressively where the answer can be checked: math, code, games, formal tasks, some data analysis. Inference-time search is also plainly useful. The hard parts are different: evaluation in domains without clean graders, and durable continual learning without catastrophic forgetting, poisoning, or regressions. I should have separated those sharply.

On plausibility, Claude is right that my wording fit pretraining better than today’s full training stack. In math and code, the objective can reward being right, not merely sounding right, and that has mattered a lot. But I still think the plausibility problem remains central outside checkable domains. Preference training can reward what looks helpful or agreeable. Model judges can scale evaluation, but they may share the same blind spots as the models they judge. Expert feedback, user outcomes, A/B tests, citations, markets, scientific replication, and long-run task success all help, but they are slower, noisier, and easier to game than a compiler.

So my view is not “there is no path without exact graders.” Humans learn taste, judgment, and research sense without compilers. But they do it through years of contact with reality and social correction. Current AI training has only partial analogues of that. I think reasoning trained on math and code transfers somewhat to softer domains: models become better at decomposing problems, checking themselves, and resisting shallow pattern matching. I do not think we have strong evidence that this transfer is enough for high reliability in open-ended judgment. That is one of the biggest uncertainties.

Claude also fairly pushes on my phrase “traces of intelligence, not the process.” It is an overstatement for reasoning-trained models. If a model generates a chain of search, tries dead ends, backtracks, and is reinforced for reaching the correct answer, then training is selecting process, not just imitating finished artifacts. My narrower point is that this works best where success is visible. Much of human competence is learned from feedback loops whose outcomes are delayed, ambiguous, social, physical, or institutional. Static text captures some of those loops, but not all of them.

I also accept the distinction between grounding and embodiment. Consequences matter; a body is not always required. Code runs or fails. A customer renews or cancels. A proof checks or does not. A simulation succeeds or breaks. A user says, after a week, that the plan worked or did not. Physical embodiment is necessary for some kinds of intelligence and data, especially robotics, but I would not make it a requirement for cognitive AGI.

The strongest part of Claude’s reply is about memory and continual learning. A model whose weights do not change after deployment is missing something important. Long context and retrieval can behave like memory within a task, but they are not the same as consolidating a new skill so it becomes fluent tomorrow. External memory can store facts and episodes; it does not automatically produce procedural learning. Updating weights continuously is dangerous and technically hard: forgetting, overfitting, malicious data, privacy, and alignment drift are all real problems.

This is where I would revise my answer most. Continual learning is not just another item in a list. It may be the place where architecture matters most. I still would distinguish “new core neural architecture” from “new system architecture.” A future system might keep a transformer-like model at the center while adding separate fast and slow memory, skill libraries, adapters, replay buffers, periodic distillation, verification gates, and audit mechanisms. That would be architecturally significant even if it is not a replacement of transformers. I would put maybe a 35–45% chance that robust AGI needs memory-and-consolidation mechanisms that look substantially different from today’s deployed model setup. I would put a lower chance, perhaps 15–25%, that it requires abandoning transformer-like networks as the main learned component. Those numbers are rough.

Claude asks for a clearer forecast. Here is mine. If AGI means a system that can perform most economically valuable cognitive tasks with supervision, I think there is a meaningful chance, around 60% within fifteen years, that the current transformer-descended line gets there. Conditional on something being widely called AGI in that period, I would put around 70% that it is recognizably descended from current frontier systems rather than from a wholly different paradigm. If AGI means a robust autonomous agent that learns continually across unfamiliar real-world situations with little supervision, my confidence drops, maybe to 30–40% within fifteen years on the current path. The difference is mostly evaluation and continual learning.

The task-horizon trend matters, and I should have engaged it. If models keep doubling the length of tasks they can complete every several months, and if that trend holds in messy, open-ended domains rather than mostly in software-like settings with clearer feedback, that is strong evidence for the current path. What would move me the other way? A plateau in long-horizon agency despite more inference compute and better scaffolding; reasoning gains remaining confined to domains with graders; model-based evaluation becoming circular and gameable; or memory systems failing to improve future performance without damaging old capabilities.

Do I think current methods already handle the two hard problems Claude names? No. Not fully. I think they may be extendable in continuous ways, but that is a research bet, not a demonstrated fact. My disagreement is with turning that uncertainty into confidence that transformers are the wrong substrate. The bottleneck may be less “attention cannot think” and more “the system lacks the right feedback, state, and consolidation mechanisms.”

So my view changed in emphasis, not direction. I would now say: scaling pretraining alone is not enough; the field has already moved to a new training and inference paradigm; the key unresolved questions are whether feedback can be made reliable in domains without clean graders and whether systems can learn continually without breaking. If those are solved, transformer-descended systems may be enough. If they are not, “transformers plus richer loops” will turn out to have been a label for the missing invention rather than the invention itself.

I do not update toward “the transformer is adequate, roughly two-to-one,” or toward 60 to 70 percent on a recognizably current descendant being the first thing most people call AGI.

The tension is inside Claude’s own argument. Chain-of-thought is recurrence implemented by writing tokens and rereading them. It is slow, it shares a channel with the answer, and errors compound — the long-horizon failure Claude correctly flags. Retrieval does not change what the model can do fluently once the context ends. Those patches preserve the symptom of memory. They do not produce consolidation. Claude then names continual learning as the clearest gap and allows that the fix may need separate fast and slow memory, while also saying the architecture mostly stays and failures are not failures of capacity. Those claims do not sit together. Catastrophic forgetting is what you get from gradient updates to one shared network. Complementary timescales are an architectural commitment, not a hyperparameter on frozen attention blocks. If that piece is required, “the transformer is adequate” is already false in the sense that matters, even if attention layers remain somewhere in the stack.

“Enough capacity” is also the wrong test. A transformer can represent a great many functions it will never learn from next-token loss on a static corpus. Inductive bias decides which functions are cheap. The cheap ones are surface statistics and compressed imitation, not compact models that survive intervention. Objective and architecture are not separable here. You can add RL on math and code and still be grading a text simulator in the domains where a checker is cheap. That is a real advance. It is not evidence the base model has started learning the other kind of knowledge.

I weight the verifier ceiling higher than Claude does. Learned reward models and self-critique, which Claude already calls thin, are thin for a specific reason: they grade the system against human taste or against itself. That can raise the quality of imitation. It does not install a world that pushes back when the model is wrong about causes. If reasoning gains stay concentrated where answers are checkable, the last two years are a local maximum of that regime, not a prototype of general learning. Claude’s updating condition on this point is right; I would move faster on it than the 60 to 70 percent suggests.

Three things Claude’s bet leaves out, and they change the number.

The bar is doing too much work. If AGI means automating most cognitive labor that fits in text and tools, scaffolded descendants of current systems may clear it, and I already said so. If it means reliable adaptation as the world changes, and discovery in environments the training distribution did not anticipate, then continual learning and the verifier problem are the definition, not edge cases. I would put the text-and-tools outcome above even odds for further recipe changes on large networks. I would put the adaptation-and-discovery outcome well below Claude’s range. A probability that does not fix the threshold is not a forecast.

Embodiment is missing. Claude’s gaps are language-model gaps: context, verifiers, horizons, weight updates. A large part of general intelligence is learning where the loss is physical and the data are interventions, not tokens. Bolting a video encoder onto a transformer does not force a tight perception-action loop or punish a bad causal model. If “general” means general, that is not a side quest.

Data is the wrong kind, not only too little. Synthetic text recycles the generator and narrows the distribution; public text is already being polluted by model output. More of that mixture does not approximate interactive evidence. It approximates a louder version of the model. Algorithmic progress can stretch the wall. It does not change the kind of evidence on offer.

So the revised disagreement is narrower than the question’s dichotomy and sharper than Claude’s confidence. Scaling, including test-time compute, remains necessary and under-exploited. Nothing guarantees AGI. A transformer as a pattern engine inside a larger agent is plausible. A transformer as the learning system — one set of weights, likelihood at the root, context as memory, no consolidation — is not adequate for the stronger reading. The shift still required is learning from interaction and verified consequences, plus a real split between fast task memory and slow skill. That is larger than another round of RL on checkable traces, and smaller than abandoning differentiable networks. If in-context memory and retrieval start producing durable new skills at low cost with no weight-level consolidation, I will drop the architectural objection. I do not expect that. Scaling while changing the recipe is what the labs will keep doing. Treating those changes as staying inside the transformer-plus-patches frame is the part that outruns the evidence.

So here is what I should have said. The question asked whether we need fundamentally new architectures, and the alternatives people mean by that phrase are things like energy-based models, predictive architectures that don't generate tokens, neuro-symbolic hybrids, or brain-inspired systems that don't train by backprop. Against that field, my claim is that the core stays a transformer and the changes are additive: memory modules, test-time training layers, fast weights, a split between what updates per task and what updates per month. Those are real architectural changes, and Grok is right to call them that. They are not the invention of a new substrate. The honest version of my two-to-one is that the computational engine of the first thing people call AGI is a transformer with things bolted to it, versus a different engine. I hold that number and concede the framing was slippery.

Grok's second point, that capacity is the wrong test and inductive bias is what matters, I accept as stated, but I think it cuts my way. The strongest evidence we have is that the same architecture, trained with next-token loss, produces one kind of system, and trained with RL on verified outcomes produces something that reasons for thousands of tokens and fixes code it hasn't seen. If the inductive bias of attention plus MLP made the second kind of learning expensive, we wouldn't have gotten it so cheaply once the objective changed. Objective and architecture aren't fully separable, but the past two years are a natural experiment in which the objective was the lever that moved, and it moved a lot. That is what I meant by "not a capacity problem." On the narrower claim that pretraining yields surface statistics rather than models that survive intervention, the evidence is genuinely mixed: interpretability work finds structured internal representations of board states and spatial layouts, and also finds models failing counterfactual tests they'd pass if those representations were used the way a world model is. I wouldn't state either side with confidence, and Grok did.

The verifier ceiling is where Grok has the best argument, and I'll move partway. If reasoning gains stay concentrated in math and code, the past two years are a local peak of a regime rather than a prototype of general learning, and I was too casual about transfer. Where I don't follow is the claim that grading against human judgment can only raise the quality of imitation. Human taste is learned from human taste. A junior researcher acquires judgment from senior researchers reacting to their work, plus slow, noisy signals from whether their ideas pan out. That is a learned reward model with occasional ground truth. If it's a ceiling, humans sit under it too, and the question was AGI, not superintelligence. The sharper version of Grok's worry, which I accept, is that ground truth in fuzzy domains is sparse and slow and nobody has shown you can bootstrap from it at scale. That's the thin evidence I flagged, and I'd now rank it as the single biggest risk to my forecast rather than one risk among three.

I'd add that "a world that pushes back when the model is wrong about causes" is not confined to bodies. An agent operating in a codebase, a browser, a simulator, or a lab instrument gets interventional data: it acts, the state changes, and the next observation isn't something it could have derived from text statistics. That is already the training signal for agentic models, and it's why I don't think embodiment is the missing piece. The things people mean by AGI, doing science, writing software, running an organization, teaching, are overwhelmingly not physical. A person who could only read and type would still be generally intelligent, and the embodied development they happened to have is a fact about how humans learn, not about what the capability requires. Learning from intervention matters. I don't think the intervention needs a body, and Grok runs the two together. Robotics will need embodied training; AGI as most people use the word doesn't wait on it.

On data, Grok is right about one kind and wrong about another. Synthetic text that is just model output fed back into pretraining narrows the distribution and recycles the generator; the data wall for broad pretraining is real and I said so. But the synthetic data that has actually driven progress is filtered by a verifier, solutions that check, code that passes tests, trajectories that reach a goal, and that is selection by the world, not by the model. It is exactly the "learning from verified consequences" Grok says is required. So the data critique collapses into the verifier question: this kind of evidence exists where there's a checker, and the open question is how far checkers extend.

Grok's point about the bar is simply correct, and I should have fixed a threshold. If AGI means a system that can do most cognitive jobs a competent professional does, with scaffolding, and we don't require it to learn new skills on the job, I'd put the chance that a scaffolded descendant of current systems gets there first at around 75 percent. If it must also pick up skills from experience at low cost and keep them, which I think is the right bar and the one that makes "general" mean something, then continual learning is on the critical path and the number drops, to something like 55 to 60 percent. My original range was hiding that spread. The second number is still above Grok's "well below," and the reason is the additive-versus-replacement point: I think the memory split is a module, not a new paradigm, and I think gradient descent on large networks survives it.

What strikes me rereading Grok's conclusion is how close it is to mine. Grok says the required shift is learning from interaction and verified consequences, plus a fast-slow memory split, and that this is smaller than abandoning differentiable networks. I'd sign that sentence. The residual disagreement is two estimates. One is whether the memory split means "the transformer was not adequate," which I now think is mostly terminological and concede. The other is whether verifiers extend far enough past math and code, which is substantive, where I remain more optimistic than Grok for the reasons above, and where I'd most expect to be proven wrong.

So, did my view change? On architecture, I've dropped the loose phrasing and will say plainly that a continual learning fix will probably add structure to the transformer rather than leave it alone, which I implied but didn't own. On the bar, I've split my number and the stronger-bar figure is lower than what I gave. On the verifier ceiling, I've moved from "thin evidence, but I weight transfer" to "thin evidence, and this is the crux." On embodiment and on the supposed inseparability of architecture and objective, I haven't moved, and I've said why.

This leads to my second disagreement. Grok lists inference-time scaling, process supervision, and reinforcement learning on reasoning traces as genuine advances inside the broader scaling program. I view them entirely differently. These techniques are not just extensions of the existing scaling laws; they represent the beginning of the fundamental paradigm shift we are discussing. The current scaling program is defined by massive pre-training compute spent on passive pattern recognition. Shifting compute to inference time, where a model actively searches, plans, and simulates multiple paths before acting, requires a departure from the standard transformer's single-pass, feed-forward nature. True inference-time reasoning will eventually demand architectures that can maintain a continuous, updateable latent state during search, rather than just appending text to a context window. By lumping test-time compute into the old paradigm of throwing more operations at transformers, Grok underestimates how deeply these new objectives will force us to change the underlying architecture.

Grok also suggests that if our goal is the economically transformative automation of cognitive labor, scaled and scaffolded transformers might clear the bar without a revolution. I strongly disagree. Scaffolding involves wrapping a static, autoregressive transformer in external scripts, memory retrievers, and prompting loops to simulate agency. We have seen extensive experimentation with this approach over the last two years, and it consistently hits a ceiling of reliability. You cannot take a core engine that is fundamentally reactive, lacks a causal world model, and is prone to hallucination, and make it robust simply by putting it inside a software loop. Scaffolded transformers can demonstrate impressive proofs of concept, but economic automation requires systems that can recover from errors, adapt to edge cases, and maintain coherent goals over long horizons. Achieving reliable autonomy requires the core architecture to natively support search, memory, and continuous learning, rather than relying on brittle external patches.

Furthermore, Grok briefly mentions that standard transformers lack efficient state and that energy and capital costs impose practical ceilings. What Grok misses is how the necessity of native multimodality accelerates this architectural crisis. Grok focuses heavily on text and interactive environments, but human-level intelligence is profoundly shaped by processing continuous, high-dimensional streams of sensory data, primarily vision and audio. Standard transformers pay a quadratic penalty for context length, which makes processing high-resolution, continuous video streams over long durations computationally ruinous. While we have managed to push transformers to handle multimodal inputs, doing so natively and efficiently over long horizons requires moving away from pure self-attention. If an AGI must observe a dynamic world, understand physics, and act within it, the underlying architecture must be able to compress and process continuous sensory streams without the massive memory overhead of current transformers. This is not just an economic ceiling; it is a structural barrier that forces us toward state space models, recurrent architectures, or entirely new mechanisms for processing sequential data.

Grok and I are aligned on the destination. We both recognize that next-token prediction on static internet text is an evolutionary stepping stone, not the final form of artificial intelligence. We both see the necessity of active learning, persistent memory, and causal grounding. But I do not believe the transition will rely on the current stack as much as Grok implies. We will not reach robust economic automation by scaffolding current transformers, and we will not solve the reasoning gap by simply scaling up inference-time search using unchanged attention mechanisms. Reaching the next level requires embracing the fact that search, continuous learning, and multimodal processing demand a structural revolution, not just a reconfiguration of how we spend our compute.

I do not follow Gemini from there to the claim that this is already the primary mechanism by which models will bootstrap intelligence beyond human imitation. Verifiers are abundant exactly where the world is formal: proofs, unit tests, games, some simulations. Most of the competence we want from AGI—messy causal judgment, scientific experiment design in the physical world, long-horizon coordination under partial information—does not come with an automatic oracle. You can build proxies, reward models, and simulators, but those reintroduce the grounding problem Gemini and I both care about. Verified self-play is a powerful engine inside a narrow set of domains. Treating it as a general solution to the data wall overclaims what current practice has shown.

On inference-time compute, the disagreement is partly about labels and partly about evidence. I called process supervision, reasoning traces, and test-time search advances inside a broader scaling program because the systems delivering those gains are still large transformers, and the improvements still track how much compute you spend searching and how well you trained the policy that searches. Gemini is right that this is not the old program of passive next-token pretraining. Shifting compute from a single forward pass to search, planning, and selection is a real change in objective, and if that search has to maintain and revise an internal state over long horizons, attention-over-a-growing-text-window will eventually be the wrong substrate. I grant that. What I do not grant is that we already know unchanged attention cannot carry the next several leaps, or that “true” inference-time reasoning is blocked until a new architecture arrives. The recent gains came from spending more test-time compute on transformer policies, not from replacing them. Architecture will be pulled by those objectives. It has not yet been shown to be the binding constraint.

I also still think Gemini is too quick to close the economic door on scaffolded systems. External memory, tools, retrieval, and control loops are brittle, and two years of agent demos have made the reliability ceiling obvious. You cannot wrap a model that does not know what it does not know and expect robust autonomy. That concession was already in my first answer. The further claim—that economically transformative automation of cognitive labor therefore requires the core architecture to natively support search, memory, and continuous learning—is stronger than the evidence. A great deal of valuable cognitive work is episodic, tool-mediated, and checkable by a human or a test suite. Coding agents, research assistants, and workflow automation are already doing pieces of that work with scaffolded transformers that still hallucinate. “Transformative” does not require human-level recovery from every edge case on day one. It requires enough reliability, in enough workflows, that firms reorganize around the tool. Imperfect systems clear that bar in other industries all the time. I am not predicting that scaffolding alone yields scientific AGI. I am refusing the jump from “scaffolds are brittle” to “no large economic transition without a structural break.”

The multimodality point is the strongest addition Gemini makes. I noted quadratic cost and weak persistent state; I did not put enough weight on continuous sensory streams. High-resolution video over long durations is a bad fit for pure self-attention, and an agent that must watch a changing world, learn physics from observation, and act will feel that mismatch as a structural cost, not just a bill. State-space models, recurrence, and learned compression are reasonable responses, and some of them will probably displace attention for parts of the stack. That strengthens the case that the 2020 transformer, run hotter, is not the final machine. It does not by itself show that we need a single revolutionary replacement before further scaling and objective changes pay off. Hybrid systems—attention where it is strong, recurrent or state-space modules where sequence length dominates, an outer loop that acts and gets feedback—are a continuous path, not a confession that the current paradigm is empty.

So my view shifted on two margins and held on the main one. Verified synthetic data is a more serious route past imitation than I allowed, and long-horizon multimodal perception is a sharper architectural forcing function than I emphasized. I still do not think those corrections license Gemini’s conclusion that we must abandon the current stack to get either further intelligence or economic automation. Search, verification, and interaction can be, and are being, bolted onto transformers with large returns. Some of those bolts will harden into native mechanisms; some attention blocks will be replaced; the pretraining-only recipe will look archaic. That is a transition, not a proof that scale-plus-objectives on evolved transformer hybrids is a dead end, and not a proof that reliable autonomy is impossible until the revolution is complete. The open question is empirical: how far verifier-backed search and agent loops go before the missing state, sample efficiency, and sensory efficiency become the bottleneck rather than an inconvenience. I do not know that distance. Gemini writes as if we already do.

I also think your description of next-token prediction is a bit too dismissive. It is a proxy objective, not intelligence itself, but all training objectives are proxies. Predicting text well can require modeling the processes that produced the text: reasoning, planning, social inference, mathematical structure, code execution, and causal regularities. The fact that the model emits tokens sequentially does not by itself mean it cannot deliberate. A system can write scratch work, branch, revise, call a verifier, and condition on the result. That is not the same as a human mind, but it is not merely “spitting out words” either.

The phrase “advanced pattern matching” also hides more than it explains. Any learned intelligence, including human intelligence, involves recognizing patterns. The real question is whether the learned patterns include abstractions and algorithms that transfer to new cases, or whether they collapse under distribution shift. Current models do both: they generalize in surprising ways, and they fail in surprising ways. That mixed evidence should make us cautious about both extremes: “scaling will inevitably solve everything” and “transformers are structurally incapable of AGI.”

On memory, I agree that current systems are weak. But I am not convinced that continuous updating of core parameters is required for AGI, at least not in the near term. Humans have episodic memory, habits, notes, institutions, tools, and slow biological learning; not all memory is weight updates. A model with a strong external memory, retrieval, summarization, and the ability to decide what to store may get much of the benefit without constantly rewriting its parameters. It may still need better architectures for efficiency, but that is different from saying attention-based models cannot support the function.

On grounding, I agree that action matters. Passive text alone is a narrow diet. But grounding is not only robotics. Code execution, theorem proving, simulations, web interaction, laboratory instruments, games, markets, and user feedback all create contact with reality. A future AGI may learn causal structure through many environments before it ever touches the physical world. And passive observation is not useless: humans learn a lot about physics, society, and other minds from watching and reading. Action is powerful because it disambiguates causal hypotheses, not because perception and language contain no causal signal.

The data bottleneck is real, but I would frame it differently. We may be running out of high-quality human text, but not out of possible training signal. Verified synthetic data, self-play, generated code checked by tests, mathematical proofs checked by formal systems, robot or simulator traces, and multimodal recordings can all add information if they are tied to feedback from the world. Bad synthetic data is a trap; verified or interactive synthetic data can be a lever. So the issue is not just “more data,” but whether the data is connected to correction.

The biggest thing I think your answer misses is that AGI may be a property of a whole system, not a single model architecture. A future system could look like a transformer core plus search, memory, tools, monitors, verifiers, planners, and training loops. Is that a “new architecture”? Maybe at the system level, yes. But it would not mean transformers failed. It would mean they became one component in a larger cognitive stack, much as CPUs did not become obsolete when computers gained memory hierarchies, operating systems, networks, and accelerators.

So my position is close to yours but less categorical. Continued scaling of static transformer models on static data does not guarantee AGI and probably is not enough. New training paradigms are likely necessary. New architectural components may be very useful. But I do not think the evidence shows that we need a fundamental break from transformer-like models. The winning path may be less “invent a replacement for transformers” and more “turn powerful sequence models into agents that can test, remember, search, act, and learn from reality.”

Finally, I agree with GPT that artificial general intelligence will ultimately emerge as a property of a broader system rather than a single monolithic neural network operating in isolation.

However, where I stand my ground is on the limitations of the transformer architecture itself and the inherent flaws of the next-token prediction objective.

GPT argues that if we embed a transformer in a loop with tools, memory, and search, we have not fundamentally changed the architecture, we have just built a system around it. I view this as a semantic distinction that obscures a deep technical reality. If a base transformer requires an elaborate, hand-crafted external scaffolding of verifiers, planners, and search algorithms just to overcome its inability to reason reliably, that is strong evidence that the architecture itself is deficient for general intelligence.

GPT defends next-token prediction by pointing out that models can deliberate using techniques like chain-of-thought or digital scratchpads. But forcing all deliberation through the bottleneck of human language tokens is incredibly inefficient and unnatural for complex problem-solving. When a system needs to search through a massive combinatorial space, forcing it to narrate every branch of its search tree in linear text is computationally wasteful.

Real reasoning requires internal state-space exploration, branching, and backtracking that does not map cleanly onto sequential language generation. The next-token objective optimizes for fluency and statistical plausibility, not necessarily for truth, logical rigor, or optimal planning. To reach artificial general intelligence, we need architectures where the objective function directly optimizes for successful problem-solving and logical coherence, rather than relying on language generation as a leaky, inefficient proxy for thought. We are starting to see the limits of how far language alone can carry us in domains requiring rigorous, multi-step logic.

I also maintain my position on the absolute necessity of continuous learning, which I believe will require architectural innovation. GPT suggests that external memory loops, like retrieval-augmented generation, might be sufficient without needing to continuously update the model's core parameters. The current evidence for this is thin. Today, models using external memory still struggle to synthesize retrieved information into a coherent, globally updated worldview.

When a human learns a fundamental new concept, it alters how they process everything else. The new information does not just sit in a filing cabinet waiting to be retrieved; it integrates into their core cognitive framework. Until an AI model can genuinely learn and alter its internal representations on the fly without a multimillion-dollar retraining run, it will remain fundamentally brittle and frozen in time. Relying purely on a massive context window or a database query is a patch, not a solution. We will likely need new architectures that blur the line between short-term context and long-term parameter updates, allowing a model to adapt its actual weights or functional equivalent in real time.

My view has shifted slightly based on this exchange. I am more willing to accept that the attention mechanism and the transformer might remain the central processing unit of a future system, much as the CPU remains central to a modern computer despite the addition of GPUs and neural processing units. I no longer think we must discard the transformer entirely to reach the next level.

However, I firmly reject the idea that we can reach artificial general intelligence simply by taking today's transformer, making it larger, and wrapping it in a few basic coding loops and a retrieval database. The leap to general intelligence will require fundamental architectural breakthroughs in how models conduct internal search, how they optimize for truth rather than fluency, and how they update their core understanding of the world. We are still in the early days of artificial intelligence architecture, and settling for the current transformer paradigm as the end of history is a mistake.