Kamal Ghorui
The world thought Google was caught off guard by the AI revolution. It was not. And the chips it has been quietly building since 2016 may be about to change everything.
Introduction: When everyone wrote google off
November 2022 was a humbling month to be Google. OpenAI had just released ChatGPT to the public, and the world lost its mind. Within days, it was clear that something fundamentally different had arrived: a conversational AI that could write, reason, explain and engage in a way that felt surprisingly human. The tech press declared a new era; And almost everyone agreed on one thing: Google – the company that had essentially invented the modern internet, dominated how the world finds information for two decades and called itself an AI-first company – had been caught sleeping.What’s ironic was that Google had the researchers, the data and the infrastructure, yet it was second to a company that most people did not hear about. Google had published an academic paper in 2017 that introduced the Transformer architecture that made ChatGPT possible in the first place. And yet here was a scrappy startup, backed by Microsoft’s $1 billion, doing what Google had apparently been too cautious or too comfortable to do: ship a product that showed ordinary people what AI could actually feel like.
Inside Google’s offices, the internal alarm was real and reports said that “Code Red” was declared. Co-founders Larry Page and Sergey Brin reportedly checked back in, and the company accelerated the development of Bard, its own AI chatbot, and began fast-tracking AI features across its product suite. From the outside, it looked like panic. Like a giant scrambling to catch up.Google stumbled with its initial launch. It made Bard from scratch and launched Gemini. Nobody talked about at the time what Google had been quietly building for nearly a decade underneath all of it. Google became confident, and CEO Sundar Pichai gave a hint of what was brewing behind the scenes.In a recent interview, Google CEO Sundar Pichai pulled back the curtain on how the company actually experienced that moment, and his description sounds nothing like a company in crisis.Pichai said in an interview: It was obviously very invert-focused in that moment. To me, it was very clear in that moment, “Hey, the Overton window shifted.” I felt like the company was built for that moment. The vertical thing, it’s not an accident or something. It was a very intentful. We were in the seventh version of TPUs. I remember it might have been 2016 Google I/O where we announced the TPUs and spoke about we are building AI data centers. This was 2016. The company was operating in an AI-first way. We had deeply internalized this shift. To me, we were behind in terms of frontier LLM models, but we had all the capabilities internally, and we had to execute to meet the moment. But the exciting part was when I look at it from a full stack, we had the research teams, we had the infrastructure teams, we had all the platforms.The capabilities Pichai was referring to included the research teams, the infrastructure, the platforms. But the most important one was the one that almost no one outside the company fully understood at the time: The chip.
Part I: The Chip Game
While the rest of the tech world was buying Nvidia, Google was playing a different game all along. Nvidia’s GPU chips became the essential hardware of the modern AI era and they proved to be the picks and shovels of the gold rush. Demand for Nvidia’s H100 chips skyrocketed, outstripping supply so dramatically that access to them became a competitive advantage in its own right. As Companies queued up, prices soared and Nvidia’s market capitalisation crossed a trillion dollars, then two trillion, then briefly touched three. Jensen Huang became one of the most celebrated CEOs on the planet. Nvidia was not just a chip company anymore. It was the infrastructure layer that the entire AI industry was built on. Soon, it became the first company to reach the $4 trillion market cap.
Google, meanwhile, had been doing something different since 2016. TPUs, or Tensor Processing Units, are chips designed by Google specifically for the kind of mathematical operations that AI models require. Unlike Nvidia’s GPUs, which are general-purpose processors adapted for AI workloads, TPUs are purpose-built from the ground up for one thing: running neural networks efficiently.The first generation was announced at Google I/O in 2016, almost as a footnote in a broader AI presentation. Few people outside the industry paid much attention. But Google kept building. By the time ChatGPT changed the world in late 2022, Google was already on its seventh generation of TPUs with years of iterative development, architectural refinement and hard-won engineering experience that no amount of money could simply buy overnight.What this meant, in practical terms, was that when Google’s most sophisticated AI models finally arrived in the form of Gemini, Gemini Nano, Gemini Pro and Gemini Ultra, and the successive versions that followed were not just capable, they were fast, accurate and up to date. And they ran on infrastructure that Google owned, controlled and had been perfecting for nearly a decade. The rest of the industry had been building on rented land while Google had been quietly laying its own foundation the whole time.
Part II: The Inference problem
For a while, the AI conversation was dominated by one metric: how big is your model, and how well does it perform on benchmarks. Training: the process of feeding enormous amounts of data to an AI model so that it learns patterns, relationships and reasoning capabilities, was treated as the primary challenge. Moreover, all the companies competed on the size of their training runs, the sophistication of their architectures, and their performance on standardised tests. A better-trained model meant a smarter AI. And a smarter AI meant winning.What the industry slowly and somewhat painfully discovered is that training is only half the problem. The other half is inference: the process of actually running the model in real time to answer a user’s question.
In easier words, when you type something into ChatGPT or Google’s Gemini and receive a response, that response is being generated through inference: the model is processing your input and producing an output, in real time, at scale, for potentially millions of users simultaneously.What it meant at that time was that Inference was hard, computationally intensive, time-sensitive and expensive. A model that produces brilliant answers but takes thirty seconds to generate them is not useful in a consumer product. Users expect responses in seconds. The chip that enables fast, efficient inference is therefore becomes an essential cog.This is where Google’s TPUs became a genuine competitive weapon. Google used Nvidia’s GPUs for training but used TPUs, particularly well-suited to inference workloads, to generate quick answers. Their architecture, purpose-built for the matrix multiplication operations that underpin neural network computation, allows them to process inference requests with a speed and efficiency that general-purpose GPUs struggle to match at scale. Google’s models, running on Google’s TPUs, in Google’s data centres, delivered responses on Google’s products with a speed that started turning heads.And then Google did something that no one had quite anticipated: it opened the doors. Rather than keeping its TPU infrastructure exclusively for internal use, Google began offering access to its chips through Google Cloud, allowing external companies, including startups, enterprises, AI labs, to rent TPU capacity for their own workloads. A piece of hardware that had been built to give Google an internal advantage was now a commercial product. The chip had become a business – and Google Cloud came out to be another fastest growing vertical for Google.
Part III: The day Google scared Nvidia
The moment that crystallised just how serious the TPU threat had become arrived without much warning. Reports emerged that Meta, one of the largest consumers of AI computing infrastructure in the world and a company that had been one of Nvidia’s most significant customers, had signed a deal with Google to use TPUs for certain workloads.The market reacted immediately. Nvidia’s share price dropped and billions of dollars in market capitalisation evaporated in a single day. The signal was clear: if Meta was diversifying away from Nvidia toward Google’s custom silicon, the assumption that Nvidia had a permanent lock on AI infrastructure was no longer safe.Nvidia pushed back, and it did so loudly. It announced in a post on X (formerly Twitter).“We’re delighted by Google’s success — they’ve made great advances in AI and we continue to supply to Google,” the company said in a statement that managed to sound both gracious and dismissive at the same time. “NVIDIA is a generation ahead of the industry — it’s the only platform that runs every AI model and does it everywhere computing is done. NVIDIA offers greater performance, versatility, and fungibility than ASICs, which are designed for specific AI frameworks or functions,” it added.The reference to ASICs (Application-Specific Integrated Circuits), the category that TPUs fall into, was pointed. Nvidia’s argument was essentially this: yes, purpose-built chips can be very fast at the specific thing they are designed for. But general-purpose platforms that can run anything, anywhere, on any framework, are ultimately more valuable. Versatility beats specialisation.It is a reasonable argument. It is also, notably, the argument of a company that felt the ground shift beneath it.Nvidia also made a significant strategic move around this period, acquiring Groq, a chip startup that had built a reputation for extraordinarily fast inference performance. The acquisition was widely read as a direct response to the inference challenge: if the next competitive battleground was not training but serving models quickly and cheaply at scale, Nvidia wanted the best inference hardware in its portfolio.
Part IV: Google strikes back with TPU v8
When everyone thought the war was cooling down, Google’s answer came in the form of its latest generation of custom silicon, and it was designed to address both sides of the AI compute equation simultaneously. The company announced TPU v8 in two distinct configurations, each targeting a different part of the AI workload spectrum.
The first, TPU 8t, is built for massive-scale training. These chips can handle training – the kind of enormous, months-long computation required to build the next generation of frontier AI models. It is Google’s answer to the question of whether its custom chips can compete with Nvidia’s best hardware when it comes to the raw, sustained power required to train models at the frontier.The second, TPU 8i, is built for something different: high-performance, low-latency agentic inference. This is the chip designed for a world where AI is not just answering simple questions but operating as an autonomous agent, including planning multi-step tasks, executing actions, interacting with external systems, and doing all of it quickly enough that users and enterprise systems do not notice the delay. Together, the two chips represent something more significant than a hardware update: They represent Google’s clearest articulation yet of what it is trying to be. Not a search company with an AI strategy. A full-stack AI company: one that owns the research, the models, the chips, the data centres, the cloud platform and the consumer products.
Sundar Pichai’s comment about the Overton window is worth returning to, because it captures something important about what Google has been doing and what it is now saying openly. The Overton window is a concept from political theory that describes the range of ideas the public is willing to consider acceptable at any given moment. Pichai used it to describe the moment ChatGPT changed what the world thought AI could and should do. The window shifted. Suddenly, people were ready for AI in a way they had not been before.Google’s argument was implicit in Pichai’s words and increasingly explicit in the company’s product and hardware announcements: it was never behind. It was waiting for the window to open. And when it did, it had everything it needed: the research, the infrastructure, the chips and the models.What is harder to dispute is where Google stands now. Its Gemini models are competitive with the best in the world; its TPU infrastructure is attracting customers that Nvidia considered its own; its cloud business is growing, and now its chips, which are multiple generations in the making, built in the years when nobody was paying attention, are now at the centre of one of the most profound technology competitions of the modern era.
