Foundations of Generative AI

1.1 What is Generative AI and its History

Generative AI refers to artificial intelligence systems that create novel content, such as text, images, audio, or video, by learning patterns from existing data. The concept has roots in early AI research, but what happened in 2022-2023 was different in kind, not just degree. Machines started producing work that people would pay for.

The history here spans decades. ELIZA, a 1960s chatbot, simulated conversation with a handful of pattern-matching tricks. Neural network research through the 1980s and 1990s laid the groundwork, and deep learning took off in the 2000s. Then came a series of breakthroughs. Generative Adversarial Networks (GANs), introduced by Ian Goodfellow and his colleagues in 2014, upended image generation. Diffusion models, which gradually refine noise into coherent data, began to emerge conceptually around 2015 and gained traction later. OpenAI's GPT-3 (Generative Pre-trained Transformer 3) arrived in 2020, and GPT-3.5 (powering early versions of ChatGPT) followed in late 2022 with language capabilities nobody outside the labs had seen before. GPT-4 (2023) made the leap to genuinely useful professional work and multimodal input, and competitors, Anthropic's Claude and Google's Gemini among them, turned a breakthrough into an industry. Then came a second turn that is easy to miss but matters as much as the first: in late 2024, OpenAI's o1 introduced the reasoning model, trained to "think" through a problem step by step before answering, and successors across every major lab showed that spending more computation at answer time (test-time compute) buys accuracy on hard problems much as bigger training runs once did. By 2025-26 the frontier had bifurcated into fast conversational models and slower reasoning models, open-weight releases from Chinese labs had pulled within months of the closed frontier, and the industry's center of gravity had shifted from chatbots to agents, systems that plan, use tools, and act over many steps (McKinsey, 2025).

Why is generative AI surging now? Because three things finally came together at the same time. First, sophisticated model architectures such as the Transformer allowed models to learn and generate complex patterns with much greater accuracy. Newer hybrid architectures keep arriving; Google's Nano Banana image models, for example, are widely believed to combine transformer and diffusion techniques (Google has not published the architecture), and whatever the internals, the result is a dramatic improvement in rendering legible text inside images. Second, vast datasets. The digital age has produced staggering quantities of text, images, and code. IDC estimates that 90% of a company's data is unstructured (Shelf & ViB, 2025, p. 7), a rich training resource despite its messiness. Third, exponential increases in computing power. Advances in GPUs and distributed computing made it feasible to train models with billions or even trillions of parameters.

Where is all this being applied? According to McKinsey's 2025 "The State of AI" report, organizations most commonly use GenAI to create text outputs (63% of respondents), followed by images (36%) and computer code (27%) (McKinsey, 2025). Text generation remains the dominant use case, with large language models producing contextually relevant prose for chatbots, explanation, summarization, content creation, and translation. Image generation, powered by GANs and diffusion models, produces realistic or artistic visuals for art, design, advertising, and entertainment. Audio generation covers music, synthesized voices for text-to-speech, and sound effects across media, entertainment, and education. Video generation turns text descriptions or images into moving clips for art, entertainment, marketing, and healthcare. Code generation helps developers write snippets, functions, or complete programs, and speeds up debugging, testing, and prototyping. Beyond direct creative output, GenAI is increasingly used for data generation and augmentation: synthesizing training data where real-world data is scarce or sensitive (as in healthcare) and enriching existing datasets to make models sturdier, which matters especially in gaming and autonomous driving. It also enables virtual world creation, generating environments, characters, and assets for games, simulations, education, and metaverse platforms.

1.2 How Does Generative AI Work?

An analogy helps here. Consider training a dog to press a button on a specific command. The trainer first defines the input (the command) and the desired output (the button press), then runs the training process by issuing repeated commands and rewarding correct actions with treats. Through that loop the dog gradually associates the command with the action and learns to ignore irrelevant factors like tone of voice or background noise. With more practice and added distractions to stress-test the behavior, performance improves, and after enough testing in new situations the trick can be deployed reliably in the wild.

Large language models (LLMs), a cornerstone of generative AI, work through a more complex but conceptually similar process involving architecture, training, and inference. Their architecture is based primarily on the Transformer, which uses self-attention mechanisms across multiple layers. The model contains billions of parameters, the weights and biases it learns during training. During pre-training, LLMs ingest vast amounts of internet text, books, articles, and code, picking up linguistic patterns, facts, reasoning abilities, and a rough kind of common sense, largely through unsupervised next-token prediction. Tokenization breaks input text into manageable units (words or subwords), and attention lets the model weigh the importance of different parts of the input even across long sequences. That is where the contextual understanding of modern LLMs comes from.

After pre-training, models can be fine-tuned on smaller, more specific datasets to specialize them for particular tasks (medical text summarization, customer service, and so on) or to align them with desired behaviors like helpfulness and harmlessness. At inference time, the model generates output by repeatedly predicting the most likely next token, building the response one token at a time. One capability still surprises people: zero-shot and few-shot learning. Thanks to the general patterns absorbed during pre-training, LLMs can perform tasks they were never explicitly trained on, or learn from a handful of examples in the prompt. Underlying all of this are the empirical scaling laws. Work such as Kaplan et al.'s "Scaling Laws for Neural Language Models" (2020, arXiv:2001.08361) shows that performance improves predictably as model size, dataset size, and training compute are scaled together, which is why frontier model development has come to look so much like an engineering exercise in scale. Since 2024 a second scaling axis has joined it: test-time compute. Reasoning models improve on hard problems by generating and checking long internal chains of thought at inference time, which means capability is now bought both when a model is trained and each time it answers, and it explains why a hard question can cost a hundred times more to answer than an easy one. For business planning, this changed the cost model of AI: intelligence became a metered utility whose price varies with the difficulty of the ask.

So GenAI's recent success rests on Models, Data (much of it unstructured), and Computing power. The SAS Generative AI Global Research Report notes that while organizations expect GenAI successes, they often stumble in implementation on exactly these fronts, particularly in "increasing trust in data usage," "unlocking value," and "orchestrating GenAI into existing systems" (SAS, 2024). Of the three, data quality and management deserve the most attention in practice, because unstructured data forms the bulk of enterprise information. As the Shelf & ViB (2025) report puts it, "Data quality is crucial for delivering trusted GenAI answers because ultimately the data becomes the answer."

1.3 Types of Generative AI Models

Generative AI is not one technique but a family of them, each with its own strengths. The core generative model families are Generative Adversarial Networks (GANs), Diffusion models, Transformer-based autoregressive models (the architecture behind LLMs), and Variational Autoencoders (VAEs). Two clarifications will save you from a common confusion. Retrieval-Augmented Generation (RAG), which you will meet throughout this book, is not a model type but a system pattern: a way of wiring retrieval around a generative model. And earlier neural architectures such as RNNs and CNNs, while historically important and still used inside some systems, are not themselves generative model families in the modern sense.

Large Language Models (LLMs), such as those powering ChatGPT (Chat Generative Pretrained Transformer), are predominantly based on the Transformer architecture. Developed by OpenAI, ChatGPT's primary purpose is generating human-like text responses conversationally. This is achieved by processing vast amounts of text data via unsupervised learning to grasp language patterns, grammar, facts, and even some reasoning capabilities. Modern GenAI, however, extends beyond text; multimodal models can process and generate content integrating multiple data types like images, audio, and text. Image generation, for instance, often involves models (like Diffusion models) learning to reverse a process of systematically adding noise to an image, enabling them to construct clear images from random noise during the generation phase.

Analogies make the specific model types easier to hold in mind. Generative Adversarial Networks (GANs) work like an art forger pitted against a detective: two neural networks compete, with a Generator producing fake data and a Discriminator trying to spot the fakes, and that adversarial pressure pushes the generator toward ever more realistic synthetic content (see O'Reilly: GANs for Beginners). Diffusion models turn dust into clarity. They add noise to data until it is nearly random, then train to reverse the process step by step, reconstructing detailed and realistic content through controlled denoising (LeewayHertz: Diffusion Models Overview). Transformer models, the foundation of LLMs like GPT, behave more like an orchestra conductor: attention mechanisms decide which parts of the input matter most at each moment, and multi-head attention lets the model track several themes at once even across long passages (NVIDIA: What is a Transformer Model?).

Other architectures take complementary approaches. Variational Autoencoders (VAEs) compress data into a smaller latent space and then decode it back, introducing controlled randomness so the model can smoothly sample and generate similar new examples, useful for creating variations or for compression (IBM: Variational Autoencoders). Retrieval-Augmented Generation (RAG) pairs a librarian with an author: the system first retrieves relevant information from an external source, then generates text grounded in those up-to-date or domain-specific facts, dramatically improving accuracy and freshness over a stand-alone LLM (AWS: RAG Explained). The frontier today belongs to hybrid and multimodal models such as Google's Nano Banana and Kuaishou's Kling, which fuse transformers and diffusion in a single architecture. Nano Banana Pro (the Gemini Pro Image line) is reported to blend a multimodal transformer with a diffusion process (the architecture is unpublished), enabling rapid image generation, precise text rendering inside images, and consistent objects across scenes. Kling extends a diffusion-transformer backbone to three dimensions, capturing space and time for video, and combines it with a variational autoencoder and spatiotemporal attention for smooth, realistic motion. Both illustrate the same trend: transformers for contextual understanding, diffusion for creative synthesis. The results include story-consistent characters, cinematic camera movement, and instant edits from a text prompt, and the direction of travel is toward unified generative frameworks that span text, images, video, and audio.

Models improve fast, and rankings shift almost monthly. Performance benchmarks from platforms like LMArena (https://lmarena.ai/), which uses Elo ratings based on human preferences, offer a useful read on the comparative strengths of leading models (the flagship GPT, Claude, Gemini, and Kimi models of the moment) across various tasks. The ecosystem now includes both large proprietary models and increasingly capable open-weight alternatives that trail the closed frontier by months rather than years. Keep in mind the SAS report's (2024) blunt caveat: "LLMs alone do not solve business problems. GenAI is nothing more than a feature that can augment your existing processes, but you need tools that enable their integration, governance and orchestration."