AI Fundamentals

A training memory optimization that discards intermediate layer activations, recalculating them during the backward pass.

A mathematical formula introducing non-linear thresholds to model outputs, allowing networks to learn complex representations.

A variant of the Adam optimizer that decouples weight decay from gradient updates to improve regularization.

An LLM system that plans, calls tools, observes results, and iterates in a loop to complete tasks rather than answering once.

Tracing, logging, and monitoring for LLM systems — seeing what the model was asked, what it did, what it cost, and where it failed.

An independent parallel compute sub-unit inside self-attention layers that tracks relationships between words.

An unsupervised neural network designed to compress inputs into low-dimensional latent spaces and reconstruct them.

The algorithm that calculates loss gradients backwards through neural layers to update model weights.

Processing LLM requests asynchronously in bulk at ~50% discount — the right mode for any AI workload that doesn't need an answer in seconds.

A 16-bit floating-point format that matches the dynamic range of FP32, preventing underflow issues.

A high-performance quantization library providing lightweight CUDA wrappers for 8-bit and 4-bit optimizers.

A prompting strategy instructing models to generate step-by-step reasoning paths before outputting final answers.

An emergent capability where models yield higher accuracy by breaking problems down into sequential reasoning steps.

The discipline of deciding what goes into an LLM's context window — instructions, retrieved data, memory, tool results — and what stays out.

The ranking pipeline sorting external retrieval chunks to ensure high-density information fits in LLM context windows.

The maximum amount of text (in tokens) an LLM can consider at once, covering the prompt, conversation history, and its own output.

An iteration-level scheduling algorithm that groups active inference requests dynamically rather than waiting for full sequence completion.

A metric that measures the angular distance between two vectors in a high-dimensional space.

A secondary retrieval pipeline step that scores document-query pairs using deep attention classification.

The token-by-token generation stage of LLM inference — sequential, memory-bandwidth-bound, and the reason output tokens cost more than input.

Direct Preference Optimization — preference tuning without a reward model or RL loop: train directly on chosen-vs-rejected response pairs.

Mathematical formulas used to calculate similarity scores between vector coordinates.

Systematic testing of LLM outputs against defined criteria — the AI equivalent of a test suite, and the thing that separates demos from products.

A fully-connected neural network layer processing token vectors independently after attention calculations.

A prompting technique providing multiple input-output examples to guide model generation formatting.

Further training a pretrained model on your own examples to change its behavior, style, or task performance.

An IO-aware attention algorithm that computes exact attention without materializing the full N×N matrix — making long context windows computationally practical.

An optimized self-attention algorithm designed for Hopper GPUs, exploiting asynchronous execution and FP8 precision.

A half-precision 16-bit floating-point data format providing wide dynamic range for deep learning.

Compressing model weights to lower-precision numbers (8-bit, 4-bit) so models need less memory and run faster with minimal quality loss.

A data-parallel training strategy that shards model states (parameters, gradients, and optimizer states) across distributed GPU pools.

The parallel processors that train and serve AI models — and the supply-constrained line item that shapes every AI infrastructure decision.

A training technique that aggregates gradients across multiple micro-batches before performing an optimization step.

A safety training strategy that caps maximum gradient values to prevent numerical instability.

An attention mechanism that groups query heads together to share single key-value projection heads, optimizing memory footprints.

When an LLM confidently generates false information — a statistical property of how these models work, not a bug that will be patched out.

Hierarchical Navigable Small World — the graph-based index behind most vector databases; logarithmic-feeling search across millions of embeddings, paid for in RAM.

An approximate nearest neighbor vector search index based on multi-layered graph networks.

A search technique combining keyword-based lexical retrieval (BM25) and vector-based semantic retrieval.

The capacity of LLMs to recognize patterns and adapt behaviors based on examples provided in the prompt.

Running a trained model to produce outputs in production — where nearly all AI compute cost and latency engineering lives.

An 8-bit integer data format used to reduce model VRAM footprints and accelerate compute stages.

A vector search index combining Inverted File indexing and Product Quantization for memory efficiency.

Kahneman-Tversky Optimization — preference tuning from simple thumbs-up/down signals instead of paired comparisons, weighted asymmetrically like human loss aversion.

Training a small, cheap model to imitate a large one's outputs — the standard route to frontier-quality behavior on a narrow task at a fraction of the cost.

The attention key/value tensors an LLM stores per token during generation — the memory bottleneck of inference and the mechanism behind prompt caching discounts.

An optimization strategy that discards low-attention key-value states to stay within VRAM bounds.

Storing the KV cache in 8- or 4-bit precision instead of FP16 — the lever that decides how many long-context conversations fit on one GPU.

A regularization technique that normalizes activations across a single token's feature dimensions.

A training framework that adjusts optimizer step sizes dynamically over training epochs.

A neural network trained on massive text corpora to predict the next token — the foundation under chatbots, coding assistants, and AI agents.

Programmatic checks around an LLM's inputs and outputs that enforce safety, format, and policy rules the model alone can't guarantee.

A trainable low-rank decomposition matrix injected into transformer attention layers during model tuning.

An open standard for connecting AI models to tools and data sources, replacing per-app custom integrations with one reusable interface.

A training method combining half-precision (FP16/BF16) and single-precision (FP32) to speed up GPU operations.

A sparse model architecture that uses a routing network to activate specific subset expert parameters per token.

A optimization technique that removes low-contribution weights from neural networks to reduce parameter counts.

An attention mechanism where all query heads share a single key-value head to minimize KV cache storage overhead.

Models that process and generate multiple data types — text, images, audio, video — in a single system rather than text alone.

vLLM's virtual-memory trick for the KV cache: store attention state in fixed-size pages instead of contiguous blocks, eliminating the fragmentation that wasted most GPU memory.

A memory allocation algorithm that partitions the KV cache into non-contiguous virtual pages to prevent fragmentation.

A distributed training and inference model that partitions layer blocks sequentially across different GPU nodes.

The first stage of LLM inference: the entire prompt is processed in parallel to build the KV cache before any output token appears.

Architecture that fetches relevant external data at query time and injects it into an LLM's context before generating an answer.

The second-stage filter in retrieval pipelines: a cross-encoder rescores the top candidates so only genuinely relevant chunks reach the LLM's context.

An LLM trained to generate internal chain-of-thought before answering, trading latency and tokens for large gains on math, code, and planning.

A specialized model that re-evaluates the relevance of document chunks retrieved by search stages.

A distributed training communication algorithm where GPUs share gradients in a logical ring topology, optimizing network bandwidth utilization.

Rotary Position Embeddings — the technique that tells transformers where tokens are by rotating query/key vectors, and the knob behind every context-window extension.

A positional encoding method that applies a rotation matrix to Query and Key vectors in self-attention layers.

Caching LLM responses by meaning rather than exact text — similar questions hit the cache even when worded differently.

A dynamic text chunking method that splits documents based on semantic transitions rather than character counts.

A text chunking method that generates overlapping segments to preserve context across boundaries.

A mathematical state where extremely large input logits result in near-zero gradients during training.

An inference acceleration technique that uses a small draft model to generate candidate tokens verified in parallel by a target model.

An optimization method that estimates gradients using mini-batches to update parameters iteratively.

Sending tokens to the client as the model generates them, so users see output immediately instead of waiting for the full response.

Supervised Fine-Tuning — teaching a pretrained model to follow instructions by imitating curated example responses; the first and most accessible post-training stage.

A high-priority context block defining the behavior, constraints, and operational persona of a language model.

An attack where malicious instructions hidden in content an LLM processes hijack the model's behavior — the top security risk for LLM applications.

The parameter controlling randomness in LLM output — low values pick likely tokens for consistency, high values spread probability for variety.

An intra-model parallelism technique that splits individual matrix multiplication weights across multiple GPUs within a single layer.

NVIDIA's library for squeezing maximum LLM inference speed out of NVIDIA GPUs by compiling models into highly-optimized engines — top performance, at the cost of a heavier build step and NVIDIA lock-in.

The subword units LLMs read and write; the billing and capacity unit for context windows, API pricing, and rate limits.

The mechanism that lets an LLM invoke external functions — the model outputs a structured call, your code executes it, results return to the model.

Tokens Per Second — LLM speed measured two ways: per-request streaming speed users see, and aggregate GPU throughput that sets your unit economics.

NVIDIA's production inference server for serving any model from any framework at scale — a general-purpose serving platform, distinct from LLM-specialized engines like vLLM.

Time to First Token — the delay between sending a request and the first output token arriving; the metric users actually feel.

A database optimized for storing embeddings and finding nearest neighbors fast, using approximate indexes like HNSW.

Numeric vector representations of text, images, or code that place semantically similar items close together in vector space.

High-dimensional coordinate lists that capture the semantic meanings of words, sentences, or images.

The open-source LLM inference engine that made high-throughput serving practical — its PagedAttention manages the KV cache like virtual memory, dramatically raising GPU utilization.

The strategy of assigning initial random numeric values to neural weights to prevent gradient explosion.

A memory optimization protocol that shards parameters, gradients, and optimizer states across data-parallel ranks.

A prompting approach where a model generates answers directly without seeing task examples.