Tensor

A tensor is a multi-dimensional array containing elements of a single data type. It is the single most fundamental object in modern machine learning: every piece of data that flows through a neural network — the input image, the token embeddings, the weight matrices, the intermediate activations, and the gradients computed during backpropagation — is stored and manipulated as a tensor.

The term is borrowed from physics and differential geometry, but in deep learning the working definition is deliberately simpler and more concrete. As the official PyTorch documentation states, “a torch.Tensor is a multi-dimensional matrix containing elements of a single data type.” You can think of it as a generalization of familiar mathematical objects: a single number, a list of numbers, a grid of numbers, and beyond.

The Rank of a Tensor

The rank (also called the number of dimensions, or ndim) of a tensor is how many indices you need to address a single element. This is the concept that unifies scalars, vectors, and matrices under one abstraction.

• Rank 0 — Scalar: a single number, e.g. a loss value. Shape ().
• Rank 1 — Vector: an ordered list of numbers, e.g. a single word embedding. Shape (d,).
• Rank 2 — Matrix: a grid, e.g. a weight matrix or a batch of embeddings. Shape (rows, cols).
• Rank 3 and higher — Tensor: the general case. A batch of RGB images is rank 4; the key/query/value activations inside a Transformer are typically rank 4 (batch, heads, sequence, head_dim).

The Three Defining Attributes

Every tensor is fully described by three properties, and understanding them is the key to debugging most deep-learning code:

• Shape — the size along each dimension (e.g. torch.Size([32, 512, 768])). Shape mismatches are the single most common source of runtime errors in model code.
• dtype — the numerical type of every element. All elements share one type. Common choices are float32, bfloat16, and increasingly float8.
• device — where the tensor physically lives: CPU RAM, a specific CUDA GPU, or another accelerator. Operations require their tensors to be on the same device.

Why Tensors Define AI Hardware

Tensors are not just a software convenience — they dictate how AI hardware is built. Neural networks are, at their core, enormous sequences of tensor operations, dominated by matrix multiplication. Because those operations are massively parallel and regular, they map perfectly onto specialized silicon.

NVIDIA Tensor Cores are dedicated hardware units that perform a small matrix-multiply-and-accumulate (D = A × B + C) in a single operation, and they are the reason modern GPUs are so effective for AI. Google’s TPUs are organized around a systolic array built for exactly the same purpose. Frameworks like PyTorch and TensorFlow exist primarily to build a computation graph of tensor operations, dispatch them to this hardware, and — via autograd — automatically compute the gradients needed to train the model.

2025–2026: Low-Precision Tensors and the Precision Frontier

The most active area of tensor evolution today is numerical precision. Training and serving frontier models is bottlenecked by memory bandwidth and capacity, so the industry has moved aggressively toward representing tensors with fewer bits per element:

• BF16 / FP16 remain the default for stable training, halving memory versus FP32.
• FP8 (8-bit floating point, e.g. float8_e4m3fn and float8_e5m2) is now a first-class dtype in PyTorch and is used for both training and inference. NVIDIA’s Hopper and Blackwell Transformer Engine dynamically manages FP8 tensors to preserve accuracy.
• FP4 (4-bit floating point) is the headline capability of the NVIDIA Blackwell architecture’s 2nd-generation Transformer Engine, enabling dramatically higher inference throughput for trillion-parameter and Mixture-of-Experts models by shrinking each tensor element to just four bits.

This precision race is why a “tensor” in 2026 is a far richer object than it was five years ago: the same multi-dimensional array must now be reasoned about not only in terms of its shape and device, but in terms of how few bits each element can safely use without degrading model quality. Every gain in low-precision tensor formats translates directly into larger context windows, cheaper inference, and the ability to train the next generation of frontier models.

import torch

# Create a tensor from data (the recommended constructor)
x = torch.tensor([[1.0, 2.0, 3.0],
                  [4.0, 5.0, 6.0]])

# Every tensor has three defining attributes
print(x.shape)   # torch.Size([2, 3])  -> dimensions
print(x.dtype)   # torch.float32       -> data type
print(x.device)  # cpu                 -> where it lives

# Rank = number of dimensions (ndim)
scalar = torch.tensor(3.14)          # rank 0
vector = torch.tensor([1, 2, 3])     # rank 1
matrix = torch.zeros(4, 4)           # rank 2
batch  = torch.randn(32, 3, 224, 224)  # rank 4: (N, C, H, W)
print(scalar.ndim, vector.ndim, matrix.ndim, batch.ndim)  # 0 1 2 4

# Move a tensor to the GPU for accelerated compute
if torch.cuda.is_available():
    x = x.to("cuda")

# Autograd: track operations to compute gradients automatically
w = torch.randn(3, 1, requires_grad=True)
y = (x.cpu() @ w).sum()
y.backward()
print(w.grad.shape)  # torch.Size([3, 1])

# 2025 low-precision training: 8-bit floating point tensors
# (native on NVIDIA Hopper/Blackwell Tensor Cores)
lp = x.cpu().to(torch.float8_e4m3fn)
print(lp.dtype)  # torch.float8_e4m3fn