PyTorch
- Category
- AI and LLMs
- Docs outline
- 16 sections
Official docs outline
docs.pytorch.org- Tensors
torch.Tensor, dtypes, devices, shapes, views and strides, in-place ops, indexing.
- Autograd mechanics
Reverse-mode automatic differentiation, requires_grad, computation graph, backward, no_grad, gradient accumulation, custom Functions.
- torch.nn modules
nn.Module, parameters, layers, loss functions, forward, state_dict, train/eval mode, initialization.
- torch.optim
Optimizers (SGD, Adam, AdamW), learning rate schedulers, zero_grad/step, parameter groups.
- Data loading
Dataset, DataLoader, samplers, batching, collate_fn, workers, shuffling.
- Saving and loading
torch.save/load, state_dict checkpoints, weights_only safety, resuming training.
- CUDA semantics
GPU devices, asynchronous execution, streams, memory management, pinned memory, moving data to device.
- Automatic mixed precision
autocast, GradScaler, float16 vs bfloat16, speed and memory tradeoffs.
- Broadcasting semantics
Shape broadcasting rules for tensor operations.
- torch.compile
TorchDynamo graph capture, Inductor, graph breaks, recompilation, speeding up models.
- Distributed training
torch.distributed, process groups, backends (NCCL, Gloo), collectives, init_process_group.
- DistributedDataParallel
Data parallel training across GPUs/nodes, gradient all-reduce, DistributedSampler, launching with torchrun.
- Fully Sharded Data Parallel
Sharding parameters, gradients and optimizer states, memory-efficient large-model training.
- Profiler
torch.profiler, tracing CPU/GPU ops, finding bottlenecks, TensorBoard traces.
- Quantization
Post-training and quantization-aware training, int8, reduced precision inference.
- torch.export
Ahead-of-time capture of full graphs, ExportedProgram, dynamic shapes, deployment.
Each section links to the official page. Titles and summaries by Extra Credit; the docs belong to their authors.
