xFormers is a PyTorch-based library which hosts flexible Transformer building blocks. They are interoperable and optimized building blocks, which can be optionally combined to create some state of the art models (e.g. MHA attention mechanisms, sparsity-aware or long-context variants).