This paper introduces TaylorShift, a novel reformulation of the attention mechanism using Taylor softmax that enables computing full token-to-token interactions in linear time. We analytically determine the crossover points where employing TaylorShift becomes more efficient than traditional attention, aligning closely with empirical measurements.

A comprehensive benchmark of more than 30 transformer models for vision to evaluate their efficiency, considering various performance metrics.