Publications

TaylorShift: Shifting the Complexity of Self-Attention from Squared to Linear (and Back) using Taylor-Softmax

This paper introduces TaylorShift, a novel reformulation of the attention mechanism using Taylor softmax that enables computing full token-to-token interactions in linear time. We analytically and empirically determine the crossover points where employing TaylorShift becomes more efficient than traditional attention. TaylorShift outperforms the traditional transformer architecture in 4 out of 5 tasks.

Tobias Christian Nauen, Sebastian Palacio, Andreas Dengel

Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers

A comprehensive benchmark and analysis of more than 45 transformer models for image classification to evaluate their efficiency, considering various performance metrics. We find the optimal architectures to use and uncover that model-scaling is more efficient than image scaling.

Tobias Christian Nauen, Sebastian Palacio, Federico Raue, Andreas Dengel

Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers

Stochastic Control with Signatures

This paper proposes a new method to parameterize open loop controls in stochastic optimal control problems using path signatures. We show that these controls are dense in the space of all admissible controls and establish conditions for stability of the controlled dynamics and target functional.

Peter Bank, Christian Bayer, Paul Peter Hager, Sebastian Riedel, Tobias Christian Nauen

Stochastic Optimal Control using Signatures

We consider a stochastic control problem and try to solve it using the signature method.

Tobias Christian Nauen, Sebastian Riedel

Stochastic Optimal Control using Signatures

Explaining Graph Neural Networks

We extend and test KEdge, an interpretable-by-design approach for graph neural networks, and compare it to gradient-based attribution techniques.

Tobias Christian Nauen, Thorben Funke, Avishek Anand

Explaining Graph Neural Networks