Arber Zela | ELLIS Institute Tübingen

About Me

I am a Postdoctoral Researcher at the ELLIS Insitute Tübingen, Germany. Before I was an ELLIS Ph.D. student at the University of Freiburg advised by Frank Hutter and co-advised by Yee Whye Teh at the University of Oxford. Since 2025 I am also an ELLIS member.

Research Interests

AutoML: Neural Architecture Search, Structural Pruning, Benchmarking
In-Context Learning: Linear RNNs, Reasoning, Tabular and Time-Series Foundation Models
Global Optimization: Bayesian Optimization, Multi-objective Optimization, Bandits

News

[Jan. 2026] "Improving LLM-based Global Optimization with Search Space Partitioning" is accepted at ICLR 2026.
[Nov. 2025] "TempoPFN: Towards Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting" is accepted at AITD@EurIPS 2025.
[Oct. 2025] I successfully defended my PhD Thesis with summa cum laude.
[Sep. 2025] "DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products" is accepted at NeurIPS 2025.
[May 2025] "Regularized Neural Ensemblers" is accepted at AutoML Conference 2025.
[Mar. 2025] "DeltaProduct: Increasing the Expressivity of DeltaNet Through Products of Householders" is accepted at FM-WILD Workshop at ICLR 2025 as an Oral.
[Feb. 2025] 2 papers accepted at ICLR 2025. "Multi-objective Differentiable Neural Architecture Search" as a Poster and "Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues" as an Oral.
[Oct. 2024] 3 papers accepted at NeurIPS 2024 workshops: "Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues" (Oral), "GAMformer" (Poster), and "Ensembling Finetuned Language Models for Text Classification" (Poster).
[Sep. 2024] "HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models" was accepted to NeurIPS 2024 Datasets and Benchmarks Track.
[Jun. 2024] "Multi-objective Differentiable Neural Architecture Search" was accepted to the WANT workshop at ICML 2024.

Recent Talks / Seminars

[Nov. 2024] "Searching for Calibrated, Efficient, and Expressive Neural Networks" at University of Basel (Slides)
[Mar. 2024] "Multi-objective Differentiable Neural Architecture Search" at the AutoML Seminars (Youtube) (Slides)

Publications

ICLR

Improving LLM-based Global Optimization with Search Space Partitioning

Andrej Schwanke*, Lyubomir Ivanov*, David Salinas, Fabio Ferreira, Aaron Klein, Frank Hutter, Arber Zela* (*equal contribution)

International Conference on Learning Representation 2026

PDF Code
NeurIPS

DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products

Julien Siems*, Timur Carstensen*, Arber Zela, Frank Hutter, Massimiliano Pontil, Riccardo Grazzi* (*equal contribution)

Advances in Neural Information Processing Systems 2025

PDF
AITD@EurIPS

TempoPFN: Towards Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting

Vladyslav Moroshan*, Julien Siems*, Arber Zela, Timur Carstensen, Frank Hutter (*equal contribution)

EurIPS 2025 Workshop on AI for Tabular Data

PDF
AutoML-Conf

Regularized Neural Ensemblers

Sebastian Pineda Arango*, Maciej Janowski*, Lennart Purucker, Arber Zela, Frank Hutter, Josif Grabocka (*equal contribution)

AutoML Conference 2025

PDF
ICLR (Oral)

Unlocking State-Tracking in Linear RNNs through Negative Eigenvalues

Riccardo Grazzi*, Julien Siems*, Jörg Franke, Arber Zela, Frank Hutter, Massimiliano Pontil (*equal contribution)

International Conference on Learning Representation 2025 (Oral)

PDF
ICLR

Multi-objective Differentiable Neural Architecture Search

Rhea Sanjay Sukthanker*, Arber Zela*, Benedikt Staffler, Samuel Dooley, Josif Grabocka, Frank Hutter (*equal contribution)

International Conference on Learning Representation 2025 (Poster)

PDF Code
JCM

Toward Artificial Intelligence in Oncology and Cardiology: A Narrative Review of Systems, Challenges, and Opportunities

Visar Vela, Ali Yasin Sonay, Perparim Limani, Lukas Graf, Besmira Sabani, Diona Gjermeni, Andi Rroku, Arber Zela, Era Gorica, Hector Rodriguez Cetina Biefer, Uljad Berdica, Euxhen Hasanaj, Adisa Trnjanin, Taulant Muka, Omer Dzemali

Journal of Clinical Medicine 14(21), 2025

PDF
TLR@NeurIPS

GAMformer: In-Context Learning for Generalized Additive Models

Andreas Müller*, Julien Siems*, Harsha Nori, David Salinas, Arber Zela, Rich Caruana, Frank Hutter (*equal contribution)

NeurIPS 2024 Workshop on Table Representation Learning

PDF
FITML@NeurIPS

Ensembling Finetuned Language Models for Text Classification

Sebastian Pineda Arango, Maciej Janowski, Lennart Purucker, Arber Zela, Frank Hutter, Josif Grabocka

NeurIPS 2024 Workshop on Fine-Tuning in Machine Learning

PDF
NeurIPS DBT

HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models

Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler, Aaron Klein, Lennart Purucker, Jörg KH Franke, Frank Hutter

Advances in Neural Information Processing Systems 2024, Datasets and Benchmarks Track

PDF Code
IJCV

Neural architecture search for dense prediction tasks in computer vision

Rohit Mohan, Thomas Elsken, Arber Zela, Jan Hendrik Metzen, Benedikt Staffler, Thomas Brox, Abhinav Valada, Frank Hutter

International Journal of Computer Vision 2023
ICLR

NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy

Yash Mehta*, Colin White*, Arber Zela, Arjun Krishnakumar, Guri Zabergja, Shakiba Moradian, Mahmoud Safari, Kaicheng Yu, Frank Hutter (*equal contribution)

International Conference on Learning Representation 2022

PDF Code
NeurIPS DBT

NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies

Arjun Krishnakumar*, Colin White*, Arber Zela*, Renbo Tu, Mahmoud Safari, Frank Hutter (*equal contribution)

Advances in Neural Information Processing Systems 2022, Datasets and Benchmarks Track

PDF Code
NeurIPS DBT

JAHS-Bench-201: A Foundation For Research On Joint Architecture And Hyperparameter Search

Archit Bansal, Danny Stoll, Maciej Janowski, Arber Zela, Frank Hutter

Advances in Neural Information Processing Systems 2022, Datasets and Benchmarks Track

PDF Code
ICLR

Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks

Arber Zela*, Julien Siems*, Lucas Zimmer*, Jovita Lukasik, Margret Keuper, Frank Hutter (*equal contribution)

International Conference on Learning Representation 2022

PDF Code
NeurIPS

Neural Ensemble Search for Uncertainty Estimation and Dataset Shift

Sheheryar Zaidi*, Arber Zela*, Thomas Elsken, Chris C Holmes, Frank Hutter, Yee Whye Teh (*equal contribution)

Advances in Neural Information Processing Systems 2021

PDF Code
NeurIPS

How Powerful are Performance Predictors in Neural Architecture Search?

Colin White, Arber Zela, Binxin Ru, Yang Liu, Frank Hutter

Advances in Neural Information Processing Systems 2021

PDF Code
UDL@ICML 2021

Multi-headed Neural Ensemble Search

Ashwin Raaghav Narayanan, Arber Zela, Tonmoy Saikia, Thomas Brox, Frank Hutter

ICML 2021 Workshop on Uncertainty and Robustness in Deep Learning

PDF
IJCNN

Smooth Variational Graph Embeddings for Efficient Neural Architecture Search

Jovita Lukasik, David Friede, Arber Zela, Frank Hutter, Margret Keuper

International Joint Conference on Neural Networks 2021

PDF
ICLR

NAS-Bench-1Shot1: Benchmarking and dissecting one-shot neural architecture search

Arber Zela*, Julien Siems*, Frank Hutter (*equal contribution)

International Conference on Learning Representation 2020

PDF Code
ICLR (Oral)

Understanding and Robustifying Differentiable Architecture Search

Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, Frank Hutter

International Conference on Learning Representation 2020 (Oral)

PDF Code
ICCV

AutoDispNet: Improving Disparity Estimation With AutoML

Tonmoy Saikia, Yassine Marrakchi, Arber Zela, Frank Hutter, Thomas Brox

International Conference on Computer Vision 2019

PDF Code