Arber Zela | University of Freiburg

About Me

I am a Ph.D. student at the University of Freiburg advised by Prof. Frank Hutter. Since 2020 I am also an ELLIS PhD student co-advised by Prof. Yee Whye Teh at the University of Oxford. I am currently living close to Basel, Switzerland and working in Freiburg, Germany.

Research Interests

Neural Architecture Search: multi-objective optimization, structural pruning, benchmarking
In-Context Learning: Linear RNNs, Reasoning, Test-Time Scaling

News

[Mar. 2025] “DeltaProduct: Increasing the Expressivity of DeltaNet Through Products of Householders” is accepted at FM-WILD Workshop at ICLR 2025.
[Feb. 2025] 2 papers accepted at ICLR 2025. “Multi-objective Differentiable Neural Architecture Search” as a Poster and “Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues” as an Oral.
[Oct. 2024] 3 papers accepted at NeurIPS 2024 workshops: “Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues” (Oral), “GAMformer” (Poster), and “Ensembling Finetuned Language Models for Text Classification” (Poster).
[Sep. 2024] “HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models” was accepted to NeurIPS 2024 Datasets and Benchmarks Track.
[Jun. 2024] “Multi-objective Differentiable Neural Architecture Search” was accepted to the WANT workshop at ICML 2024.

Recent Talks / Seminars

[Nov. 2024] “Searching for Calibrated, Efficient, and Expressive Neural Networks” at University of Basel (Slides)
[Mar. 2024] “Multi-objective Differentiable Neural Architecture Search” at the AutoML Seminars (Youtube) (Slides)

Publications

FM-Wild@ICLR

DeltaProduct: Increasing the Expressivity of DeltaNet Through Products of Householders

Julien Siems*, Timur Carstensen*, Arber Zela, Frank Hutter, Massimiliano Pontil, Riccardo Grazzi* (*equal contribution)

ICLR Workshop on Foundations Models in the Wild

PDF

ICLR (Oral)

Unlocking State-Tracking in Linear RNNs through Negative Eigenvalues

Riccardo Grazzi*, Julien Siems*, Jörg Franke, Arber Zela, Frank Hutter, Massimiliano Pontil (*equal contribution)

International Conference on Learning Representation 2025 (Oral)

PDF

ICLR

Multi-objective Differentiable Neural Architecture Search

Rhea Sanjay Sukthanker*, Arber Zela*, Benedikt Staffler, Samuel Dooley, Josif Grabocka, Frank Hutter (*equal contribution)

International Conference on Learning Representation 2025 (Poster)

PDF Code

TLR@NeurIPS

GAMformer: In-Context Learning for Generalized Additive Models

Andreas Müller*, Julien Siems*, Harsha Nori, David Salinas, Arber Zela, Rich Caruana, Frank Hutter (*equal contribution)

NeurIPS 2024 Workshop on Table Representation Learning

PDF

FITML@NeurIPS

Ensembling Finetuned Language Models for Text Classification

Sebastian Pineda Arango, Maciej Janowski, Lennart Purucker, Arber Zela, Frank Hutter, Josif Grabocka

NeurIPS 2024 Workshop on Fine-Tuning in Machine Learning

PDF

NeurIPS DBT

HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models

Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler, Aaron Klein, Lennart Purucker, Jörg KH Franke, Frank Hutter

Advances in Neural Information Processing Systems 2024, Datasets and Benchmarks Track

PDF Code

IJCV

Neural architecture search for dense prediction tasks in computer vision

Rohit Mohan, Thomas Elsken, Arber Zela, Jan Hendrik Metzen, Benedikt Staffler, Thomas Brox, Abhinav Valada, Frank Hutter

International Journal of Computer Vision 2023

ICLR

NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy

Yash Mehta*, Colin White*, Arber Zela, Arjun Krishnakumar, Guri Zabergja, Shakiba Moradian, Mahmoud Safari, Kaicheng Yu, Frank Hutter (*equal contribution)

International Conference on Learning Representation 2022

PDF Code

NeurIPS DBT

NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies

Arjun Krishnakumar*, Colin White*, Arber Zela*, Renbo Tu, Mahmoud Safari, Frank Hutter (*equal contribution)

Advances in Neural Information Processing Systems 2022, Datasets and Benchmarks Track

PDF Code

NeurIPS DBT

JAHS-Bench-201: A Foundation For Research On Joint Architecture And Hyperparameter Search

Archit Bansal, Danny Stoll, Maciej Janowski, Arber Zela, Frank Hutter

Advances in Neural Information Processing Systems 2022, Datasets and Benchmarks Track

PDF Code

ICLR

Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks

Arber Zela*, Julien Siems*, Lucas Zimmer*, Jovita Lukasik, Margret Keuper, Frank Hutter (*equal contribution)

International Conference on Learning Representation 2022

PDF Code

NeurIPS

Neural Ensemble Search for Uncertainty Estimation and Dataset Shift

Sheheryar Zaidi*, Arber Zela*, Thomas Elsken, Chris C Holmes, Frank Hutter, Yee Whye Teh (*equal contribution)

Advances in Neural Information Processing Systems 2021

PDF Code

NeurIPS

How Powerful are Performance Predictors in Neural Architecture Search?

Colin White, Arber Zela, Binxin Ru, Yang Liu, Frank Hutter

Advances in Neural Information Processing Systems 2021

PDF Code

UDL@ICML 2021

Multi-headed Neural Ensemble Search

Ashwin Raaghav Narayanan, Arber Zela, Tonmoy Saikia, Thomas Brox, Frank Hutter

ICML 2021 Workshop on Uncertainty and Robustness in Deep Learning

PDF

IJCNN

Smooth Variational Graph Embeddings for Efficient Neural Architecture Search

Jovita Lukasik, David Friede, Arber Zela, Frank Hutter, Margret Keuper

International Joint Conference on Neural Networks 2021

PDF

ICLR

NAS-Bench-1Shot1: Benchmarking and dissecting one-shot neural architecture search

Arber Zela*, Julien Siems*, Frank Hutter (*equal contribution)

International Conference on Learning Representation 2020

PDF Code

ICLR (Oral)

Understanding and Robustifying Differentiable Architecture Search

Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, Frank Hutter

International Conference on Learning Representation 2020 (Oral)

PDF Code

ICCV

AutoDispNet: Improving Disparity Estimation With AutoML

Tonmoy Saikia, Yassine Marrakchi, Arber Zela, Frank Hutter, Thomas Brox

International Conference on Computer Vision 2019

PDF Code