Arber Zela

About Me

I am a Postdoctoral Researcher at LIONS, EPFL. Before I was an ELLIS Ph.D. student at the University of Freiburg advised by Frank Hutter and co-advised by Yee Whye Teh at the University of Oxford. Since 2025 I am also an ELLIS member.

Research Interests

  • Adaptive Computation

    Reasoning, looping, test-time training

  • Global Optimization

    Hyperparameter optimization, multi-objective optimization, architecture search

  • Artificial Life

    Cellular automata, evolutionary computation

News

  1. Sep 2026 "TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting" is accepted at NeurIPS 2026.
  2. Jun 2026 "Evolving Many Worlds: Towards Open-Ended Discovery in Petri Dish NCA via Population-Based Training" was accepted for Oral presentation at ALife 2026.
  3. May 2026 I joined Volkan Cevher's LIONS lab at EPFL as a postdoc.
  4. Jan 2026 "Improving LLM-based Global Optimization with Search Space Partitioning" is accepted at ICLR 2026.
  5. Nov 2025 "TempoPFN: Towards Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting" is accepted at AITD@EurIPS 2025.
  6. Oct 2025 I successfully defended my PhD Thesis with summa cum laude.
  7. Sep 2025 "DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products" is accepted at NeurIPS 2025.
  8. May 2025 "Regularized Neural Ensemblers" is accepted at AutoML Conference 2025.
  9. Mar 2025 "DeltaProduct: Increasing the Expressivity of DeltaNet Through Products of Householders" is accepted at FM-WILD Workshop at ICLR 2025 as an Oral.
  10. Feb 2025 2 papers accepted at ICLR 2025. "Multi-objective Differentiable Neural Architecture Search" as a Poster and "Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues" as an Oral.
  11. Oct 2024 3 papers accepted at NeurIPS 2024 workshops: "Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues" (Oral), "GAMformer" (Poster), and "Ensembling Finetuned Language Models for Text Classification" (Poster).
  12. Sep 2024 "HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models" was accepted to NeurIPS 2024 Datasets and Benchmarks Track.
  13. Jun 2024 "Multi-objective Differentiable Neural Architecture Search" was accepted to the WANT workshop at ICML 2024.

Recent Talks / Seminars

  • Nov 2024
    Searching for Calibrated, Efficient, and Expressive Neural Networks
    University of Basel
  • Mar 2024
    Multi-objective Differentiable Neural Architecture Search
    AutoML Seminars

Publications

27 papers · Google Scholar

2026

  1. NeurIPS

    TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting

    Vladyslav Moroshan*, Julien Siems*, Arber Zela, Timur Carstensen, Frank Hutter (*equal contribution)

    Advances in Neural Information Processing Systems 2026

  2. arXiv

    Spherical Fixed-Point Reasoners: Adaptive Loop Halting on the Hypersphere

    Arber Zela, Artavazd Maranjyan, Pol Puigdemont, David Berghaus, Valentyn Boreiko, Volkan Cevher

    arXiv preprint 2026

  3. arXiv

    Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

    Julien Siems*, Riccardo Grazzi*, Korbinian Pöppel*, Jaisidh Singh, Arber Zela, Timur Carstensen, Jenia Jitsev, Frank Hutter, Volkan Cevher, Antonio Orvieto, Aaron Klein (*equal contribution)

    arXiv preprint 2026

  4. ALIFE (Oral)

    Evolving Many Worlds: Towards Open-Ended Discovery in Petri Dish NCA via Population-Based Training

    Uljad Berdica*, Jakob Foerster, Frank Hutter, Arber Zela* (*equal contribution)

    The 2026 Artificial Life Conference (Oral)

  5. ICLR

    Improving LLM-based Global Optimization with Search Space Partitioning

    Andrej Schwanke*, Lyubomir Ivanov*, David Salinas, Fabio Ferreira, Aaron Klein, Frank Hutter, Arber Zela* (*equal contribution)

    International Conference on Learning Representation 2026

  6. arXiv

    Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch

    Fabio Ferreira*, Lucca Wobbe, Arjun Krishnakumar, Frank Hutter*, Arber Zela* (*equal supervision)

    arXiv preprint 2026

  7. arXiv

    Multi-Objective Hierarchical Optimization with Large Language Models

    Andrej Schwanke, Lyubomir Ivanov, David Salinas, Frank Hutter, Arber Zela

    arXiv preprint 2026

2025

  1. NeurIPS

    DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products

    Julien Siems*, Timur Carstensen*, Arber Zela, Frank Hutter, Massimiliano Pontil, Riccardo Grazzi* (*equal contribution)

    Advances in Neural Information Processing Systems 2025

  2. AutoML-Conf

    Regularized Neural Ensemblers

    Sebastian Pineda Arango*, Maciej Janowski*, Lennart Purucker, Arber Zela, Frank Hutter, Josif Grabocka (*equal contribution)

    AutoML Conference 2025

  3. ICLR (Oral)

    Unlocking State-Tracking in Linear RNNs through Negative Eigenvalues

    Riccardo Grazzi*, Julien Siems*, Jörg Franke, Arber Zela, Frank Hutter, Massimiliano Pontil (*equal contribution)

    International Conference on Learning Representation 2025 (Oral)

  4. ICLR

    Multi-objective Differentiable Neural Architecture Search

    Rhea Sanjay Sukthanker*, Arber Zela*, Benedikt Staffler, Samuel Dooley, Josif Grabocka, Frank Hutter (*equal contribution)

    International Conference on Learning Representation 2025 (Poster)

  5. JCM

    Toward Artificial Intelligence in Oncology and Cardiology: A Narrative Review of Systems, Challenges, and Opportunities

    Visar Vela, Ali Yasin Sonay, Perparim Limani, Lukas Graf, Besmira Sabani, Diona Gjermeni, Andi Rroku, Arber Zela, Era Gorica, Hector Rodriguez Cetina Biefer, Uljad Berdica, Euxhen Hasanaj, Adisa Trnjanin, Taulant Muka, Omer Dzemali

    Journal of Clinical Medicine 14(21), 2025

2024

  1. TLR@NeurIPS

    GAMformer: In-Context Learning for Generalized Additive Models

    Andreas Müller*, Julien Siems*, Harsha Nori, David Salinas, Arber Zela, Rich Caruana, Frank Hutter (*equal contribution)

    NeurIPS 2024 Workshop on Table Representation Learning

  2. FITML@NeurIPS

    Ensembling Finetuned Language Models for Text Classification

    Sebastian Pineda Arango, Maciej Janowski, Lennart Purucker, Arber Zela, Frank Hutter, Josif Grabocka

    NeurIPS 2024 Workshop on Fine-Tuning in Machine Learning

  3. NeurIPS DBT

    HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models

    Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler, Aaron Klein, Lennart Purucker, Jörg KH Franke, Frank Hutter

    Advances in Neural Information Processing Systems 2024, Datasets and Benchmarks Track

2023

  1. IJCV

    Neural architecture search for dense prediction tasks in computer vision

    Rohit Mohan, Thomas Elsken, Arber Zela, Jan Hendrik Metzen, Benedikt Staffler, Thomas Brox, Abhinav Valada, Frank Hutter

    International Journal of Computer Vision 2023

2022

  1. ICLR

    NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy

    Yash Mehta*, Colin White*, Arber Zela, Arjun Krishnakumar, Guri Zabergja, Shakiba Moradian, Mahmoud Safari, Kaicheng Yu, Frank Hutter (*equal contribution)

    International Conference on Learning Representation 2022

  2. NeurIPS DBT

    NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies

    Arjun Krishnakumar*, Colin White*, Arber Zela*, Renbo Tu, Mahmoud Safari, Frank Hutter (*equal contribution)

    Advances in Neural Information Processing Systems 2022, Datasets and Benchmarks Track

  3. NeurIPS DBT

    JAHS-Bench-201: A Foundation For Research On Joint Architecture And Hyperparameter Search

    Archit Bansal, Danny Stoll, Maciej Janowski, Arber Zela, Frank Hutter

    Advances in Neural Information Processing Systems 2022, Datasets and Benchmarks Track

  4. ICLR

    Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks

    Arber Zela*, Julien Siems*, Lucas Zimmer*, Jovita Lukasik, Margret Keuper, Frank Hutter (*equal contribution)

    International Conference on Learning Representation 2022

2021

  1. NeurIPS

    Neural Ensemble Search for Uncertainty Estimation and Dataset Shift

    Sheheryar Zaidi*, Arber Zela*, Thomas Elsken, Chris C Holmes, Frank Hutter, Yee Whye Teh (*equal contribution)

    Advances in Neural Information Processing Systems 2021

  2. NeurIPS

    How Powerful are Performance Predictors in Neural Architecture Search?

    Colin White, Arber Zela, Binxin Ru, Yang Liu, Frank Hutter

    Advances in Neural Information Processing Systems 2021

  3. UDL@ICML 2021

    Multi-headed Neural Ensemble Search

    Ashwin Raaghav Narayanan, Arber Zela, Tonmoy Saikia, Thomas Brox, Frank Hutter

    ICML 2021 Workshop on Uncertainty and Robustness in Deep Learning

  4. IJCNN

    Smooth Variational Graph Embeddings for Efficient Neural Architecture Search

    Jovita Lukasik, David Friede, Arber Zela, Frank Hutter, Margret Keuper

    International Joint Conference on Neural Networks 2021

2020

  1. ICLR

    NAS-Bench-1Shot1: Benchmarking and dissecting one-shot neural architecture search

    Arber Zela*, Julien Siems*, Frank Hutter (*equal contribution)

    International Conference on Learning Representation 2020

  2. ICLR (Oral)

    Understanding and Robustifying Differentiable Architecture Search

    Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, Frank Hutter

    International Conference on Learning Representation 2020 (Oral)

2019

  1. ICCV

    AutoDispNet: Improving Disparity Estimation With AutoML

    Tonmoy Saikia, Yassine Marrakchi, Arber Zela, Frank Hutter, Thomas Brox

    International Conference on Computer Vision 2019

No publications match your search.