Tabular Foundation Models (TFMs)

Tabular foundation models are pretrained transformers that make predictions on a new table without training on it. They are pretrained once on millions of synthetic datasets, often generated from structural causal models, and at inference you give them your labeled rows as context and your unlabeled rows as queries; a single forward pass produces predictions, with no gradient descent on your data. Gradient boosting instead adapts by optimization, building new trees for every dataset. The idea comes from Prior-Fitted Networks, which approximate Bayesian prediction.

The article calls TabPFN, TabICL and TabFM the most benchmarked and documented, and names TabDPT, LimiX and Mitra as other models using the same in-context-learning idea.

On TabArena, which has 51 datasets and 16 model families ranked by Elo, the article reports that mid-2026 top single models by Elo are tabular foundation models, roughly 240 Elo ahead of the best single gradient boosting configuration. It adds two caveats: rankings depend on the protocol (without ensembling, CatBoost moves back ahead of neural approaches), and some TabPFN numbers come from Prior Labs itself, which TabArena's maintainers flag as a conflict-of-interest caveat. The TabICLv2 authors say their model beats heavily tuned XGBoost, CatBoost or LightGBM on about 80% of TabArena datasets, which is a claim by the model's authors.

The incumbent: additive ensembles of shallow trees, each fit to the pseudo-residuals of the current model. XGBoost uses second-order gradients and regularization, CatBoost uses ordered boosting, symmetric trees and native categorical handling, and LightGBM grows trees leaf-wise with gradient-based sampling. All three are permissively licensed and run well on CPU.

Cost shifts from training to inference: a TFM has almost no training step, but every prediction is a forward pass over the whole context, so memory and latency grow with the training set, and a GPU is advised beyond toy scale. In the article's one small test (569 rows, a free CPU Colab session), TabPFN scored higher accuracy than XGBoost and CatBoost but took about 48 seconds to predict, against hundredths of a second for CatBoost, which is a single illustrative run. GBMs have no real size ceiling, while each TFM has one tied to its architecture and cell budget. TFMs work near their defaults, so they save tuning time. CatBoost handles categoricals natively, and TFMs do so inside their preprocessing. Both families can be explained with Shapley values, though TFMs need dedicated tools. Licensing can decide the matter: GBMs are permissive, TabPFN and TabFM weights are non-commercial unless licensed, and TabICL is permissive.

Prefer gradient boosting when the training set is large or growing, serving is latency-sensitive or CPU-only, you need unrestricted commercial licensing, or you already have tuning and monitoring built around trees. Prefer a tabular foundation model for small to medium datasets inside the model's validated range, quick strong baselines without hyperparameter search, scarce data, batch or offline inference, and cases where the licence or a hosted API works for you. The article's bottom line is that gradient boosting is not being replaced yet, only scoped.

The article this page is based on.