Machine Learning Models for Tabular Data logopeterindia.net
Knowledge Hub

Machine Learning Models: Logistic Regression, Random Forest, CatBoost & TabPFN-3.5

The main model families for tabular data — from logistic regression and random forests to CatBoost and tabular foundation models like TabPFN-3.5 — with papers, libraries, benchmarks and guidance on choosing.

79 topics•9 categories•Updated October 2026

How the Model Families Compare

8 entries

Logistic regression is a fast, interpretable linear baseline that is easy to calibrate and explain. Random forests average many decision trees and are robust, need little tuning, and give feature importances. Gradient-boosted trees (CatBoost, XGBoost, LightGBM) usually lead on tabular data after tuning; CatBoost is notable for handling categorical features natively with ordered boosting. Tabular foundation models such as TabPFN are pretrained on synthetic tasks and predict in a single forward pass with no per-dataset training, which is strongest on small and medium tables, while tuned boosted trees remain hard to beat on very large or unusual data. Start with a simple baseline, then compare trees and a foundation model with cross-validation on your own data.

01Which algorithm should I use? (scikit-learn map)Cheat sheet for picking an estimator. 02Why do tree-based models still outperform deep learning on tabular data?The benchmark paper that framed the debate. 03TabArena: A Living Benchmark for Machine Learning on Tabular DataContinuously updated benchmark comparing tuned and default models, including foundation models. 04TabArena leaderboardLive leaderboard. 05TabArena on GitHubBenchmark code and results. 06TALENT benchmarkLarge collection of tabular datasets and methods. 07TabReD: Analyzing Pitfalls in Tabular ML BenchmarksBenchmark with realistic temporal splits. 08OpenMLOpen platform of datasets, tasks and benchmark runs.

Logistic Regression & Linear Models

9 entries

Interpretable baselines.

09Logistic regression (scikit-learn guide)User guide to logistic regression and regularisation. 10LogisticRegression (API)Estimator reference. 11Probability calibrationMake predicted probabilities trustworthy. 12Support vector machinesSVM guide. 13Nearest neighborsk-nearest neighbours guide. 14Naive BayesNaive Bayes classifiers. 15Explainable Boosting Machines (InterpretML)Glass-box models with strong accuracy. 16An Introduction to Statistical LearningFree textbook covering linear and tree models. 17The Elements of Statistical LearningFree advanced textbook.

Decision Trees & Random Forests

6 entries

Bagged tree ensembles.

18Random Forests (Breiman, 2001)The original paper. 19Forests of randomized trees (scikit-learn)Random forests and extra-trees guide. 20RandomForestClassifier (API)Estimator reference. 21Decision trees (scikit-learn)Decision tree guide. 22Ensemble methods (scikit-learn)Bagging, boosting and stacking. 23Permutation vs impurity feature importancesInterpreting forests.

Gradient Boosting: CatBoost, XGBoost, LightGBM

10 entries

The usual winners on tabular data.

24CatBoost documentationGradient boosting with native categorical feature support. 25CatBoost on GitHubSource code and releases. 26CatBoost: unbiased boosting with categorical featuresThe ordered boosting paper. 27CatBoost: gradient boosting with categorical features supportEngineering paper. 28XGBoost documentationScalable tree boosting. 29XGBoost: A Scalable Tree Boosting SystemThe XGBoost paper. 30LightGBM documentationFast, histogram-based gradient boosting. 31LightGBM on GitHubSource code (moved to the lightgbm-org organisation). 32LightGBM paper (NeurIPS 2017)A highly efficient gradient boosting decision tree. 33HistGradientBoosting (scikit-learn)Native scikit-learn boosting.

Tabular Foundation Models: TabPFN-3.5 & Others

12 entries

Pretrained models that predict by in-context learning.

34TabPFN-3.5: Technical ReportPrior Labs' September 2026 report describing the flagship model and its Fast, Plus and Thinking variants. 35Prior LabsCompany behind TabPFN; the hosted API and product news. 36Prior Labs documentationQuickstart for the Python package and API. 37TabPFN on GitHubOpen-source Python package (pip install tabpfn). 38TabPFN extensionsInterpretability, feature selection and other add-ons. 39Prior Labs on Hugging FaceModel checkpoints; newer weights are gated and licensed for non-commercial use, so check terms before production use. 40TabPFN v2 on Hugging FaceThe widely cited version 2 classifier checkpoint. 41Accurate predictions on small data with a tabular foundation model (Nature)TabPFN v2 in Nature, 2025. 42TabPFN: A Transformer That Solves Small Tabular Classification Problems in a SecondThe original TabPFN paper. 43TabPFN (Wikipedia)Overview and version history. 44TabICL: A Tabular Foundation Model for In-Context Learning on Large DataOpen alternative aimed at larger tables. 45TabICL on GitHubCode and checkpoints.

Tabular Deep Learning

6 entries

Neural networks for tables.

46Revisiting Deep Learning Models for Tabular Data (FT-Transformer)Strong MLP and transformer baselines. 47rtdl (Yandex Research)Code for tabular deep learning research. 48TabNetAttentive interpretable tabular learning. 49TabNet (PyTorch)PyTorch implementation. 50TabRTabular deep learning meets nearest neighbours. 51TabMEfficient ensembling of MLPs for tabular data.

AutoML, Tuning & Explainability

11 entries

Automate selection and explain results.

52AutoGluonAutoML that ensembles boosted trees, neural nets and foundation models. 53AutoGluon on GitHubSource code. 54auto-sklearnAutoML on scikit-learn. 55TPOTGenetic-programming AutoML. 56H2O AutoMLAutoML in H2O. 57OptunaHyperparameter optimisation. 58SHAPExplain model predictions with Shapley values. 59Interpretable Machine Learning (Molnar)Free book on explaining models. 60InterpretMLInterpretable models and explanations. 61Model evaluation (scikit-learn)Metrics and scoring. 62Cross-validation and model selection (scikit-learn)Validate and tune models.

Libraries, Data & Courses

5 entries

Where to practise.

63scikit-learnThe standard Python machine learning library. 64Kaggle: Intro to Machine LearningFree short course. 65Kaggle: Intermediate Machine LearningFree course on pipelines, categorical data and boosting. 66Machine Learning Specialization (Coursera)Andrew Ng's updated course series. 67UCI Machine Learning RepositoryClassic datasets for experiments.

Related Pages

12 entries

Neighbouring pages on this site.

68Machine LearningMachine learning hub. 69ML FrameworksMachine learning frameworks. 70AutoML ToolsAutoML tools. 71Feature Engineering ToolsFeature engineering tools. 72Hyperparameter Optimisation ToolsHyperparameter tuning. 73Explainable AI FrameworksExplainability frameworks. 74Python Libraries for XAIPython explainability libraries. 75Causal AI ModelsCausal inference and discovery. 76Deep LearningDeep learning hub. 77Data ScienceData science hub. 78Artificial IntelligenceAI hub. 79Feature Engineering: Approaches & ToolsHow to prepare model inputs: encoding, scaling, selection and domain features, with libraries and feature stores.

Frequently Asked Questions

Which machine learning model is best for tabular data?

There is no single winner. Tuned gradient-boosted trees such as CatBoost, XGBoost and LightGBM are the usual strong choice, random forests are robust with little tuning, logistic regression is the interpretable baseline, and tabular foundation models like TabPFN are strong on small to medium tables. Compare them with cross-validation on your own data.

What is TabPFN-3.5?

TabPFN-3.5 is Prior Labs' flagship tabular foundation model, described in a technical report published in September 2026. It is pretrained on synthetic tasks and predicts in a single forward pass without per-dataset training, with Fast, Plus and Thinking variants. Prior Labs reports state-of-the-art results on TabArena, but independent validation is still limited, and the newer open weights carry a non-commercial licence, so check the terms before production use.

Why is CatBoost popular?

CatBoost handles categorical features natively using ordered target statistics, reducing target leakage, often gives strong results with default settings, and supports GPU training and good tooling.

When should I still use logistic regression or random forests?

Use logistic regression when you need transparency, calibrated probabilities and a fast baseline, and random forests for a robust nonlinear model with minimal tuning and built-in importances.

How current is this page?

Links were checked in October 2026. TabPFN-3.5 claims come largely from Prior Labs and independent validation is still limited. Sites that block automated checks are kept and may need a manual click.