Please note: This PhD seminar will take place in DC 2314 and online.
William Loh, PhD candidate
David R. Cheriton School of Computer Science
Supervisor: Professor Pascal Poupart
Basis transformers have shown some successes in multi-task regression for multimodal and heterogeneous tabular data. However, they have yet to demonstrate their ability to produce effective tabular encodings through pretraining.
In this work, we investigate pretraining a basis transformer on over three million tables with the goal of learning representations that best capture the semantic context represented by each row of data. To reach this goal, we introduce a new loss function for transforming a regression task into a multi-label classification task, and use distance correlation for improved self-supervised pretraining. When evaluated on realistic tabular data, a pretrained basis transformer outperformed various foundation model baselines and competitive classical approaches on zero-shot and regular regression tasks.
To attend this PhD seminar in person, please go to DC 2314. You can also attend virtually on MS Teams.