Please note: This PhD defence will take place online.
Liam Hebert, PhD candidate
David R. Cheriton School of Computer Science
Supervisors: Professors Robin Cohen, Lukasz Golab
Context is crucial for understanding the world and making informed decisions. While existing transformer architectures excel at contextualizing information locally, such as other words in a sentence, they fail to factor in broader, macro-level contexts. We identify available yet underutilized macro contexts in three use cases: online discussions, federated learning, and recommender systems. For each, we motivate the need to leverage existing macro context and propose context-aware solutions based on the transformer architecture.
In online discussion boards, the rich conversational and multimodal macro context in which a comment is made is often overlooked. This is especially pertinent in hate speech detection. Classical solutions that examine the text of individual comments in isolation fail to account for this context, leading to ambiguity and misinterpretation. For instance, the comment “Ew, that’s gross!”' has a different interpretation depending on whether it’s in response to food or sensitive issues like LGBTQ rights. Furthermore, images that accompany text can also provide crucial context. We propose Graphormer and mDT, novel deep learning model architectures based on graph transformer networks, which incorporate this valuable context when evaluating the hatefulness of individual comments. Our experimental results demonstrate a 7% F1 improvement over existing baselines that do not utilize this context, and a 21% F1 improvement over previous graph-based methods.
Second, we tackle the context-agnostic paradigm of federated learning. The prevalent Federated Averaging (FedAvg) method mechanically averages model weights, failing to account for the crucial macro-level context of heterogeneous-agent environments, leading to a suboptimal, one-size-fits-all model. For example, autonomous driving agents exploring rural roads acquire different knowledge than those in urban settings, and this environmental context is lost in the process. We propose FedFormer, a novel federation strategy that leverages transformer attention to enable each agent to weigh and selectively incorporate insights from its peers in a context-dependent manner. In turn, FedFormer enables a more effective, efficient federation that respects and adapts to environmental diversity while preserving privacy. Our experiments, conducted across heterogeneous environments from MetaWorld, demonstrate improvements of 1.48x to 3.41x over FedAvg.
Finally, in recommender systems, we explore recent work that replaces ID-based collaborative filtering models that focus on macro-level user similarity with text-only architectures that prioritize local textual representations of items. We demonstrate that a well-tuned, ID-only baseline is far more competitive than previously assumed, establishing the continued value of macro-level collaborative signals. This finding motivates our solution, Flare, a novel hybrid architecture that synergistically fuses robust collaborative embeddings with the rich semantic context from item descriptions via a Perceiver transformer. Furthermore, Flare is designed to incorporate the additional macro-context of explicit user intent through natural-language critiquing, bridging the gap between historical preferences and immediate goals and enabling enhanced user agency. Our experiments on the Amazon Product Reviews datasets show 1.7x and 2.53x increases in recall@1 and recall@10, respectively, compared to existing approaches.