Interchangeable Token Embeddings for Extendable Vocabulary and Alpha-Equivalence

Date

2025-01-01

Author

Işık, İlker
Cinbiş, Ramazan Gökberk
Gol, Ebru Aydin

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

36
views

0
downloads

Cite This

Language models lack the notion of interchangeable tokens: symbols that are semantically equivalent yet distinct, such as bound variables in formal logic. This limitation prevents generalization to larger vocabularies and hinders the model’s ability to recognize alpha-equivalence, where renaming bound variables preserves meaning. We formalize this machine learning problem and introduce alpha-covariance, a metric for evaluating robustness to such transformations. To tackle this task, we propose a dual-part token embedding strategy: a shared component ensures semantic consistency, while a randomized component maintains token distinguishability. Compared to a baseline that relies on alpha-renaming for data augmentation, our approach demonstrates improved generalization to unseen tokens in linear temporal logic solving, propositional logic assignment prediction, and copying with an extendable vocabulary, while introducing a favorable inductive bias for alphaequivalence. Our findings establish a foundation for designing language models that can learn interchangeable token representations, a crucial step toward more flexible and systematic reasoning in formal domains. Our code and project page are available at necrashter.github.io/interchangeabletoken-embeddings

URI

https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=105023640640&origin=inward
https://hdl.handle.net/11511/118151

Collections

Department of Computer Engineering, Conference / Seminar