Refurbished embeddings with integrated guidance networks for efficient context-length scaling

Date

2025-8-27

Author

Çavuşoğlu, Ali Devrim Ekin

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

981
views

0
downloads

Cite This

This thesis presents REIGN (Refurbished Embeddings with Integrated Guidance Networks), a framework for efficient representation learning of very long documents. Unlike Transformer-based encoders constrained by token length, REIGN adopts a hierarchical approach: a frozen Guidance Network (GN) first produces fixed-size chunk embeddings, which are then aggregated by a lightweight encoder trained with a SimCLR-style contrastive objective. This decoupled design enables semantic understanding of documents spanning hundreds of thousands of tokens without backpropagating through token-level models. On the GoodWiki-Long benchmark, REIGN improves retrieval quality by +3 percentage points (pp) in nDCG@10 over truncated GN baselines, while also offering significant computational savings. A caching mechanism further accelerates training by reusing GN embeddings, making REIGN highly effective for scalable document retrieval under resource constraints.

Subject Keywords

Long Document Embeddings, Contrastive Learning, Hierarchical Encoding, Guidance Networks, Document Retrieval, Large Context NLP, Token-Free Training, HDF5 Caching

URI

https://hdl.handle.net/11511/116043

Collections

Graduate School of Natural and Applied Sciences, Thesis