Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages.

Sun, 01 Jan 2023 00:00:00 +0100

Tomasz Limisiewicz, Jirí Balhar, David Marecek:
Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages. ACL (Findings) 2023: 5661-5681