mirrors/Odin
1
0
Fork 0
mirror of https://github.com/odin-lang/Odin.git synced 2026-06-04 01:34:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #3785 from mgavioli/new_i18n_API

Browse Source 
Separate I18N calls for immutable and for pluraliseable strings. Fixes #3687
This commit is contained in:
Jeroen van Rijn
2024-06-19 11:50:24 +02:00
committed by GitHub
parent 7064166da2 322b7f1f2f
commit 2797dc6452
3 changed files with 179 additions and 127 deletions
Download Patch File Download Diff File Expand all files Collapse all files

78
core/text/i18n/doc.odin
View File

@@ -1,31 +1,44 @@
/*
The `i18n` package is flexible and easy to use.
The `i18n` package is a flexible and easy to use way to localise applications.
It has one call to get a translation: `get`, which the user can alias into something like `T`.
It has two calls to get a translation: `get()` and `get_n()`, which the user can alias into something like `T` and `Tn`
with statements like:
T :: i18n.get
Tn :: i18n.get_n.
`get`, referred to as `T` here, has a few different signatures.
All of them will return the key if the entry can't be found in the active translation catalog.
`get()` is used for retrieving the translation of sentences which **never** change in form,
like for instance "Connection established" or "All temporary files have been deleted".
Note that the number (singular, dual, plural, whatever else) is not relevant: the sentence is fixed and it will have only one possible translation in any other language.
- `T(key)` returns the translation of `key`.
- `T(key, n)` returns a pluralized translation of `key` according to value `n`.
`get_n()` is used for retrieving the translations of sentences which change according to the number of items referenced.
The various signatures of `get_n()` have one more parameter, `n`, which will receive that number and be used
to select the correct form according to the pluralizer attached to the message catalogue when initially loaded;
for instance, to summarise a rather complex matter, some languages use the singular form when referring to 0 items and some use the (only in their case) plural forms;
also, languages may have more or less quantifier forms than a single singular form and a universal plural form:
for instance, Chinese has just one form for any quantity, while Welsh may have up to 6 different forms for specific different quantities.
- `T(section, key)` returns the translation of `key` in `section`.
- `T(section, key, n)` returns a pluralized translation of `key` in `section` according to value `n`.
Both `get()` and `get_n()`, referred to as `T` and `Tn` here, have several different signatures.
All of them will return the key if the entry can't be found in the active translation catalogue.
By default lookup take place in the global `i18n.ACTIVE` catalogue for ease of use, unless a specific catalogue is supplied.
By default lookup take place in the global `i18n.ACTIVE` catalog for ease of use.
If you want to override which translation to use, for example in a language preview dialog, you can use the following:
- `T(key)` returns the translation of `key`.
- `T(key, catalog)` returns the translation of `key` from explictly supplied catalogue.
- `T(section, key)` returns the translation of `key` in `section`.
- `T(section, key, catalog)` returns the translation of `key` in `section` from explictly supplied catalogue.
- `T(key, n, catalog)` returns the pluralized version of `key` from explictly supplied catalog.
- `T(section, key, n, catalog)` returns the pluralized version of `key` in `section` from explictly supplied catalog.
- `Tn(key, n)` returns the translation of `key` according to number of items `n`.
- `Tn(key, n, catalog)` returns the translation of `key` from explictly supplied catalogue.
- `Tn(section, key, n)` returns the translation of `key` in `section` according to number of items `n`.
- `Tn(section, key, n, catalog)` returns the translation of `key` in `section` according to number of items `n` from explictly supplied catalogue.
If a catalog has translation contexts or sections, then omitting it in the above calls looks up in section "".
The default pluralization rule is n != 1, which is to say that passing n == 1 (or not passing n) returns the singular form.
Passing n != 1 returns plural form 1.
The default pluralization rule is `n != 1`, which is to say that passing `n == 1` returns the singular form (in slot 0).
Passing `n != 1` returns the plural form in slot 1 (if any).
Should a language not conform to this rule, you can pass a pluralizer procedure to the catalog parser.
This is a procedure that maps an integer to an integer, taking a value and returning which plural slot should be used.
This is a procedure that maps an integer to an integer, taking a quantity and returning which plural slot should be used.
You can also assign it to a loaded catalog after parsing, of course.
@@ -34,24 +47,21 @@ Example:
import "core:fmt"
import "core:text/i18n"
T :: i18n.get
T :: i18n.get
Tn :: i18n.get_n
mo :: proc() {
using fmt
err: i18n.Error
/*
Parse MO file and set it as the active translation so we can omit `get`'s "catalog" parameter.
*/
// Parse MO file and set it as the active translation so we can omit `get`'s "catalog" parameter.
i18n.ACTIVE, err = i18n.parse_mo(#load("translations/nl_NL.mo"))
defer i18n.destroy()
if err != .None { return }
/*
These are in the .MO catalog.
*/
// These are in the .MO catalog.
println("-----")
println(T(""))
println("-----")
@@ -60,13 +70,11 @@ Example:
println(T("Hellope, World!"))
println("-----")
// We pass 1 into `T` to get the singular format string, then 1 again into printf.
printf(T("There is %d leaf.\n", 1), 1)
printf(Tn("There is %d leaf.\n", 1), 1)
// We pass 42 into `T` to get the plural format string, then 42 again into printf.
printf(T("There is %d leaf.\n", 42), 42)
printf(Tn("There is %d leaf.\n", 42), 42)
/*
This isn't in the translation catalog, so the key is passed back untranslated.
*/
// This isn't in the translation catalog, so the key is passed back untranslated.
println("-----")
println(T("Come visit us on Discord!"))
}
@@ -76,19 +84,13 @@ Example:
err: i18n.Error
/*
Parse QT file and set it as the active translation so we can omit `get`'s "catalog" parameter.
*/
// Parse QT file and set it as the active translation so we can omit `get`'s "catalog" parameter.
i18n.ACTIVE, err = i18n.parse_qt(#load("translations/nl_NL-qt-ts.ts"))
defer i18n.destroy()
if err != .None {
return
}
if err != .None { return }
/*
These are in the .TS catalog. As you can see they have sections.
*/
// These are in the .TS catalog. As you can see they have sections.
println("--- Page section ---")
println("Page:Text for translation =", T("Page", "Text for translation"))
println("-----")
@@ -99,8 +101,8 @@ Example:
println("-----")
println("--- apple_count section ---")
println("apple_count:%d apple(s) =")
println("\t 1 =", T("apple_count", "%d apple(s)", 1))
println("\t 42 =", T("apple_count", "%d apple(s)", 42))
println("\t 1 =", Tn("apple_count", "%d apple(s)", 1))
println("\t 42 =", Tn("apple_count", "%d apple(s)", 42))
}
*/
package i18n

188
core/text/i18n/i18n.odin
View File

@@ -10,23 +10,13 @@ package i18n
*/
import "core:strings"
/*
TODO:
- Support for more translation catalog file formats.
*/
/*
Currently active catalog.
*/
// Currently active catalog.
ACTIVE: ^Translation
// Allow between 1 and 255 plural forms. Default: 10.
MAX_PLURALS :: min(max(#config(ODIN_i18N_MAX_PLURAL_FORMS, 10), 1), 255)
/*
The main data structure. This can be generated from various different file formats, as long as we have a parser for them.
*/
// The main data structure. This can be generated from various different file formats, as long as we have a parser for them.
Section :: map[string][]string
Translation :: struct {
@@ -37,34 +27,24 @@ Translation :: struct {
}
Error :: enum {
/*
General return values.
*/
// General return values.
None = 0,
Empty_Translation_Catalog,
Duplicate_Key,
/*
Couldn't find, open or read file.
*/
// Couldn't find, open or read file.
File_Error,
/*
File too short.
*/
// File too short.
Premature_EOF,
/*
GNU Gettext *.MO file errors.
*/
// GNU Gettext *.MO file errors.
MO_File_Invalid_Signature,
MO_File_Unsupported_Version,
MO_File_Invalid,
MO_File_Incorrect_Plural_Count,
/*
Qt Linguist *.TS file errors.
*/
// Qt Linguist *.TS file errors.
TS_File_Parse_Error,
TS_File_Expected_Context,
TS_File_Expected_Context_Name,
@@ -85,73 +65,142 @@ DEFAULT_PARSE_OPTIONS :: Parse_Options{
}
/*
Several ways to use:
- get(key), which defaults to the singular form and i18n.ACTIVE catalog, or
- get(key, number), which returns the appropriate plural from the active catalog, or
- get(key, number, catalog) to grab text from a specific one.
*/
get_single_section :: proc(key: string, number := 1, catalog: ^Translation = ACTIVE) -> (value: string) {
/*
A lot of languages use singular for 1 item and plural for 0 or more than 1 items. This is our default pluralize rule.
*/
plural := 1 if number != 1 else 0
Returns the first translation string for the passed `key`.
It is also aliased with `get()`.
if catalog.pluralize != nil {
plural = catalog.pluralize(number)
}
return get_by_slot(key, plural, catalog)
Two ways to use it:
- get(key), which defaults to the `i18n.ACTIVE` catalogue, or
- get(key, catalog) to grab text from a specific loaded catalogue
Inputs:
- key: the string to translate
- catalog: the catalogue to use for the translation (defaults to i18n.ACTIVE)
Returns: the translated string, or the original `key` if no translation was found.
*/
get_single_section :: proc(key: string, catalog: ^Translation = ACTIVE) -> (value: string) {
return get_by_slot(key, 0, catalog)
}
/*
Several ways to use:
- get(section, key), which defaults to the singular form and i18n.ACTIVE catalog, or
- get(section, key, number), which returns the appropriate plural from the active catalog, or
- get(section, key, number, catalog) to grab text from a specific one.
*/
get_by_section :: proc(section, key: string, number := 1, catalog: ^Translation = ACTIVE) -> (value: string) {
/*
A lot of languages use singular for 1 item and plural for 0 or more than 1 items. This is our default pluralize rule.
*/
plural := 1 if number != 1 else 0
Returns the first translation string for the passed `key` in a specific section or context.
It is also aliases with `get()`.
if catalog.pluralize != nil {
plural = catalog.pluralize(number)
}
return get_by_slot(section, key, plural, catalog)
Two ways to use it:
- get(section, key), which defaults to the `i18n.ACTIVE` catalogue, or
- get(section, key, catalog) to grab text from a specific loaded catalogue
Inputs:
- section: the catalogue section (sometimes also called 'context') in which to look up the translation
- key: the string to translate
- catalog: the catalogue to use for the translation (defaults to i18n.ACTIVE)
Returns: the translated string, or the original `key` if no translation was found.
*/
get_by_section :: proc(section, key: string, catalog: ^Translation = ACTIVE) -> (value: string) {
return get_by_slot(section, key, 0, catalog)
}
get :: proc{get_single_section, get_by_section}
/*
Several ways to use:
- get_by_slot(key), which defaults to the singular form and i18n.ACTIVE catalog, or
- get_by_slot(key, slot), which returns the requested plural from the active catalog, or
- get_by_slot(key, slot, catalog) to grab text from a specific one.
Returns the translation string for the passed `key` in a specific plural form (if present in the catalogue).
It is also aliased with `get_n()`.
Two ways to use it:
- get_n(key, quantity), which returns the appropriate plural from the active catalogue, or
- get_n(key, quantity, catalog) to grab text from a specific loaded catalogue
Inputs:
- key: the string to translate
- quantity: the quantity of item to be used to select the correct plural form
- catalog: the catalogue to use for the translation (defaults to i18n.ACTIVE)
Returns: the translated string, or the original `key` if no translation was found.
*/
get_single_section_with_quantity :: proc(key: string, quantity: int, catalog: ^Translation = ACTIVE) -> (value: string) {
/*
A lot of languages use singular for 1 item and plural for 0 or more than 1 items. This is our default pluralize rule.
*/
slot := 1 if quantity != 1 else 0
if catalog.pluralize != nil {
slot = catalog.pluralize(quantity)
}
return get_by_slot(key, slot, catalog)
}
/*
Returns the translation string for the passed `key` in a specific plural form (if present in the catalogue)
in a specific section or context.
It is also aliases with `get_n()`.
Two ways to use it:
- get(section, key, quantity), which returns the appropriate plural from the active catalogue, or
- get(section, key, quantity, catalog) to grab text from a specific loaded catalogue
Inputs:
- section: the catalogue section (sometime also called 'context') from which to lookup the translation
- key: the string to translate
- qantity: the quantity of item to be used to select the correct plural form
- catalog: the catalogue to use for the translation (defaults to i18n.ACTIVE)
Returns: the translated string, or the original `key` if no translation was found
*/
get_by_section_with_quantity :: proc(section, key: string, quantity: int, catalog: ^Translation = ACTIVE) -> (value: string) {
/*
A lot of languages use singular for 1 item and plural for 0 or more than 1 items. This is our default pluralize rule.
*/
slot := 1 if quantity != 1 else 0
if catalog.pluralize != nil {
slot = catalog.pluralize(quantity)
}
return get_by_slot(section, key, slot, catalog)
}
get_n :: proc{get_single_section_with_quantity, get_by_section_with_quantity}
/*
Two ways to use:
- get_by_slot(key, slot), which returns the requested plural from the active catalogue, or
- get_by_slot(key, slot, catalog) to grab text from a specific loaded catalogue.
If a file format parser doesn't (yet) support plural slots, each of the slots will point at the same string.
- section: the catalogue section (sometime also called 'context') from which to lookup the translation
Inputs:
- key: the string to translate.
- slot: the translation slot to choose (slots refer to plural forms specific for each language and their meaning changes from catalogue to catalogue).
- catalog: the catalogue to use for the translation (defaults to i18n.ACTIVE)
Returns: the translated string, or the original `key` if no translation was found.
*/
get_by_slot_single_section :: proc(key: string, slot := 0, catalog: ^Translation = ACTIVE) -> (value: string) {
get_by_slot_single_section :: proc(key: string, slot: int, catalog: ^Translation = ACTIVE) -> (value: string) {
return get_by_slot_by_section("", key, slot, catalog)
}
/*
Several ways to use:
- get_by_slot(key), which defaults to the singular form and i18n.ACTIVE catalog, or
Two ways to use:
- get_by_slot(key, slot), which returns the requested plural from the active catalog, or
- get_by_slot(key, slot, catalog) to grab text from a specific one.
If a file format parser doesn't (yet) support plural slots, each of the slots will point at the same string.
Inputs:
- section: the catalogue section (sometime also called 'context') from which to lookup the translation
- key: the string to translate.
- slot: the translation slot to choose (slots refer to plural forms specific for each language and their meaning changes from catalogue to catalogue).
- catalog: the catalogue to use for the translation (defaults to i18n.ACTIVE)
Returns: the translated string or the original `key` if no translation was found.
*/
get_by_slot_by_section :: proc(section, key: string, slot := 0, catalog: ^Translation = ACTIVE) -> (value: string) {
get_by_slot_by_section :: proc(section, key: string, slot: int, catalog: ^Translation = ACTIVE) -> (value: string) {
if catalog == nil || section not_in catalog.k_v {
/*
Return the key if the catalog catalog hasn't been initialized yet, or the section is not present.
*/
// Return the key if the catalog catalog hasn't been initialized yet, or the section is not present.
return key
}
/*
Return the translation from the requested slot if this key is known, else return the key.
*/
// Return the translation from the requested slot if this key is known, else return the key.
if translations, ok := catalog.k_v[section][key]; ok {
plural := min(max(0, slot), len(catalog.k_v[section][key]) - 1)
return translations[plural]
@@ -161,7 +210,6 @@ get_by_slot_by_section :: proc(section, key: string, slot := 0, catalog: ^Transl
get_by_slot :: proc{get_by_slot_single_section, get_by_slot_by_section}
/*
Same for destroy:
- destroy(), to clean up the currently active catalog catalog i18n.ACTIVE
- destroy(catalog), to clean up a specific catalog.
*/

40
tests/core/text/i18n/test_core_text_i18n.odin
View File

@@ -5,6 +5,7 @@ import "core:testing"
import "core:text/i18n"
T :: i18n.get
Tn :: i18n.get_n
Test :: struct {
section: string,
@@ -47,7 +48,8 @@ test_custom_pluralizer :: proc(t: ^testing.T) {
{"", "Message1/plural", "This is message 1", 1},
{"", "Message1/plural", "This is message 1 - plural A", 1_000_000},
{"", "Message1/plural", "This is message 1 - plural B", 42},
// This isn't in the catalog, so should ruturn the key.
// This isn't in the catalog, so should return the key.
{"", "Come visit us on Discord!", "Come visit us on Discord!", 1},
},
})
@@ -61,11 +63,11 @@ test_mixed_context :: proc(t: ^testing.T) {
plural = nil,
tests = {
// These are in the catalog.
{"", "Message1", "This is message 1 without Context", 1},
{"Context", "Message1", "This is message 1 with Context", 1},
{"", "Message1", "This is message 1 without Context",-1},
{"Context", "Message1", "This is message 1 with Context", -1},
// This isn't in the catalog, so should ruturn the key.
{"", "Come visit us on Discord!", "Come visit us on Discord!", 1},
{"", "Come visit us on Discord!", "Come visit us on Discord!", -1},
},
})
}
@@ -90,15 +92,15 @@ test_nl_mo :: proc(t: ^testing.T) {
plural = nil, // Default pluralizer
tests = {
// These are in the catalog.
{"", "There are 69,105 leaves here.", "Er zijn hier 69.105 bladeren.", 1},
{"", "Hellope, World!", "Hallo, Wereld!", 1},
{"", "There are 69,105 leaves here.", "Er zijn hier 69.105 bladeren.", -1},
{"", "Hellope, World!", "Hallo, Wereld!", -1},
{"", "There is %d leaf.\n", "Er is %d blad.\n", 1},
{"", "There are %d leaves.\n", "Er is %d blad.\n", 1},
{"", "There is %d leaf.\n", "Er zijn %d bladeren.\n", 42},
{"", "There are %d leaves.\n", "Er zijn %d bladeren.\n", 42},
// This isn't in the catalog, so should ruturn the key.
{"", "Come visit us on Discord!", "Come visit us on Discord!", 1},
{"", "Come visit us on Discord!", "Come visit us on Discord!", -1},
},
})
}
@@ -111,15 +113,15 @@ test_qt_linguist :: proc(t: ^testing.T) {
plural = nil, // Default pluralizer
tests = {
// These are in the catalog.
{"Page", "Text for translation", "Tekst om te vertalen", 1},
{"Page", "Also text to translate", "Ook tekst om te vertalen", 1},
{"installscript", "99 bottles of beer on the wall", "99 flessen bier op de muur", 1},
{"Page", "Text for translation", "Tekst om te vertalen", -1},
{"Page", "Also text to translate", "Ook tekst om te vertalen", -1},
{"installscript", "99 bottles of beer on the wall", "99 flessen bier op de muur", -1},
{"apple_count", "%d apple(s)", "%d appel", 1},
{"apple_count", "%d apple(s)", "%d appels", 42},
// These aren't in the catalog, so should ruturn the key.
{"", "Come visit us on Discord!", "Come visit us on Discord!", 1},
{"Fake_Section", "Come visit us on Discord!", "Come visit us on Discord!", 1},
{"", "Come visit us on Discord!", "Come visit us on Discord!", -1},
{"Fake_Section", "Come visit us on Discord!", "Come visit us on Discord!", -1},
},
})
}
@@ -133,16 +135,16 @@ test_qt_linguist_merge_sections :: proc(t: ^testing.T) {
options = {merge_sections = true},
tests = {
// All of them are now in section "", lookup with original section should return the key.
{"", "Text for translation", "Tekst om te vertalen", 1},
{"", "Also text to translate", "Ook tekst om te vertalen", 1},
{"", "99 bottles of beer on the wall", "99 flessen bier op de muur", 1},
{"", "Text for translation", "Tekst om te vertalen", -1},
{"", "Also text to translate", "Ook tekst om te vertalen", -1},
{"", "99 bottles of beer on the wall", "99 flessen bier op de muur", -1},
{"", "%d apple(s)", "%d appel", 1},
{"", "%d apple(s)", "%d appels", 42},
// All of them are now in section "", lookup with original section should return the key.
{"Page", "Text for translation", "Text for translation", 1},
{"Page", "Also text to translate", "Also text to translate", 1},
{"installscript", "99 bottles of beer on the wall", "99 bottles of beer on the wall", 1},
{"Page", "Text for translation", "Text for translation", -1},
{"Page", "Also text to translate", "Also text to translate", -1},
{"installscript", "99 bottles of beer on the wall", "99 bottles of beer on the wall", -1},
{"apple_count", "%d apple(s)", "%d apple(s)", 1},
{"apple_count", "%d apple(s)", "%d apple(s)", 42},
},
@@ -175,7 +177,7 @@ test :: proc(t: ^testing.T, suite: Test_Suite, loc := #caller_location) {
if err == .None {
for test in suite.tests {
val := T(test.section, test.key, test.n, cat)
val := test.n > -1 ? Tn(test.section, test.key, test.n, cat): T(test.section, test.key, cat)
testing.expectf(t, val == test.val, "Expected key `%v` from section `%v`'s form for value `%v` to equal `%v`, got `%v`", test.key, test.section, test.n, test.val, val, loc=loc)
}
}