all Python 3unicodedatafunction

unicodedata.normalize()

normalization

Brings a string to one of the Unicode normal forms (NFC, NFD, NFKC, NFKD) so that strings that look the same become equal.

Syntax

unicodedata.normalize(form, unistr, /)

'й' can be one code point (U+0439, the NFC form) or two, 'и' plus a combining breve (the NFD form): that is how text arrives from macOS, file names and PDF copy-paste. The strings look identical, yet == is False and len() gives 1 and 2. Normalize to NFC before comparing, searching or storing; NFKC/NFKD also change meaning ('ﬁ²' becomes 'fi2') and suit search, not storage.

Examples

import unicodedata
a = 'й'  # одна кодовая точка U+0439
b = 'и\u0306'  # и + комбинируемое бреве
print(a == b, len(a), len(b))  # → False 1 2
print(unicodedata.normalize('NFC', b) == a)  # → True
print(len(unicodedata.normalize('NFD', a)))  # → 2

import unicodedata
print(unicodedata.normalize('NFKC', 'ﬁ²'))  # → fi2 — совместимая форма меняет смысл
print(unicodedata.normalize('NFC', 'ﬁ²'))  # → ﬁ²

unicodedata.normalize()

Syntax

Examples

See also