CharBusters

10 Unicode myths

������ ����, @���

CharBusters

10 Unicode myths

Tomasz Nurkiewicz, @tnurkiewicz

काचं शक्नोम्यत्तुम् । नोपहिनस्ति माम् ॥
Μπορῶ νὰ φάω σπασμένα γυαλιὰ χωρὶς νὰ πάθω τίποτα 🇬🇷
ᛁᚳ᛫ᛗᚨᚷ᛫ᚷᛚᚨᛋ᛫ᛖᚩᛏᚪᚾ᛫ᚩᚾᛞ᛫ᚻᛁᛏ᛫ᚾᛖ᛫ᚻᛖᚪᚱᛗᛁᚪᚧ᛫ᛗᛖ᛬
⠊⠀⠉⠁⠝⠀⠑⠁⠞⠀⠛⠇⠁⠎⠎⠀⠁⠝⠙⠀⠊⠞⠀⠙⠕⠑⠎⠝⠞⠀⠓⠥⠗⠞⠀⠍⠑
Я можу їсти скло, і воно мені не зашкодить. 🇺🇦
მინას ვჭამ და არა მტკივა. 🇬🇪
Կրնամ ապակի ուտել և ինծի անհանգիստ չըներ։ 🇦🇲
నేను గాజు తినగలను మరియు అలా చేసినా నాకు ఏమి ఇబ్బంది లేదు 🇮🇳
איך קען עסן גלאָז און עס טוט מיר נישט װײ
ᠪᠢ ᠰᠢᠯᠢ ᠢᠳᠡᠶᠦ ᠴᠢᠳᠠᠨᠠ ᠂ ᠨᠠᠳᠤᠷ ᠬᠣᠤᠷᠠᠳᠠᠢ ᠪᠢᠰᠢ 🇲🇳
我能吞下玻璃而不伤身体。🇨🇳
Jag kan äta glas utan att skada mig 🇸🇪

I don’t need to worry about Unicode

Myth 0

History of IT

(or how we estimate badly)

Parkinson's Law

Work expands so as to fill the time available for its completion

Hofstadter's Law

It always takes longer than you expect, even when you take into account Hofstadter's Law

IPv4

1978

640 KiB

Bill Gates IBM, 1981

Y2K38

512K problem

GPS 2019

ASCII

1963

1 character = 1 byte

Myth 1

Let’s talk about "ą"

ą

Content-type: text/html; charset=utf-8

🇵🇱

🇨🇿

🇫🇷

🇷🇺

🇩🇪 🇬🇷 🇪🇸

Unicode 1.0

modern [characters], whose number is undoubtedly far below 2¹⁴ = 16 384

1988

Unicode code points

a = U+0061 =    97
å = U+00E5 =   229
ą = U+0105 =   261
鑫= U+946B = 37995

1 character = 1 char

Myth 2

[...] undoubtedly far below 2¹⁴ = 16 384

1988

🇨🇳

讓我來! 让我来!

CJK

88 thousand characters in Unicode 12.0

𝄞

U+1D11E (119 070)

Unicode 2.0

1996

Correct Java type for one character is...?

1. byte
2. char
3. int
4. String

String

						codePointAt(int)         : int
						codePoints()             : IntStream
						codePointCount(int, int) : int
					

Unicode

vs

UTF-*

a

U+0061

ą

U+0105

鑫

U+946B

𝄞

U+1D11E

🤔

"𝄞".length() == 2

"(𝄞)".substring(0, 2)

(?

Surrogate pairs

var life = "🏭" + "🏖";

				        StringBuilder rev = new StringBuilder();
				        for (int i = life.length() - 1; i >= 0; i--)
				            rev.append(life.charAt(i));
					

var life = "🏭🏖";

"?🏭?"

new StringBuilder("🏭🏖")
          .reverse()
          .toString()
					

Java is UTF-16

Myth 3

Java 8

    					private final char value[];
    				

Java 9+

						private final byte[] value;
					

Java 9+

						public int indexOf(int ch, int fromIndex) {
						    return isLatin1() 
						      ? StringLatin1.indexOf(value, ch, fromIndex) 
						      : StringUTF16.indexOf(value, ch, fromIndex);
						}
					

String.getBytes()

Unicode is unambiguous

Myth 4

"ą".equals("ą")

Tiny tail

Normalizer.normalize("ą", Form.NFKC)
					

1 character ≤ 1 int

Myth 5

Let’s talk about emoji

🇵🇱

🇵+🇱

🇵🇱

👧🏽

1 character ≤ 2 ints

Myth 6

👩🏾‍🚀

👨‍👩‍👧‍👦

String.length() is useful

Myth 7

public int length()

Returns the length [...] equal to the number of Unicode code units in the string.

Code unit 😱

The minimal bit combination that can represent a unit of encoded text.

"a".length()	  // 1
"ą".length()	  // 1
"ą".length()	  // 2
"𝄞".length()	  // 2
					

"👰".length()    // 2
"🇵🇱".length()    // 4
"👩🏾‍🚀".length()    // 7
"👨‍👩‍👧‍👦".length() 	// 11
					

"T̢̗̮͉͈̠̣͆͆̎͐̌͒͢ȍ̵͑̾͒͂͛̄̔͢҉̡̦͙͎̱̹͍͎͖̪̮̙̪͔̺͕̞̰̤̯m̍ͩ̓͋ͫ̑҉̵̷͓̦̩̭̗̩̫̺e̵̦̫̭̫̬͉̞̪̹̓̆̈́͊̂̃̀͡ǩ̸̴̢̛̫̦̬̪̘̱̖̼̺͕͇͕̞͓̮̭̯ͣ̌̂̏ͨͤͬ͛̏̋̉̀".length()

119

import java.text.BreakIterator;

int graphemes(String str) {
   BreakIterator graphemeCounter = BreakIterator
           .getCharacterInstance(Locale.US);
   graphemeCounter.setText(str);
   int graphemeCount = 0;
   while (graphemeCounter.next() != BreakIterator.DONE) {
       graphemeCount++;
   }
   return graphemeCount;
}
					

What is character?

• Code point
• Code unit
• Grapheme cluster
• Glyph

UTF

UTF is an [...] mapping from every Unicode code point [...] to a unique byte sequence

Whitespace is straightforward

Myth 8

How many different types of whitespaces there are?

All of them:

Space, tab, enter...

						IntStream.rangeClosed(0, 0x10FFFF)
						         .filter(Character::isDefined)
						         .count();
					

						IntStream.rangeClosed(0, 0x10FFFF)
						         .filter(Character::isWhitespace)
						         .count();
					

Character.isWhitespace()

25 characters

String.trim()

32 characters

Pattern.compile("\\s")

6 characters

Upper case is simple

Myth 9

🇹🇷

🇹+🇷

"i".toUpperCase(tr_TR)

i → İ

• jira.atlassian.com/browse/CONFSERVER-7168 (Confluence)
• blogs.msdn.microsoft.com/anutthara/2005/12/05/avoiding-the-turkish-i-issue/ (.NET)
• bz.apache.org/bugzilla/show_bug.cgi?id=38787 (BCEL)
• bugzilla.redhat.com/show_bug.cgi?id=1408950 (Fedora)

• JDK-6220064 - REGRESSION: SSL connections fail with Turkish input locale
• JDK-6240755 - Swing rendering error for Turkish locale on XP look and feel
• JDK-6240963 - XSLT transforms broken in Turkish locale in JDK 1.5.0+
• JDK-6341798 - XMLDecoder fails when using Turkish Locale
• JDK-8023943 - Method description fix for String.toLower/UpperCase() methods
• JDK-4624534 - JarEntry.getCertificates() returns null on Turkish locale (tr_TR)
• JDK-6867345 - Turkish regional options cause NPE in sun.security.x509.AlgorithmId.algOID
• JDK-6972386 - issues with String.toLowerCase/toUpperCase
• JDK-6353473 - REGRESSION: NameGenerator.captialize doesn't work properly with Turkish locale

Unicode is harmless

Myth 10

effective. Power لُلُصّبُلُلصّبُررً ॣ ॣh ॣ ॣ 冗

జ్ఞా

<⚫️>👈🏻

‏‎

sıkışınca ≠ sikişince

Zaten sen sıkışınca konuyu değiştiriyorsun.

Zaten sen sikişınce konuyu değiştiriyorsun.

sıkışınca ≠ sikişince

Anyhow, whenever you can't answer an argument, you change the subject.

Anyhow, whenever they are f***ing you, you change the subject.

Romanization

Zażółć gęślą jaźń

👇

Zazolc gesla jazn

Pangram

The quick brown fox jumps over the lazy dog

Jeżu klątw, spłódź Finom część gry hańb

Conclusions

Ińtërnâtiônàlizætión☃⛄️

Which encoding is the best?

References