Unicode and UTF-8

Code points, variable-length encoding, surrogate pairs, grapheme clusters — and why string length lies

1 / 24

steps

Act 1 of 8·Bytes Are Not Text

step 1

a file holds bytes — nothing else

decoded as UTF-8caféC3 A9 is one code point

decoded as Windows-1252cafÃ©reads C3 and A9 as two separate characters

decoded as ISO-8859-5cafУЉsame bytes, Cyrillic table

There is no way to look at bytes and know their encoding — it is metadata that travels separately, in a Content-Type header, an XML declaration, or an assumption. Get it wrong and you get mojibake. This is why "just store text" was never simple, and why UTF-8 everywhere was such a relief.

Speednormal

A file, a socket, a database column — none of them hold text. They hold bytes. Text only appears when something decides how to interpret those bytes, and that decision lives outside the bytes themselves.

The layers

Bytesmeaningless alone

Code pointU+0000–U+10FFFF

UTF-81–4 bytes · prefixed

UTF-162 or 4 · surrogates

Graphemewhat humans count

ZWJU+200D glues emoji

NFC / NFDnormalize to compare

Active Recall

Question 1 / 3

What makes UTF-8 "self-synchronizing"?

Under the Hood

Loading visualizer…

DNSresolving host

TCPconnecting

TLShandshake

HTTPrequesting

DSAverse

Unicode and UTF-8

Unicode and UTF-8