What is UTF-8?#

  • UTF is an encoding standard to display raw things from computer to human.
  • Firstly, when u store anything on ur computer via disc/ram/anything in terms of text or video or image etc, all these will be stored in binary(1s or 0s) under the hood.
  • Each of these binaries are tagged with each letter, for EG:
"H" - 72(in decimal) 1001000(in binary)
"E" - 69(in decimal) 1000101(in binary)
etcc
  • Like the above, the entire letters(english, japanese, tamil etc), symbols, everything has been mapped to some unique number, this is called Unicode(initially it was ascii and then extended ascii).
  • This Unicode is created, primarily so that humans can view raw text, instead of plain binary, instead of showing the binary values to the user, there are some mapped ids to each letters and then those letters will be showed to the user(notepad, vscode, browser etccc).
  • All this Unicode, ASCII are just a representation of characters, emojis etc.
  • To represent this Unicode, there should be a common format right, thats where utf-8, utf-16 comes into play.

NOTE: Initially there was only ASCII, where it represents only english alphabets and symbols with total 128 letters, and extended ASCII with 256 letters, later we included every language and every language symbol and placed into a common thing called Unicode.

UTF representation#

Initially when ASCII was the only standard, UTF was represented with 1byte(8bits), but mostly 7bits are used.

1Byte

Once Unicode has been introduced, there are some characters or emoji that took more than 8bits, in those scenarios. This comes into complex representation for eg: character A is +0041, this could easily fit into 1byte, emoji 😀 is +1F600, this could not fit into 1byte. considering the above, you should represent unicode in computers dynamically, A should get only 1byte, and 😀 should get 4byte, also the decoder should able to determine these bytes wrt to its associated characters. This is where utf-8 introduced, it dynamically structures how much bytes a character is, so that if any text decoder is in middle of any byte, it can able to determine how much is the length of the byte and how long it should traverse and decode in order to decode that character.

0xxxxxxx - this means the character is of single byte. 110xxxxx 10xxxxxx - this means the character is of two bytes, first byte first two chars(11) represent the total length of the bytes followed by 0 stating its the end of the length representation, second byte you would see (10) this means its the continuation byte, for eg: lets say ur text editor or some program lands in the middle of any byte from the memory and it got this 10xxxxxx, now the decoder that follows utf-8 clearly knows ok its a continuation byte i need to traverse back. 1110xxxx 10xxxxxx 10xxxxxx - this is just character with 3bytes, same as above. 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx - this is just character with 3bytes, same as above. if u see here the maximum number of characters that utf8 supports 21bits, which is 2^21.

EG

Lets take this emoji example: 😀 -> U+1F600(unicode id)
decimal -> 128512
binary -> 0001 1111 0110 0000 0000

if letter is 2bytes it would represent
110xxxxx 10xxxxxx

for our above example 128512 would fit in 4bytes

binary -> 0001 1111 0110 0000 0000

it will be filled from right to left
from right most to left 1st6bits(000000) would fit in the right most below
from right most to left 2st6bits(011000) would fit in the right most below
from right most to left 3rd6bits(011111) would fit in the right most below

11110000 10011111 10011000 10000000