Semiconductor & Computer Engineering
This page does not attempt to describe what is Unicode but rather how it works around mIRC. The technical terms are omitted on purpose.
Before Unicode (mIRC 7.x), mIRC supported code pages, handling various language used in the world, code pages are encoding on 8 bits (byte), 8 bits can be used to represent 256 values, mIRC also supported others encoding for Japanese system for example. Basically, code pages are all based on ASCII, which defines 128 characters, assigned to the first 128 values represented in 8 bits, and then each code page adds the required characters for the language, é for the french or a Greek letter for Greek people.
You can see Unicode as a new codepage, but which defines 1,114,112 characters, including all languages. This is much better for IRC, which can be used from all over the world.
They are different ways to implement the handling of Unicode in a application, mIRC uses the utf16 encoding internally, before it was handled using US-ASCII.
Two mains reasons for this:
- The most frequently used character, the first 65635 ones, can be stored with 16 bits, as a result, routine dealing with Unicode are going to be faster.
- It uses less memory than utf32.
Drawback: each character in a script is depicted as a 16 bits unit, $asc and $chr cannot be used with characters over 65635, but you can form those others characters in your script by combining two 16 bits characters together:
//var -s %a = $chr(55384), %b = $chr(56320) | echo -a %a $+ %b 𦀀
You can also express those characters with their utf8 representation using $utfdecode, which decode utf8:
;you can use utf8 to form the character 65536 for example, which is four bytes in utf8 (f0 90 80 80): //var -s %a $utfdecode($chr($base(f0,16,10)) $+ $chr($base(90,16,10)) $+ $chr($base(80,16,10)) $+ $chr($base(80,16,10))) 𐀀
$utfencode can be used to encode text to utf8:
//var -s %a $utfencode(é) é
The scripting language still somewhat support code pages, you can decode text to utf8 while the bytes in the text are interpreted in the given code page.
Each code page has a number (Gdi charset):
- 000 - ANSI_CHARSET
- 001 - DEFAULT_CHARSET
- 002 - SYMBOL_CHARSET
- 077 - MAC_CHARSET
- 128 - SHIFTJIS_CHARSET
- 129 - HANGEUL_CHARSET
- 130 - JOHAB_CHARSET
- 134 - GB2312_CHARSET
- 136 - CHINESEBIG5_CHARSET
- 161 - GREEK_CHARSET
- 162 - TURKISH_CHARSET
- 163 - VIETNAMESE_CHARSET
- 177 - HEBREW_CHARSET
- 178 - ARABIC_CHARSET
- 186 - BALTIC_CHARSET
- 204 - RUSSIAN_CHARSET
- 222 - THAI_CHARSET
- 238 - EASTEUROPE_CHARSET
- 255 - OEM_CHARSET
Note: GDI charsets 1 and 255 are system dependent and are therefore expected to return different results across different machines. Values not on the table are treated as a reference to DEFAULT_CHARSET, equivalent to using C = 1.
For example, if you want to get the text (FROM GREEK TO UTF8), which used the ISO-8859-7 (GREEK) encoding for greek letters, in utf8, you need to encode that to utf8, interpreting the bytes as per in the GREEK code page, and then to decode that to utf8: $utfdecode($utfencode(text,161))
If you want to send the text in GREEK over IRC, mIRC will encode the bytes internally so you must encode the text in utf8, and then decode to utf8, interpreting the bytes as per in the GREEK code page: /raw -n privmsg #chan $utfdecode($utfencode(text),161)
It must be noted that some commands and identifiers will encode your text to utf8, changing the integrity of your input. Each character is represented as 16 bits internally, the array of character is not an array of byte, it's an array of 16 bit (two bytes). But let's take a look at "/msg #chan é". é is the code point 233, which fits in one byte, but mIRC will encode your byte 233 to utf8, giving the two bytes 165 169. Now in this case there is little value not encoding in utf8, on IRC you usually don't really care about the integrity of your bytes. However there are others features where it's not so simple. $sha1(é) for example can be tricky, this is going to give the hash of the two bytes. The reason this is happening is because of the conversion, the hashing algorithm works on an array of byte, so mIRC has to take your character (represented in an array of two bytes) and convert them to single bytes, that's why the utf8 conversion happens. And this is happening pretty much everywhere.
/raw -n can be used for IRC, it sends the data to the server without encode the characters in the range 0-255 to utf8.
/sockwrite -u can be used to the same effect, won't encode characters in the range 0-255 to utf8.