From WikiChip
Difference between revisions of "mirc/unicode"
< mirc

m
Line 13: Line 13:
 
* It uses less memory than utf32.
 
* It uses less memory than utf32.
  
−
Drawback: each character in a script is depicted as a 16 bits unit, {{mIRC|$asc}} and {{mIRC|$chr}} cannot be used with characters over 65635, but you can form those others characters in your script by combining two 16 bits characters together:
+
Drawback: each character in a script is depicted as a 16 bits unit, {{mIRC|$asc}} and {{mIRC|$chr}} cannot be used with characters over 65535.
 +
 
 +
Characters over 65535 are encoded in utf16 as two 16 bits unit called surrogates but mIRC does not *support* this.
  
 
<source lang="mIRc">
 
<source lang="mIRc">
−
//var -s %a = $chr(55384), %b = $chr(56320) | echo -a %a $+ %b
+
//var -s %a = $chr(55384), %b = $chr(56320) | echo -a %a $+ %b -- $len(%a $+ %b)
  
 
  𦀀
 
  𦀀
 +
</source>What happens here is that mIRC pass the two surrogate to /echo, and /echo will call a function to display the text, this function is a Windows' function and it is this function that will 'decode' from utf16.
 +
As you may have noted, $len() is 2, showing that mirc sees two characters.
 +
It may be safe to assume that all rendering functions will display the character however note that some usages of some functions in mIRC, such as $regsubex, won't handle the surrogate as expected.
 +
<source lang="mIRc">
 +
//echo -ag $regsubex(aa,/(a)/gu,$chr($gettok(55357 56607,\n,32))) vs $replace(ab,a,$chr(55357),b,$chr(56607))
 
</source>
 
</source>
 +
 +
For $regsubex this is because mIRC is using the 8 bits pcre api: when replacing it has to convert your 16 bits unit ($chr(55357) in the first iteration) to utf8.
 +
 +
In fact mIRC does convert the two surrogates to their utf8 representation, except that this is kind of illegal.
 +
 +
Surrogates are not really characters, they are just code points used to form others characters, therefore it's illegal to decode thoses bytes to that character.
 +
 +
We say illegal because the general algorithm for utf8 can still be used, and is used by mIRC here, to encode the lone surrogate.
 +
 +
So basically, when mIRC decodes what $regsubex generated, to utf8, it recognizes the illegal sequence and simply return the characters corresponding to the byte, so you get 3 characters per surrogate.
  
 
You can also express those characters with their utf8 representation using {{mIRC|$utfdecode}}, which decode utf8:
 
You can also express those characters with their utf8 representation using {{mIRC|$utfdecode}}, which decode utf8:
Line 33: Line 50:
  
 
<source lang="mIRc">
 
<source lang="mIRc">
−
//var -s %a $utfencode(é)
+
//var -s %a $utfencode(è)
  
−
é
+
è
 
</source>
 
</source>
  

Revision as of 11:21, 23 June 2018

Commands & Identifiers
Basics
Events
Matching Tools
Data Storage
Control Structures
GUI Scripting
Sockets
Advanced Scripting
Additional Resources
Security
Other
v · d · e

This page does not attempt to describe what is Unicode but rather how it works around mIRC. The technical terms are omitted on purpose.

Before Unicode (mIRC 7.x), mIRC supported code pages, handling various language used in the world, code pages are encoding on 8 bits (byte), 8 bits can be used to represent 256 values, mIRC also supported others encoding for Japanese system for example. Basically, code pages are all based on ASCII, which defines 128 characters, assigned to the first 128 values represented in 8 bits, and then each code page adds the required characters for the language, é for the french or a Greek letter for Greek people.

You can see Unicode as a new codepage, but which defines 1,114,112 characters, including all languages. This is much better for IRC, which can be used from all over the world.

They are different ways to implement the handling of Unicode in a application, mIRC uses the utf16 encoding internally, before it was handled using US-ASCII.

Two mains reasons for this:

  • The most frequently used character, the first 65635 ones, can be stored with 16 bits, as a result, routine dealing with Unicode are going to be faster.
  • It uses less memory than utf32.

Drawback: each character in a script is depicted as a 16 bits unit, $asc and $chr cannot be used with characters over 65535.

Characters over 65535 are encoded in utf16 as two 16 bits unit called surrogates but mIRC does not *support* this. 

//var -s %a = $chr(55384), %b = $chr(56320) | echo -a %a $+ %b -- $len(%a $+ %b)
 
 𦀀
What happens here is that mIRC pass the two surrogate to /echo, and /echo will call a function to display the text, this function is a Windows' function and it is this function that will 'decode' from utf16.

As you may have noted, $len() is 2, showing that mirc sees two characters. It may be safe to assume that all rendering functions will display the character however note that some usages of some functions in mIRC, such as $regsubex, won't handle the surrogate as expected. 

//echo -ag $regsubex(aa,/(a)/gu,$chr($gettok(55357 56607,\n,32))) vs $replace(ab,a,$chr(55357),b,$chr(56607))

For $regsubex this is because mIRC is using the 8 bits pcre api: when replacing it has to convert your 16 bits unit ($chr(55357) in the first iteration) to utf8.

In fact mIRC does convert the two surrogates to their utf8 representation, except that this is kind of illegal.

Surrogates are not really characters, they are just code points used to form others characters, therefore it's illegal to decode thoses bytes to that character.

We say illegal because the general algorithm for utf8 can still be used, and is used by mIRC here, to encode the lone surrogate.

So basically, when mIRC decodes what $regsubex generated, to utf8, it recognizes the illegal sequence and simply return the characters corresponding to the byte, so you get 3 characters per surrogate.

You can also express those characters with their utf8 representation using $utfdecode, which decode utf8: 

;you can use utf8 to form the character 65536 for example, which is four bytes in utf8 (f0 90 80 80):
//var -s %a $utfdecode($chr($base(f0,16,10)) $+ $chr($base(90,16,10)) $+ $chr($base(80,16,10)) $+ $chr($base(80,16,10)))
 
𐀀

$utfencode can be used to encode text to utf8: 

//var -s %a $utfencode(è)
 
è

The scripting language still somewhat support code pages, you can decode text to utf8 while the bytes in the text are interpreted in the given code page.

Each code page has a number (Gdi charset):

  • 000 - ANSI_CHARSET
  • 001 - DEFAULT_CHARSET
  • 002 - SYMBOL_CHARSET
  • 077 - MAC_CHARSET
  • 128 - SHIFTJIS_CHARSET
  • 129 - HANGEUL_CHARSET
  • 130 - JOHAB_CHARSET
  • 134 - GB2312_CHARSET
  • 136 - CHINESEBIG5_CHARSET
  • 161 - GREEK_CHARSET
  • 162 - TURKISH_CHARSET
  • 163 - VIETNAMESE_CHARSET
  • 177 - HEBREW_CHARSET
  • 178 - ARABIC_CHARSET
  • 186 - BALTIC_CHARSET
  • 204 - RUSSIAN_CHARSET
  • 222 - THAI_CHARSET
  • 238 - EASTEUROPE_CHARSET
  • 255 - OEM_CHARSET

Note: GDI charsets 1 and 255 are system dependent and are therefore expected to return different results across different machines. Values not on the table are treated as a reference to DEFAULT_CHARSET, equivalent to using C = 1.

For example, if you want to get the text (FROM GREEK TO UTF8), which used the ISO-8859-7 (GREEK) encoding for greek letters, in utf8, you need to encode that to utf8, interpreting the bytes as per in the GREEK code page, and then to decode that to utf8: $utfdecode($utfencode(text,161))

If you want to send the text in GREEK over IRC, mIRC will encode the bytes internally so you must encode the text in utf8, and then decode to utf8, interpreting the bytes as per in the GREEK code page: /raw -n privmsg #chan $utfdecode($utfencode(text),161)

It must be noted that some commands and identifiers will encode your text to utf8, changing the integrity of your input. Each character is represented as 16 bits internally, the array of character is not an array of byte, it's an array of 16 bit (two bytes). But let's take a look at "/msg #chan é". é is the code point 233, which fits in one byte, but mIRC will encode your byte 233 to utf8, giving the two bytes 165 169. Now in this case there is little value not encoding in utf8, on IRC you usually don't really care about the integrity of your bytes. However there are others features where it's not so simple. $sha1(é) for example can be tricky, this is going to give the hash of the two bytes. The reason this is happening is because of the conversion, the hashing algorithm works on an array of byte, so mIRC has to take your character (represented in an array of two bytes) and convert them to single bytes, that's why the utf8 conversion happens. And this is happening pretty much everywhere.

/raw -n can be used for IRC, it sends the data to the server without encode the characters in the range 0-255 to utf8.

/sockwrite -u can be used to the same effect, won't encode characters in the range 0-255 to utf8.

Retrieved from "https://en.wikichip.org/w/index.php?title=mirc/unicode&oldid=79477"