Perl

De Organic Design

Perl and UTF-8

This excellent blog item on Perl's UTF-8 handling clears up the confusion that can arise when dealing with UTF-8 in Perl. The main complexity is that Perl has an internal character encoding. This encoding allows Perl to hold Unicode characters in strings and treat them as single characters. This is useful as Unicode characters may be represented by multiple bytes in a Unicode encoding such as UTF-8. This means there are two types of strings: Those with characters in its internal encoding, and those not. The latter typically can be treated as binary data ("octets"), though strictly speaking they are octets only if they come from outside a program. i.e., not strings defined in source code.

One approach that can work is to have binmode set on all filehandles and stdout/stdin. Then Encode::decode() at the point data enters the program, and Encode::encode() on its way out. This lets you avoid the complexity of I/O layers, and means you have a defined boundary where translation occurs.

Remember also to always use "UTF-8" rather than "utf8" as the encoding in the Encode module as only the former produces correct UTF-8, while the latter is something Perl specific and is not UTF-8.

See also Reading and writing UTF-8 examples with IO::All.


See also