|What character encoding should I use when creating XML documents?|
The best choice in most cases is either utf-8 or utf-16. Advantages of
these encodings include:
- The best portability. These encodings are more widely supported by
XML processors than any others, meaning that your documents will have the best
possible chance of being read correctly, no matter where they end up.
- Full international character support. Both utf-8 and utf-16 cover the
full Unicode character set, which includes all of the characters from all major
national, international and industry character sets.
- Efficient. utf-8 has the smaller storage requirements for documents
that are primarily composed of characters from the Latin alphabet. utf-16 is
more efficient for encoding Asian languages. But both encodings cover all
languages without loss.
The only drawback of utf-8 or utf-16 is that they are not the native
text file format for most systems, meaning that some text file editors
and viewers can not be directly used.
A second choice of encoding would be any of the others listed in the
table above. This works best when the xml encoding is the same as the default
system encoding on the machine where the XML document is being prepared,
because the document will then display correctly as a plain text file. For UNIX
systems in countries speaking Western European languages, the encoding will
usually be iso-8859-1.
A word of caution for Windows users: The default character set on
Windows systems is windows-1252, not iso-8859-1. While Xerces-C++ does
recognize this Windows encoding, it is a poor choice for portable XML data
because it is not widely recognized by other XML processing tools. If you are
using a Windows-based editing tool to generate XML, check which character set
it generates, and make sure that the resulting XML specifies the correct name
|How do I transcode to/from something besides the local code page?|
XMLString::transcode() will transcode from XMLCh to the local code page, and
other APIs which take a char* assume that the source text is in the local
code page. If this is not true, you must transcode the text yourself. You
can do this using local transcoding support on your OS, such as Iconv on
Unix or IBM's ICU package. However, if your transcoding needs are simple,
you can achieve better portability by using the Xerces-C++ parser's
transcoder wrappers. You get a transcoder like this:
Call XMLPlatformUtils::fgTransServer->MakeNewTranscoderFor() and provide
the name of the encoding you wish to create a transcoder for. This will
return a transcoder to you, which you own and must delete when you are
through with it.
NOTE: You must provide a maximum block size that you will pass to the transcoder
at one time, and you must pass blocks of characters of this count or smaller when
you do your transcoding. The reason for this is that this is really an
internal API and is used by the parser itself to do transcoding. The parser
always does transcoding in known block sizes, and this allows transcoders to
be much more efficient for internal use since it knows the max size it will
ever have to deal with and can set itself up for that internally. In
general, you should stick to block sizes in the 4 to 64K range.
The returned transcoder is something derived from XMLTranscoder, so they
are all returned to you via that interface.
This object is really just a wrapper around the underlying transcoding
system actually in use by your version of Xerces-C++, and does whatever is
necessary to handle differences between the XMLCh representation and the
representation used by that underlying transcoding system.
The transcoder object has two primary APIs, transcodeFrom() and
transcodeTo(). These transcode between the XMLCh format and the encoding you
These APIs will transcode as much of the source data as will fit into the
outgoing buffer you provide. They will tell you how much of the source they
ate and how much of the target they filled. You can use this information to
continue the process until all source is consumed.
char* data is always dealt with in terms of bytes, and XMLCh data is
always dealt with in terms of characters. Don't mix up which you are dealing
with or you will not get the correct results, since many encodings don't
have a one to one relationship of characters to bytes.
When transcoding from XMLCh to the target encoding, the transcodeTo()
method provides an 'unrepresentable flag' parameter, which tells the
transcoder how to deal with an XMLCh code point that cannot be converted
legally to the target encoding, which can easily happen since XMLCh is
Unicode and can represent thousands of code points. The options are to use a
default replacement character (which the underlying transcoding service will
choose, and which is guaranteed to be legal for the target encoding), or to
throw an exception.
Here is an example:
// Create an XMLTranscoder that is able to transcode between
// Unicode and UTF-8.
XMLTranscoder* t = XMLPlatformUtils::fgTransService->makeNewTranscoderFor(
"UTF-8", failReason, 16*1024);
// Source string is in Unicode, want to transcode to UTF-8
// Source string in UTF-8, want to transcode to Unicode.
An even simpler way to transcode to a different encoding is
to use the TranscodeToStr and TranscodeFromStr wrapper classes
which represent a one-time transcoding and encapsulate all the
memory management. Refer to the API Reference for more information.