Home > Products > downCast > Support > TechNote 1
TechNote 1 Converting Unicode to RTF |
This document describes how downCast converts incoming Unicode to RTF character data. It describes the roles of stdfonts.config
and *.enc
files in that process.
In RTF, some Unicode codepoints have a special symbol representation. An incoming Unicode character is therefore checked against this list of special RTF characters:
U+0009
(tab character)U+000A
(line break)U+00AD
(optional hyphen)U+2028
(line break)U+2011
(non-breaking hyphen)These characters are immediately translated into their RTF symbol equivalent and bypass any other processing.
Any character which is not a special character will be tried to be encoded in the current encoding. The most suitable encoding for a character is determined by checking which font-family
it is hinted to be part of by checking the value of the CSS font-family
property. With this font name, the most suitable entry in stdfonts.config
is searched and its codepage setting is used to select one of the encodings (built-in or custom).
Using the inverse mapping of this current encoding, the result character index is determined for the Unicode character at hand, and that character is written to the RTF output file.
Now, it may well be that the font-family
property value was just a hint at which font to use, but the current character can of course be any Unicode character and needn't be part of the determined font (encoding). When this happens, downCast looks for the @font-search-list
option in the stdfonts.config
file. The value of this option is an ordered list of font names to be searched for any Unicode character that was not found in the current encoding. The first font whose encoding does contain the Unicode character at hand is chosen, its associated encoding is used for translating the incoming Unicode character to its RTF counterpart, and the RTF character is written wrapped in an RTF font change sequence.
The kind of font change sequence can be selected by setting the -ilx-renderhint-fontswitch
property in stdfonts.config
appropriately for the respective font, i.e. either an ordinary font switch using the \f
symbol or a SYMBOL
field destination.
If none of the fonts in the font search list contain the requested Unicode character, the character is written in Unicode notation with an underscore '_' character as replacement (which will be displayed by RTF rendering applications which do not support the \u
Unicode character RTF symbol).
stdfonts.config
RTF files need to specify which encoding a font to be used is using and what properties it has. This is used by a rendering application to determine the best matching font on a platform where the exact specified font is not available. Additionally, the encoding a font is in is used by the rendering application to correctly interpret the characters found in the RTF file.
However, this mechanism does not support custom fonts with a special mapping of their constituing characters to a Unicode codepoint. This is what the stdfonts.config file is for. downCast comes with a default file embedded in the application JAR. You may extend and/or override it by providing a custom stdfonts.config file, to be located in the {Encodings Folder}. Here, you can specify standard font properties based on the font name, specifically any custom encoding resp. codepage this font uses..
stdfonts.config can be found at the following location in the package hierarchy in the downcast.jar JAR file:
{JAR Resource Location Path}/config/stdfonts.config
You can modify this file to your liking and requirements, or supply an override file in the location mentioned above. Follows an informal description of the file format and the necessary properties, followed by the search algorithm used by downCast to find the properties for a given font in a CSS-like rule.
The following special properties are used in the stdfonts.config file:
Determines the general RTF font family a font belongs to based on its design. An RTF rendering application will use this information to find a font with similar appearance when an exact match cannot be found.
Supported values: roman, swiss, symbol, modern, script, decor, tech, bidi
This indicates the Windows codepage the font uses for its encoding.
Supported values: codepageAsInteger, -1, 10000, -1000, -1001, -1002, -1004
The special values have the following meaning:
Uses the default encoding. This is the best choice for normal fonts.
Identifies the Mac Roman encoding.
Identifies the Private Use Area mapper.
Identifies the standard encoding of the Symbol font.
Identifies the encoding of the Wingdings font.
Identifies the encoding of the Zapf Dingbats font.
Specifies the Unicode codepoint offset for this particular font. On platforms like Macintosh and Windows, fonts that have no Unicode mapping defined like Webdings or Hoefler Text Ornaments, will be mapped 1:1 into the PUA. Normally, this is the area of U+F000…U+F0FF, but by using the U-xxxx notation below, you can set the offset anywhere you require.
Supported values: normal | private | U-xxxx
with private being equivalent to U-F000, which is the Unicode codepoint offset (should be in the Private Use Area (PUA)) where this mapping starts, and normal being equivalent to U-0000, which is also the default if the property is not specified.
When downCast encounters a Unicode character to render to RTF, it first looks whether this character is part of the encoding of the current font. If it is, it is written according to RTF specifications. However, when this Unicode character is not part of the encoding of the current font, downCast tries to look up a font in the names specified using the @font-search-list option in order. The first one it finds will be used to write the character to RTF. However, for a subsequent RTF reader to pick this up correctly, downCast must write a switch of font for this specific character. This property specifies which method downCast should use for this, if possible:
downCast will write a simple RTF font switch {\fx c}.
downCast will write the character using a SYMBOL field. This is only possible for single-byte-fonts.
downCast decides how to best write the character.
When downCast needs to write a character, it can do it in two ways: either just the character for the current font's encoding, or additionally as the original Unicode codepoint. By specifying one of the following values for a font, you can tell downCast which method it should use )if possible):
downCast will always write the character in the current encoding and its Unicode equivalent
downCast will not write the Unicode equivalent
downCast decides how to best write the character.
The following general options are available:
lets you specify a comma-separated list of font names in which downCast will search for an incoming Unicode character to be output to RTF if it is not part of the current encoding. This lets you specify precedences, e.g. you may want to list the actually installed fonts on your particular system first. If downCast does not find a match in the listed fonts, it will try all fonts defined in stdfonts.config. If it then still does not find a match, it will use Unicode notation with an underscore '_' as replacement character.
The file structure is line based. Each line identifies a set of font names with a set of properties:
fontlist ::= propertyset fontlist ::= font ( ', ' font)* font ::= fontname | '"' fontname '"' propertyset ::= '\-ilx-rtf-font-family: ' ffval '; \-ilx-codepage: ' [0-9]+ ';' ' \-ilx-unicode-offset: ' uoval '; \-ilx-renderhint-fontswitch: ' rhfs '; \-ilx-renderhint-unicode: ' rhuc ';' ffval ::= 'roman' | 'swiss' | 'symbol' | 'modern' | 'script' | 'decor' | 'tech' | 'bidi' uoval ::= 'normal' | 'private' | 'U-' [0-9A-F]{4} rhfs ::= 'font' | 'field' | 'auto' rhuc ::= 'always' | 'never' | 'auto' fontname ::= name of font
Note that you must use CSS style escapes (or numerical character entities of the form &#...;) to generate Unicode characters for specifying font names using characters outside the ASCII range. Examples for this can be seen in the file packaged default stdfonts.config file.
All lines starting with // denote a comment line, as do empty lines.
To avoid having to explicitly define every font in the stdfonts.config file which might ever occur in a stylesheet, downCast employs a multi-stage search algorithm for a matching property definition entry as follows:
First, a potentially existing user supplied stdfonts.config is prepended to the default one supplied by downCast. Within this concatenated, big file, the following search algorithm is employed:
A search for the exact name (considering case) is performed. The first matching entry is is used if it exists.
A search for the exact name, but ignoring case, is performed. The first matching entry is is used if it exists.
A search for a font name is performed that matches the start of the actual name. So if the characteristics for "Univers Bold" are requested, and there is an entry "Univers" in stdfonts.config, then its properties are used. Case is ignored.
A search for a font name is performed that is contained in the actual name. So if the characteristics for "L Univers 44" are requested, and there is an entry "Univers" in stdfonts.config, then its properties are used because the string "Univers" is contained in the actual font name. Case is ignored.
downCast comes complete with virtually all default encodings you can use in RTF resp. Word, including many two-byte ones. This means that normally, you do not need to provide a custom encoding.
The default encodings are hard-coded with optimizations done for each specific encoding to provide efficient access, since the mapping functions are called for each character that passes through downCast. These default encodings are therefore not directly user-accessible. However, there are sometimes occasions where you'll need to use a custom encoding, especially when you are using custom fonts.
downCast provides a custom encoding loader and handler which lets you specify your own mappings from character codepoint in the font to Unicode by means of a simple text file. Both, one-byte and two-byte encodings can be specified in this way.
To create a custom encoding, you need to create an ASCII text file with the extension .encoding which specifies both the mapping of the individual codepoints to Unicode and also states which codepage it implements. You can also give it a name for easily spotting it in the UI portions of the application. downCast looks for custom encoding files in the {Encodings Folder} at startup. All encodings it finds are added to the internal set of default encodings. By specifying a codepage in a custom encoding that has a default equivalent, you may override any of the factory-supplied encodings.
Since the mapping is built on the fly, specific optimizations cannot be performed and the use of custom encodings may slow-down processing slightly.
A custom encoding per se is not tied to anything but the codepage it implements. To tie a codepage to a specific font, you need to extend or override the stdfonts.config file. In this mapping file, you simply list the font's name and associate it with a codepage using the keywords as described.
It is recommended to use codepage values greater than 40000 for custom encodings, as downCast will not use these codepages internally. Which you use for custom encodings is up to you. downCast reserves the range from 32000 to 35000 for internal use, so you should not use these. Also note that when you override a default encoding, every font that is specified to use that encoding will use the custom one.
File names can be arbitrary, must however have an extension of .encoding. They should be placed into the {Encodings Folder} for downCast to scan and load them automatically at startup.
The file structure is simple: one mapping entry per line, and all lines starting with #, // or ; are treated as comments. To create a two-byte encoding, separate the two bytes by a comma.
A mapping entry has the form (notation similar to BNF):
mapping ::= <srcbyte> [',' <srcbyte>] '=' <unicodechar>
with:
srcbyte ::= hexNumber | decimalNumber unicodechar ::= hexNumber | decimalNumber hexNumber ::= ('0x' | '0X' | '$')[0-9A-Fa-f]+ decimalNumber ::= [0-9]+
Follows a rather silly example, which maps what in codepage 1252 fonts is a space to the at-sign:
@codepage 42001 @encodingname Silly Encoding $20=$40
Options. Two special options are supported:
This specifies the codepage this encoding represents.
You can specify either an existing encoding to override its definition, or create custom codepages for specific fonts, in which case you should choose codepage number higher than 40000.
This is a descriptive name for the encoding so you can easily spot it in downCast's UI.
It is recommended to use codepage values greater than 40000 for custom encodings, as downCast will not use these codepages internally. Which you use for custom encodings is up to you.
downCast reserves the range from 32000 to 35000 for internal use, so you should not use these. Also note that when you override a default encoding, every font that is specified to use that encoding will use the custom one.
Author: Christian Roth
Last revised: 2005-09-19 16:06
© 2003-2007 infinity-loop GmbH Send mail to Webmaster Anbieterkennzeichnung §6 TDG