News Products Support Download Licenses Company

Home > Products > downCast > Support > TechNote 1

 

TechNote 1

Converting Unicode to RTF
using Font and Encoding information

Summary

This document describes how downCast converts incoming Unicode to RTF character data. It describes the roles of stdfonts.config and *.enc files in that process.

Flow steps & graph

Special characters

In RTF, some Unicode codepoints have a special symbol representation. An incoming Unicode character is therefore checked against this list of special RTF characters:

These characters are immediately translated into their RTF symbol equivalent and bypass any other processing.

Current Encoding

Any character which is not a special character will be tried to be encoded in the current encoding. The most suitable encoding for a character is determined by checking which font-family it is hinted to be part of by checking the value of the CSS font-family property. With this font name, the most suitable entry in stdfonts.config is searched and its codepage setting is used to select one of the encodings (built-in or custom).

Using the inverse mapping of this current encoding, the result character index is determined for the Unicode character at hand, and that character is written to the RTF output file.

Unicode character not part of found encoding

Now, it may well be that the font-family property value was just a hint at which font to use, but the current character can of course be any Unicode character and needn't be part of the determined font (encoding). When this happens, downCast looks for the @font-search-list option in the stdfonts.config file. The value of this option is an ordered list of font names to be searched for any Unicode character that was not found in the current encoding. The first font whose encoding does contain the Unicode character at hand is chosen, its associated encoding is used for translating the incoming Unicode character to its RTF counterpart, and the RTF character is written wrapped in an RTF font change sequence.

The kind of font change sequence can be selected by setting the -ilx-renderhint-fontswitch property in stdfonts.config appropriately for the respective font, i.e. either an ordinary font switch using the \f symbol or a SYMBOL field destination.

If none of the fonts in the font search list contain the requested Unicode character, the character is written in Unicode notation with an underscore '_' character as replacement (which will be displayed by RTF rendering applications which do not support the \u Unicode character RTF symbol).

Support files

1. stdfonts.config

RTF files need to specify which encoding a font to be used is using and what properties it has. This is used by a rendering application to determine the best matching font on a platform where the exact specified font is not available. Additionally, the encoding a font is in is used by the rendering application to correctly interpret the characters found in the RTF file.

However, this mechanism does not support custom fonts with a special mapping of their constituing characters to a Unicode codepoint. This is what the stdfonts.config file is for. downCast comes with a default file embedded in the application JAR. You may extend and/or override it by providing a custom stdfonts.config file, to be located in the {Encodings Folder}. Here, you can specify standard font properties based on the font name, specifically any custom encoding resp. codepage this font uses..

stdfonts.config can be found at the following location in the package hierarchy in the downcast.jar JAR file:

{JAR Resource Location Path}/config/stdfonts.config

You can modify this file to your liking and requirements, or supply an override file in the location mentioned above. Follows an informal description of the file format and the necessary properties, followed by the search algorithm used by downCast to find the properties for a given font in a CSS-like rule.

1.1. Properties and Values

The following special properties are used in the stdfonts.config file:

1.1.1. -ilx-rtf-font-family

Determines the general RTF font family a font belongs to based on its design. An RTF rendering application will use this information to find a font with similar appearance when an exact match cannot be found.

Supported values: roman, swiss, symbol, modern, script, decor, tech, bidi

1.1.2. -ilx-codepage

This indicates the Windows codepage the font uses for its encoding.

Supported values: codepageAsInteger, -1, 10000, -1000, -1001, -1002, -1004

The special values have the following meaning:

-1

Uses the default encoding. This is the best choice for normal fonts.

10000

Identifies the Mac Roman encoding.

-1000

Identifies the Private Use Area mapper.

-1001

Identifies the standard encoding of the Symbol font.

-1002

Identifies the encoding of the Wingdings font.

-1004

Identifies the encoding of the Zapf Dingbats font.

1.1.3. -ilx-unicode-offset

Specifies the Unicode codepoint offset for this particular font. On platforms like Macintosh and Windows, fonts that have no Unicode mapping defined like Webdings or Hoefler Text Ornaments, will be mapped 1:1 into the PUA. Normally, this is the area of U+F000…U+F0FF, but by using the U-xxxx notation below, you can set the offset anywhere you require.

Supported values: normal | private | U-xxxx

with private being equivalent to U-F000, which is the Unicode codepoint offset (should be in the Private Use Area (PUA)) where this mapping starts, and normal being equivalent to U-0000, which is also the default if the property is not specified.

1.1.4. -ilx-renderhint-fontswitch

When downCast encounters a Unicode character to render to RTF, it first looks whether this character is part of the encoding of the current font. If it is, it is written according to RTF specifications. However, when this Unicode character is not part of the encoding of the current font, downCast tries to look up a font in the names specified using the @font-search-list option in order. The first one it finds will be used to write the character to RTF. However, for a subsequent RTF reader to pick this up correctly, downCast must write a switch of font for this specific character. This property specifies which method downCast should use for this, if possible:

font

downCast will write a simple RTF font switch {\fx c}.

field

downCast will write the character using a SYMBOL field. This is only possible for single-byte-fonts.

auto

downCast decides how to best write the character.

1.1.5. -ilx-renderhint-unicode

When downCast needs to write a character, it can do it in two ways: either just the character for the current font's encoding, or additionally as the original Unicode codepoint. By specifying one of the following values for a font, you can tell downCast which method it should use )if possible):

always

downCast will always write the character in the current encoding and its Unicode equivalent

never

downCast will not write the Unicode equivalent

auto

downCast decides how to best write the character.

1.2. Options

The following general options are available:

@font-search-list

lets you specify a comma-separated list of font names in which downCast will search for an incoming Unicode character to be output to RTF if it is not part of the current encoding. This lets you specify precedences, e.g. you may want to list the actually installed fonts on your particular system first. If downCast does not find a match in the listed fonts, it will try all fonts defined in stdfonts.config. If it then still does not find a match, it will use Unicode notation with an underscore '_' as replacement character.

1.3. File structure

The file structure is line based. Each line identifies a set of font names with a set of properties:

fontlist ::= propertyset
fontlist ::= font ( ', ' font)*
font ::= fontname | '"' fontname '"'
propertyset ::= '\-ilx-rtf-font-family: ' ffval '; \-ilx-codepage: ' [0-9]+ ';'
                ' \-ilx-unicode-offset: ' uoval '; \-ilx-renderhint-fontswitch: ' rhfs '; \-ilx-renderhint-unicode: ' rhuc ';' 
ffval ::= 'roman' | 'swiss' | 'symbol' | 'modern' | 'script' | 'decor' | 'tech' | 'bidi'
uoval ::= 'normal' | 'private' | 'U-' [0-9A-F]{4}
rhfs ::= 'font' | 'field' | 'auto'
rhuc ::= 'always' | 'never' | 'auto'
fontname ::= name of font

Note

Note that you must use CSS style escapes (or numerical character entities of the form &#...;) to generate Unicode characters for specifying font names using characters outside the ASCII range. Examples for this can be seen in the file packaged default stdfonts.config file.

All lines starting with // denote a comment line, as do empty lines.

1.4. Matching Algorithm

To avoid having to explicitly define every font in the stdfonts.config file which might ever occur in a stylesheet, downCast employs a multi-stage search algorithm for a matching property definition entry as follows:

First, a potentially existing user supplied stdfonts.config is prepended to the default one supplied by downCast. Within this concatenated, big file, the following search algorithm is employed:

  1. A search for the exact name (considering case) is performed. The first matching entry is is used if it exists.

  2. A search for the exact name, but ignoring case, is performed. The first matching entry is is used if it exists.

  3. A search for a font name is performed that matches the start of the actual name. So if the characteristics for "Univers Bold" are requested, and there is an entry "Univers" in stdfonts.config, then its properties are used. Case is ignored.

  4. A search for a font name is performed that is contained in the actual name. So if the characteristics for "L Univers 44" are requested, and there is an entry "Univers" in stdfonts.config, then its properties are used because the string "Univers" is contained in the actual font name. Case is ignored.

2. Custom Encodings

2.1. How it works

downCast comes complete with virtually all default encodings you can use in RTF resp. Word, including many two-byte ones. This means that normally, you do not need to provide a custom encoding.

The default encodings are hard-coded with optimizations done for each specific encoding to provide efficient access, since the mapping functions are called for each character that passes through downCast. These default encodings are therefore not directly user-accessible. However, there are sometimes occasions where you'll need to use a custom encoding, especially when you are using custom fonts.

downCast provides a custom encoding loader and handler which lets you specify your own mappings from character codepoint in the font to Unicode by means of a simple text file. Both, one-byte and two-byte encodings can be specified in this way.

To create a custom encoding, you need to create an ASCII text file with the extension .encoding which specifies both the mapping of the individual codepoints to Unicode and also states which codepage it implements. You can also give it a name for easily spotting it in the UI portions of the application. downCast looks for custom encoding files in the {Encodings Folder} at startup. All encodings it finds are added to the internal set of default encodings. By specifying a codepage in a custom encoding that has a default equivalent, you may override any of the factory-supplied encodings.

Since the mapping is built on the fly, specific optimizations cannot be performed and the use of custom encodings may slow-down processing slightly.

2.2. Associating a Font with an Encoding

A custom encoding per se is not tied to anything but the codepage it implements. To tie a codepage to a specific font, you need to extend or override the stdfonts.config file. In this mapping file, you simply list the font's name and associate it with a codepage using the keywords as described.

It is recommended to use codepage values greater than 40000 for custom encodings, as downCast will not use these codepages internally. Which you use for custom encodings is up to you. downCast reserves the range from 32000 to 35000 for internal use, so you should not use these. Also note that when you override a default encoding, every font that is specified to use that encoding will use the custom one.

2.3. File format

File names can be arbitrary, must however have an extension of .encoding. They should be placed into the {Encodings Folder} for downCast to scan and load them automatically at startup.

The file structure is simple: one mapping entry per line, and all lines starting with #, // or ; are treated as comments. To create a two-byte encoding, separate the two bytes by a comma.

A mapping entry has the form (notation similar to BNF):

mapping ::= <srcbyte> [',' <srcbyte>] '=' <unicodechar>

with:

srcbyte ::= hexNumber | decimalNumber
unicodechar ::= hexNumber | decimalNumber
hexNumber ::= ('0x' | '0X' | '$')[0-9A-Fa-f]+
decimalNumber ::= [0-9]+

Follows a rather silly example, which maps what in codepage 1252 fonts is a space to the at-sign:

@codepage 42001
@encodingname Silly Encoding
$20=$40

Options. Two special options are supported:

@codepage decimalNumber

This specifies the codepage this encoding represents.

You can specify either an existing encoding to override its definition, or create custom codepages for specific fonts, in which case you should choose codepage number higher than 40000.

@encodingname asciistring

This is a descriptive name for the encoding so you can easily spot it in downCast's UI.

Note

It is recommended to use codepage values greater than 40000 for custom encodings, as downCast will not use these codepages internally. Which you use for custom encodings is up to you.

downCast reserves the range from 32000 to 35000 for internal use, so you should not use these. Also note that when you override a default encoding, every font that is specified to use that encoding will use the custom one.

 

 

back to downCast Support Page


Author: Christian Roth
Last revised: 2005-09-19 16:06


© 2003-2007 infinity-loop GmbHSend mail to WebmasterAnbieterkennzeichnung §6 TDG