News Products Support Download Licenses Company

Home > Products > upCast > Support > TechNote 2

 

TechNote 2

Fonts, Encodings and Unicode

Summary

This document describes how upCast converts incoming RTF characters to Unicode characters, the internal storage format. It describes the roles of stdfonts.config and *.enoding files in that process.

Flow steps & graph

Determining the encoding

When a character is read from the RTF file, upCast first determines its type. If this is a Unicode character (possibly with alternative representation), upCast always prefers the Unicode character and uses it as-is.

Otherwise, upCast tries to determine the current encoding with which it needs to interpret the incoming character codepoint. This happens in a 3-step decision cascade:

  1. If the RTF explicitly specified an encoding to use for the character at hand (RTF symbols: \cchs, \cpg) , this is used.
  2. If the RTF specified font information for the character at hand, upCast tries to determine the encoding for that font by using the font name as index into the stdfonts.config font property mapping file. If a font matching by name is found, its encoding property is retrieved from that file and the associated encoding is used (either one of the built-in encodings or a custom encoding).
    The built-in encodings and their associated codepages are listed in upCast under Help > System Information… .
  3. If neither an encoding has been specified explicitly and no font has been set or its encoding could not be determined, upCast uses the default encoding for the RTF file which is a required property.

Character Translation

Having determined the encoding to use, upCast now uses the codepoint retrieved from the RTF file as index into the encoding – which essentially is a codepoint-to-Unicode mapping table – and uses whatever Unicode character that is specified in that encoding. This Unicode character is used from now on internally and for serialization some time later in an export filter.

Support files

1. stdfonts.config

RTF files need to specify which encoding a font to be used is using and what properties it has. This is used by a rendering application to determine the best matching font on a platform where the exact specified font is not available. Additionally, the encoding a font is in is used by the rendering application to correctly interpret the characters found in the RTF file.

However, this mechanism does not support custom fonts with a special mapping of their constituing characters to a Unicode codepoint. This is what the stdfonts.config file is for. upCast comes with a default file embedded in the application JAR. You may extend and/or override it by providing a custom stdfonts.config file, to be located in the {Encodings Folder}. Here, you can specify standard font properties based on the font name, specifically any custom encoding resp. codepage this font uses..

stdfonts.config can be found at the following location in the package hierarchy in the upcast.jar JAR file:

{JAR Resource Location Path}/config/stdfonts.config

You can modify this file to your liking and requirements, or supply an override file in the location mentioned above. Follows an informal description of the file format and the necessary properties, followed by the search algorithm used by upCast to find the properties for a given font in a CSS-like rule.

1.1. Properties and Values

The following special properties are used in the stdfonts.config file:

1.1.1. -ilx-rtf-font-family

Determines the general RTF font family a font belongs to based on its design. An RTF rendering application will use this information to find a font with similar appearance when an exact match cannot be found.

Supported values: roman, swiss, symbol, modern, script, decor, tech, bidi

1.1.2. -ilx-codepage

This indicates the Windows codepage the font uses for its encoding.

Supported values: codepageAsInteger, -1, 10000, -1000, -1001, -1002, -1004

The special values have the following meaning:

-1

Uses the default encoding. This is the best choice for normal fonts.

10000

Identifies the Mac Roman encoding.

-1000

Identifies the Private Use Area mapper.

-1001

Identifies the standard encoding of the Symbol font.

-1002

Identifies the encoding of the Wingdings font.

-1004

Identifies the encoding of the Zapf Dingbats font.

1.1.3. -ilx-unicode-offset

Specifies the Unicode codepoint offset for this particular font. On platforms like Macintosh and Windows, fonts that have no Unicode mapping defined like Webdings or Hoefler Text Ornaments, will be mapped 1:1 into the PUA. Normally, this is the area of U+F000…U+F0FF, but by using the U-xxxx notation below, you can set the offset anywhere you require.

Supported values: normal | private | U-xxxx

with private being equivalent to U-F000, which is the Unicode codepoint offset (should be in the Private Use Area (PUA)) where this mapping starts, and normal being equivalent to U-0000, which is also the default if the property is not specified.

1.1.4. -ilx-renderhint-fontswitch

Note

This property is only relevant for downCast, upCast's companion product.

When downCast encounters a Unicode character to render to RTF, it first looks whether this character is part of the encoding of the current font. If it is, it is written according to RTF specifications. However, when this Unicode character is not part of the encoding of the current font, downCast tries to look up a font in the names specified using the @font-search-list option in order. The first one it finds will be used to write the character to RTF. However, for a subsequent RTF reader to pick this up correctly, downCast must write a switch of font for this specific character. This property specifies which method downCast should use for this, if possible:

font

downCast will write a simple RTF font switch {\fx c}.

field

downCast will write the character using a SYMBOL field. This is only possible for single-byte-fonts.

auto

downCast decides how to best write the character.

1.1.5. -ilx-renderhint-unicode

Note

This property is only relevant for downCast, upCast's companion product.

When downCast needs to write a character, it can do it in two ways: either just the character for the current font's encoding, or additionally as the original Unicode codepoint. By specifying one of the following values for a font, you can tell upCast which method it should use )if possible):

always

downCast will always write the character in the current encoding and its Unicode equivalent

never

downCast will not write the Unicode equivalent

auto

downCast decides how to best write the character.

1.2. Options

Note

This property is only relevant for downCast, upCast's companion product.

The following general options are available:

@font-search-list

lets you specify a comma-separated list of font names in which downCast will search for an incoming Unicode character to be output to RTF if it is not part of the current encoding. This lets you specify precedences, e.g. you may want to list the actually installed fonts on your particular system first. If downCast does not find a match in the listed fonts, it will try all fonts defined in stdfonts.config. If it then still does not find a match, it will use Unicode notation with an underscore '_' as replacement character.

1.3. File structure

The file structure is line based. Each line identifies a set of font names with a set of properties:

stdfonts ::= fontlist '=' propertyset
fontlist ::= font ( ', ' font)*
font ::= fontname | '"' fontname '"'
propertyset ::= '\-ilx-rtf-font-family: ' ffval '; \-ilx-codepage: ' [0-9]+ ';'
                ' \-ilx-unicode-offset: ' uoval '; \-ilx-renderhint-fontswitch: ' rhfs '; \-ilx-renderhint-unicode: ' rhuc ';' 
ffval ::= 'roman' | 'swiss' | 'symbol' | 'modern' | 'script' | 'decor' | 'tech' | 'bidi'
uoval ::= 'normal' | 'private' | 'U-' [0-9A-F]{4}
rhfs ::= 'font' | 'field' | 'auto'
rhuc ::= 'always' | 'never' | 'auto'
fontname ::= name of font

Note

Note that you must use CSS style escapes (or numerical character entities of the form &#...;) to generate Unicode characters for specifying font names using characters outside the ASCII range. Examples for this can be seen in the file packaged default stdfonts.config file.

All lines starting with // denote a comment line, as do empty lines.

1.4. Matching Algorithm

To avoid having to explicitly define every font in the stdfonts.config file which might ever occur in a stylesheet, upCast employs a multi-stage search algorithm for a matching property definition entry as follows:

First, a potentially existing user supplied stdfonts.config is prepended to the default one supplied by upCast. Within this concatenated, big file, the following search algorithm is employed:

  1. A search for the exact name (considering case) is performed. The first matching entry is is used if it exists.

  2. A search for the exact name, but ignoring case, is performed. The first matching entry is is used if it exists.

  3. A search for a font name is performed that matches the start of the actual name. So if the characteristics for "Univers Bold" are requested, and there is an entry "Univers" in stdfonts.config, then its properties are used. Case is ignored.

  4. A search for a font name is performed that is contained in the actual name. So if the characteristics for "L Univers 44" are requested, and there is an entry "Univers" in stdfonts.config, then its properties are used because the string "Univers" is contained in the actual font name. Case is ignored.

2. Custom Encodings

2.1. How it works

upCast comes complete with virtually all default encodings you can use in RTF resp. Word, including many two-byte ones. This means that normally, you do not need to provide a custom encoding.

The default encodings are hard-coded with optimizations done for each specific encoding to provide efficient access, since the mapping functions are called for each character that passes through upCast. These default encodings are therefore not directly user-accessible. However, there are sometimes occasions where you'll need to use a custom encoding, especially when you are using custom fonts.

upCast provides a custom encoding loader and handler which lets you specify your own mappings from character codepoint in the font to Unicode by means of a simple text file. Both, one-byte and two-byte encodings can be specified in this way.

To create a custom encoding, you need to create an ASCII text file with the extension .encoding which specifies both the mapping of the individual codepoints to Unicode and also states which codepage it implements. You can also give it a name for easily spotting it in the UI portions of the application. upCast looks for custom encoding files in the {Encodings Folder} at startup. All encodings it finds are added to the internal set of default encodings. By specifying a codepage in a custom encoding that has a default equivalent, you may override any of the factory-supplied encodings.

Since the mapping is built on the fly, specific optimizations cannot be performed and the use of custom encodings may slow-down processing slightly.

2.2. Associating a Font with an Encoding

A custom encoding per se is not tied to anything but the codepage it implements. To tie a codepage to a specific font, you need to extend or override the stdfonts.config file. In this mapping file, you simply list the font's name and associate it with a codepage using the keywords as described.

It is recommended to use codepage values greater than 40000 for custom encodings, as upCast will not use these codepages internally. Which you use for custom encodings is up to you. upCast reserves the range from 32000 to 35000 for internal use, so you should not use these. Also note that when you override a default encoding, every font that is specified to use that encoding will use the custom one.

2.3. File format

File names can be arbitrary, must however have an extension of .encoding. They should be placed into the {Encodings Folder} for downCast to scan and load them automatically at startup.

The file structure is simple: one mapping entry per line, and all lines starting with #, // or ; are treated as comments. To create a two-byte encoding, separate the two bytes by a comma.

A mapping entry has the form (notation similar to BNF):

mapping ::= <srcbyte> [',' <srcbyte>] '=' <unicodechar>

with:

srcbyte ::= hexNumber | decimalNumber
unicodechar ::= hexNumber | decimalNumber
hexNumber ::= ('0x' | '0X' | '$')[0-9A-Fa-f]+
decimalNumber ::= [0-9]+

Follows a rather silly example, which maps what in codepage 1252 fonts is a space to the at-sign:

@codepage 42001
@encodingname Silly Encoding
$20=$40

Options. Two special options are supported:

@codepage decimalNumber

This specifies the codepage this encoding represents.

You can specify either an existing encoding to override its definition, or create custom codepages for specific fonts, in which case you should choose codepage number higher than 40000.

@encodingname asciistring

This is a descriptive name for the encoding so you can easily spot it in downCast's UI.

Note

It is recommended to use codepage values greater than 40000 for custom encodings, as upCast will not use these codepages internally. Which you use for custom encodings is up to you.

upCast reserves the range from 32000 to 35000 for internal use, so you should not use these. Also note that when you override a default encoding, every font that is specified to use that encoding will use the custom one.

 

back to upCast Support Page


Author: Christian Roth
Last revised: 2006-03-21 13:58


© 2003-2007 infinity-loop GmbHSend mail to WebmasterAnbieterkennzeichnung §6 TDG