Character Sets

Supported character sets

mnoGoSearch supports almost all known 8 bit character sets as well as some multi-byte charsets including Korean euc-kr, Chinese big5 and gb2312, Japanese shift-jis, euc-jp and iso-2022-jp, as well as utf8.

mnoGoSearch also supports the following Macintosh chatacter sets: MacCE, MacCroatian, MacGreek, MacRoman, MacTurkish, MacIceland, MacRomania, MacThai, MacArabic, MacHebrew, MacCyrillic, MacGujarati.

Several languages in one database

It is often necessary to deal with several languages simultaneously. Number of supported languages depends on choice of character set that mnoGoSearch will use to store data. Character set is specified with LocalCharset command.

UTF-8 mode

When UTF-8 is specified in LocalCharset command, you may work with any languages supported in Unicode. That means you may use any number of over 650 languages. However, using UTF-8 may consume large amount of disk space (up to twice for some languages), leading to slower indexation and search.

non-UTF-8 mode

Since every character set includes latin characters, any character set supports at least two languages - English and one or more other languages. US-ASCII is an exception - it supports only Latin characters.

Note

When using mnoGoSearch in standard (non-UTF-8) mode, you may use as many languages as you like if they all belong to same language group.

Table 11.1. Language groups

E.g. in case you search engine is configured to use LocalCharset from the 5th group (Cyrillic), you may index servers containing documents in Bulgarian, Byelorussian, Macedonian, Russian, Serbian and Ukrainian. Indexing a multi-language document in UTF-8 is possible as well; however indexer will extract and save only cyrillic content from the page. To provide support for over 650 languages, please use LocalCharset utf-8 command.

Character set conversion

indexer converts all documents to the character set specified in the LocalCharset indexer.conf command. Internally conversion is implemented using Unicode. Please note that some conversion may loose some data. For example, conversion between any Greek and Russian charsets looses all national characters. This does not matter for a single language sites. If you want to build multi-lingual search engine use UTF8 character set as a LocalCharset.

Character set conversion at search time

You may use BrowserCharset command to choose a charset which will be used to display search results. BrowserCharset may differ from LocalCharset.

Character sets aliases

Each charset is recognized by a number of its aliases. Web servers can return the same charset in different notation. For example, iso-8859-2, iso8859-2, latin2 are the same charsets. There is support for charsets names aliases which search engine can understand:

Table 11.2. Charsets aliases