yaz-icu - Man Page

yaz-icu [-c config] [-p opt] [-s] [-x] [infile]

yaz-icu is a utility which demonstrates the ICU chain module of yaz​. (yaz/icu​.h)​.

The utility can be used in two ways​. It may read some text using an XML configuration for configuring ICU and show text analysis​. This mode is triggered by option -c which specifies the configuration to be used​. The input file is read from standard input or from a file if infile is specified​.

The utility may also show ICU information​. This is triggered by option -p​.

-c config

Specifies the file containing ICU chain configuration which is XML based​.

-p type

Specifies extra information to be printed about the ICU system​. If type is c then ICU converters are printed​. If type is l, then available locales are printed​. If type is t, then available transliterators are printed​.

-s

Specifies that output should include sort key as well​. Note that sort key differs between ICU versions​.

-x

Specifies that output should be XML based rather than "text" based​.

The ICU chain configuration specifies one or more rules to convert text data into tokens​. The configuration format is XML based​.

The toplevel element must be named icu_chain​. The icu_chain element has one required attribute locale which specifies the ICU locale to be used in the conversion steps​.

The icu_chain element must include elements where each element specifies a conversion step​. The conversion is performed in the order in which the conversion steps are specified​. Each conversion element takes one attribute: rule which serves as argument to the conversion step​.

The following conversion elements are available:

casemap

Converts case (and rule specifies how):

l

Lower case using ICU function u_strToLower​.

u

Upper case using ICU function u_strToUpper​.

t

To title using ICU function u_strToTitle​.

f

Fold case using ICU function u_strFoldCase​.

display

This is a meta step which specifies that a term/token is to be displayed​. This term is retrieved in an application using function icu_chain_token_display (yaz/icu​.h)​.

transform

Specifies an ICU transform rule using a transliterator Identifier​. The rule attribute is the transliterator Identifier​. See ICU Transforms​^[1] for more information​.

transliterate

Specifies a rule-based transliterator​. The rule attribute is the custom transformation rule to be used​. See ICU Transforms​^[1] for more information​.

tokenize

Breaks / tokenizes a string into components using ICU functions ubrk_open, ubrk_setText, ​.​. ​. The rule is one of:

l

Line​. ICU: UBRK_LINE​.

s

Sentence​. ICU: UBRK_SENTENCE​.

w

Word​. ICU: UBRK_WORD​.

c

Character​. ICU: UBRK_CHARACTER​.

t

Title​. ICU: UBRK_TITLE​.

join

Joins tokens into one string​. The rule attribute is the joining string, which may be empty​. The join conversion element was added in YAZ 4​.2​.49​.

The following command analyzes text in file text using ICU chain configuration chain​.xml:

    cat text | yaz-icu -c chain​.xml

The chain​.xml might look as follows:

<icu_chain locale="en">
  <transform rule="[:Control:] Any-Remove"/>
  <tokenize rule="w"/>
  <transform rule="[[:WhiteSpace:][:Punctuation:]] Remove"/>
  <transliterate rule="xy > z;"/>
  <display/>
  <casemap rule="l"/>
</icu_chain>

yaz(7)

ICU Home​^[2]

ICU Transforms​^[1]

Index Data

1. ICU Transforms
   https://unicode-org.github.io/icu/userguide/transforms/general/
2. ICU Home
   https://github.com/unicode-org/icu