4.24 Character properties

SWI-Prolog offers two comprehensive predicates for classifying characters and character codes. These predicates are defined as built-in predicates to exploit the C-character classification's handling of locale (handling of local character sets). These predicates are fast, logical and deterministic if applicable.

In addition, there is the library library(ctypes) providing compatibility with some other Prolog systems. The predicates of this library are defined in terms of code_type/2.

code_type(?Code, ?Type)

char_type(?Char, ?Type)

Tests or generates alternative Types or Char/Codes. The character types are inspired by the standard C <ctype.h> primitives. Neither the classes nor the case conversions are sensitive to the active locale: they are derived from the Unicode Character Database at build time (unicode_syntax_version) rather than from the C library's <wctype.h>, so a program classifies text the same way regardless of LC_CTYPE and of the platform's C library. This also means the classes agree with the classification used by read/1 and friends, which is where the types prolog_var_start, prolog_atom_start, prolog_identifier_continue and prolog_symbol come from. The case (to_lower, to_upper, lower(Upper) and upper(Lower)) still use the C library and are thus still locale conversions (to_lower, to_upper, lower(Upper) and upper(Lower)) use the Unicode simple case mapping, which is one code point to one code point and thus length preserving; upcase_atom('straße', X) yields 'STRAßE' rather than 'STRASSE'. Because the mapping is Unicode's rather than the C library's, it does not follow the Turkish and Azeri dotted/dotless i rules even under a tr_TR locale. Foreign code can reach the same classifier and the same case mapping through PL_ctype_flags(), PL_toupper(), PL_tolower() and PL_totitle().

Every type is derived from the Unicode Character Database, from the files UnicodeData.txt (general categories and the simple case mappings), DerivedCoreProperties.txt (Alphabetic, Uppercase, Lowercase, XID_Start, XID_Continue), PropList.txt (White_Space, Sentence_Terminal, Pattern_Syntax) and EastAsianWidth.txt (display width), for the Unicode version named by the unicode_syntax_version flag. The derivations are:

Type	Derived from
alnum	alpha, or category Nd, Nl or No
alpha	property Alphabetic
ascii	code point < 128; not a Unicode notion
cntrl	category Cc or Cf
csym	ASCII letter, digit or _; see below
csymf	ASCII letter or _; see below
decimal	category Nd
digit	U+0030..U+0039; POSIX fixes this to ASCII
end_of_file	the code -1
end_of_line	the Unicode line terminators
graph	alnum or punct
lower	property Lowercase
newline	U+000A
paren	categories Ps and Pe, paired
pattern_syntax	property Pattern_Syntax (UAX #31)
period	property Sentence_Terminal
print	graph, or white space that is not Cc
prolog_end_of_line	the Unicode line terminators
prolog_layout	property Pattern_White_Space (UAX #31)
punct	category P* or S*, not Alphabetic
quote	', ", ` and categories Pi, Pf
space	property White_Space
to_lower, to_upper	the simple case mappings
upper	property Uppercase
white	White_Space minus end_of_line
width	property East_Asian_Width (UAX #11)
xdigit	0..9, a..f, A..F

The types prolog_var_start, prolog_atom_start, prolog_identifier_continue, prolog_symbol and prolog_solo are the reader's own classification; see section 2.15.1.9.

Note that the mode (-,+) is only efficient if the Type has a parameter, e.g., char_type(C, digit(8)). If Type is a atomic, the whole unicode range (0..0x1ffff) is generated and tested against the character classification function.

alnum

Char is alpha or a numeric character (Unicode general category Nd, Nl or No). Note that this is wider than digit: e.g. 0’
u0660 (ARABIC-INDIC DIGIT ZERO) and 0’
u00BD (VULGAR FRACTION ONE HALF) are alnum but not digit.

alpha

Char has the Unicode Alphabetic property. This covers the letters of all scripts, not just those that have case, as well as letter-like numbers such as the Roman numerals.

csym

Char is a character of a C identifier: an ASCII letter, an ASCII digit or the underscore (_).

csymf

Char is a valid first character of a C identifier: an ASCII letter or the underscore (_).

These two are ASCII because C identifiers are: C89 admits nothing else, and the extended identifiers of C99 and later are written as universal character names (\u00E9) rather than as source characters. Prolog identifiers do range over all of Unicode; use prolog_atom_start, prolog_var_start and prolog_identifier_continue for those, or alnum and alpha for a script-independent notion of a word character.

ascii

Char is a 7-bit ASCII character (0..127). This is the one type that is defined by the code point range rather than by a Unicode property.

white

Char is white space that stays within a line: the space characters minus the seven line terminators (U+000A.. U+000D, U+0085, U+2028 and U+2029). That leaves U+0009 TAB, U+0020 SPACE, U+00A0 NO-BREAK SPACE, U+1680, U+2000..U+200A, U+202F, U+205F and U+3000 IDEOGRAPHIC SPACE. This is what POSIX calls blank.

cntrl

Char is a control character (Unicode general category Cc, i.e. 0..31, 127 and 128..159) or a format character (category Cf, e.g. U+00AD SOFT HYPHEN, U+200B ZERO WIDTH SPACE and U+FEFF). U+0009..U+000D and U+0085 are both cntrl and space. Note that U+2028 LINE SEPARATOR and U+2029 PARAGRAPH SEPARATOR are space, not cntrl.

digit

Char is in 0 ... 9. POSIX requires the digit class to hold exactly the ten ASCII digits, so the decimal digits of other scripts are alnum and decimal, but not digit. See also decimal.

digit(Weight)

Char is a digit with value Weight. I.e. char_type(X, digit(6)) yields X = ’6’. Useful for parsing numbers.

xdigit(Weight)

Char is a hexadecimal digit with value Weight. I.e. char_type(a, xdigit(X)) yields X = ’10’. Useful for parsing numbers. As with digit(Weight) this is ASCII only; there is no Unicode-wide counterpart because hexadecimal notation is a source syntax rather than a script property.

decimal

Char is a decimal digit in any script. This implies it has the Unicode general category Nd).

decimal(Weight)

Char is a decimal digit in any script with Weight 0 ... 9.

print

Char is a printable character: graph, or white space that is not a control character (U+0020, U+00A0, U+2028, U+3000, ... but not tab or newline).

graph

Char produces a visible mark on a page when printed, i.e. alnum or punct. Note that the space is not included! POSIX requires graph and space to be disjoint, which is why U+00A0 NO-BREAK SPACE is print but not graph.

lower

Char is a lowercase letter.

lower(Upper)

Char is a lowercase version of Upper. Only true if Char is lowercase and Upper uppercase.

to_lower(Upper)

Char is a lowercase version of Upper. For non-letters, or letter without case, Char and Lower are the same. See also upcase_atom/2 and downcase_atom/2.

upper

Char is an uppercase letter.

upper(Lower)

Char is an uppercase version of Lower. Only true if Char is uppercase and Lower lowercase.

to_upper(Lower)

Char is an uppercase version of Lower. For non-letters, or letter without case, Char and Lower are the same. See also upcase_atom/2 and downcase_atom/2.

punct

Char is a punctuation character. This is a graph character that is not a letter or digit.

space

Char has the Unicode White_Space property: the ASCII layout characters (tab, newline, vertical tab, form feed, carriage return and space), U+0085 NEL, U+00A0 NO-BREAK SPACE, U+1680, the U+2000..U+200A spaces, U+2028, U+2029, U+202F, U+205F and U+3000. Note that <wctype.h> implementations disagree here: glibc denies that U+00A0 is white space and Darwin denies that U+0085 is. Note also that U+00A0 is deliberately not layout for the reader; see prolog_layout.

end_of_file

Char is -1.

end_of_line

Char terminates a line: the seven code points Unicode defines as line terminator, being U+000A LF, U+000B VT, U+000C FF, U+000D CR, U+0085 NEL, U+2028 LINE SEPARATOR and U+2029 PARAGRAPH SEPARATOR. Equivalently, space that is not white: white space either stays within a line or ends it. prolog_end_of_line is the set the reader acts on, which happens to be the same today but is defined by the Prolog syntax rather than by Unicode.

newline

Char is a newline character (10).

period

Char counts as the end of a sentence. This is the Unicode Sentence_Terminal property: besides ., ! and ? it holds the Armenian, Arabic, N'Ko, Devanagari, ideographic and fullwidth sentence enders, 170 code points in Unicode 17.

quote

Char is a quote character: ", ', ` or one of the Unicode initial/final quotation marks (categories Pi and Pf). Note that this holds for both sides of a pair, where quote(Close) below holds only for the opening one.

paren(Close)

Char is an opening bracket and Close is its matching close. Covers the three ASCII bracket pairs (), [] and {}, plus every Unicode Ps/Pe pair (about 60 pairs in Unicode 17, including angle, corner, ceiling, floor, mathematical, ornamental, fullwidth and CJK brackets). The mapping is reversible: with Close bound, Char unifies with the matching open.

quote(Close)

Char is an opening quotation mark and Close is its matching close. The ASCII quotes ’, ", and ‘ have Close = Char; Unicode Pi/Pf quote pairs (the guillemets, the standard left/right curly single and double quotes, and the single/double angle and reversed quotation marks) have Close different from Char. The mapping is reversible.

width(Width)

Width is the number of columns for fixed-width usage used by Char. True for all printable characters. Most characters require 1 column. Unicode combining characters require no space (they follow the base character). Many Asian characters and the Emojis require 2 columns. These values are used by stream_property/2 for the position(-Pos) property.

prolog_layout

Char is a Prolog layout character: a member of the Unicode Pattern_White_Space set used by read_term/2 to separate tokens. The eleven code points are U+0009..U+000D, U+0020, U+0085, U+200E, U+200F, U+2028 and U+2029. Locale- independent; pinned to unicode_syntax_version. prolog_end_of_line is the seven-element line-terminator subset.

prolog_end_of_line

Char ends a line of Prolog source text. Covers the seven line-terminator-like Pattern_White_Space code points: U+000A (LF), U+000B (VT), U+000C (FF), U+000D (CR), U+0085 (NEL), U+2028 (LINE SEPARATOR), and U+2029 (PARAGRAPH SEPARATOR). The same set terminates % comments and increments the source line counter. It is the reader's own definition and is kept separate from end_of_line, which is Unicode's, although the two currently agree. See section 2.15.1.9.

prolog_var_start

Char can start a Prolog variable name.

prolog_atom_start

Char can start a unquoted Prolog atom that is not a symbol.

prolog_identifier_continue

Char can continue a Prolog variable name or atom.

prolog_symbol

Char is a Prolog symbol character. Sequences of Prolog symbol characters glue together to form an unquoted atom. Examples are =.., \=, etc.

prolog_solo

Char is a Prolog solo character: a punctuation code point that forms an atom on its own and never combines with neighbouring symbol characters. In ASCII the solo set is !, ; and %; the same flag carries over to non-ASCII code points via the Unicode syntax map (see section 2.15.1.9). Solo characters are written unquoted by writeq/1 and are accepted as single-character atoms by the reader.

pattern_syntax

Char has the Unicode Pattern_Syntax property (UAX #31 R3). This is the immutable set of punctuation and symbol code points whose classification is guaranteed not to change across Unicode versions. Used by write_canonical/1 and the pattern_syntax_solo option of write_term/2 to decide which single-character atoms can be printed bare with round-trip safety across Unicode upgrades.

4.24.1 Case conversion

There is nothing in the Prolog standard for converting case in textual data. The SWI-Prolog predicates code_type/2 and char_type/2 can be used to test and convert individual characters. We have started some additional support:

downcase_atom(+AnyCase, -LowerCase)

Converts the characters of AnyCase into lowercase as char_type/2 does (i.e. based on the defined locale if Prolog provides locale support on the hosting platform) and unifies the lowercase atom with LowerCase.

upcase_atom(+AnyCase, -UpperCase)

Converts, similar to downcase_atom/2, an atom to uppercase.

4.24.2 White space normalization

normalize_space(-Out, +In)

Normalize white space in In. All leading and trailing white space is removed. All non-empty sequences of white space characters are replaced by a single space (\u0020) character. Out uses the same conventions as with_output_to/2 and format/3.

A character counts as white space if it satisfies code_type(Code, space), i.e., the C library function iswspace(). This depends on the locale. Using a Unicode locale this includes the Unicode separator characters such as \u00a0 (no-break space) and \u3000 (ideographic space), while in the C locale only the ASCII white space characters are recognised. Note that the bidi format characters \u200e and \u200f are not white space and are thus preserved.

4.24.3 Language-specific comparison

This section deals with predicates for language-specific string comparison operations.

collation_key(+Atom, -Key)

Create a Key from Atom for locale-specific comparison. The key is defined such that if the key of atom A precedes the key of atom B in the standard order of terms, A is alphabetically smaller than B using the sort order of the current locale.

The predicate collation_key/2 is used by locale_sort/2 from library(sort). Please examine the implementation of locale_sort/2 as an example of using this call.

The Key is an implementation-defined and generally unreadable string. On systems that do not support locale handling, Key is simply unified with Atom.

locale_sort(+List, -Sorted)

Sort a list of atoms using the current locale. List is a list of atoms or string objects (see section 5.2). Sorted is unified with a list containing all atoms of List, sorted to the rules of the current locale. See also collation_key/2 and setlocale/3.