Basic text handling.

Topics
	Unicode character database and algorithms

Files
file	text_phrasing.hpp
	The text_phrasing type.

Data Structures
class	std::char_traits< hi::character >
	Character-traits for hi::character. More...

Enumerations
enum class	v1::text_phrasing : uint8_t {   v1::text_phrasing::regular = 0 , v1::text_phrasing::emphesis = 1 , v1::text_phrasing::strong = 2 , v1::text_phrasing::code = 3 ,   v1::text_phrasing::abbreviation = 4 , v1::text_phrasing::quote = 5 , v1::text_phrasing::keyboard = 6 , v1::text_phrasing::highlight = 7 ,   v1::text_phrasing::math = 8 , v1::text_phrasing::example = 9 , v1::text_phrasing::unarticulated = 10 , v1::text_phrasing::title = 11 ,   v1::text_phrasing::success = 12 , v1::text_phrasing::warning = 13 , v1::text_phrasing::error = 14 }
	Text phrasing. More...

Functions
template<typename It , std::sentinel_for< It > ItEnd> requires (std::is_same_v<std::iter_value_t<It>, character>)
void	hi::v1::fixup_script (It first, ItEnd last, iso_15924 default_script) noexcept
	Fixup the iso_15924 script in text.
template<std::ranges::range R> requires (std::is_same_v<std::ranges::range_value_t<R>, character>)
void	hi::v1::fixup_script (R &str, iso_15924 default_script) noexcept
text	hi::v1::to_text (gstring_view str, character_attributes default_attributes) noexcept
template<character_attribute... Args>
text	hi::v1::to_text (gstring_view str, Args const &...args) noexcept
text	hi::v1::to_text (std::string_view str, unicode_normalize_config config, character_attributes default_attributes) noexcept
text	hi::v1::to_text (std::string_view str, character_attributes default_attributes) noexcept
template<character_attribute... Args>
text	hi::v1::to_text (std::string_view str, unicode_normalize_config config, Args const &...args) noexcept
template<character_attribute... Args>
text	hi::v1::to_text (std::string_view str, Args const &...args) noexcept
text	hi::v1::to_text_with_markup (gstring_view str, character_attributes default_attributes) noexcept
text	hi::v1::to_text_with_markup (std::string_view str, character_attributes default_attributes) noexcept
template<character_attribute... Attributes>
text	hi::v1::to_text_with_markup (gstring_view str, Attributes const &...attributes) noexcept
template<character_attribute... Attributes>
text	hi::v1::to_text_with_markup (std::string_view str, Attributes const &...attributes) noexcept
constexpr gstring	hi::v1::to_gstring (text_view str) noexcept
constexpr std::string	hi::v1::to_string (text_view str) noexcept
constexpr std::wstring	hi::v1::to_wstring (text_view str) noexcept
constexpr std::u32string	hi::v1::to_u32string (text_view str) noexcept
template<typename It , std::sentinel_for< It > ItEnd> requires (std::is_same_v<std::iter_value_t<It>, character>)
void	hi::v1::set_attributes (It first, ItEnd last, character_attributes attributes) noexcept
	Change the attributes on a piece of text.
template<typename It , std::sentinel_for< It > ItEnd, character_attribute... Args> requires (std::is_same_v<std::iter_value_t<It>, character>)
void	hi::v1::set_attributes (It first, ItEnd last, Args const &...args) noexcept
	Change the attributes on a piece of text.
template<std::ranges::range R> requires (std::is_same_v<std::ranges::range_value_t<R>, character>)
void	hi::v1::set_attributes (R &str, character_attributes attributes) noexcept
	Change the attributes on text.
template<std::ranges::range R, character_attribute... Args> requires (std::is_same_v<std::ranges::range_value_t<R>, character>)
void	hi::v1::set_attributes (R &str, Args const &...args) noexcept
	Change the attributes on text.
text	hi::v1::to_text (std::integral auto const &value) noexcept
	Convert integer to string.
text	hi::v1::to_text (std::floating_point auto const &value) noexcept
	Convert floating point to string.
template<std::integral T>
T	hi::v1::from_text (text_view str, int base=10)
	Convert a string to an integer.
template<std::floating_point T>
T	hi::v1::from_text (text_view str)
	Convert a string to an floating point.

Detailed Description

Characters

A hi::character contains the following information:

• The full unicode grapheme cluster.
• The phrasing.
• The language.
• The text-style.

The information is compressed into a 64 bit integer type so that it can be used in a std::basic_string and std::basic_string_view.

Grapheme Cluster

In linguistics, a grapheme is the smallest functional unit of a writing system. In other words a grapheme is a character from the view point of a user. A grapheme cluster is a set of Unicode code-points that together form a single grapheme.

A grapheme cluster consists of a base-character followed by zero or more combining marks; combining into an accented character or an emoji.

A grapheme cluster in HikoGUI is stored as a 24 bit integer. The values 0 through 10ffff is a single base-character code-point with zero combining marks. Values higher than this number are an index into an append-only table of grapheme-cluster which so far have been used by the application, which results in a maximum of about 24 Milion different clusters.

Phrasing

The phrasing is an enum describing the way a character is semantically used in the text. Examples of phrasings are: regular, emphesis, strong, code, abbreviation, citation, etc.

This is used to select a sub-style from a style.

Language and Script

The language and script of the run of text this character is part of.

There are several reasons to have language stored for each character:

• It is used to select a sub-style, possibly selecting different font, size, weight or color which is better fitting for different scripts and languages. For example when mixing English with Japanese text.
• Support for mixing languages when doing spell and grammar checks
• Better pronounciation of text for mixed languages.
• Text shaping rules for ligatures, glyph substitution and positioning are language and script depended.

Only the 16-bit language code nees to be stored with a character. The script code can be determine from the base code-point of the current character and surrounding haracters using a Unicode algorithm.

Text-theme and text-style

The text-theme how the text needs to be displayed. The text-style is selected from a text-theme using the: phrasing, language and script.

We store the theme in the character so that during text editing we can modify the language, phrasing and theme separately.

The text-theme is a 16 bit index into append-only table of text-theme which so far have been used by the application, which results in a maximum of about 65000 different text-themes.

Markup

String literals and text in translation files may contain markup codes to control how text is displayed.

The markup in string literals are surrounded by square brackets '[' and ']'. The commands are:

Code	Description
'[' [a-z] ']'	Select phrasing.
'[' [0-9]+ ']'	Select text-theme.
'[' <ietf language tag> ']'	Select language.
'[' '.' ']'	Set phrasing, text-theme and language back to default.
'[' '['	Literal '~'.

All other combination with brackets is reserved.

When converting text back to string with markup the canonical output is as follows:

• Commands are only output when there is a change of attributes.
• When multiple attributes change at the same time the commands are output in the order:
  • theme
  • phrasing
  • language
• If default attributes are passed to the convertion function then the "default" command must be used when able, but not for the first character (no attributes changed).

Phrasing

Command	Phrasing	Description
r	regular	Plain text
e	emphesis	Emphesized, accented voice (often italic)
s	strong	Strong, louder voice (often bold)
c	code	A piece of code (often in non-proportional font)
a	abbreviation	An abbreviation. (may have double underline to a description)
b	bold	Non-semantic bold
i	italic	Non-semantic italic
k	keyboard / control	Shown as a push button
h	highlight / mark	Highlight text (often yellow background)
m	math	Using math font
x	example	Console output (Often in non-proportional bitmap-like font)
u	unarticulated	(Often underlined)
q	quote / citation	A quote from someone (often a swirly italic or cursive font)
l	link	A hyper link (often blue and underlined)

Theme

Command	Theme	Description
0	User-interface	Text theme to use for labels on the GUI.
1	Text-field	Text theme to use for multi-line text fields.

2 - 9 | reserved | 10 - 65535 | custom |

Language

Enumeration Type Documentation

text_phrasing

enum class v1::text_phrasing : uint8_t

strong

Text phrasing.

The underlying value must be between 0 through 15.

Enumerator
regular	Regular, normal text.
emphesis	Emphesised text; spoken as if the text is special importance, significant or promonent. Often formatted in italic.
strong	Strong text; spoken louder, as if the text is not to be missed. Often formatted in bold.
code	Text is a piece of programming-code; a variable name, a function name. Often formatted in a constant-width font, with a greater weight and in a different color and possible background block, than the surrounding text.
abbreviation	An abbreviation. Sometimes formatted with a double underline and hovering will show the expansion of the abbreviation.
quote	The text is quoted from somewhere. Often formatted using a more italic / cursive font, with a lower weight.
keyboard	Used in help text to show which key or button to press. Often formatted with a background that looks raised up like a button. With the text in inverted color.
highlight	The text is marked or highlighted as if being marked by a highlight pen. Often formatted with a yellow background.
math	Text formatted as math. Often formatted using a special math font.
example	Used in help text to show an example. Often formatted using a non-proportional font with a low resultion bitmap-like style. Used for displaying console output.
unarticulated	Unarticulated. Often formatted using an underlying line.
title	Format a heading Often in bold, larger font and on a line by itself.
success	Format a "good" message Often in bright green.
warning	Format a warning message Often in bright yellow.
error	Format a "bad" message Often in bright red.

Function Documentation

fixup_script()

template<typename It , std::sentinel_for< It > ItEnd>
requires (std::is_same_v<std::iter_value_t<It>, character>)

void hi::v1::fixup_script ( It first, ItEnd last, iso_15924 default_script )

inlinenoexcept

Fixup the iso_15924 script in text.

Check the characters in text and make sure the script-attribute does not contradict the Unicode script table. And if the script-attribute for the character was not set, then determine the script.

from_text() [1/2]

template<std::floating_point T>

T hi::v1::from_text

(

text_view

str

)

Convert a string to an floating point.

This function bypasses std::locale

Template Parameters

T	The integer type.

Parameters

str	The string is an integer.

Returns

The integer converted from a string.

from_text() [2/2]

template<std::integral T>

T hi::v1::from_text	(	text_view	str,
		int	base = 10 )

Convert a string to an integer.

This function bypasses std::locale

Template Parameters

T	The integer type.

Parameters

str	The string is an integer.
base	The base radix of the string encoded integer.

Returns

The integer converted from a string.

set_attributes() [1/4]

template<typename It , std::sentinel_for< It > ItEnd, character_attribute... Args>
requires (std::is_same_v<std::iter_value_t<It>, character>)

void hi::v1::set_attributes ( It first, ItEnd last, Args const &... args )

inlinenoexcept

Change the attributes on a piece of text.

set_attributes() [2/4]

template<typename It , std::sentinel_for< It > ItEnd>
requires (std::is_same_v<std::iter_value_t<It>, character>)

void hi::v1::set_attributes ( It first, ItEnd last, character_attributes attributes )

inlinenoexcept

Change the attributes on a piece of text.

set_attributes() [3/4]

template<std::ranges::range R, character_attribute... Args>
requires (std::is_same_v<std::ranges::range_value_t<R>, character>)

void hi::v1::set_attributes ( R & str, Args const &... args )

inlinenoexcept

Change the attributes on text.

set_attributes() [4/4]

template<std::ranges::range R>
requires (std::is_same_v<std::ranges::range_value_t<R>, character>)

void hi::v1::set_attributes ( R & str, character_attributes attributes )

inlinenoexcept

Change the attributes on text.

to_text() [1/2]

text hi::v1::to_text ( std::floating_point auto const & value )

noexcept

Convert floating point to string.

This function bypasses std::locale.

Parameters

value

The signed or unsigned integer value.

Returns

The integer converted to a decimal string.

to_text() [2/2]

text hi::v1::to_text ( std::integral auto const & value )

noexcept

Convert integer to string.

This function bypasses std::locale.

Parameters

value

The signed or unsigned integer value.

Returns

The integer converted to a decimal string.