FLTK 1.4.4

Loading...
Searching...
No Matches
Unicode and UTF-8 functions

fl global Unicode and UTF-8 handling functions declared in <FL/fl_utf8.h> More...

## Macros
#define | ERRORS_TO_CP1252 1
| Set to 1 to turn bad UTF-8 bytes in the 0x80-0x9f range into the Unicode index for Microsoft's CP1252 character set.
#define | ERRORS_TO_ISO8859_1 1
| Set to 1 to turn bad UTF-8 bytes into ISO-8859-1.
#define | NBC 0xFFFF + 1
#define | STRICT_RFC3629 0
| A number of Unicode code points are in fact illegal and should not be produced by a UTF-8 converter.

## Functions
int | fl_access (const char *f, int mode)
| Cross-platform function to test a files access() with a UTF-8 encoded name or value.
int | fl_chdir (const char *path)
| Cross-platform function to change the current working directory, given as a UTF-8 encoded string.
int | fl_chmod (const char *f, int mode)
| Cross-platform function to set a files mode() with a UTF-8 encoded name or value.
int | fl_close_fd (int fd)
| Cross-platform function to close a file descriptor.
int | fl_execvp (const char *file, char *const *argv)
FILE * | fl_fopen (const char *f, const char *mode)
| Cross-platform function to open files with a UTF-8 encoded name.
char * | fl_getcwd (char *buf, int len)
| Cross-platform function to get the current working directory as a UTF-8 encoded value.
char * | fl_getenv (const char *v)
| Cross-platform function to get environment variables with a UTF-8 encoded name or value.
char | fl_make_path (const char *path)
| Cross-platform function to recursively create a path in the file system.
void | fl_make_path_for_file (const char *path)
| Cross-platform function to create a path for the file in the file system.
int | fl_mkdir (const char *f, int mode)
| Cross-platform function to create a directory with a UTF-8 encoded name.
unsigned int | fl_nonspacing (unsigned int ucs)
| Returns true if the Unicode character `ucs` is non-spacing.
int | fl_open (const char *fname, int oflags,...)
| Cross-platform function to open files with a UTF-8 encoded name.
int | fl_open_ext (const char *fname, int binary, int oflags,...)
| Cross-platform function to open files with a UTF-8 encoded name.
int | fl_putenv (const char *var)
| Cross-platform function to write environment variables with a UTF-8 encoded name or value.
int | fl_rename (const char *f, const char *n)
| Cross-platform function to rename a filesystem object using UTF-8 encoded names.
int | fl_rmdir (const char *f)
| Cross-platform function to remove a directory with a UTF-8 encoded name.
int | fl_stat (const char *f, struct stat *b)
| Cross-platform function to stat() a file using a UTF-8 encoded name or value.
int | fl_system (const char *cmd)
| Cross-platform function to run a system command with a UTF-8 encoded string.
int | fl_tolower (unsigned int ucs)
| Returns the Unicode lower case value of `ucs`.
int | fl_toupper (unsigned int ucs)
| Returns the Unicode upper case value of `ucs`.
unsigned | fl_ucs_to_Utf16 (const unsigned ucs, unsigned short *dst, const unsigned dstlen)
| Convert a single 32-bit Unicode codepoint into an array of 16-bit characters.
int | fl_unlink (const char *fname)
| Cross-platform function to unlink() (that is, delete) a file using a UTF-8 encoded filename.
char * | fl_utf2mbcs (const char *s)
| Converts UTF-8 string `s` to a local multi-byte character string.
const char * | fl_utf8back (const char *p, const char *start, const char *end)
| Move `p` backward until it points to the start of a UTF-8 character.
int | fl_utf8bytes (unsigned ucs)
| Return the number of bytes needed to encode the given UCS4 character in UTF-8.
unsigned | fl_utf8decode (const char *p, const char *end, int *len)
| Decode a single UTF-8 encoded character starting at p.
int | fl_utf8encode (unsigned ucs, char *buf)
| Write the UTF-8 encoding of ucs into buf and return the number of bytes written.
unsigned | fl_utf8from_mb (char *dst, unsigned dstlen, const char *src, unsigned srclen)
| Convert a filename from the locale-specific multibyte encoding used by Windows to UTF-8 as used by FLTK.
unsigned | fl_utf8froma (char *dst, unsigned dstlen, const char *src, unsigned srclen)
| Convert an ISO-8859-1 (ie normal c-string) byte stream to UTF-8.
unsigned | fl_utf8fromwc (char *dst, unsigned dstlen, const wchar_t *src, unsigned srclen)
| Turn "wide characters" as returned by some system calls (especially on Windows) into UTF-8.
const char * | fl_utf8fwd (const char *p, const char *start, const char *end)
| Move `p` forward until it points to the start of a UTF-8 character.
int | fl_utf8len (char c)
| Returns the byte length of the UTF-8 sequence with first byte `c`, or -1 if `c` is not valid.
int | fl_utf8len1 (char c)
| Returns the byte length of the UTF-8 sequence with first byte `c`, or 1 if `c` is not valid.
int | fl_utf8locale ()
| Return true if the "locale" seems to indicate that UTF-8 encoding is used.
int | fl_utf8strlen (const char *text, int len)
| Return the length in bytes of a UTF-8 string.
int | fl_utf8test (const char *src, unsigned srclen)
| Examines the first `srclen` bytes in `src` and returns a verdict on whether it is UTF-8 or not.
unsigned | fl_utf8to_mb (const char *src, unsigned srclen, char *dst, unsigned dstlen)
| Convert the UTF-8 used by FLTK to the locale-specific encoding used for filenames (and sometimes used for data in files).
unsigned | fl_utf8toa (const char *src, unsigned srclen, char *dst, unsigned dstlen)
| Convert a UTF-8 sequence into an array of 1-byte characters.
unsigned | fl_utf8toUtf16 (const char *src, unsigned srclen, unsigned short *dst, unsigned dstlen)
| Convert a UTF-8 sequence into an array of 16-bit characters.
unsigned | fl_utf8towc (const char *src, unsigned srclen, wchar_t *dst, unsigned dstlen)
| Converts a UTF-8 string into a wide character string.
int | fl_utf_nb_char (const unsigned char *buf, int len)
| Returns the number of Unicode chars in the UTF-8 string.
int | fl_utf_strcasecmp (const char *s1, const char *s2)
| UTF-8 aware strcasecmp - converts to Unicode and tests.
int | fl_utf_strncasecmp (const char *s1, const char *s2, int n)
| UTF-8 aware strncasecmp - converts to lower case Unicode and tests.
int | fl_utf_tolower (const unsigned char *str, int len, char *buf)
| Converts the string `str` to its lower case equivalent into buf.
int | fl_utf_toupper (const unsigned char *str, int len, char *buf)
| Converts the string `str` to its upper case equivalent into buf.
int | fl_wcwidth (const char *src)
| extended wrapper around fl_wcwidth_(unsigned int ucs) function.
int | fl_wcwidth_ (unsigned int ucs)
| Wrapper to adapt Markus Kuhn's implementation of wcwidth() for FLTK.

## Detailed Description

fl global Unicode and UTF-8 handling functions declared in <FL/fl_utf8.h>

## Macro Definition Documentation

## ◆ ERRORS_TO_CP1252

#define ERRORS_TO_CP1252 1

Set to 1 to turn bad UTF-8 bytes in the 0x80-0x9f range into the Unicode index for Microsoft's CP1252 character set.

You should also set ERRORS_TO_ISO8859_1. With this a huge amount of more available text (such as all web pages) are correctly converted to Unicode.

## ◆ ERRORS_TO_ISO8859_1

#define ERRORS_TO_ISO8859_1 1

Set to 1 to turn bad UTF-8 bytes into ISO-8859-1.

If this is zero they are instead turned into the Unicode REPLACEMENT CHARACTER, of value 0xfffd. If this is on fl_utf8decode() will correctly map most (perhaps all) human-readable text that is in ISO-8859-1. This may allow you to completely ignore character sets in your code because virtually everything is either ISO-8859-1 or UTF-8.

## ◆ STRICT_RFC3629

#define STRICT_RFC3629 0

A number of Unicode code points are in fact illegal and should not be produced by a UTF-8 converter.

Turn this on will replace the bytes in those encodings with errors. If you do this then converting arbitrary 16-bit data to UTF-8 and then back is not an identity, which will probably break a lot of software.

## Function Documentation

## ◆ fl_access()

int fl_access | ( | const char * | f,
|  | int | mode )

Cross-platform function to test a files access() with a UTF-8 encoded name or value.

This function is especially useful on the Windows platform where the standard access() function fails with UTF-8 encoded non-ASCII filenames.

Windows defines the mode values 0 for existence, 2 for writable, 4 for readable, and 6 of readable and writable. On other systems, the modes X_OK, W_OK, and R_OK are usually defined as 1, 2, and 4.

Upon successful completion, the value 0 is returned on all platforms.

Parameters

  [in] | f | the UTF-8 encoded filename
  [in] | mode | the mode to test

Returns
  the return value of _waccess() on Windows or access() on other platforms.

## ◆ fl_chdir()

int fl_chdir | ( | const char * | path | ) |

Cross-platform function to change the current working directory, given as a UTF-8 encoded string.

This function is especially useful on the Windows platform where the standard _wchdir() function needs a `path` in UTF-16 encoding.

The `path` is converted to a system specific encoding if necessary and the system specific `chdir(converted_path)` function is called.

The function returns 0 on success and -1 on error. Depending on the platform, `errno` may be set if an error occurs.

Note
  The possible errno values are platform specific. Refer to the documentation of the platform specific chdir() function.

If the function is not implemented on a particular platform the default implementation returns -1 and `errno` is not set.

If the `path` is `NULL` the function returns -1, but `errno` is not changed. This is a convenience feature of fl_chdir() as opposed to chdir().

Parameters

  [in] | path | the target directory for chdir (may be `NULL`)

Returns
  0 if successful, -1 on error (errno may be set)

## ◆ fl_chmod()

int fl_chmod | ( | const char * | f,
|  | int | mode )

Cross-platform function to set a files mode() with a UTF-8 encoded name or value.

This function is especially useful on the Windows platform where the standard chmod() function fails with UTF-8 encoded non-ASCII filenames.

Parameters

  [in] | f | the UTF-8 encoded filename
  [in] | mode | the mode to set

Returns
  the return value of _wchmod() on Windows or chmod() on other platforms.

## ◆ fl_close_fd()

int fl_close_fd | ( | int | fd | ) |

Cross-platform function to close a file descriptor.

Returns
  0 in case of success, or -1 in case of error.

## ◆ fl_fopen()

FILE * fl_fopen | ( | const char * | f,
|  | const char * | mode )

Cross-platform function to open files with a UTF-8 encoded name.

This function is especially useful on the Windows platform where the standard fopen() function fails with UTF-8 encoded non-ASCII filenames.

Parameters

  f | the UTF-8 encoded filename
  mode | same as the second argument of the standard fopen() function

Returns
  a FILE pointer upon successful completion, or NULL in case of error.

See also
  fl_open().

## ◆ fl_getcwd()

char * fl_getcwd | ( | char * | buf,
|  | int | len )

Cross-platform function to get the current working directory as a UTF-8 encoded value.

This function is especially useful on the Windows platform where the standard _wgetcwd() function returns UTF-16 encoded non-ASCII filenames.

If `buf` is `NULL` a buffer of size ``(len+1) is allocated, filled with the current working directory, and returned. In this case the buffer must be released by the caller with free() to prevent memory leaks.

Parameters

  [in] | buf | the buffer to populate (may be NULL)
  [in] | len | the length of the buffer

Returns
  the CWD encoded as UTF-8

## ◆ fl_getenv()

char * fl_getenv | ( | const char * | v | ) |

Cross-platform function to get environment variables with a UTF-8 encoded name or value.

This function is especially useful on the Windows platform where non-ASCII environment variables are encoded as wide characters. The returned value of the variable is encoded in UTF-8 as well.

On platforms other than Windows this function calls getenv directly. The return value is returned as-is.

The return value is a pointer to an implementation defined buffer:

- an internal buffer that is (re)allocated as needed (Windows) or
- the string in the environment itself (Unix, Linux, MaOS) or
- any other implementation (other platforms). This string must be considered read-only and must not be freed by the caller.

If the resultant string is to be used later it must be copied to a safe place. The next call to fl_getenv() or any other environment changes may overwrite the string.

Note
  This function is not thread-safe.

Parameters

  [in] | v | the UTF-8 encoded environment variable

Returns
  the environment variable in UTF-8 encoding, or NULL in case of error.

## ◆ fl_make_path()

char fl_make_path | ( | const char * | path | ) |

Cross-platform function to recursively create a path in the file system.

This function creates a `path` in the file system by recursively creating all directories.

Parameters

  [in] | path | a Unix style ('/' forward slashes) absolute or relative pathname

Returns
  1 if the path was created, 0 if creating the path failed at some point

## ◆ fl_make_path_for_file()

void fl_make_path_for_file | ( | const char * | path | ) |

Cross-platform function to create a path for the file in the file system.

This function strips the filename from the given `path` and creates a path in the file system by recursively creating all directories.

## ◆ fl_mkdir()

int fl_mkdir | ( | const char * | f,
|  | int | mode )

Cross-platform function to create a directory with a UTF-8 encoded name.

This function is especially useful on the Windows platform where the standard _wmkdir() function expects UTF-16 encoded non-ASCII filenames.

Parameters

  [in] | f | the UTF-8 encoded filename
  [in] | mode | the mode of the directory

Returns
  the return value of _wmkdir() on Windows or mkdir() on other platforms.

## ◆ fl_nonspacing()

unsigned int fl_nonspacing | ( | unsigned int | ucs | ) |

Returns true if the Unicode character `ucs` is non-spacing.

Non-spacing characters in Unicode are typically combining marks like tilde (~), diaeresis (¨), or other marks that are added to a base character, for instance 'a' (base character) + '¨' (combining mark) = 'ä' (German Umlaut).

- http://unicode.org/glossary/#base_character
- http://unicode.org/glossary/#nonspacing_mark
- http://unicode.org/glossary/#combining_character

## ◆ fl_open()

int fl_open | ( | const char * | fname,
|  | int | oflags,
|  |  | ... )

Cross-platform function to open files with a UTF-8 encoded name.

This function is especially useful on the Windows platform where the standard open() function fails with UTF-8 encoded non-ASCII filenames.

Parameters

  [in] | fname | the UTF-8 encoded filename
  [in] | oflags | other arguments are as in the standard open() function

Returns
  a file descriptor upon successful completion, or -1 in case of error.

See also
  fl_fopen(), fl_open_ext(), fl_close_fd(int fd).

## ◆ fl_open_ext()

int fl_open_ext | ( | const char * | fname,
|  | int | binary,
|  | int | oflags,
|  |  | ... )

Cross-platform function to open files with a UTF-8 encoded name.

In comparison with fl_open(), this function allows to control whether the file is opened in binary (a.k.a. untranslated) mode. This is especially useful on the Windows platform where files are by default opened in text (translated) mode.

Parameters

  [in] | fname | the UTF-8 encoded filename
  [in] | binary | if non-zero, the file is to be accessed in binary (a.k.a. untranslated) mode.
  [in] | oflags,... | these arguments are as in the standard open() function. Setting `oflags` to zero opens the file for reading.

Returns
  a file descriptor upon successful completion, or -1 in case of error.

## ◆ fl_putenv()

int fl_putenv | ( | const char * | var | ) |

Cross-platform function to write environment variables with a UTF-8 encoded name or value.

This function is especially useful on the Windows platform where non-ASCII environment variables are encoded as wide characters.

The given argument `var` must be encoded in UTF-8 in the form "name=value". The `'name'` part must conform to platform dependent restrictions on environment variable names.

The string given in `var` is copied and optionally converted to the required encoding for the platform. On platforms other than Windows this function calls putenv directly.

The return value is zero on success and non-zero in case of error. The value in case of error is platform specific and returned as-is.

Note
  The copied string is allocated on the heap and "lost" on some platforms, i.e. calling fl_putenv() to change environment variables frequently may cause memory leaks. There may be an option to avoid this in a future implementation.
  This function is not thread-safe.

Parameters

  [in] | var | the UTF-8 encoded environment variable `'name=value'`

Returns
  0 on success, non-zero in case of error.

## ◆ fl_rename()

int fl_rename | ( | const char * | f,
|  | const char * | n )

Cross-platform function to rename a filesystem object using UTF-8 encoded names.

This function is especially useful on the Windows platform where the standard _wrename() function expects UTF-16 encoded non-ASCII filenames.

Parameters

  [in] | f | the UTF-8 encoded filename to change
  [in] | n | the new UTF-8 encoded filename to set

Returns
  the return value of _wrename() on Windows or rename() on other platforms.

## ◆ fl_rmdir()

int fl_rmdir | ( | const char * | f | ) |

Cross-platform function to remove a directory with a UTF-8 encoded name.

This function is especially useful on the Windows platform where the standard _wrmdir() function expects UTF-16 encoded non-ASCII filenames.

Parameters

  [in] | f | the UTF-8 encoded filename to remove

Returns
  the return value of _wrmdir() on Windows or rmdir() on other platforms.

## ◆ fl_stat()

int fl_stat | ( | const char * | f,
|  | struct stat * | b )

Cross-platform function to stat() a file using a UTF-8 encoded name or value.

This function is especially useful on the Windows platform where the standard stat() function fails with UTF-8 encoded non-ASCII filenames.

Parameters

  [in] | f | the UTF-8 encoded filename
  | b | the stat struct to populate

Returns
  the return value of _wstat() on Windows or stat() on other platforms.

## ◆ fl_system()

int fl_system | ( | const char * | cmd | ) |

Cross-platform function to run a system command with a UTF-8 encoded string.

This function is especially useful on the Windows platform where non-ASCII program (file) names must be encoded as wide characters.

On platforms other than Windows this function calls system() directly.

Parameters

  [in] | cmd | the UTF-8 encoded command string

Returns
  the return value of _wsystem() on Windows or system() on other platforms.

## ◆ fl_ucs_to_Utf16()

unsigned fl_ucs_to_Utf16 | ( | const unsigned | ucs,
|  | unsigned short * | dst,
|  | const unsigned | dstlen )

Convert a single 32-bit Unicode codepoint into an array of 16-bit characters.

These are used by some system calls, especially on Windows.

`ucs` is the value to convert.

`dst` points at an array to write, and `dstlen` is the number of locations in this array. At most `dstlen` words will be written, and a 0 terminating word will be added if `dstlen` is large enough. Thus this function will never overwrite the buffer and will attempt return a zero-terminated string if space permits. If `dstlen` is zero then `dst` can be set to NULL and no data is written, but the length is returned.

The return value is the number of 16-bit words that would be written to `dst` if it is large enough, not counting any terminating zero.

If the return value is greater than `dstlen` it indicates truncation, you should then allocate a new array of size return+1 and call this again.

Unicode characters in the range 0x10000 to 0x10ffff are converted to "surrogate pairs" which take two words each (in UTF-16 encoding). Typically, setting `dstlen` to 2 will ensure that any valid Unicode value can be converted, and setting `dstlen` to 3 or more will allow a NULL terminated sequence to be returned.

## ◆ fl_unlink()

int fl_unlink | ( | const char * | fname | ) |

Cross-platform function to unlink() (that is, delete) a file using a UTF-8 encoded filename.

This function is especially useful on the Windows platform where the standard function expects UTF-16 encoded non-ASCII filenames.

Parameters

  fname | the filename to unlink

Returns
  the return value of _wunlink() on Windows or unlink() on other platforms.

## ◆ fl_utf8back()

const char * fl_utf8back | ( | const char * | p,
|  | const char * | start,
|  | const char * | end )

Move `p` backward until it points to the start of a UTF-8 character.

If it already points at the start of one then it is returned unchanged. Any UTF-8 errors are treated as though each byte of the error is an individual character.

start is the start of the string and is used to limit the backwards search for the start of a UTF-8 character.

end is the end of the string and is assumed to be a break between characters. It is assumed to be greater than p.

If you wish to decrement a UTF-8 pointer, pass p-1 to this.

## ◆ fl_utf8bytes()

int fl_utf8bytes | ( | unsigned | ucs | ) |

Return the number of bytes needed to encode the given UCS4 character in UTF-8.

Returns number of bytes that utf8encode() will use to encode the character `ucs`.

Parameters

  [in] | ucs | UCS4 encoded character

Returns
  number of bytes required

## ◆ fl_utf8decode()

unsigned fl_utf8decode | ( | const char * | p,
|  | const char * | end,
|  | int * | len )

Decode a single UTF-8 encoded character starting at p.

The resulting Unicode value (in the range 0-0x10ffff) is returned, and len is set to the number of bytes in the UTF-8 encoding (adding len to p will point at the next character).

If `p` points at an illegal UTF-8 encoding, including one that would go past end, or where a code uses more bytes than necessary, then (unsigned char)p is translated as though it is in the Microsoft CP1252 character set and len is set to 1. Treating errors this way allows this to decode almost any ISO-8859-1 or CP1252 text that has been mistakenly placed where UTF-8 is expected, and has proven very useful.

If you want errors to be converted to error characters (as the standards recommend), adding a test to see if the length is unexpectedly 1 will work:

if (*p & 0x80) { // what should be a multibyte encoding
code = fl_utf8decode(p,end,&len);
if (len<2) code = 0xFFFD; // Turn errors into REPLACEMENT CHARACTER
} else { // handle the 1-byte UTF-8 encoding:
code = *p;
len = 1;
}
fl_utf8decode
unsigned fl_utf8decode(const char *p, const char *end, int *len)
Decode a single UTF-8 encoded character starting at p.
Definition fl_utf8.cxx:724

Direct testing for the 1-byte case (as shown above) will also speed up the scanning of strings where the majority of characters are ASCII.

## ◆ fl_utf8encode()

int fl_utf8encode | ( | unsigned | ucs,
|  | char * | buf )

Write the UTF-8 encoding of ucs into buf and return the number of bytes written.

Up to 4 bytes may be written. If you know that `ucs` is less than 0x10000 then at most 3 bytes will be written. If you wish to speed this up, remember that anything less than 0x80 is written as a single byte.

If ucs is greater than 0x10ffff this is an illegal character according to RFC 3629. These are converted as though they are 0xFFFD (REPLACEMENT CHARACTER).

RFC 3629 also says many other values for `ucs` are illegal (in the range 0xd800 to 0xdfff, or ending with 0xfffe or 0xffff). However I encode these as though they are legal, so that utf8encode/fl_utf8decode will be the identity for all codes between 0 and 0x10ffff.

## ◆ fl_utf8from_mb()

unsigned fl_utf8from_mb | ( | char * | dst,
|  | unsigned | dstlen,
|  | const char * | src,
|  | unsigned | srclen )

Convert a filename from the locale-specific multibyte encoding used by Windows to UTF-8 as used by FLTK.

Up to `dstlen` bytes are written to `dst`, including a null terminator. The return value is the number of bytes that would be written, not counting the null terminator. If greater or equal to `dstlen` then if you malloc a new array of size n+1 you will have the space needed for the entire string. If `dstlen` is zero then nothing is written and this call just measures the storage space needed.

On Unix or on Windows when a UTF-8 locale is in effect, this does not change the data. You may also want to check if fl_utf8test() returns non-zero, so that the filesystem can store filenames in UTF-8 encoding regardless of the locale.

## ◆ fl_utf8froma()

unsigned fl_utf8froma | ( | char * | dst,
|  | unsigned | dstlen,
|  | const char * | src,
|  | unsigned | srclen )

Convert an ISO-8859-1 (ie normal c-string) byte stream to UTF-8.

It is possible this should convert Microsoft's CP1252 to UTF-8 instead. This would translate the codes in the range 0x80-0x9f to different characters. Currently it does not do this.

Up to `dstlen` bytes are written to `dst`, including a null terminator. The return value is the number of bytes that would be written, not counting the null terminator. If greater or equal to `dstlen` then if you malloc a new array of size n+1 you will have the space needed for the entire string. If `dstlen` is zero then nothing is written and this call just measures the storage space needed.

`srclen` is the number of bytes in `src` to convert.

If the return value equals `srclen` then this indicates that no conversion is necessary, as only ASCII characters are in the string.

## ◆ fl_utf8fromwc()

unsigned fl_utf8fromwc | ( | char * | dst,
|  | unsigned | dstlen,
|  | const wchar_t * | src,
|  | unsigned | srclen )

Turn "wide characters" as returned by some system calls (especially on Windows) into UTF-8.

Up to `dstlen` bytes are written to `dst`, including a null terminator. The return value is the number of bytes that would be written, not counting the null terminator. If greater or equal to `dstlen` then if you malloc a new array of size n+1 you will have the space needed for the entire string. If `dstlen` is zero then nothing is written and this call just measures the storage space needed.

`srclen` is the number of words in `src` to convert. On Windows this is not necessarily the number of characters, due to there possibly being "surrogate pairs" in the UTF-16 encoding used. On Unix wchar_t is 32 bits and each location is a character.

On Unix if a `src` word is greater than 0x10ffff then this is an illegal character according to RFC 3629. These are converted as though they are 0xFFFD (REPLACEMENT CHARACTER). Characters in the range 0xd800 to 0xdfff, or ending with 0xfffe or 0xffff are also illegal according to RFC 3629. However I encode these as though they are legal, so that fl_utf8towc will return the original data.

On Windows "surrogate pairs" are converted to a single character and UTF-8 encoded (as 4 bytes). Mismatched halves of surrogate pairs are converted as though they are individual characters.

## ◆ fl_utf8fwd()

const char * fl_utf8fwd | ( | const char * | p,
|  | const char * | start,
|  | const char * | end )

Move `p` forward until it points to the start of a UTF-8 character.

If it already points at the start of one then it is returned unchanged. Any UTF-8 errors are treated as though each byte of the error is an individual character.

start is the start of the string and is used to limit the backwards search for the start of a UTF-8 character.

end is the end of the string and is assumed to be a break between characters. It is assumed to be greater than p.

This function is for moving a pointer that was jumped to the middle of a string, such as when doing a binary search for a position. You should use either this or fl_utf8back() depending on which direction your algorithm can handle the pointer moving. Do not use this to scan strings, use fl_utf8decode() instead.

## ◆ fl_utf8len()

int fl_utf8len | ( | char | c | ) |

Returns the byte length of the UTF-8 sequence with first byte `c`, or -1 if `c` is not valid.

This function is helpful for finding faulty UTF-8 sequences.

See also
  fl_utf8len1

## ◆ fl_utf8len1()

int fl_utf8len1 | ( | char | c | ) |

Returns the byte length of the UTF-8 sequence with first byte `c`, or 1 if `c` is not valid.

This function can be used to scan faulty UTF-8 sequences, albeit ignoring invalid codes.

See also
  fl_utf8len

## ◆ fl_utf8locale()

int fl_utf8locale | ( | void |  | ) |

Return true if the "locale" seems to indicate that UTF-8 encoding is used.

If true the fl_utf8to_mb and fl_utf8from_mb don't do anything useful.

It is highly recommended that you change your system so this does return true. On Windows this is done by setting the "codepage" to CP_UTF8. On Unix this is done by setting $LC_CTYPE to a string containing the letters "utf" or "UTF" in it, or by deleting all $LC* and $LANG environment variables. In the future it is likely that all non-Asian Unix systems will return true, due to the compatibility of UTF-8 with ISO-8859-1.

## ◆ fl_utf8strlen()

int fl_utf8strlen | ( | const char * | text,
|  | int | len )

Return the length in bytes of a UTF-8 string.

Parameters

  [in] | text | encoded in UTF-8
  [in] | len | number of Unicode characters, -1 to test until the end of text

Returns
  number of bytes that make up the Unicode string

See also
  fl_utf_nb_char(const unsigned char *buf, int len)

## ◆ fl_utf8test()

int fl_utf8test | ( | const char * | src,
|  | unsigned | srclen )

Examines the first `srclen` bytes in `src` and returns a verdict on whether it is UTF-8 or not.

- Returns 0 if there is any illegal UTF-8 sequences, using the same rules as fl_utf8decode(). Note that some UCS values considered illegal by RFC 3629, such as 0xffff, are considered legal by this.
- Returns 1 if there are only single-byte characters (ie no bytes have the high bit set). This is legal UTF-8, but also indicates plain ASCII. It also returns 1 if `srclen` is zero.
- Returns 2 if there are only characters less than 0x800.
- Returns 3 if there are only characters less than 0x10000.
- Returns 4 if there are characters in the 0x10000 to 0x10ffff range.

Because there are many illegal sequences in UTF-8, it is almost impossible for a string in another encoding to be confused with UTF-8. This is very useful for transitioning Unix to UTF-8 filenames, you can simply test each filename with this to decide if it is UTF-8 or in the locale encoding. My hope is that if this is done we will be able to cleanly transition to a locale-less encoding.

## ◆ fl_utf8to_mb()

unsigned fl_utf8to_mb | ( | const char * | src,
|  | unsigned | srclen,
|  | char * | dst,
|  | unsigned | dstlen )

Convert the UTF-8 used by FLTK to the locale-specific encoding used for filenames (and sometimes used for data in files).

Unfortunately due to stupid design you will have to do this as needed for filenames. This is a bug on both Unix and Windows.

Up to `dstlen` bytes are written to `dst`, including a null terminator. The return value is the number of bytes that would be written, not counting the null terminator. If greater or equal to `dstlen` then if you malloc a new array of size n+1 you will have the space needed for the entire string. If `dstlen` is zero then nothing is written and this call just measures the storage space needed.

If fl_utf8locale() returns true then this does not change the data.

## ◆ fl_utf8toa()

unsigned fl_utf8toa | ( | const char * | src,
|  | unsigned | srclen,
|  | char * | dst,
|  | unsigned | dstlen )

Convert a UTF-8 sequence into an array of 1-byte characters.

If the UTF-8 decodes to a character greater than 0xff then it is replaced with '?'.

Errors in the UTF-8 sequence are converted as individual bytes, same as fl_utf8decode() does. This allows ISO-8859-1 text mistakenly identified as UTF-8 to be printed correctly (and possibly CP1252 on Windows).

`src` points at the UTF-8 sequence, and `srclen` is the number of bytes to convert.

Up to `dstlen` bytes are written to `dst`, including a null terminator. The return value is the number of bytes that would be written, not counting the null terminator. If greater or equal to `dstlen` then if you malloc a new array of size n+1 you will have the space needed for the entire string. If `dstlen` is zero then nothing is written and this call just measures the storage space needed.

## ◆ fl_utf8toUtf16()

unsigned fl_utf8toUtf16 | ( | const char * | src,
|  | unsigned | srclen,
|  | unsigned short * | dst,
|  | unsigned | dstlen )

Convert a UTF-8 sequence into an array of 16-bit characters.

These are used by some system calls, especially on Windows.

`src` points at the UTF-8, and `srclen` is the number of bytes to convert.

`dst` points at an array to write, and `dstlen` is the number of locations in this array. At most `dstlen-1` words will be written there, plus a 0 terminating word. Thus this function will never overwrite the buffer and will always return a zero-terminated string. If `dstlen` is zero then `dst` can be null and no data is written, but the length is returned.

The return value is the number of 16-bit words that would be written to `dst` if it were long enough, not counting the terminating zero. If the return value is greater or equal to `dstlen` it indicates truncation, you can then allocate a new array of size return+1 and call this again.

Errors in the UTF-8 are converted as though each byte in the erroneous string is in the Microsoft CP1252 encoding. This allows ISO-8859-1 text mistakenly identified as UTF-8 to be printed correctly.

Unicode characters in the range 0x10000 to 0x10ffff are converted to "surrogate pairs" which take two words each (this is called UTF-16 encoding).

## ◆ fl_utf8towc()

unsigned fl_utf8towc | ( | const char * | src,
|  | unsigned | srclen,
|  | wchar_t * | dst,
|  | unsigned | dstlen )

Converts a UTF-8 string into a wide character string.

This function generates 32-bit wchar_t (e.g. "ucs4" as it were) except on Windows where it is equivalent to fl_utf8toUtf16 and returns UTF-16.

`src` points at the UTF-8, and `srclen` is the number of bytes to convert.

`dst` points at an array to write, and `dstlen` is the number of locations in this array. At most `dstlen-1` wchar_t will be written there, plus a 0 terminating wchar_t.

The return value is the number of wchar_t that would be written to `dst` if it were long enough, not counting the terminating zero. If the return value is greater or equal to `dstlen` it indicates truncation, you can then allocate a new array of size return+1 and call this again.

Notice that sizeof(wchar_t) is 2 on Windows and is 4 on Linux and most other systems. Where wchar_t is 16 bits, Unicode characters in the range 0x10000 to 0x10ffff are converted to "surrogate pairs" which take two words each (this is called UTF-16 encoding). If wchar_t is 32 bits this rather nasty problem is avoided.

Note that Windows includes Cygwin, i.e. compiled with Cygwin's POSIX layer (cygwin1.dll, –enable-cygwin), either native (GDI) or X11.

## ◆ fl_utf_nb_char()

int fl_utf_nb_char | ( | const unsigned char * | buf,
|  | int | len )

Returns the number of Unicode chars in the UTF-8 string.

See also
  fl_utf8strlen(const char *text, int len)

## ◆ fl_utf_strcasecmp()

int fl_utf_strcasecmp | ( | const char * | s1,
|  | const char * | s2 )

UTF-8 aware strcasecmp - converts to Unicode and tests.

Returns
  result of comparison

Return values

  0 | if the strings are equal
  1 | if s1 is greater than s2
  -1 | if s1 is less than s2

## ◆ fl_utf_strncasecmp()

int fl_utf_strncasecmp | ( | const char * | s1,
|  | const char * | s2,
|  | int | n )

UTF-8 aware strncasecmp - converts to lower case Unicode and tests.

Parameters

  s1,s2 | the UTF-8 strings to compare
  n | the maximum number of UTF-8 characters to compare

Returns
  result of comparison

Return values

  0 | if the strings are equal
  >0 | if s1 is greater than s2
  <0 | if s1 is less than s2

## ◆ fl_utf_tolower()

int fl_utf_tolower | ( | const unsigned char * | str,
|  | int | len,
|  | char * | buf )

Converts the string `str` to its lower case equivalent into buf.

Warning: to be safe buf length must be at least 3 * len [for 16-bit Unicode]

## ◆ fl_utf_toupper()

int fl_utf_toupper | ( | const unsigned char * | str,
|  | int | len,
|  | char * | buf )

Converts the string `str` to its upper case equivalent into buf.

Warning: to be safe buf length must be at least 3 * len [for 16-bit Unicode]

## ◆ fl_wcwidth()

int fl_wcwidth | ( | const char * | src | ) |

extended wrapper around fl_wcwidth_(unsigned int ucs) function.

Parameters

  [in] | src | pointer to start of UTF-8 byte sequence

Returns
  width of character in columns

Depending on build options, this function may map C1 control characters (0x80 to 0x9f) to CP1252, and return the width of that character instead. This is not the same behaviour as fl_wcwidth_(unsigned int ucs) .

Note that other control characters and DEL will still return -1, so if you want different behaviour, you need to test for those characters before calling fl_wcwidth(), and handle them separately.

## ◆ fl_wcwidth_()

int fl_wcwidth_ | ( | unsigned int | ucs | ) |

Wrapper to adapt Markus Kuhn's implementation of wcwidth() for FLTK.

Parameters

  [in] | ucs | Unicode character value

Returns
  width of character in columns

See http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c for Markus Kuhn's original implementation of wcwidth() and wcswidth() (defined in IEEE Std 1002.1-2001) for Unicode.

WARNING: this function returns widths for "raw" Unicode characters. It does not even try to map C1 control characters (0x80 to 0x9F) to CP1252, and C0/C1 control characters and DEL will return -1. You are advised to use fl_width(const char* src) instead.