Commit f1afb531 authored by Egmont Koblinger's avatar Egmont Koblinger

BiDi specs, v0.1

parent 3d4e9f89
Pipeline #14285 passed with stage
in 50 seconds
---
title: Introduction
layout: default
nav_order: 10
permalink: /
---
# Introduction
{:.no_toc}
* TOC
{:toc}
## Foreword
Handling right-to-left (RTL) or bi-directional (BiDi) text is an often
requested feature for various terminal emulators. As I'll show later, I
am not satisfied with any existing documentation or implementation I
could find in this topic. Therefore I decided to come up with a new
proposal (built on top of prior work by others), and implement it in the
VTE terminal emulation widget.
I aim to come up with a design that is clean, simple, practical, follows
best practices, and is easily implementable by terminal emulators and
terminal-based applications.
Some areas I deliberately leave open for the time being. By implementing
the basics in a few key emulators and apps, we'll gather real life
experience for the missing bits. Ideally this specification will receive
subsequent versions that are (reasonably) backwards compatible with
earlier ones.
This specification is still work in progress. Especially during the
early days of adoption, it might receive backward incompatible updates.
The topic of BiDi in terminals has been unsolved for decades, I'd much
rather have occasional breakages during the first few years of adoption
than carry a faulty decision and its consequences for decades to come.
In the (hopefully unlikely) event of spotting a bug or bad design, the
specification will be fixed and implementations will be expected to
adjust reasonably quickly.
## Conventions in this document
### Fake RTL
It's a standard convention in examples to use lowercase letters for LTR
and uppercase English letters for "fake RTL", since obviously readers of
this document and developers of relevant software aren't expected to
read any of these scripts or speak any of these languages. Example:
this is an english sentence ending in SDROW WERBEH EMOS.
ECMA TR/53 uses opposite casing.
Whenever I say "English", as well as "Arabic" or "Hebrew" (pretty
randomly) throughout the document (except when speaking of Arabic
shaping), please treat them as "any LTR script" and "any RTL script".
### Applications, utilities
Whenever I say "application" ("app" for short) or "utility", unless
otherwise stated or otherwise obvious from context, I refer to an
application or utility running *inside* the terminal emulator (e.g.
bash, cat, mc, vim, emacs, tmux, cowsay... you name it). I tend to use
"utility" for simpler ones and "application" for the fullscreen ones
(that is, using the entire canvas of the terminal emulator).
### Terminal, emulator
I use "terminal", "emulator" and "terminal emulator" interchangeably for
the very same thing. It's typical loose wording to say "terminal" when
it refers to a graphical emulator and not a hardware one.
---
title: RTL and BiDi Introduction
layout: default
nav_order: 20
has_children: true
permalink: /bidi-intro/
---
# RTL and BiDi Introduction
---
title: Think RTL
layout: default
nav_order: 10
parent: RTL and BiDi Introduction
permalink: /bidi-intro/think-rtl.html
---
# Think RTL
People who grow up seeing RTL scripts don't just read/write, they also
*think* in that direction most of the time. For example, if two pictures
are placed next to each other depicting the old and new state of
something, the old one should go to the right (the first place), and the
new one to the left.
As a rule of thumb, if an application was designed to have a certain UI,
it should be entirely flipped for RTL scripts. The left-side navbar
should become right-side, the "File Edit ..." menu should be "File" on
the very right, followed by "Edit" to its left, etc.
It's a good practice to try to get rid of any left-to-right
preconception. Never think of LTR as "normal" and RTL as "reverse" or
"exceptional". They should be treated equally. Try not to think of
memory addresses, elements of an array, characters of a logical string
being stored from left to right (even though that's how the BiDi spec
and this document denotes them for practical purposes); try to think of
them for example as top-down, or close-far along the Z axis to help
yourself spot the cases where you'd subconsciously take LTR granted as
the default. Or if you'd prefer to still think of them as left-to-right,
think of it as "LTR" within a different namespace, unrelated to the
"LTR" or "RTL" properties in the displayed content's namespace.
Given an overall direction, it's okay to speak of "normal direction"
(content that matches the overall direction) and "foreign (reverse)
direction" (LTR content embedded in an RTL context, or vice versa).
While LTR scripts are solely LTR most of the time, RTL needs to come
hand-in-hand with BiDi, probably there's no such thing as pure RTL text
direction. Numbers are written LTR, that is, in foreign direction. Maybe
that's the only exception if let's say you read a novel \[info needed\].
In the digital world, especially terminal emulation (shell prompt,
commands etc.) it's pretty unlikely for a text to be RTL only, it
probably contains quite a few English words which need to be written
LTR.
---
title: RTL UI handling in general
layout: default
nav_order: 20
parent: RTL and BiDi Introduction
permalink: /bidi-intro/rtl-ui.html
---
# RTL UI handling in general
In all well designed systems, an overall RTL user interface is just a
*display* property. Let's take an example: with English settings your
graphical application consists of three columns, from left to right: 1,
2, 3. You probably build up an HTML DOM, or a widget hierarchy with them
being the children of their container in this order.
Whatever contents you have in the left column, you want that to be in
the right column in RTL locales, and vice versa. The visual order should
be 3 on the left, 2 in the center, 1 on the right. However, the logical
order is still 1, 2, 3 (your eyes should scan the screen from right to
left), and so should be in your software, in the DOM or widget
hierarchy. The application's business logic shouldn't require any
change. It's solely a display property telling the browser or the
graphical toolkit to lay them out in opposite order.
(Story: About 10 years ago, in HTML/CSS an element could have a
background image aligned to the left, to the right, or to the left with
an additional offset. It was not possible to align it to the right with
an additional offset. If an LTR-only webpage used left alignment with an
offset, it couldn't easily be mirrored to RTL, a different technical
solution had to be chosen (involving modifications to the DOM, CSS and
JS code). This limitation was soon after fixed in CSS.)
Those who RTL-ify a terminal based application should also be able to
easily follow this principle. They should be able to add RTL support as
a display-only property, not affecting the business logic. The only way
to do that is if we make terminal emulators fully symmetrical for the
two display directions.
This diff is collapsed.
---
title: Why terminals are a truly special story
layout: default
nav_order: 40
parent: RTL and BiDi Introduction
permalink: /bidi-intro/why-terminals-are-special.html
---
# Why terminals are a truly special story
Word processors, browsers etc. see the entire document, and display a
part thereof. Other parts of the text might be scrolled out, obscured by
a popup etc., these are no problem. The entire text's visual layout can
still be constructed, and only the relevant parts end up being
displayed.
A terminal emulator is only aware of the part that it displays, and not
what's "outside" or "underneath". E.g. if a line of a text file contains
100 characters, the text viewer or editor is in "unwrap" mode, and the
emulator window is 80 characters wide, then the emulator has no idea
about the remaining 20 characters of that line.
In most terminal-based applications cropping is essential. These apps,
unlike graphical ones, cannot enforce a minimum window size. Cropping is
required to squeeze any text into a smaller viewport. A text viewer or
editor needs to crop all the lines that don't currently fit in the view
(e.g. the 100 character line to an 80 character string in the previous
example), a file manager needs to crop all the long filenames, etc. With
BiDi in the game, cropping is tricky. Cropping the in-memory string and
then applying the BiDi algorithm produces faulty visual layout; cropping
needs to be performed on the BiDi algorithm's result. The BiDi algorithm
cannot be run on partial data, it must be run on the entire paragraph,
and hence can only be run by someone who is aware of the entire
paragraph. The only such component is the application.
The same applies for shaping, too. Shaping depends on the neighbor
characters which are not always known by the terminal emulator. Hence it
can only be done by the emitting application, either by printing the
desired "presentation form" variants, or by inventing other means of
sending the result to the terminal emulator.
In many other cases, however, a simple utility producing some output
doesn't want to and cannot reasonably deal with BiDi, it just wants the
terminal emulator do it. Let's take possibly the simplest utilities,
like echo or cat, and imagine that these (or their BiDi-aware
counterparts) would need to check whether their output goes to a
terminal, query its width, run the BiDi algorithm, run the shaping
algorithm, wrap the text (taking care of TAB characters and friends),
align the text... and the result would still not be okay if one resizes
the terminal afterwards. Clearly not a feasible approach.
There isn't a single mode of operation that can cover all use cases,
we've just seen that we'll need two substantially different modes.
With graphical applications, it's the responsibility of one single
application to do BiDi rendering, i.e. to convert the external data it
handles (e.g. document, web page) along with its own UI to the
pixel-by-pixel user-visible representation. In case of the terminal
emulator, it's the joint responsibility of two components: the emulator,
and the application inside. The exact responsibility of each party and
the interface between them needs to be well thought out.
Split responsibility is fragile: the two parties have to *exactly* agree
on the details. What if, for example, they implement a different version
of the BiDi algorithm, or use a different Unicode character database?
Some of the BiDi done by the app and at the same some other parts done
by the terminal sounds bad design. Ideally at any point in time the
entire responsibility should be at one place -- either completely at the
terminal, or completely at the app.
Web pages, graphical applications have different means for handling
non-text elements (overall RTL UI by reversing the order of children)
versus text (whatever the BiDi algorithm dictates). Terminal emulators
have to flatten these two into the same level, as terminal-based apps
can only implement any UI using text. This one level has to fulfill the
requirements of both.
Terminal based apps often have a pretty dense UI, sometimes adjacent UI
elements only being separated by different attributes (typically
different background color), without a space between. Think of e.g. the
bottom bar of htop or mc (`9PullDn10Quit` as a single "word"). So the
problem with any approach relying solely on the *resolved embedding
levels* (we'll see later) is a real issue.
Terminals are a legacy story of 50+ years, with additional layers
(screen handling libraries etc.) written during these times, none of
which had BiDi or shaping in mind. We'd need to add BiDi support in a
backwards compatible way, without redesigning everything from the
grounds up.
We must be careful not to end up with a standard that no one implements
for decades (as it happened to ECMA TR/53), but with one that's simple
enough and good enough for many terminal emulators and applications to
adopt.
---
title: The Recommendation
layout: default
nav_order: 30
has_children: true
permalink: /recommendation/
---
# The Recommendation
---
title: Paragraphs
layout: default
nav_order: 10
parent: The Recommendation
permalink: /recommendation/paragraphs.html
---
# Paragraphs
The BiDi algorithm operates on *paragraphs* of texts.
In terminal emulators, the reasonable definition of paragraphs, hence
the one we will go with, is the segments delimited by explicit newline
characters (I'll sometimes refer to these as *hard newlines*). A
paragraph appears as one or more rows of the emulator.
In the unlikely case that an emulator doesn't yet have this concept, it
will have to introduce and track it for the sake of BiDi.
Most terminal emulators already track which of the lines were terminated
by a hard newline and which by a character wrapping to the next line.
They use this information e.g. for selecting text on double or triple
mouse click, copying the selected text to the clipboard, or when
autodetecting URLs. Emulators that rewrap their contents on resize also
use this very same concept of paragraphs for this operation.
On the *alternate* screen, most apps (or libraries) produce output where
each line ends in a hard newline, hence paragraphs become equivalent to
lines. This is just a convention, not a technical necessity. Even on the
*normal* screen, paragraphs typically consist of a few lines only.
---
title: BiDi is display-only
layout: default
nav_order: 20
parent: The Recommendation
permalink: /recommendation/display-only.html
---
# BiDi is display-only
Terminal emulators have an in-memory concept about their contents. This
is called "data layer" by ECMA TR/53, I tend to call it "model". All the
incoming data (characters, escape sequences) operate on this layer. My
BiDi proposal only makes a few necessary additions and tiny corrections
to this layer, but leaves the emulation behavior essentially unchanged,
backwards compatible.
Then every once in a while (conforming to the human eye's perception)
terminal emulators update their display. (Updating the display after
every change in the model would be unbearably expensive, and BiDi would
probably make it even significantly worse.)
Traditionally, without BiDi support, the in-memory cells are simply
displayed from left to right. This is what we're about to change. With
BiDi support, the cells can be shuffled in various ways for display
purposes, plus might get mirrored. This operation is called
"presentation process" in TR/53, I tend to call it "transformation",
"mapping" or "shuffling" (the latter one being loose wording because
shuffling is the key component, but there's more to this story such as
mirroring certain glyphs).
The result of this transformation is called the "presentation layer" by
TR/53, I'll often call this "view", although see soon for a better
definition.
Note that I'm not making any distinction between the "character view"
consisting of characters (Unicode codepoints) along with attributes
(such as the mirroring property) after the BiDi algorithm laid out the
model's cells in their visual order, versus the "pixel view" as the
rendered canvas with beautiful glyphs visible to the user; and
apparently neither does TR/53 for its "presentation layer". I'm sure it
won't cause any confusion. It's up for terminal emulators to come up
with internal terminology for these two if they wish to.
The result of the transformation is sufficient for displaying the
contents to the user. Some meta operations, however, such as selecting
with the mouse (for copy-pasting), might need to look back at the
*model* and the *transformation* (to operate on the *model* (logical
order) rather than the *view* (visual order) in our forthcoming
*implicit* mode). As such, it's probably unfortunate to think of the
*view* as just the *result* of the *model* undergoing some
*transformation*. It's probably better to redefine the *view* as the
*model plus transformation* (that is, automatically having access not
just to the result, but also to everything contained in the model and
everything done by the transformation).
Various pieces of existing documentation about terminals often use the
words "left" or "right". To be pedantic, from now on they should be
interpreted as "towards preceding columns" and "towards subsequent
columns" since the in-memory *model* doesn't have intrinsic left or
right directions; or if you prefer to still think of in-memory order
being represented from left to right, it should be emphasized that these
"left" and "right" words of the specifications from now on operate on
the *model*, and actually may end up being displayed differently.
On *conformance level 1*, my recommendation essentially leaves the
model, i.e. the emulation logic unmodified. The only tiny changes are
that the concept of *paragraph* needs to be introduced (in the unlikely
case that it isn't already), in some new cases a paragraph needs to be
broken into two, and a few new bits need to be tracked and remembered
for each paragraph.
On to-be-designed *conformance level 2*, some new emulation requirements
will be introduced to track BiDi control characters (similarly to
combining accents) or other additional information, but this still
leaves the emulation behavior backward compatible.
The majority of the BiDi feature goes to how the model is transformed to
the view. The model's cells are no longer necessarily displayed from
left to right, we define several possible modes.
Each *paragraph* is in one of these modes. As mentioned in the generic
BiDi introduction, the cells are always shuffled within a single line
(and some might even get mirrored), although *how* they are shuffled
(and mirrored) might depend on the contents of the entire paragraph.
In all the modes, whenever a mouse event is reported back to an app, it
undergoes the inverse transformation (i.e. the reported column is the
*model's* column corresponding to the mouse position).
---
title: The basic modes
layout: default
nav_order: 30
parent: The Recommendation
permalink: /recommendation/basic-modes.html
---
# The basic modes
{:.no_toc}
* TOC
{:toc}
## 6 modes in the model
In the data layer (model), the following three new essential *BiDi
properties* are tracked for each *paragraph*:
- Implicit or explicit
- Direction autodetection enabled or disabled
- LTR or RTL
Other properties might also be introduced, e.g. see later the ncurses
compatibility mode for mirrored box drawing characters.
The core idea behind *implicit* and *explicit* modes is the same as in
ECMA TR/53 (discussed later), however, many details are different. *My
explicit mode (level 1)* is a small subset of *TR/53's explicit* mode.
*My implicit mode level 1* corresponds to *TR/53's implicit* mode. *My
implicit mode level 2* replaces the features that I didn't like and
didn't adopt from *TR/53's explicit* mode.
Unless otherwise specified, in this document *implicit* and *explicit*
modes refer to *my* definition.
The combination of the BiDi properties results in the following 6
possible main modes for each paragraph, as it's stored in the model:
- Implicit, LTR
- Implicit, RTL
- Implicit, autodetected direction with LTR fallback
- Implicit, autodetected direction with RTL fallback
- Explicit, LTR
- Explicit, RTL
The *autodetection* flag is ignored in *explicit* mode.
Based on these BiDi properties and possibly the paragraph's contents
too, each paragraph is then *transformed* to its view in one of 4
possible ways.
## 4 ways of transformation
For each paragraph, there are 4 possible ways the cells of its lines
could be arranged.
One of the two flags is the implicit vs. explicit property, as stored in
the model.
The other flag is the direction. When in *explicit* mode OR
*autodetection disabled*, the model's corresponding flag is used
directly. When in *implicit* mode AND *autodetection enabled*, it's
autodetected (details soon) based on the paragraph's contents, still
possibly falling back to the model's corresponding flag.
This results in the following four possible ways of laying out the
cells:
- Implicit LTR
- Implicit RTL
- Explicit LTR
- Explicit RTL
## Implicit mode (level 1)
In implicit mode, doing BiDi and shaping are the sole responsibilities
of the *terminal emulator*, to do it for applications that don't do it
themselves. The transferred data stream contains the text in its
*logical* order.
This mode is designed for *BiDi-unaware* apps, typically simple
utilities that just print their data to the standard output/error
without sophisticated screen handling.
The most basic expectation from those who ask for BiDi support is for
simple commands such as a `cat file` to show its Arabic, Hebrew etc.
contents in a readable way. This is what PuTTY, Konsole, mlterm,
Terminal.app and probably a few other emulators address to some extent,
and this is what both *ECMA's* and *my implicit* mode addresses too.
For these kinds of utilities there's no reasonable way to inject special
instructions to enter (and later leave) implicit mode. Hence, contrary
to ECMA TR/53 and ECMA 48, this implicit mode must be the default.
On level 1, BiDi control characters are discarded. Strings that rely on
them won't show up properly. In a future revision, implicit mode level 2
might be designed that remembers them to some extent (somewhat similarly
to combining accents) and takes into account when running the BiDi
algorithm.
### Autodetecting the direction
In some cases having a strict predefined paragraph direction provides
the better user experience. For example, when printing lists (e.g. the
output of an `ls` command), it's generally preferred if foreign
direction items are still aligned as normal direction ones. In some
other cases it's better to have it autodetected, typically when printing
larger amount of foreign direction text. (PuTTY does autodetection,
Konsole and Terminal.app don't. Mlterm offers a command line switch.)
Let our users and utilities choose.
For *implicit* paragraphs that have *autodetection enabled*, their base
direction is autodetected (guessed).
UAX \#9 doesn't specify an exact algorithm for autodetecting the
paragraph direction. An implementation might decide on the first strong
character (as it's done inside an FSI...PDI block), this provides a
reasonable experience if the user is typing some text and doesn't want
the overall direction to jump back and forth. Most terminal emulators
will probably just go with whatever is offered by the BiDi library of
their choice, and that's fine. (It's not necessarily a problem if the
exact autodetection method slightly varies across terminal emulators).
UAX \#9 bullet point HL1 says:
> "As another example, when the paragraph contains no strong characters,
> its direction could be determined by the levels of the paragraphs
> before and after."
In terminal emulators the paragraphs are often unrelated to each other
(e.g. output from different utilities), hence I discourage looking at
the surrounding paragraphs, I encourage handling each paragraph
independently. (Should there be a demand for looking at surrounding
paragraphs, a new escape sequence could be introduced in the future to
control this behavior, or to set up hard break points, e.g. to look at
the entire output of each command.)
If the autodetecting algorithm fails to make a guess (e.g. the paragraph
is empty, or consists of neutral characters (e.g. whitespaces) only), we
fall back to the paragraph's direction as specified in the model. (Even
for empty paragraphs a decision has to be made, so that empty input
lines can show the cursor at the expected edge, and mouse clicks can
properly be transformed. Therefore, as opposed to HTML having a single
`dir=auto` mode only, we have separate autodetection modes for both
fallback directions.)
Finally the paragraph is handled according to *implicit LTR* or
*implicit RTL* mode, whichever direction was guessed or fallen back to.
### Implicit LTR mode
The terminal emulator asks the BiDi algorithm to lay out the entire
paragraph, using LTR *paragraph direction*. This results in a mapping
from logical to visual positions. The cells are laid out according to
this mapping, aligned to the left edge of the emulator.
Glyphs are mirrored (or rather: counterpart glyphs are used) according
to the BiDi algorithm.
Arabic shaping is performed on the paragraph, before splitting to lines.
The I-beam cursor is shown at the side corresponding to the resolved
directionality of the character underneath.
### Implicit RTL mode
Like implicit LTR mode, except that RTL *paragraph direction* is used,
and the result is aligned to the right edge of the terminal.
## Explicit mode (level 1)
In explicit mode, doing BiDi is the sole responsibility of the
*application*, the terminal emulator doesn't do any (it may still do
overall RTL direction, though). The transferred data stream contains the
text in its *visual* order (either pure LTR or pure RTL).
The emulator doesn't perform shaping either, this is also the sole
responsibility of the application, using "presentation form" characters.
(Level 2 might add some improved shaping control in a future revision of
this document.)
This mode is primarily for sophisticated *BiDi-aware* apps.
However, this mode is also useful for damage control in apps that don't
do any BiDi at all, but have a complex UI that could fall apart in
implicit mode when encountering foreign direction text. By switching to
explicit mode these apps still won't do BiDi (in fact, foreign direction
text snippets will be reversed, i.e. much harder to read), but at least
the overall UI will remain intact.
The *autodetection* flag is ignored, as it doesn't make sense with
*visual* order.
### Explicit LTR mode
This is exactly what most terminal emulators (the ones that don't do any
RTL or BiDi at all) do now. Everything goes from left to right.
### Explicit RTL mode
Cells are laid out in the exact reverse order than in explicit LTR. The
model's first column is displayed in the rightmost cell, etc.
Mirrorable glyphs are mirrored.
The I-Beam shaped cursor is displayed on the right side of its cell.
---
title: Erased cells
layout: default
nav_order: 40
parent: The Recommendation
permalink: /recommendation/erased-cells.html
---
# Erased cells
Most terminal emulators distinguish *erased* (this is the ECMA-48
terminology; can also be called *unused* or *empty*) cells from spaces,
e.g. they don't copy-paste anything from there. The screen initially
consists of erased cells, they appear at the bottom when scrolling, they
appear on the escape sequence that clears to the end of the line etc.
In cooked mode (e.g. when typing to `cat`), when the user enters
something but then erases it by pressing Backspace, technically the
terminal emulator receives space characters to overwrite the previous
ones. That is, initially erased cells get replaced by spaces.
Luckily, the BiDi algorithm places trailing logical whitespaces to the
trailing side of the rendered string. That is, when shuffling the cells
according to the BiDi algorithm, this difference of erased cells vs.
space characters remains invisible to the user.
In both *implicit* and *explicit* modes, erased cells are mapped to the
end (according to the paragraph direction) of the line, in the order
according to this direction. (This matters e.g. if the cursor is there,
or the mouse is clicked.) For example: if the terminal is 80 characters
wide, a line has RTL direction, and in this line the first 60 cells of
the model contain actual characters and the last 20 cells are erased,
then the first 60 cells are mapped somehow to the rightmost 60 columns,
the model's 61st cell is mapped to the visual 61st cell from the right
(20th from the left), the model's last (80th) cell is mapped to the
leftmost column.
Trailing erased cells cannot only occur at the end of paragraphs, they
can also occur at end of lines mid-paragraph. One example is when the
window is made wider and the emulator doesn't rewrap the lines; it
probably adds erased cells there. Another example is when a double wide
character didn't fit in the last column, so had to be wrapped to the
next line, leaving a single erased cell at the end of the previous line.
Such erased cells should be ignored when feeding the paragraph's
contents to the BiDi algorithm (just as they should be ignored when
copy-pasting, searching etc.), and should also be skipped when rendering
the result (so that each character remains in its row, and still no wide
character is cut in half).
---
title: Combining characters
layout: default
nav_order: 50
parent: The Recommendation
permalink: /recommendation/combining.html
---
# Combining characters
{:.no_toc}
* TOC
{:toc}
This page is for *explicit* modes, in *implicit* modes the handling of
combining characters is obvious.
## Nonspacing and enclosing marks (Mn, Me)
This BiDi specification doesn't change the essentials of terminal
emulation behavior. As a direct consequence, in all of the modes, a
nonspacing or enclosing mark is stored in the same character cell as the
preceding base character of the input stream. The cells are typically
expected to be rendered on their own, that is, no combining accent
should move to another cell for display purposes, not even with BiDi in
the game.
Example: The Hebrew word שָׁלוֹם (Shalom) has the following logical order,
denoting the base characters and combining accents as they belong
together:
0 1 2 3 4 5 6
U+05e9 U+05b8 U+05c1 U+05dc U+05d5 U+05b9 U+05dd
‭└╴שָׁ╶───────────────┘ └╴ל╶─┘ └╴וֹ╶────────┘ └╴ם╶─┘‬
If an application wishes to print this word in *explicit LTR* mode, it
has to reverse the order of the base letters but keep every combining
accent *after* its base letter, that is, it needs to emit:
6 4 5 3 0 1 2
U+05dd U+05d5 U+05b9 U+05dc U+05e9 U+05b8 U+05c1
‭└╴ם╶─┘ └╴וֹ╶────────┘ └╴ל╶─┘ └╴שָׁ╶───────────────┘‬
This is *not* what the BiDi algorithm produces by default, check e.g. at
the online [C](https://unicode.org/cldr/utility/bidic.jsp) or
[JAVA](https://unicode.org/cldr/utility/bidi.jsp) reference
implementation.
The FriBidi library implements our desired behavior if
FRIBIDI_FLAG_REORDER_NSM is passed to fribidi_reorder_line(), which is
the default behavior.
In *explicit RTL* mode, an app would output the BiDi algorithm's result
in reverse memory order (assuming that the BiDi algorithm produces
visual LTR order, that is, the beginning of the array corresponds to the
left of the screen, the end of the array corresponds to the right). Care
should be taken to restore the position of combining accents that belong
to LTR glyphs. The FriBidi library doesn't provide out of the box
support for this. \[TODO: file a feature request.\]
## Spacing marks (Mc)
Spacing marks are used in Devanagari and several other scripts, I
believe all of them are LTR ones. Similarly to nonspacing marks, they
add a combining symbol to the base one, but in addition they also
increase the overall width by one character cell.
The added combining symbol does not necessarily show on the right of the
base one, it can show on its left, too, see e.g. U+093F. Still, the
logical order is the base character followed by the mark. This visual
swapping is unrelated to the BiDi story.
Even without introducing BiDi, it's unclear how such characters should
be handled in terminal emulation, e.g. whether they should combine in
the *data layer* or in the *presentation layer*. See e.g. [VTE issue
584160](https://bugzilla.gnome.org/show_bug.cgi?id=584160).
With BiDi, it's not entirely clear what the proper output order should
be when outputting reverse string, that is, when printing such LTR text
in an *explicit RTL* paragraph (e.g. an overall Arabic text has some
embedded Devanagari word), or, theoretically, if in *explicit LTR* mode
a RTL text with such spacing marks would be emitted.
Due to such marks not existing in RTL scripts, and visual RTL not being
a common thing, I'm not sure if there's any best practice here that we
could follow. I find "Approach A" to be the cleaner overall design, but
input from BiDi experts is desired to make a final decision.
### Approach A