-Next-Gen Multimedia Standard - Proposed Design Document
+Terminal Emulator Multimedia Standard - Proposed Design
=======================================================
+Version: 1
+
+
Purpose
-------
-TODO: Crib text from the first message of
-https://gitlab.freedesktop.org/terminal-wg/specifications/issues/12 as
-to why people want images in their terminals.
+Multiple standards exist to incorporate image data in text-based
+terminals and terminal emulators. Few standards have wide adoption
+despite frequent user requests for these features and hardware support
+for several of the standards.
+
+A group including developers of several widely-used terminal emulators
+has been working on defining the needs and limitations for a standard
+that can be implemented in current-gen terminal emulators. The
+discussion has been primarily captured here:
+https://gitlab.freedesktop.org/terminal-wg/specifications/issues/12
+
+This document collects many of the reported desires and practical
+constraints of that discussion into a proposed standard that
+encompasses three independent new features:
+
+1. A method to transfer multimedia data for immediate display within
+ the screen cell grid ("Direct Multimedia").
+
+2. A method to transfer multimedia data to a terminal-managed cache,
+ and later display that data within the screen cell grid ("Cached
+ Multimedia").
+
+3. A method to assign cell data to different layers with options for
+ both layer and cell transparency ("Layers").
+
+A terminal may implement any combination of these features
+independently of each other. If all features are supported, then all
+of the design goals outlined in this document can be met.
+
+The same mechanisms that can put raster-based images on the screen are
+also readily generalizable to other media types such as vector-based
+images and animations. This document is thus a "multimedia" proposal
+rather than a "simple images" proposal.
-The same mechanism that can put raster-based images on the screen is
-easily generalizable to other media types such as vector-based images,
-animations, and embedded GUI widgets. This document is thus a
-"multimedia" proposal, not just "simple images".
Acknowledgements
https://gitlab.com/klamonte/jexer
+
Design Goals - Core
-------------------
* Have no ambiguity. If two terminal or application developers can
read this document and reach different conclusions on what should be
on the screen, then an error exists in this document that must be
- fixed.
+ corrected.
- Every feature must be straightforward to validate via automated
unit testing.
- Erroneous sequences must have defined expected results.
- Every operation must act atomically: either everything worked
- (image is on screen, cursor has moved, etc.) or nothing did.
+ (image is on screen, cursor has moved, terminal state has changed,
+ etc.) or nothing did.
+
+* Integrate with existing ECMA-48 / ANSI X3.64 defined sequences:
+
+ - Operations on Tiles/Cells containing text will have the same
+ effect when applied to Tiles/Cells containing image data.
+
+ - Existing sequences are given new parameters to cover needed
+ features rather than entirely new sequences introduced.
* Be straightforward to implement in non-"physical" terminals,
including:
Out Of Scope
------------
-The following items are out of scope for this standard:
+The following items are out of scope:
* Bidirectional output. Applications are expected to generate Tiles
and place them on screen where they need. The cursor response to
- image sequences are defined as left-to-right, consistent with
- ECMA-48 / ANSI X3.64 sequences. An independent BIDI standard is
- free to apply whatever solution will work for ECMA-48 sequences to
- the sequences described in this document.
+ image sequences are defined as left-to-right-top-to-bottom,
+ consistent with ECMA-48 / ANSI X3.64 sequences. An independent BIDI
+ standard is free to apply whatever solution will work for ECMA-48 /
+ ANSI X3.64 sequences to the sequences described in this document.
-* Capabilities. This standard defines a limited number of terminal
- reports. These are not intended to be used as a general-purpose
- capabilities model.
+* Capabilities. This standard defines a limited number of new
+ terminal reports and responses. These are not intended to be used
+ as a general-purpose capabilities model.
Z - The layer that text or multimedia is placed on. This proposal
uses a right-hand coordinate system with (X, Y, Z) = (1, 1, 1)
- defined as the top-left corner on the default layer: positive Z
+ defined as the top-left corner on the default layer; positive Z
projects "away" from the user and "into" or "behind" the screen.
Rendering the Cells on the screen must produce the same result as
- painter's algorithm (see Rendering section below).
+ painter's algorithm (see "Layers - Rendering" section below).
Cell - A fixed-width-and-height rectangle on the screen. The cells of
the screen are arranged in a grid of X columns and Y rows. A
- Cell has dimensions of cellWidth and cellHeight, which can be
- measured in either pixels or points. Every Cell has a
- coordinate of (X, Y, Z).
+ Cell has dimensions of cellWidth and cellHeight pixels. Every
+ Cell has a coordinate of (X, Y) (or (X, Y, Z) when the terminal
+ supports the layers feature).
Tile - One or more contiguous Cells with data to be displayed. The
data can be text or image data, but not both. A Tile has width
glyphs, and two Cells wide for "fullwidth" glyphs as used in
Asian langauges, emojis, and symbols. This standard does not
preclude Tiles from encompassing entire grapheme clusters.
+ Note that ECMA-48 / ANSI X3.64 operations are performed against
+ Tiles, not Cells: if a 2-Cell-wide Tile is deleted via
+ backspace, then the cursor will decrement on screen by two
+ columns.
Layer - A screen-sized grid of Cells that have the same Z coordinate.
Layers are drawn to the screen in descending Z order. Layers
may have optional additional attributes such as transparency.
+ Layer support is an orthogonal (independent) option to
+ multimedia support. It is acceptable for terminals to support
+ multimedia without layers and vice versa.
+
+
+
+
+All Features - Detection
+------------------------
+
+Applications can detect support for these features using Primary
+Device Attributes (DA) and DECID (ESC Z, or 0x9A).
+
+Terminals that support this standard will repond with additional
+parameter(s): "224" for direct multimedia, "225" for cached
+multimedia, and "226" for layers. A recap of the parameters xterm
+supports is listed below, with these new feature responses included:
+
+| VT220 (and higher) Response | Description |
+|-----------------------------|--------------------------------------------|
+| 1 | 132-columns |
+| 2 | Printer |
+| 3 | ReGIS graphics |
+| 4 | Sixel graphics |
+| 6 | Selective erase |
+| 8 | User-defined keys |
+| 9 | National Replacement Character sets |
+| 1 5 | Technical characters |
+| 1 6 | Locator port |
+| 1 7 | Terminal state interrogation |
+| 1 8 | User windows |
+| 2 1 | Horizontal scrolling |
+| 2 2 | ANSI color, e.g., VT525 |
+| 2 8 | Rectangular editing |
+| 2 9 | ANSI text locator (i.e., DEC Locator mode) |
+| 2 2 4 | Direct Multimedia Version 1 |
+| 2 2 5 | Cached Multimedia Version 1 |
+| 2 2 6 | Layers |
+
+
+
+Direct Multimedia - Summary
+---------------------------
+
+
+
+Direct Multimedia - Required Support For Existing Sequences
+-----------------------------------------------------------
+
+A terminal with direct multimedia feature must support the following
+defined xterm sequences:
+
+| Sequence | Description |
+|----------------|-----------------------------------------------------|
+| CSI 16 t | Responds with CSI 6 ; cellHeight ; cellWidth t |
+| CSI 18 t | Responds with CSI 8 ; rows ; columns t |
+
+
+
+Direct Multimedia - New Sequences
+---------------------------------
+
+
+
+Direct Multimedia - Error Handling
+----------------------------------
+
+
+
+Direct Multimedia - Cursor Position
+-----------------------------------
+
+
+
+Direct Multimedia - Wire Format
+-------------------------------
+
+
+
+Direct Multimedia - Examples
+----------------------------
+
+
+
+Cached Multimedia - Summary
+---------------------------
+
+
+
+
+Pixel data that has scrolled off the displayed screen and into the
+scrollback buffer is required to be persistent even if the cache entry
+containing that image data has been evicted by the terminal or removed
+by the application.
+
+
+
+Cached Multimedia - Required Support For Existing Sequences
+-----------------------------------------------------------
+
+A terminal with cached multimedia feature must support the following
+defined xterm sequences:
+
+| Sequence | Description |
+|----------------|-----------------------------------------------------|
+| CSI 16 t | Responds with CSI 6 ; cellHeight ; cellWidth t |
+| CSI 18 t | Responds with CSI 8 ; rows ; columns t |
+
+
+
+Cached Multimedia - New Sequences
+---------------------------------
+
+
+
+Cached Multimedia - Error Handling
+----------------------------------
+
+
+
+Cached Multimedia - Cursor Position
+-----------------------------------
+
+
+
+Cached Multimedia - Scrollback
+------------------------------
+
-Rendering
----------
+Cached Multimedia - Wire Format
+-------------------------------
-A terminal will display its Cells such that the screen will look as if
-it was rendered in the following pseudo-code manner:
+
+
+Cached Multimedia - Examples
+----------------------------
+
+
+
+
+Layers - Summary
+----------------
+
+Layers introduce the concept of a layer "Z" coordinate to the existing
+rows ("Y") by columns ("X") grid. Put another way, the
+two-dimensional grid of columns-by-rows becomes a three-dimensional
+cube of columns-by-rows-by-layers. For this document, the column,
+row, and layer coordinates are referred to as X, Y, and Z. This
+cartesian coordinate system is right-handed, with the Z axis pointing
+"away" from the user "into" the screen.
+
+An application treats the Z coordinate exactly as it does X and Y
+(rows and columns) coordinates:
+
+ * If it attemps to set Z to a value less than 1, then Z is set to 1.
+
+ * If it attempts to set Z to a value greater than the number of
+ layers, then Z is set to the number of layers.
+
+New sequences are provided to set and query Z, Y, X, to set and query
+the screen cube size, and control visibility of Cells in-front-of
+other Cells.
+
+Operations that act on more than one Cell are defined such to act on
+all layers simultaneously by default.
+
+
+
+Layers - Number of Layers
+-------------------------
+
+A terminal is required to provide between 1 and a finite number of
+layers.
+
+The number of layers may be different between the primary and
+alternate screens.
+
+An application may request that the terminal allocate additional
+layers. The terminal is free to honor or ignore such requests as it
+sees fit.
+
+The scrollback buffer is permitted, and recommended, to contain only a
+"flattened" single layer.
+
+
+
+Layers - Terminal State
+-----------------------
+
+The terminal maintains a complex state at all times. This state
+includes variables such as cursor position, foreground/background
+color, attributes to apply to the next displayed character, and so on.
+The layers feature adds more variables to the state, and these
+variables are required to be stored with DECSC (ESC 7) and restored
+with DECRC (ESC 8). The new variables are listed below:
+
+| Mnemonic | Description | Default value |
+|----------|-----------------------------|---------------|
+| Z | Cursor position Z | 1 |
+| MAL | Manipulate all layers | on / enabled |
+| TFT | Text foreground transparent | false |
+| TBT | Text background transparent | false |
+
+
+
+Layers - Required Support For Existing Sequences
+------------------------------------------------
+
+A terminal with layers feature must support the standard VT100/VT102
+sequences defined in their respective manuals.
+
+
+
+Layers - New Sequences
+----------------------
+
+A terminal with layer feature must support the following new
+sequences:
+
+| Sequence | Command | Description |
+|-------------------|-------------|----------------------------------------|
+| CSI ? z ; y ; x H | CUPZ | Move cursor to (x, y, z) |
+| CSI ? z ; y ; x H | SLA | Set layer alpha |
+| CSI ? 3 0 0 1 h | DECSET 3001 | Disable Manipulate All Layers (MAL) |
+| CSI ? 3 0 0 1 l | DECRST 3001 | Enable Manipulate All Layers (MAL) |
+| CSI ? l ; h ; w t | RSZCUBE | Resize cube to (layers, height, width) |
+
+Default parameters and ranges are listed below:
+
+| Command | Position / Variable | Default Value | Minumum | Maximum |
+|---------|---------------------|---------------|---------|-----------|
+| CUPZ | 1 / z | 1 | 1 | # layers |
+| CUPZ | 2 / y | 1 | 1 | # rows |
+| CUPZ | 3 / x | 1 | 1 | # columns |
+| SLA | 1 / alpha | 255 | 0 | 255 |
+| RSZCUBE | 1 / l | 1 | 1 | varies |
+| RSZCUBE | 2 / h | 80 | 1 | varies |
+| RSZCUBE | 3 / w | 24 | 1 | varies |
+
+The terminal must also support the following new queries:
+
+| Query | Response | Description |
+|-----------------------------------------|------------------------------------|
+| CSI ? 1 0 0 n | CSI ? z ; y ; x n | Report cursor Z, Y, X position |
+| CSI ? 1 8 t | CSI ? 8 ; l ; h ; w t | Report the text area cube layers, height, width |
+
+
+The terminal must support the following new Set Graphics Rendition
+(SGR) character attributes commands:
+
+| SGR Parameter | Description |
+|---------------|---------------------------------------------|
+| 230 | Set text foreground color to transparent |
+| 239 | Set text foreground color to solid (opaque) |
+| 240 | Set text background color to transparent |
+| 249 | Set text background color to solid (opaque) |
+
+
+
+
+Layers - Error Handling
+-----------------------
+
+No additional error reporting is provided for layer feature.
+
+
+
+Layers - Rendering
+------------------
+
+A terminal with layer feature will display its Cells such that the
+screen will appear as if it was rendered in the manner of the
+pseudo-code below:
```
for each layer Z, in descending order from maxZ to minZ:
+
for each row Y, in ascending order from minY to maxY:
+
for each column X, in ascending order from minX to maxX:
- draw tile at (X, Y, Z)
+
+ if tile at (X, Y, Z) background color is solid:
+ draw rectangle of background color with layer alpha
+
+ if tile at (X, Y, Z) foreground color is solid:
+ if tile at (X, Y, Z) is glyph:
+ draw glyph with foreground color with layer alpha
+ else
+ draw pixel data of tile as red/green/blue/alpha pixels with
+ layer alpha
+
advance X by tile width
next column
+
advance Y by 1
next row
+
decrease Z by 1
next layer
```
-Terminal State
---------------
-
-
-
-Terminal Reports
-----------------
-
-
-
-Error Handling
---------------
+Layers - Integration With Existing Sequences
+--------------------------------------------
+
+Sequences that insert characters/lines, delete characters/lines, or
+modify larger regions are changed to act upon multiple layers as
+defined below. By default, MAL (Modify All Layers) is off/unset, and
+Z is 1, so if the application never changes MAL or Z then these
+sequences will produce the same visible output as a terminal without
+layer support.
+
+A terminal is not required to support all of these sequences; however,
+for those sequences it does support, if it supports the layers feature
+then the sequences must behave as shown below:
+
+| Sequence | Command | Additional behavior |
+|------------|-------------|------------------------------------------|
+| BS (0x08) | Backspace | Only current layer affected if MAL=off |
+| DEL (0x7F) | Delete | Only current layer affected if MAL=off |
+| IND (0x84) | Index | Only current layer affected if MAL=off |
+| RI (0x8D | Reverse Index | Only current layer affected if MAL=off |
+| ESC # 3 | DECDHL | Cells on all layers always affected |
+| ESC # 4 | DECDHL | Cells on all layers always affected |
+| ESC # 5 | DECSWL | Cells on all layers always affected |
+| ESC # 6 | DECDWL | Cells on all layers always affected |
+| ESC # 8 | DECALN | All layers > 1 cleared; Z, MAL, TFT, TBT reset to default |
+| ESC 7 | DECSC | Also store Z, MAL, TFT, TBT |
+| ESC 8 | DECRC | Also restore Z, MAL, TFT, TBT |
+| ESC c | RIS | All layers > 1 cleared; Z, MAL, TFT, TBT reset to default |
+| CSI @ | ICH | Only current layer affected if MAL=off |
+| CSI J | ED | Only current layer affected if MAL=off |
+| CSI K | EL | Only current layer affected if MAL=off |
+| CSI ? K | DECSEL | Only current layer affected if MAL=off |
+| CSI L | IL | Only current layer affected if MAL=off |
+| CSI M | DL | Only current layer affected if MAL=off |
+| CSI X | ECH | Only current layer affected if MAL=off |
+| CSI M | DL | Only current layer affected if MAL=off |
+| CSI P | DCH | Only current layer affected if MAL=off |
+| CSI R | DECSTBM | Cells on all layers always affected |
+| CSI $ t | DECARA | Only current layer affected if MAL=off |
+| CSI $ v | DECCRA | Only current layer affected if MAL=off |
+| CSI x | DECSACE | Cells on all layers always affected |
+| CSI $ x | DECFRA | Only current layer affected if MAL=off |
+| CSI $ z | DECERA | Only current layer affected if MAL=off |
+
+The VT52 sub-mode commands:
+
+| Sequence | Command | Additional behavior |
+|------------|-------------|------------------------------------------|
+| ESC J | ED | Only current layer affected if MAL=off |
+| ESC K | EL | Only current layer affected if MAL=off |
+
+
+
+Layers - Use With Multiplexers
+------------------------------
+
+Layers are inteded to provide a means for multiplexers to pass on the
+job of multimedia support to the "outer" or host terminal. The
+proposed mechanics of that is outlined in the pseudo-code below:
+```
+for each inner terminal in descending order from maxZ to minZ:
+ emit CUPZ(inner terminal Z, inner terminal Y, inner terminal X)
-Cursor Position
----------------
+ draw inner terminal text with standard VT100/VT102/xterm sequences
+ for each multimedia sequence emitted by the inner terminal:
+ emit CUP(inner terminal Y, inner terminal X)
+ emit multimedia sequences to outer terminal
+ next multimedia sequence
+ decrease Z by 1
+next inner terminal
+```
+The method above may not be effective for complex multi-terminal
+screen layouts, but is hoped to work well for many simple cases.
-Wire Formats
-------------
+Layers - Examples
+-----------------
-Optimizations
--------------
+References
+----------
-Examples
---------
+* xterm control sequences:
+* ECMA-48:
git://git.nikiroo.be / fanfix-jexer.git / commitdiff