From 9a1dfe2bd9a0a1310b05ea0cd5aade5c2536b547 Mon Sep 17 00:00:00 2001 From: Kevin Lamonte Date: Tue, 10 Sep 2019 00:06:19 -0500 Subject: [PATCH] simplified proposal --- docs/images2.md | 495 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 495 insertions(+) create mode 100644 docs/images2.md diff --git a/docs/images2.md b/docs/images2.md new file mode 100644 index 0000000..34c1341 --- /dev/null +++ b/docs/images2.md @@ -0,0 +1,495 @@ +Terminal Emulator Images Standard - Proposed Design - Simplified +================================================================ + +Version: 1 + + + +Purpose +------- + +See the [original proposal](images.md) for purpose, design goals, and +definitions. + +This document is an updated proposal to address feedback on the first +proposal, which included: "overengineered", "hopelessly +overengineered", and "unnecessarily complex." + +I perceive this feedback as a positive: it is far easier to imagine a +feature and remove it, than to fail to picture it and need it later. +The original proposal was a superset of every image format referenced, +and generalized beyond to multimedia. This proposal is sharply +reduced from that to: "put this pixel rectangle from the image, into +that cell-based rectangle with specific scaling policy". It is mostly +a subset of the iTerm2 protocol, with specifications for what happens +to the cursor, and more precise definitions of the +"preserveAspectRatio" equivalent options. + + + +Tradeoffs +--------- + +Simplifying the original proposal will significantly reduce +complexity, but also eliminates features. The major tradeoffs offered +in this revised proposal are: + +1. Elimination of the layers feature, and with it the ability to place + images behind text. In this proposal, a Cell on the screen will + show either a (part of a) visible image, or a (part of a) text + glyph, but never both. + +2. Elimination of the "url" option, and with it the ability for an + application to specify a filename or other method for the terminal + to find the file data on the local machine. Image data must always + be passed inline with the sequences. + +3. Elimination of response codes, and with it: + + - The ability for multiplexers to blindly pass on the sequences to + their host terminal. + + - The ability for applications to reliably detect success or + failure of image display operations. + +4. Elimination of pixel-oriented image placement operations, and with + it the ability of applications to pass on image calculations to the + terminal. An application which requires pixel-perfect rendering + must generate the pixels it needs, aligned such to be displayed at + the top-left corner of the text Cell rectangle. + + + +Summary +------- + +This revised document proposes two independent new features: + +1. A method to transfer image data for immediate display within the + screen Cell grid ("Direct Images"). + +2. A method to transfer image data to a terminal-managed cache, and + later display that data within the screen Cell grid ("Cached + Images"). + +The only difference between the first and second feature is the +presence of an ID key. Direct images do not use an ID key, while +cached images use a store operation with ID key followed by one or +more display operations with ID key. + +Images are applied to text Cells, and once set handled the same way +text Cells are handled: erasing a line erases the image Cells on that +line, inserting a character will shift image Cells on that row over, +scrolling will shift the image up, and so on. Therefore, terminals +will need to be prepared for the scenario that every Cell on the +display is a separate image, with a separate display scaling option +that will need to be re-applied automatically if font metrics change. + + + +All Features - Detection +------------------------ + +Applications can detect support for these features using Primary +Device Attributes (DA) and DECID (ESC Z, or 0x9A). + +Terminals that support this standard will repond with additional +parameter(s): "224" for direct images and "225" for cached images. A +recap of the parameters xterm supports is listed below, with these new +feature responses included: + +| VT220 (and higher) Response | Description | +|-----------------------------|--------------------------------------------| +| 1 | 132-columns | +| 2 | Printer | +| 3 | ReGIS graphics | +| 4 | Sixel graphics | +| 6 | Selective erase | +| 8 | User-defined keys | +| 9 | National Replacement Character sets | +| 1 5 | Technical characters | +| 1 6 | Locator port | +| 1 7 | Terminal state interrogation | +| 1 8 | User windows | +| 2 1 | Horizontal scrolling | +| 2 2 | ANSI color, e.g., VT525 | +| 2 8 | Rectangular editing | +| 2 9 | ANSI text locator (i.e., DEC Locator mode) | +| 2 2 4 | Direct Images Version 1 | +| 2 2 5 | Cached Images Version 1 | + + + +Direct Images - Summary +----------------------- + +Non-text data (images) can be sent to the terminal for immediate +display in a rectangular region of text Cells. Image data is +transmitted to the terminal using a wire format described later in +this document. + +Setting a Cell to image is a destructive operation: the Cell's +original text is lost. Similarly, setting a Cell (or multiple Cells +for fullwidth glyphs or grapheme clusters) to text is a destructive +operation: the image in the Cell(s) is lost. + +Setting any part of a multi-Cell Tile to image also "breaks up" the +Tile into a range of single Cells. In other words, image data can +only be carried by a Cell, not a Tile. + + + +Direct Images - New Sequences +----------------------------- + +A terminal with direct images feature must support the following new +sequences: + +| Sequence | Description | +|--------------------------------------|-------------------------| +| OSC 1 3 3 8 ; F i l e = {args} : {data} BEL | Display image at (x, y) | +| OSC 1 3 3 8 ; F i l e = {args} : {data} ST | Display image at (x, y) | + + + +For the OSC 1 3 3 8 sequence: + +* The {args} is a set of key-value pairs (each pair separated by + semicolon (';')), followed by a colon (':'), followed by a base-64 + encoded string ({data}). + +* A key can be any alpha-numeric ASCII string ('0' - '9', 'A' - 'Z', + 'a' - 'z'). + +* A value is any printable ASCII string not containing whitespace, + colon, or semicolon ('!' - '9', '<' - '~'). + +* Any alpha-numeric key may be specified. A key that is not supported + by the terminal is ignored without error. + +* The image is processed as shown below: + + - The pixels are drawn starting at the upper-left corner of the text + cursor position. + + - If scroll is specified as 1 (enabled), then: + + a. The screen is scrolled up if the image overflows into the + bottom text row. + + b. The cursor's final position is on the same column as the + starting cursor position, and on the row immediately below the + image. + + - If scroll is omitted or specified as 0 (disabled), then: + + a. The screen is never scrolled. + + b. Pixels that would be drawn below the visible region on screen + are discarded. + + c. The cursor's final position is at the same column and row as + the starting cursor position, i.e. the cursor does not move at + all. + + - Pixels that would be drawn to the right of the visible region on + screen are discarded. + + + +The keys for the key-value pairs that must be supported by the +terminal are listed below: + +| Key | Default Value | Description | +|--------------|---------------|----------------------------------------------| +| type | "image/rgb" | mime-type describing data field | +| width | 1 | Number of Cells or pixels wide to display in | +| height | 1 | Number of Cells or pixels high to display in | +| scale | "none" | Scale/zoom option, see below | +| sourceX | 0 | Media source X position to display | +| sourceY | 0 | Media source Y position to display | +| sourceWidth | "auto" | Media width in pixels to display | +| sourceHeight | "auto" | Media height in pixels to display | +| scroll | 0 | If 0, scroll the display if needed | + +A terminal may support additional keys. If a key is specified but not +supported by the terminal, then it is ignored without error. + + + +The "type" value is a mime-type string describing the format of the +base64-encoded binary data. The terminal must support at minimum these +mime-types: + +| Type String | Description | +|---------------|--------------------------------------------------------------| +| "image/rgb" | Big-endian-encoded 24-bit red, green, blue values | +| "image/rgba" | Big-endian-encoded 32-bit red, green, blue, alpha values | +| "image/png" | PNG file data as described by (reference to PNG format) | + +A terminal may support additional types. An application can detect +terminal support for a format by: + + 1. Attempt to draw image, with "scroll" set to 1. + + 2. Check cursor position DSR 6. + + 3. If cursor has moved, then the terminal supports this image type. + + + +The "width" and "height" values are positive integers describing the +number of Cells the image will be placed in. + + + +The "scale" value can take the following values: + +| Value | Meaning | +|------------|---------------------------------------------------------------| +| "none" | No scaling along either axis. | +| "scale" | Stretch image, preserving aspect ratio, to maximum size in the target area without cropping | +| "stretch" | Stretch along both axes, distorting aspect ratio, to fill the target area | +| "crop" | Stretch along both axes, preserving aspect ration, to completely fill the target area, cropping pixels that will not fit | + + + +"sourceX", "sourceY", "sourceWidth", and "sourceHeight" define the +rectangle of pixels from the media that will be displayed on the +screen. The ranges for these values is shown below: + +| Key | Minimum Value | Maximum Value | Default Value | +|--------------|---------------|-------------------------------|---------------| +| sourceX | 0 | Media's full width - 1 | 0 | +| sourceY | 0 | Media's full height - 1 | 0 | +| sourceWidth | 1 | Media's full width - sourceX | "auto" | +| sourceHeight | 1 | Media's full height - sourceY | "auto" | + +If any of these values are specified and outside the range, no image +is displayed, and the cursor does not move. "sourceWidth" and +"sourceHeight" can be "auto", which means use the maximum available +width/height (given sourceX/sourceY) from the media's inherent +dimensions. + + + +Cached Images - Summary +----------------------- + +Non-text data (image) can be sent to the terminal for later display in +a rectangular region of text Cells. Image data is transmitted to the +terminal using the CSTORE command described below, and displayed on +screen using the CDISPLAY command. A single CSTORE command can +support many CDISPLAY commands. + +Upon display, setting a Cell to image is a destructive operation: the +Cell's original text is lost. Similarly, setting a Cell (or multiple +Cells for fullwidth glyphs or grapheme clusters) to text is a +destructive operation: the image in the Cell(s) is lost. + +Setting any part of a multi-Cell Tile to image also "breaks up" the +Tile into a range of single Cells. In other words, image data can +only be carried by a Cell, not a Tile. + + + +Cached Images - Cache/Memory Management +--------------------------------------- + +The terminal manages a cache of multimedia data on behalf of the +application. The application requests media be stored in the cache +and provides an ID. This ID is later used to request display on the +screen. + +The amount of memory and retention/eviction strategy for the cache is +wholly managed by the terminal, with the following restrictions: + +* The terminal may not remove items from the cache that have any + portion being actively displayed on the primary or alternate + screens. + +The scrollback buffer is permitted, and recommended, to contain only a +few (or zero) multimedia images. Terminals should consider retaining +only the last 2-5 screens' worth of pixel data in the scrollback +buffer. + +Applications have no control over when images are removed from the +cache, and no provision is made to generate/ensure unique IDs. + +A terminal multiplexer that passes all CSTORE/CDISPLAY commands to the +host terminal will need to parse the CSTORE and CDISPLAY sequences for +the "id" field and rewrite it to be unique for all of its inner +terminals. + + + +Cached Images - New Sequences +----------------------------- + +A terminal with cached images feature must support the following new +sequences: + +| Sequence | Command | Description | +|--------------------------------------|-----------|-------------------------| +| OSC 1 3 4 0 ; F i l e = {args} : {data} BEL | CSTORE | Display media at (x, y) | +| OSC 1 3 4 1 ; Pi ; {args} ST | CDISPLAY | Display media at (x, y) | + + + +Cached Images - CSTORE +---------------------- + +For the CSTORE command: + +* The {args} is a set of key-value pairs (each pair separated by + semicolon (';')), followed by a colon (':'), followed by a base-64 + encoded string ({data}). + +* A key can be any alpha-numeric ASCII string ('0' - '9', 'A' - 'Z', + 'a' - 'z'). + +* A value is any printable ASCII string not containing whitespace, + colon, or semicolon ('!' - '9', '<' - '~'). + + + +The keys for the key-value pairs that must be supported by the +terminal are listed below: + +| Key | Default Value | Description | +|--------------|---------------|----------------------------------------------| +| id | 0 | ID to refer to the image | +| type | "image/rgb" | mime-type describing data field | + + + +The "id" value is a non-negative integer between 0 and 999999. + + + +The "type" value is a mime-type string describing the format of the +base64-encoded binary data. The terminal must support at mimunum these +mime-types: + +| Type String | Description | +|---------------|--------------------------------------------------------------| +| "image/rgb" | Big-endian-encoded 24-bit red, green, blue values | +| "image/rgba" | Big-endian-encoded 32-bit red, green, blue, alpha values | +| "image/png" | PNG file data as described by (reference to PNG format) | + +A terminal may support additional types. An application can detect +terminal support for a format by: + + 1. Store image in cache. + + 2. Attempt to draw image, with "scroll" set to 1. + + 3. Check cursor position DSR 6. + + 4. If cursor has moved, then the terminal supports this image type. + + + +Cached Images - CDISPLAY +------------------------ + +For the CDISPLAY command: + +* Pi - a non-negative integer ID that was used in a previous CSTORE + command. + +* The {args} is a set of key-value pairs (each pair separated by + semicolon (';')), followed by a colon (':'), followed by a base-64 + encoded string. + +* A key can be any alpha-numeric ASCII string ('0' - '9', 'A' - 'Z', + 'a' - 'z'). + +* A value is any printable ASCII string not containing whitespace, + colon, or semicolon ('!' - '9', '<' - '~'). + +* Any alpha-numeric key may be specified. A key that is not supported + by the terminal is ignored without error. + +* The image pixels are processed as shown below. + + - The pixel are drawn starting at the upper-left corner of the text + cursor position. + + - If scroll is specified as 1 (enabled), then: + + a. The screen is scrolled up if the image overflows into the + bottom text row. + + b. The cursor's final position is on the same column as the + starting cursor position, and on the row immediately below the + image. + + - If scroll is omitted or specified as 0 (disabled), then: + + a. The screen is never scrolled. + + b. Pixels that would be drawn below the visible region on screen + are discarded. + + c. The cursor's final position is at the same column and row as + the starting cursor position, i.e. the cursor does not move at + all. + + - Pixels that would be drawn to the right of the visible region on + screen are discarded. + + + +The keys for the key-value pairs that must be supported by the +terminal are listed below: + +| Key | Default Value | Description | +|--------------|---------------|----------------------------------------------| +| id | 0 | ID to refer to the image | +| width | 1 | Number of Cells or pixels wide to display in | +| height | 1 | Number of Cells or pixels high to display in | +| scale | "none" | Scale/zoom option, see below | +| sourceX | 0 | Media source X position to display | +| sourceY | 0 | Media source Y position to display | +| sourceWidth | "auto" | Media width in pixels to display | +| sourceHeight | "auto" | Media height in pixels to display | +| scroll | 0 | If 1, scroll the display if needed | + +A terminal may support additional keys. If a key is specified but not +supported by the terminal, then it is ignored without error. + + + +The "width" and "height" values are positive integers describing the +number of Cells the image will be placed in. + + + +The "scale" value can take the following values: + +| Value | Meaning | +|------------|---------------------------------------------------------------| +| "none" | No scaling along either axis. | +| "scale" | Stretch image, preserving aspect ratio, to maximum size in the target area without cropping | +| "stretch" | Stretch along both axes, distorting aspect ratio, to fill the target area | +| "crop" | Stretch along both axes, preserving aspect ration, to completely fill the target area, cropping pixels that will not fit | + + + +"sourceX", "sourceY", "sourceWidth", and "sourceHeight" define the +rectangle of pixels from the media that will be displayed on the +screen. The ranges for these values is shown below: + +| Key | Minimum Value | Maximum Value | Default Value | +|--------------|---------------|-------------------------------|---------------| +| sourceX | 0 | Media's full width - 1 | 0 | +| sourceY | 0 | Media's full height - 1 | 0 | +| sourceWidth | 1 | Media's full width - sourceX | "auto" | +| sourceHeight | 1 | Media's full height - sourceY | "auto" | + +If any of these values are specified and outside the range, no image +is displayed, and the cursor does not move. "sourceWidth" and +"sourceHeight" can be "auto", which means use the maximum available +width/height (given sourceX/sourceY) from the media's inherent +dimensions. -- 2.27.0