TComboBox: some fixes:
[nikiroo-utils.git] / docs / images.md
CommitLineData
930d75f0 1Terminal Emulator Multimedia Standard - Proposed Design
5a16a542
KL
2=======================================================
3
930d75f0
KL
4Version: 1
5
6
5a16a542
KL
7
8Purpose
9-------
10
930d75f0
KL
11Multiple standards exist to incorporate image data in text-based
12terminals and terminal emulators. Few standards have wide adoption
13despite frequent user requests for these features and hardware support
14for several of the standards.
15
16A group including developers of several widely-used terminal emulators
17has been working on defining the needs and limitations for a standard
18that can be implemented in current-gen terminal emulators. The
19discussion has been primarily captured here:
20https://gitlab.freedesktop.org/terminal-wg/specifications/issues/12
21
22This document collects many of the reported desires and practical
23constraints of that discussion into a proposed standard that
24encompasses three independent new features:
25
261. A method to transfer multimedia data for immediate display within
27 the screen cell grid ("Direct Multimedia").
28
292. A method to transfer multimedia data to a terminal-managed cache,
30 and later display that data within the screen cell grid ("Cached
31 Multimedia").
32
333. A method to assign cell data to different layers with options for
34 both layer and cell transparency ("Layers").
35
36A terminal may implement any combination of these features
37independently of each other. If all features are supported, then all
38of the design goals outlined in this document can be met.
39
40The same mechanisms that can put raster-based images on the screen are
41also readily generalizable to other media types such as vector-based
42images and animations. This document is thus a "multimedia" proposal
43rather than a "simple images" proposal.
5a16a542 44
5a16a542
KL
45
46
47Acknowledgements
48----------------
49
50This proposal has been informed from the following prior work:
51
52* DEC VT300 series sixel graphics standard:
53 https://vt100.net/docs/vt3xx-gp/chapter14.html
54
55* iTerm2 image protocol:
56 https://iterm2.com/documentation-images.html
57
58* Kitty image protocol:
5172fe03 59 https://sw.kovidgoyal.net/kitty/graphics-protocol.html
5a16a542
KL
60
61* Jexer Terminal User Interface:
62 https://gitlab.com/klamonte/jexer
63
64
930d75f0 65
5a16a542
KL
66Design Goals - Core
67-------------------
68
69The core ("must-have") design goals are:
70
71* Be easy to implement in existing terminals and applications:
72
73 - Sacrifice "10%" of potential function to eliminate "90%" of
74 implementation pain. "Less is more."
75
76 - Be a strict superset of the existing iTerm2 and DEC sixel image
77 solutions. One should be able to take an existing terminal or
78 application that emits/consumes iTerm2 or sixel sequences, and
79 only change the control sequence introducer/termination to achieve
80 the same effect as a terminal/application that conforms with this
81 standard.
82
83* Have no ambiguity. If two terminal or application developers can
84 read this document and reach different conclusions on what should be
85 on the screen, then an error exists in this document that must be
930d75f0 86 corrected.
5a16a542 87
895440f3 88 - Every feature must be straightforward to validate via automated
5a16a542
KL
89 unit testing.
90
895440f3
KL
91 - Every conformant terminal must produce the same output (pixels on
92 screen) given the same input (terminal font, terminal sequences).
5a16a542
KL
93
94 - Every option must have a defined default value.
95
96 - Erroneous sequences must have defined expected results.
97
98 - Every operation must act atomically: either everything worked
930d75f0
KL
99 (image is on screen, cursor has moved, terminal state has changed,
100 etc.) or nothing did.
101
102* Integrate with existing ECMA-48 / ANSI X3.64 defined sequences:
103
104 - Operations on Tiles/Cells containing text will have the same
105 effect when applied to Tiles/Cells containing image data.
106
107 - Existing sequences are given new parameters to cover needed
108 features rather than entirely new sequences introduced.
5a16a542
KL
109
110* Be straightforward to implement in non-"physical" terminals,
111 including:
112
113 - Future versions of terminal control libraries such as ncurses and
114 termbox.
115
116 - Terminal multiplexers that support "headless" terminals (no
117 physical screen) and "multi-head" terminals (many different
118 physical screens).
119
120* Be platform-agnostic, and easy to implement on (at the least):
121 POSIX, Windows, and web.
122
123 - All features must be available even if the only means of
124 communication between the application and terminal is control
125 sequences (e.g. no shared disk, no shared memory, no shared DOM,
126 etc.).
127
128* Support graceful fallback:
129
130 - Terminal emulators and physical terminals that do not support this
131 standard should remain usable with no undefined screen artifacts,
132 even when the application blindly emits these sequences to those
133 terminals.
134
135 - This standard must able to be versioned for future enhancements.
136
137 - An application must be able to detect that its terminal supports
138 this standard, and at what version.
139
140* Support secure programming practices:
141
142 - Applications must not be able to obtain unauthorized data from
143 terminal memory, such as: images emitted by other applications
144 still present in the terminal's scrollback buffer, terminal or
145 system memory limits.
146
147 - Applications must not be able to compromise the terminal through
148 denial-of-service such as: excessive memory usage, unterminated
149 control sequences. Similarly, terminals must not be able to
150 compromise application through their responses to application
151 queries.
152
153 - Applications must not be able to manipulate the terminal into
154 performing an insecure operation such as: reading arbitrary shared
155 memory regions, reading arbitrary files on disk, deleting
156 arbitrary files on disk, etc. Similarly, terminals must not be
157 able to manipulate applications into performing insecure
158 operations.
159
160 - This standard must be implementable when the terminal has a fixed
161 maximum memory, such as a kernel-level device driver.
162
163
164
165Design Goals - Secondary
166------------------------
167
5172fe03
KL
168The secondary ("nice-to-have") design goals are listed below. These
169might not all be possible, but will kept in mind:
5a16a542
KL
170
171* Minimal redundant network traffic for on-screen data that is
172 repeated: either on screen in multiple places, or in the same place
173 but refreshed multiple times.
174
175* Asynchronous notification from terminal to application that the
176 screen has been changed by outside or user action. Examples: font
177 change, session detach/attach, user changed image preferences.
178
5172fe03
KL
179* The ability for a multiplexer to "pass-thru" the image drawing
180 sequence to its "outer" terminal, with some support for limited
181 clipping.
182
183
5a16a542
KL
184
185Out Of Scope
186------------
187
930d75f0 188The following items are out of scope:
5a16a542
KL
189
190* Bidirectional output. Applications are expected to generate Tiles
191 and place them on screen where they need. The cursor response to
930d75f0
KL
192 image sequences are defined as left-to-right-top-to-bottom,
193 consistent with ECMA-48 / ANSI X3.64 sequences. An independent BIDI
194 standard is free to apply whatever solution will work for ECMA-48 /
195 ANSI X3.64 sequences to the sequences described in this document.
5a16a542 196
930d75f0
KL
197* Capabilities. This standard defines a limited number of new
198 terminal reports and responses. These are not intended to be used
199 as a general-purpose capabilities model.
5a16a542 200
40e3b486
KL
201* Terminal Cache Management. This standard defines a means for
202 applications and terminals to communicate around cached multimedia
203 items, but terminals are free to implement whatever cache management
204 strategies they deem fit.
205
206* Reliable Transport. This standard defines a two-way
207 command/response protocol that may get out of order on unreliable
208 channels such as 3-wire RS232. Applictions that require reliable
209 transport on unreliable links may choose to use one of the many
210 successful standards available for this purpose.
211
5a16a542
KL
212
213
214Definitions
215-----------
216
217Terminal - The hardware, or a program that simulates hardware,
218 comprising a keyboard, screen, and mouse.
219
220Application - A program that utilizes the terminal for its
221 input/output with the user.
222
223Multiplexer - A special case of an application that simulates one or
224 more "inner" terminals for other applications to use,
225 and composes these inner terminals into a combined
226 screen to emit to one or more "outer" terminals that
227 obtain input/output from the user. Multiplexers are
228 thus both applications and terminals.
229
5172fe03
KL
230X - The column coordinate of a cell. This standard is 1-based (like
231 ECMA-48): the left-most column of the screen is numbered 1.
5a16a542 232
5172fe03
KL
233Y - The row coordinate of a cell. This standard is 1-based (like
234 ECMA-48): the top-most row of the screen is numbered 1.
5a16a542
KL
235
236Z - The layer that text or multimedia is placed on. This proposal
5172fe03 237 uses a right-hand coordinate system with (X, Y, Z) = (1, 1, 1)
930d75f0 238 defined as the top-left corner on the default layer; positive Z
5a16a542
KL
239 projects "away" from the user and "into" or "behind" the screen.
240 Rendering the Cells on the screen must produce the same result as
930d75f0 241 painter's algorithm (see "Layers - Rendering" section below).
5a16a542
KL
242
243Cell - A fixed-width-and-height rectangle on the screen. The cells of
244 the screen are arranged in a grid of X columns and Y rows. A
930d75f0
KL
245 Cell has dimensions of cellWidth and cellHeight pixels. Every
246 Cell has a coordinate of (X, Y) (or (X, Y, Z) when the terminal
247 supports the layers feature).
5a16a542
KL
248
249Tile - One or more contiguous Cells with data to be displayed. The
250 data can be text or image data, but not both. A Tile has width
251 of 1, 2, or more, and a coordinate of (X, Y, Z) that is the
252 same as its left-most (first) Cell's (X, Y, Z). In practice,
253 Tiles are typically one Cell wide for ASCII and Latin language
254 glyphs, and two Cells wide for "fullwidth" glyphs as used in
255 Asian langauges, emojis, and symbols. This standard does not
256 preclude Tiles from encompassing entire grapheme clusters.
930d75f0
KL
257 Note that ECMA-48 / ANSI X3.64 operations are performed against
258 Tiles, not Cells: if a 2-Cell-wide Tile is deleted via
259 backspace, then the cursor will decrement on screen by two
260 columns.
5a16a542
KL
261
262Layer - A screen-sized grid of Cells that have the same Z coordinate.
263 Layers are drawn to the screen in descending Z order. Layers
264 may have optional additional attributes such as transparency.
930d75f0
KL
265 Layer support is an orthogonal (independent) option to
266 multimedia support. It is acceptable for terminals to support
267 multimedia without layers and vice versa.
268
269
270
930d75f0
KL
271All Features - Detection
272------------------------
273
274Applications can detect support for these features using Primary
275Device Attributes (DA) and DECID (ESC Z, or 0x9A).
276
277Terminals that support this standard will repond with additional
278parameter(s): "224" for direct multimedia, "225" for cached
279multimedia, and "226" for layers. A recap of the parameters xterm
280supports is listed below, with these new feature responses included:
281
282| VT220 (and higher) Response | Description |
283|-----------------------------|--------------------------------------------|
284| 1 | 132-columns |
285| 2 | Printer |
286| 3 | ReGIS graphics |
287| 4 | Sixel graphics |
288| 6 | Selective erase |
289| 8 | User-defined keys |
290| 9 | National Replacement Character sets |
291| 1 5 | Technical characters |
292| 1 6 | Locator port |
293| 1 7 | Terminal state interrogation |
294| 1 8 | User windows |
295| 2 1 | Horizontal scrolling |
296| 2 2 | ANSI color, e.g., VT525 |
297| 2 8 | Rectangular editing |
298| 2 9 | ANSI text locator (i.e., DEC Locator mode) |
299| 2 2 4 | Direct Multimedia Version 1 |
300| 2 2 5 | Cached Multimedia Version 1 |
301| 2 2 6 | Layers |
302
303
304
305Direct Multimedia - Summary
306---------------------------
307
40e3b486
KL
308Non-text data (multimedia) can be sent to the terminal for immediate
309display in a rectangular (single-layer) region of text Cells.
310Multimedia data is transmitted to the terminal using one of two wire
311formats described later in this document.
312
313Setting a Cell to multimedia is a destructive operation: the Cell's
314original text is lost. Multimedia pixels will not overlap rendered
315text in the same Cell. To achieve pixels overlaid on text, the layers
316feature can be used.
317
318Setting any part of a multi-Cell Tile to multimedia also "breaks up"
319the Tile into a range of single Cells. In other words, multimedia can
320only be carried by a Cell, not a Tile.
321
322The pixels of a multimedia Cell are assigned to the Cell's foreground;
323multimedia Cells have no background. If a terminal supports the
324layers feature, setting a multimedia Cell's foreground transparency to
325true/enabled causes that Cell to not be displayed at all; setting its
326background transparency to either true/enabled or false/disabled has
327no visible effect.
328
329The pixels of multimedia Cells can come from two sources:
330
331 1. The application can generate pixels and send them to the terminal
332 for display at the current cursor position.
333
334 2. The application can specify a source for the multimedia and the
335 terminal will generate the pixels for display at the current
336 cursor position.
337
930d75f0
KL
338
339
340Direct Multimedia - Required Support For Existing Sequences
341-----------------------------------------------------------
342
343A terminal with direct multimedia feature must support the following
344defined xterm sequences:
345
346| Sequence | Description |
347|----------------|-----------------------------------------------------|
348| CSI 16 t | Responds with CSI 6 ; cellHeight ; cellWidth t |
349| CSI 18 t | Responds with CSI 8 ; rows ; columns t |
350
351
352
353Direct Multimedia - New Sequences
354---------------------------------
355
40e3b486
KL
356A terminal with direct multimedia feature must support the following
357new sequences:
930d75f0 358
40e3b486
KL
359| Sequence | Command | Description |
360|--------------------------------------|-------------|-------------------------|
361| OSC 1 3 3 8 ; s i x e l : {data} BEL | SIXEL | Display sixel at (x, y) |
362| OSC 1 3 3 8 ; s i x e l : {data} ST | SIXEL | Display sixel at (x, y) |
5919b89a
KL
363| OSC 1 3 3 8 ; F i l e = {args} : {data} BEL | DMDISPLAY | Display media at (x, y) |
364| OSC 1 3 3 8 ; F i l e = {args} : {data} ST | DMDISPLAY | Display media at (x, y) |
365| CSI ? 3 0 0 0 h | DECSET 3000 | Enable SCRCHANGE notification |
366| CSI ? 3 0 0 0 l | DECRST 3000 | Disable SCRCHANGE notification |
367| OSC 1 3 3 9 ; Pe ; {args} ST | DMRESP | Terminal response to DMDISPLAY |
40e3b486
KL
368| CSI ? 3 0 0 1 h | DECSET 3001 | Enable DMDISPLAY responses |
369| CSI ? 3 0 0 1 l | DECRST 3001 | Disable DMDISPLAY responses |
5919b89a
KL
370
371
372
373If SCRCHANGE is set/enabled, then the terminal will send the "CSI 6 ;
374cellHeight ; cellWidth t" when the font size has changed, and "CSI 8 ;
375rows ; columns t" when the number of rows/columns on the screen has
376changed.
377
40e3b486
KL
378
379
380For the SIXEL command:
381
382* The {data} is a sixel sequence as described in the VT330/340
383 Programmer Reference Manual, Chapter 14, available online at:
384 http://vt100.net/docs/vt3xx-gp/chapter14.html . The {data} is the
385 "P1 ; P2 ; P3 ; q s..s" portion of the Device Control String, i.e. a
386 complete sixel sequence minus the leading DCS and trailing ST.
387
388* The sixel image is processed as shown below. Note that this
389 behavior is equivalent to Sixel Scrolling mode enabled.
390
391 - The sixel active position starts at the upper-left corner of the
392 text cursor position.
393
394 - The screen is scrolled up if the image overflows into the bottom
395 text row.
396
397 - Pixels that would be drawn to the right of the visible region on
398 screen are discarded.
399
400 - The cursor's final position is on the same column as the starting
401 cursor position, and on the row immediately below the image.
402
403
404For the DMDISPLAY command:
405
5919b89a 406* The {args} is a set of key-value pairs (each pair separated by
40e3b486 407 semicolon (';')), followed by a colon (':'), followed by a base-64
5919b89a 408 encoded string ({data}).
40e3b486
KL
409
410* A key can be any alpha-numeric ASCII string ('0' - '9', 'A' - 'Z',
411 'a' - 'z').
412
413* A value is any printable ASCII string not containing whitespace,
414 colon, or semicolon ('!' - '9', '<' - '~').
415
416* Any alpha-numeric key may be specified. A key that is not supported
417 by the terminal is ignored without error.
418
419* The multimedia pixels are processed as shown below.
420
421 - The pixel are drawn starting at the upper-left corner of the text
422 cursor position.
423
9dbe0d21 424 - If scroll is specified as 1 (enabled), then:
40e3b486
KL
425
426 a. The screen is scrolled up if the image overflows into the
427 bottom text row.
428
429 b. The cursor's final position is on the same column as the
430 starting cursor position, and on the row immediately below the
431 image.
432
9dbe0d21 433 - If scroll is omitted or specified as 0 (disabled), then:
40e3b486
KL
434
435 a. The screen is never scrolled.
436
437 b. Pixels that would be drawn below the visible region on screen
438 are discarded.
439
440 c. The cursor's final position is at the same column and row as
441 the starting cursor position, i.e. the cursor does not move at
442 all.
443
444 - Pixels that would be drawn to the right of the visible region on
445 screen are discarded.
446
447
448
449The keys for the key-value pairs that must be supported by the
450terminal are listed below:
451
452| Key | Default Value | Description |
453|--------------|---------------|----------------------------------------------|
5919b89a
KL
454| type | "image/rgb" | mime-type describing data field |
455| url | "" | If set, a location containing the media data |
40e3b486
KL
456| width | 1 | Number of Cells or pixels wide to display in |
457| height | 1 | Number of Cells or pixels high to display in |
458| scale | "none" | Scale/zoom option, see below |
459| align | "nw" | Align image to edge option, see below |
5919b89a
KL
460| sourceX | 0 | Media source X position to display |
461| sourceY | 0 | Media source Y position to display |
462| sourceWidth | "auto" | Media width in pixels to display |
463| sourceHeight | "auto" | Media height in pixels to display |
9dbe0d21 464| scroll | 1 | If 1, scroll the display if needed |
40e3b486
KL
465
466A terminal may support additional keys. If a key is specified but not
467supported by the terminal, then it is ignored without error.
930d75f0 468
930d75f0
KL
469
470
5919b89a
KL
471The "type" value is a mime-type string describing the format of the
472base64-encoded binary data. The terminal must support at mimunum these
473mime-types:
474
475| Type String | Description |
476|---------------|--------------------------------------------------------------|
477| "image/rgb" | Big-endian-encoded 24-bit red, green, blue values |
478| "image/rgba" | Big-endian-encoded 32-bit red, green, blue, alpha values |
479| "image/png" | PNG file data as described by (reference to PNG format) |
480
481A terminal may support additional types. An application can detect
482terminal support for a format by: enabling terminal responses (DECSET
4833001), sending a DMDISPLAY command, and examining the terminal's
484response sequence for success or error.
485
486
487
488The "url" value is a RFC-XXXX defined Universal Resource Located,
489encoded in RFC-XXXX form as a printable ASCII string not containing:
490whitespace, colon (':'), semicolon (';'), or equals ('=').
491
492A terminal is not required to support any URLs.
493
494
495
40e3b486 496The "width" and "height" values can take the following forms:
930d75f0 497
40e3b486
KL
498| Value | Meaning |
499|-------------------------------|---------------------------|
500| N (a positive integer) | Number of Cells |
501| Npx (positive integer + "px") | Number of pixels |
502| N% (positive integer + "%") | Percent of screen width or height |
503| "auto" | Number of pixels as defined by the multimedia data |
930d75f0
KL
504
505
506
40e3b486
KL
507The "scale" value can take the following values:
508
509| Value | Meaning |
510|------------|---------------------------------------------------------------|
511| "none" | No scaling along either axis. |
512| "scale" | Stretch image, preserving aspect ratio, to maximum size in the target area without cropping |
513| "stretch" | Stretch along both axes, distorting aspect ratio, to fill the target area |
514| "crop" | Stretch along both axes, preserving aspect ration, to completely fill the target area, cropping pixels that will not fit |
515
516
517
518The "align" value can take the following values:
519
520| Value | Meaning |
521|------------|-----------------------------------------------------------------|
522| "nw" | Media is placed at the top-left corner (northwest) |
523| "n" | Media is placed on the top and centered horizontally (north) |
524| "ne" | Media is placed at the top-right corner (northest) |
525| "w" | Media is placed on the left and centered vertically (west) |
526| "c" | Media is centered in the target area (center) |
527| "e" | Media is placed on the right and centered vertically (east) |
528| "sw" | Media is placed on the bottom-left corner (southwest) |
529| "s" | Media is placed on the bottom and centered horizontally (south) |
530| "se" | Media is placed on the bottom-right corner (southeast) |
531
532
533
5919b89a
KL
534"sourceX", "sourceY", "sourceWidth", and "sourceHeight" define the
535rectangle of pixels from the media that will be displayed on the
536screen. The ranges for these values is shown below:
40e3b486 537
5919b89a
KL
538| Key | Minimum Value | Maximum Value | Default Value |
539|--------------|---------------|-------------------------------|---------------|
540| sourceX | 0 | Media's full width - 1 | 0 |
541| sourceY | 0 | Media's full height - 1 | 0 |
542| sourceWidth | 1 | Media's full width - sourceX | "auto" |
543| sourceHeight | 1 | Media's full height - sourceY | "auto" |
40e3b486 544
5919b89a
KL
545If any of these values are specified and outside the range, no image
546is displayed, and the cursor does not move. "sourceWidth" and
547"sourceHeight" can be "auto", which means use the maximum available
548width/height (given sourceX/sourceY) from the media's inherent
549dimensions.
40e3b486
KL
550
551
552
553Direct Multimedia - Terminal Responses / Error Handling
554-------------------------------------------------------
555
556If DMDISPLAY reponses are enabled, then a terminal will respond to the
557DMDISPLAY display with DMRESP. DMRESP responses must be sent in the
558same sequential order as the DMDISPLAY commands they are responses to:
559the terminal may not re-order responses.
560
561No provision is made for reliable delivery. On unreliable links
562(example: 3-wire RS232), the DMDISPLAY and DMRESP command/response
563sequence may get out of order.
564
565
566
567The format of DMRESP is:
568
569* Pe - a non-negative integer error code.
570
5919b89a 571* The {args} is a set of key-value pairs (each pair separated by
40e3b486
KL
572 semicolon (';')).
573
574* A key can be any alpha-numeric ASCII string ('0' - '9', 'A' - 'Z',
575 'a' - 'z').
576
577* A value is any printable ASCII string not containing whitespace,
578 colon, or semicolon ('!' - '9', '<' - '~').
579
580
5919b89a 581
40e3b486
KL
582The Pe error codes are defined as:
583
5919b89a 584| Value | Meaning | {args} containts |
40e3b486
KL
585|-------|------------------------------------|--------------------------|
586| 0 | No error occurred, i.e. success | nothing |
5919b89a 587| 1 | Unsupported "type" | "type" value that was incorrect |
40e3b486
KL
588| 2 | Invalid value - no media displayed | "key" that was incorrect |
589| 3 | Unsupported key - media displayed | "key" that unsupported |
590| 4 | Insufficient memory | nothing |
591| 5 | Other error - no media displayed | nothing |
592| 6 | Other - media displayed | nothing |
593| 7 | Conflicting keys - no media displayed | nothing |
5919b89a 594| 8 | RESERVED FOR FUTURE USE | RESERVED FOR FUTURE USE |
40e3b486
KL
595
596Additional Pe error codes may be returned; any Pe value except 0, 3,
597and 6 must mean that the media was not displayed, and the cursor was
598not moved.
599
600If both "type" and "url" are set, no media is diaplyed, the cursor is
601not moved, and the DMRESP error code is 7.
930d75f0
KL
602
603
604
605Direct Multimedia - Examples
606----------------------------
607
608
609
610Cached Multimedia - Summary
611---------------------------
612
40e3b486
KL
613Non-text data (multimedia) can be sent to the terminal for later
614display in a rectangular (single-layer) region of text Cells.
615Multimedia data is transmitted to the terminal using the CMCACHE
616command described below, and displayed on screen using the CMDISPLAY
617command. A single CMCACHE command can support many CMDISPLAY
618commands.
619
620Upon display, setting a Cell to multimedia is a destructive operation:
621the Cell's original text is lost. Multimedia pixels will not overlap
622rendered text in the same Cell. To achieve pixels overlaid on text,
623the layers feature can be used.
624
625Setting any part of a multi-Cell Tile to multimedia also "breaks up"
626the Tile into a range of single Cells. In other words, multimedia can
627only be carried by a Cell, not a Tile.
628
629The pixels of a multimedia Cell are assigned to the Cell's foreground;
630multimedia Cells have no background. If a terminal supports the
631layers feature, setting a multimedia Cell's foreground transparency to
632true/enabled causes that Cell to not be displayed at all; setting its
633background transparency to either true/enabled or false/disabled has
634no visible effect.
635
636The pixels of multimedia Cells can come from two sources:
637
638 1. The application can generate pixels and send them to the terminal
639 for display at the current cursor position.
640
641 2. The application can specify a source for the multimedia and the
642 terminal will generate the pixels for display at the current
643 cursor position.
644
645
646
647
648Cached Multimedia - Cache/Memory Management
649-------------------------------------------
930d75f0 650
5919b89a
KL
651The terminal manages a cache of multimedia data on behalf of one or
652more applications. Applications request media be stored in the cache,
653and if successful the terminal provides an identification number that
654applications must use to request display from the cache to the screen.
930d75f0 655
5919b89a
KL
656The amount of memory and retention/eviction strategy for the cache is
657wholly managed by the terminal, with the following restrictions:
930d75f0 658
5919b89a
KL
659* The terminal may not remove items from the cache that have any
660 portion being actively displayed on the primary or alternate
661 screens.
40e3b486 662
5919b89a
KL
663* The terminal must respond to every CMCACHE command with a new unique
664 ID.
40e3b486 665
5919b89a
KL
666The scrollback buffer is permitted, and recommended, to contain only a
667few (or zero) multimedia images. Terminals should consider retaining
668only the last 2-5 screens' worth of pixel data in the scrollback
669buffer.
930d75f0
KL
670
671
672
673Cached Multimedia - Required Support For Existing Sequences
674-----------------------------------------------------------
675
676A terminal with cached multimedia feature must support the following
677defined xterm sequences:
678
679| Sequence | Description |
680|----------------|-----------------------------------------------------|
681| CSI 16 t | Responds with CSI 6 ; cellHeight ; cellWidth t |
682| CSI 18 t | Responds with CSI 8 ; rows ; columns t |
683
684
685
686Cached Multimedia - New Sequences
687---------------------------------
688
40e3b486
KL
689A terminal with cached multimedia feature must support the following new
690sequences:
691
692| Sequence | Command | Description |
693|--------------------------------------|-----------|-------------------------|
5919b89a
KL
694| CSI ? 3 0 0 0 h | DECSET 3000 | Enable SCRCHANGE notification |
695| CSI ? 3 0 0 0 l | DECRST 3000 | Disable SCRCHANGE notification |
696| OSC 1 3 4 0 ; F i l e = {args} : {data} BEL | CMCACHE | Display media at (x, y) |
697| OSC 1 3 4 1 ; Pi ; {args} ST | CMDISPLAY | Display media at (x, y) |
698| OSC 1 3 4 2 ; Pi ; Pe ; {args} ST | CMCRESP | Terminal response to CMCACHE |
699| OSC 1 3 4 3 ; Pi ; Pe ; {args} ST | CMDRESP | Terminal response to CMDISPLAY |
40e3b486
KL
700
701
702
5919b89a
KL
703If SCRCHANGE is set/enabled, then the terminal will send the "CSI 6 ;
704cellHeight ; cellWidth t" when the font size has changed, and "CSI 8 ;
705rows ; columns t" when the number of rows/columns on the screen
706changes.
707
708
40e3b486
KL
709
710Cached Multimedia - CMCACHE
711---------------------------
712
713For the CMCACHE command:
714
5919b89a
KL
715* The {args} is a set of key-value pairs (each pair separated by
716 semicolon (';')), followed by a colon (':'), followed by a base-64
717 encoded string ({data}).
40e3b486
KL
718
719* A key can be any alpha-numeric ASCII string ('0' - '9', 'A' - 'Z',
720 'a' - 'z').
721
722* A value is any printable ASCII string not containing whitespace,
723 colon, or semicolon ('!' - '9', '<' - '~').
724
5919b89a
KL
725
726
40e3b486
KL
727The keys for the key-value pairs that must be supported by the
728terminal are listed below:
729
730| Key | Default Value | Description |
731|--------------|---------------|----------------------------------------------|
732| type | "image/rgb" | mime-type describing data field |
733| url | "" | If set, a location containing the media data |
734
735
5919b89a 736
40e3b486
KL
737The "type" value is a mime-type string describing the format of the
738base64-encoded binary data. The terminal must support at mimunum these
739mime-types:
740
741| Type String | Description |
742|---------------|--------------------------------------------------------------|
743| "image/rgb" | Big-endian-encoded 24-bit red, green, blue values |
744| "image/rgba" | Big-endian-encoded 32-bit red, green, blue, alpha values |
745| "image/png" | PNG file data as described by (reference to PNG format) |
746
747A terminal may support additional types. An application can detect
748terminal support for a format by: sending a CMCACHE command, and
749examining the terminal's CMCRESP sequence for success or error.
750
751
752
5919b89a
KL
753The "url" value is a RFC-XXXX defined Universal Resource Located,
754encoded in RFC-XXXX form as a printable ASCII string not containing:
755whitespace, colon (':'), semicolon (';'), or equals ('=').
756
757A terminal is not required to support any URLs.
758
759
760
40e3b486
KL
761Cached Multimedia - CMDISPLAY
762-----------------------------
763
764For the CMDISPLAY command:
765
766* Pi - a non-negative integer media ID that was returned by a CMCRESP
767 response to a previous CMCACHE command.
768
5919b89a 769* The {args} is a set of key-value pairs (each pair separated by
40e3b486
KL
770 semicolon (';')), followed by a colon (':'), followed by a base-64
771 encoded string.
772
773* A key can be any alpha-numeric ASCII string ('0' - '9', 'A' - 'Z',
774 'a' - 'z').
775
776* A value is any printable ASCII string not containing whitespace,
777 colon, or semicolon ('!' - '9', '<' - '~').
778
779* Any alpha-numeric key may be specified. A key that is not supported
780 by the terminal is ignored without error.
781
782* The multimedia pixels are processed as shown below.
783
784 - The pixel are drawn starting at the upper-left corner of the text
785 cursor position.
786
9dbe0d21 787 - If scroll is specified as 1 (enabled), then:
40e3b486 788
9dbe0d21
KL
789 a. The screen is scrolled up if the image overflows into the
790 bottom text row.
791
792 b. The cursor's final position is on the same column as the
793 starting cursor position, and on the row immediately below the
794 image.
795
796 - If scroll is omitted or specified as 0 (disabled), then:
40e3b486 797
9dbe0d21
KL
798 a. The screen is never scrolled.
799
800 b. Pixels that would be drawn below the visible region on screen
801 are discarded.
802
803 c. The cursor's final position is at the same column and row as
804 the starting cursor position, i.e. the cursor does not move at
805 all.
40e3b486
KL
806
807 - Pixels that would be drawn to the right of the visible region on
808 screen are discarded.
809
810
811
812The keys for the key-value pairs that must be supported by the
813terminal are listed below:
814
815| Key | Default Value | Description |
816|--------------|---------------|----------------------------------------------|
9dbe0d21
KL
817| width | 1 | Number of Cells or pixels wide to display in |
818| height | 1 | Number of Cells or pixels high to display in |
819| scale | "none" | Scale/zoom option, see below |
820| align | "nw" | Align image to edge option, see below |
5919b89a
KL
821| sourceX | 0 | Media source X position to display |
822| sourceY | 0 | Media source Y position to display |
823| sourceWidth | "auto" | Media width in pixels to display |
824| sourceHeight | "auto" | Media height in pixels to display |
9dbe0d21 825| scroll | 1 | If 1, scroll the display if needed |
40e3b486
KL
826
827A terminal may support additional keys. If a key is specified but not
828supported by the terminal, then it is ignored without error.
829
830
831
832The "width" and "height" values can take the following forms:
833
834| Value | Meaning |
835|-------------------------------|---------------------------|
836| N (a positive integer) | Number of Cells |
837| Npx (positive integer + "px") | Number of pixels |
838| N% (positive integer + "%") | Percent of screen width or height |
839| "auto" | Number of pixels as defined by the multimedia data |
840
841
842
843The "scale" value can take the following values:
844
845| Value | Meaning |
846|------------|---------------------------------------------------------------|
847| "none" | No scaling along either axis. |
848| "scale" | Stretch image, preserving aspect ratio, to maximum size in the target area without cropping |
849| "stretch" | Stretch along both axes, distorting aspect ratio, to fill the target area |
850| "crop" | Stretch along both axes, preserving aspect ration, to completely fill the target area, cropping pixels that will not fit |
851
852
853
854The "align" value can take the following values:
855
856| Value | Meaning |
857|------------|-----------------------------------------------------------------|
858| "nw" | Media is placed at the top-left corner (northwest) |
859| "n" | Media is placed on the top and centered horizontally (north) |
860| "ne" | Media is placed at the top-right corner (northest) |
861| "w" | Media is placed on the left and centered vertically (west) |
862| "c" | Media is centered in the target area (center) |
863| "e" | Media is placed on the right and centered vertically (east) |
864| "sw" | Media is placed on the bottom-left corner (southwest) |
865| "s" | Media is placed on the bottom and centered horizontally (south) |
866| "se" | Media is placed on the bottom-right corner (southeast) |
867
930d75f0
KL
868
869
5919b89a
KL
870"sourceX", "sourceY", "sourceWidth", and "sourceHeight" define the
871rectangle of pixels from the media that will be displayed on the
872screen. The ranges for these values is shown below:
873
874| Key | Minimum Value | Maximum Value | Default Value |
875|--------------|---------------|-------------------------------|---------------|
876| sourceX | 0 | Media's full width - 1 | 0 |
877| sourceY | 0 | Media's full height - 1 | 0 |
878| sourceWidth | 1 | Media's full width - sourceX | "auto" |
879| sourceHeight | 1 | Media's full height - sourceY | "auto" |
880
881If any of these values are specified and outside the range, no image
882is displayed, and the cursor does not move. "sourceWidth" and
883"sourceHeight" can be "auto", which means use the maximum available
884width/height (given sourceX/sourceY) from the media's inherent
885dimensions.
886
887
888
930d75f0
KL
889Cached Multimedia - Error Handling
890----------------------------------
891
40e3b486
KL
892A terminal will always respond to the CMCACHE command with CMCRESP,
893and to the CMDISPLAY command with CMDRESP. Responses must be sent in
894the same sequential order as the CMCACHE/CMDISPLAY commands they are
895responses to: the terminal may not re-order responses.
930d75f0 896
40e3b486
KL
897No provision is made for reliable delivery. On unreliable links
898(example: 3-wire RS232), the command/response sequence may get out of
899order.
930d75f0 900
930d75f0
KL
901
902
40e3b486
KL
903Cached Multimedia - Error Handling - CMCRESP
904--------------------------------------------
930d75f0 905
40e3b486
KL
906The format of CMCRESP is:
907
908* Pi - a non-negative integer media ID. The terminal will generate a
909 new ID for every image successfully loaded into the cache. The
910 application must use this ID for CMDISPLAY commands.
911
912* Pe - a non-negative integer error code.
913
5919b89a 914* The {args} is a set of key-value pairs (each pair separated by
40e3b486
KL
915 semicolon (';')).
916
917* A key can be any alpha-numeric ASCII string ('0' - '9', 'A' - 'Z',
918 'a' - 'z').
930d75f0 919
40e3b486
KL
920* A value is any printable ASCII string not containing whitespace,
921 colon, or semicolon ('!' - '9', '<' - '~').
5a16a542
KL
922
923
5919b89a 924
40e3b486
KL
925The Pe error codes are defined as:
926
5919b89a
KL
927| Value | Meaning | {args} containts |
928|-------|----------------------------------------|--------------------------|
929| 0 | No error occurred, i.e. success | nothing |
930| 1 | Unsupported "type" | "type" value that was incorrect |
931| 2 | Invalid value - no media stored | "key" that was incorrect |
932| 3 | Unsupported key - media stored | "key" that unsupported |
933| 4 | Insufficient memory - no media stored | nothing |
934| 5 | Other error - no media stored | nothing |
935| 6 | Other - media stored | nothing |
936| 7 | Conflicting keys - no media stored | nothing |
937| 8 | RESERVED FOR FUTURE USE | RESERVED FOR FUTURE USE |
40e3b486
KL
938
939Additional Pe error codes may be returned; any Pe value except 0, 3,
940and 6 must mean that the media was not stored in the cache.
941
942If both "type" and "url" are set, no media is diaplyed, the cursor is
943not moved, and the CMCRESP error code is 7.
944
945
946
947Cached Multimedia - Error Handling - CMDRESP
948--------------------------------------------
949
950The format of CMDRESP is:
951
952* Pi - a non-negative integer media ID.
953
954* Pe - a non-negative integer error code.
955
5919b89a 956* The {args} is a set of key-value pairs (each pair separated by
40e3b486
KL
957 semicolon (';')).
958
959* A key can be any alpha-numeric ASCII string ('0' - '9', 'A' - 'Z',
960 'a' - 'z').
961
962* A value is any printable ASCII string not containing whitespace,
963 colon, or semicolon ('!' - '9', '<' - '~').
964
965
5919b89a 966
40e3b486
KL
967The Pe error codes are defined as:
968
5919b89a
KL
969| Value | Meaning | {args} containts |
970|-------|----------------------------------------|--------------------------|
971| 0 | No error occurred, i.e. success | nothing |
972| 1 | RESERVED FOR FUTURE USE | RESERVED FOR FUTURE USE |
40e3b486
KL
973| 2 | Invalid value - no media displayed | "key" that was incorrect |
974| 3 | Unsupported key - media displayed | "key" that unsupported |
975| 4 | Insufficient memory - no media displayed | nothing |
976| 5 | Other error - no media displayed | nothing |
977| 6 | Other - media displayed | nothing |
5919b89a
KL
978| 7 | RESERVED FOR FUTURE USE | RESERVED FOR FUTURE USE |
979| 8 | Media was evicted - no media displayed | nothing |
40e3b486
KL
980
981Additional Pe error codes may be returned; any Pe value except 0, 3,
982and 6 must mean that the media was not displayed.
5a16a542 983
930d75f0
KL
984
985
986Cached Multimedia - Examples
987----------------------------
988
989
990
991
992Layers - Summary
993----------------
994
995Layers introduce the concept of a layer "Z" coordinate to the existing
996rows ("Y") by columns ("X") grid. Put another way, the
997two-dimensional grid of columns-by-rows becomes a three-dimensional
998cube of columns-by-rows-by-layers. For this document, the column,
999row, and layer coordinates are referred to as X, Y, and Z. This
1000cartesian coordinate system is right-handed, with the Z axis pointing
1001"away" from the user "into" the screen.
1002
1003An application treats the Z coordinate exactly as it does X and Y
1004(rows and columns) coordinates:
1005
1006 * If it attemps to set Z to a value less than 1, then Z is set to 1.
1007
1008 * If it attempts to set Z to a value greater than the number of
1009 layers, then Z is set to the number of layers.
1010
40e3b486
KL
1011New sequences are provided to set and query Z, Y, X; to set and query
1012the screen cube size; and control visibility of Cells in-front-of
930d75f0
KL
1013other Cells.
1014
40e3b486
KL
1015Operations that can act on more than one Cell are defined such to act
1016on all layers simultaneously by default; most of these operations can
1017also be set to act only on the current layer.
930d75f0
KL
1018
1019
1020
1021Layers - Number of Layers
1022-------------------------
1023
1024A terminal is required to provide between 1 and a finite number of
1025layers.
1026
1027The number of layers may be different between the primary and
1028alternate screens.
1029
1030An application may request that the terminal allocate additional
1031layers. The terminal is free to honor or ignore such requests as it
1032sees fit.
1033
1034The scrollback buffer is permitted, and recommended, to contain only a
1035"flattened" single layer.
1036
1037
1038
1039Layers - Terminal State
1040-----------------------
1041
1042The terminal maintains a complex state at all times. This state
1043includes variables such as cursor position, foreground/background
1044color, attributes to apply to the next displayed character, and so on.
1045The layers feature adds more variables to the state, and these
1046variables are required to be stored with DECSC (ESC 7) and restored
1047with DECRC (ESC 8). The new variables are listed below:
1048
203cfc23
KL
1049| Mnemonic | Description | Default value |
1050|----------|-----------------------------|----------------|
1051| Z | Cursor position Z | 1 |
1052| MSL | Manipulate single layer | off / disabled |
1053| TFT | Text foreground transparent | false |
1054| TBT | Text background transparent | false |
930d75f0
KL
1055
1056
1057
1058Layers - Required Support For Existing Sequences
1059------------------------------------------------
1060
1061A terminal with layers feature must support the standard VT100/VT102
1062sequences defined in their respective manuals.
1063
1064
1065
1066Layers - New Sequences
1067----------------------
1068
1069A terminal with layer feature must support the following new
1070sequences:
1071
1072| Sequence | Command | Description |
1073|-------------------|-------------|----------------------------------------|
1074| CSI ? z ; y ; x H | CUPZ | Move cursor to (x, y, z) |
40e3b486
KL
1075| CSI 2 2 5 ; 1 ; Pa t | SLA | Set layer alpha |
1076| CSI ? 3 0 0 2 h | DECSET 3002 | Enable Manupulate Single Layer (MSL) |
1077| CSI ? 3 0 0 2 l | DECRST 3002 | Disable Manupulate Single Layer (MSL) |
930d75f0
KL
1078| CSI ? l ; h ; w t | RSZCUBE | Resize cube to (layers, height, width) |
1079
1080Default parameters and ranges are listed below:
1081
1082| Command | Position / Variable | Default Value | Minumum | Maximum |
1083|---------|---------------------|---------------|---------|-----------|
1084| CUPZ | 1 / z | 1 | 1 | # layers |
1085| CUPZ | 2 / y | 1 | 1 | # rows |
1086| CUPZ | 3 / x | 1 | 1 | # columns |
1087| SLA | 1 / alpha | 255 | 0 | 255 |
1088| RSZCUBE | 1 / l | 1 | 1 | varies |
40e3b486
KL
1089| RSZCUBE | 2 / h | 24 | 1 | varies |
1090| RSZCUBE | 3 / w | 80 | 1 | varies |
930d75f0
KL
1091
1092The terminal must also support the following new queries:
1093
191ed82c 1094| Query | Response | Description |
02c14905 1095|-----------------|-----------------------|--------------------------------|
191ed82c 1096| CSI ? 1 0 0 n | CSI ? z ; y ; x n | Report cursor Z, Y, X position |
930d75f0
KL
1097| CSI ? 1 8 t | CSI ? 8 ; l ; h ; w t | Report the text area cube layers, height, width |
1098
930d75f0
KL
1099The terminal must support the following new Set Graphics Rendition
1100(SGR) character attributes commands:
1101
1102| SGR Parameter | Description |
1103|---------------|---------------------------------------------|
40e3b486
KL
1104| 2 3 0 | Set text foreground color to transparent |
1105| 2 3 9 | Set text foreground color to solid (opaque) |
1106| 2 4 0 | Set text background color to transparent |
1107| 2 4 9 | Set text background color to solid (opaque) |
930d75f0
KL
1108
1109
1110
1111Layers - Error Handling
1112-----------------------
1113
1114No additional error reporting is provided for layer feature.
1115
1116
1117
1118Layers - Rendering
1119------------------
1120
1121A terminal with layer feature will display its Cells such that the
1122screen will appear as if it was rendered in the manner of the
1123pseudo-code below:
5a16a542
KL
1124
1125```
1126for each layer Z, in descending order from maxZ to minZ:
930d75f0 1127
5a16a542 1128 for each row Y, in ascending order from minY to maxY:
930d75f0 1129
5a16a542 1130 for each column X, in ascending order from minX to maxX:
930d75f0
KL
1131
1132 if tile at (X, Y, Z) background color is solid:
1133 draw rectangle of background color with layer alpha
1134
1135 if tile at (X, Y, Z) foreground color is solid:
1136 if tile at (X, Y, Z) is glyph:
1137 draw glyph with foreground color with layer alpha
1138 else
1139 draw pixel data of tile as red/green/blue/alpha pixels with
1140 layer alpha
1141
5a16a542
KL
1142 advance X by tile width
1143 next column
930d75f0 1144
5a16a542
KL
1145 advance Y by 1
1146 next row
930d75f0 1147
5a16a542
KL
1148 decrease Z by 1
1149next layer
1150```
1151
1152A terminal is free to optimize its rendering as it sees fit, so long
1153as the final screen output looks equivalent to the above method.
1154
1155
1156
930d75f0
KL
1157Layers - Integration With Existing Sequences
1158--------------------------------------------
1159
1160Sequences that insert characters/lines, delete characters/lines, or
1161modify larger regions are changed to act upon multiple layers as
8af51dab
KL
1162defined below. By default, MSL (Manipulate Single Layer) is
1163off/unset, and Z is 1, so if the application never changes MSL or Z
1164then these sequences will produce the same visible output as a
1165terminal without layer support.
930d75f0
KL
1166
1167A terminal is not required to support all of these sequences; however,
1168for those sequences it does support, if it supports the layers feature
1169then the sequences must behave as shown below:
1170
1171| Sequence | Command | Additional behavior |
1172|------------|-------------|------------------------------------------|
203cfc23
KL
1173| BS (0x08) | Backspace | Only current layer affected if MSL=on |
1174| DEL (0x7F) | Delete | Only current layer affected if MSL=on |
1175| IND (0x84) | Index | Only current layer affected if MSL=on |
1176| RI (0x8D | Reverse Index | Only current layer affected if MSL=on |
930d75f0
KL
1177| ESC # 3 | DECDHL | Cells on all layers always affected |
1178| ESC # 4 | DECDHL | Cells on all layers always affected |
1179| ESC # 5 | DECSWL | Cells on all layers always affected |
1180| ESC # 6 | DECDWL | Cells on all layers always affected |
203cfc23
KL
1181| ESC # 8 | DECALN | All layers > 1 cleared; Z, MSL, TFT, TBT reset to default |
1182| ESC 7 | DECSC | Also store Z, MSL, TFT, TBT |
1183| ESC 8 | DECRC | Also restore Z, MSL, TFT, TBT |
1184| ESC c | RIS | All layers > 1 cleared; Z, MSL, TFT, TBT reset to default |
1185| CSI @ | ICH | Only current layer affected if MSL=on |
1186| CSI J | ED | Only current layer affected if MSL=on |
1187| CSI K | EL | Only current layer affected if MSL=on |
1188| CSI ? K | DECSEL | Only current layer affected if MSL=on |
1189| CSI L | IL | Only current layer affected if MSL=on |
1190| CSI M | DL | Only current layer affected if MSL=on |
1191| CSI X | ECH | Only current layer affected if MSL=on |
1192| CSI M | DL | Only current layer affected if MSL=on |
1193| CSI P | DCH | Only current layer affected if MSL=on |
930d75f0 1194| CSI R | DECSTBM | Cells on all layers always affected |
203cfc23
KL
1195| CSI $ t | DECARA | Only current layer affected if MSL=on |
1196| CSI $ v | DECCRA | Only current layer affected if MSL=on |
930d75f0 1197| CSI x | DECSACE | Cells on all layers always affected |
203cfc23
KL
1198| CSI $ x | DECFRA | Only current layer affected if MSL=on |
1199| CSI $ z | DECERA | Only current layer affected if MSL=on |
930d75f0 1200
40e3b486
KL
1201(( TODO: add many more to the above table... ))
1202
930d75f0
KL
1203The VT52 sub-mode commands:
1204
1205| Sequence | Command | Additional behavior |
1206|------------|-------------|------------------------------------------|
203cfc23
KL
1207| ESC J | ED | Only current layer affected if MSL=on |
1208| ESC K | EL | Only current layer affected if MSL=on |
930d75f0
KL
1209
1210
1211
1212Layers - Use With Multiplexers
1213------------------------------
1214
1215Layers are inteded to provide a means for multiplexers to pass on the
1216job of multimedia support to the "outer" or host terminal. The
1217proposed mechanics of that is outlined in the pseudo-code below:
5a16a542 1218
930d75f0
KL
1219```
1220for each inner terminal in descending order from maxZ to minZ:
5a16a542 1221
930d75f0 1222 emit CUPZ(inner terminal Z, inner terminal Y, inner terminal X)
5a16a542 1223
930d75f0 1224 draw inner terminal text with standard VT100/VT102/xterm sequences
5a16a542 1225
930d75f0
KL
1226 for each multimedia sequence emitted by the inner terminal:
1227 emit CUP(inner terminal Y, inner terminal X)
1228 emit multimedia sequences to outer terminal
1229 next multimedia sequence
5a16a542 1230
930d75f0
KL
1231 decrease Z by 1
1232next inner terminal
1233```
5a16a542 1234
930d75f0
KL
1235The method above may not be effective for complex multi-terminal
1236screen layouts, but is hoped to work well for many simple cases.
5a16a542 1237
5a16a542
KL
1238
1239
930d75f0
KL
1240Layers - Examples
1241-----------------
5a16a542
KL
1242
1243
5a16a542
KL
1244
1245
930d75f0
KL
1246References
1247----------
5a16a542 1248
930d75f0 1249* xterm control sequences:
5a16a542
KL
1250
1251
930d75f0 1252* ECMA-48: