Sixel background color, and transparency #391

Closed
opened 8 months ago by dnkl · 6 comments
dnkl commented 8 months ago
Owner

There's a discussion over at https://github.com/dankamongmen/notcurses/pull/1384 about the meaning of the P2 DCS parameter, and the "Set Raster Attributes" command; how the interact and how they affect “empty” pixels.

P2 is one of the DCS parameters in the Sixel "Device Control String":

DCS P1 ; P2; P3; q s..s ST

P2 selects how the terminal draws the background color. You can use one of three values.

  • 0 or 2 (default) Pixel positions specified as 0 are set to the current background color.
  • 1 Pixel positions specified as 0 remain at their current color.

The "Set Raster Attributes" is mainly used to inform the terminal of the final image size, but also affect the background::

" Pan ; Pad; Ph; Pv

Ph and Pv define the horizontal and vertical size of the image (in pixels), respectively.

Ph and Pv do not limit the size of the image defined by the sixel data. However, Ph and Pv let you omit background sixel data from the image definition and still have a color background. They also provide a concise way for the application or terminal to encode the size of an image.

NOTE: The VT300 uses Ph and Pv to erase the background when P2 is set to 0 or 2.

The above raises a couple of questions:

What does “pixel positions specified as 0“ mean?
Each sixel character defines a 1x6 (1 column, 6 rows) pixel stripe, where a 1-bit in the character code means use the currently selected sixel color register for the corresponding pixel. You can print multiple sixels on top of each other, to set different pixels in the stripe to different colors. Pixel positions specified as 0 probably means pixels where no sixels contained a 1-bit.
What does “current background color” mean?
Sixels don’t have background colors. There are sixel color registers, that are used to paint the pixels corresponding to 1-bits in sixels. I have found no mentioning of some registers being used for something else.
“remain at their current color“ sounds like transparency...
I mean, what else could it mean?
If 0-pixels are set to the current background color when P2=0, why are they explicitly mentioning Ph/Pv being used to erase the background? Can’t you just emit empty sixels?
Perhaps it’s as simple as: it allows you to erase a large area without emitting any sixel characters at all. I.e. it saves bandwidth.

My understand of the above is this:

  • Using “Set Raster Attributes” doesn’t change how the background is painted, except that you can use it to erase a much larger area than your final sixel will cover.
  • P2=0|2 means use the current ANSI background color, as set with \E[48m.
  • P2=1` means transparency.

Unfortunately, both XTerm and mlterm disagree. However, they also disagree with each other...

XTerm: behaves as if P2=1 if no raster attributes is used. I.e. empty pixels are transparent. For P2=0 (when we do use raster attributes), it does not use the current ANSI background color, but uses sixel color register #0.

mlterm: behaves the same regardless of P2 and the presence of raster attributes; it fills empty pixels with the current ANSI background color.

See attached screenshots.

So, mlterm supports the claim that we should be using the current ANSI background color for P2=0|2. XTerm supports the claim that P2=1 means transparent.

I haven’t been able to find any support for XTerms choice of using sixel color register #0 instead of the current ANSI background color. I also do not understand why XTerm treats P2=0|2 as P2=1 when the raster attributes are omitted.

Given the above, this is what I think foot should do:

P2=0|2, no raster attributes
Use current ANSI background color for empty pixels. Final image size is determined solely by the emitted sixels.
P2=0|2, with raster attributes
Use current ANSI background color for empty pixels. Final image size is whatever is largest; the area specified by Ph and Pv, or the emitted sixels. If case Ph and Pv is larger, the extra space is also filled with the current ANSI background color.
P2=1, no raster attributes
Empty pixels are transparent. Final image size is determined solely by the emitted sixels.
P2=1, with raster attributes
Empty pixels are transparent. Final image size is whatever is largest; the area specified by Ph and Pv, or the emitted sixels. If case Ph and Pv is larger, the extra space is transparent.

Current behavior: foot ignores P2, and always fills empty pixels with the default background color (i.e. not the current background color). Final image size is handled correctly in the not-yet-merged sixel-optimize branch.

The images in the screenshots are from a hand-crafted raw "text" file. Simply cat it. Make sure the terminal window is large enough - if the output scrolls the cursor movements will be correct. The size of the X-rectangles aren't calculated, you may have to adjust the font size for it to look good (I used a 12px font). To avoid corrupting the terminal content, you may want to do something like:

echo -e '\e[?1049h' && cat sixel-p2-vs-raster-attributes.txt && sleep 10 ; echo -e '\e[?1049l'

cat:ing the file will write a green-on-red rectangle of X's (as background, to be able to see transparency), and then write a sixel image on top of it. Before writing the sixel, we set the background color to blue (with \E[48:2::0:0:255m).

There's a discussion over at https://github.com/dankamongmen/notcurses/pull/1384 about the meaning of the P2 DCS parameter, and the "Set Raster Attributes" command; how the interact and how they affect “empty” pixels. P2 is one of the DCS parameters in the Sixel "Device Control String": > `DCS P1 ; P2; P3; q s..s ST` > > P2 selects how the terminal draws the background color. You can use one of three values. > > * 0 or 2 (default) Pixel positions specified as 0 are set to the current background color. > * 1 Pixel positions specified as 0 remain at their current color. The "Set Raster Attributes" is mainly used to inform the terminal of the final image size, but also affect the background:: > `" Pan ; Pad; Ph; Pv` > > Ph and Pv define the horizontal and vertical size of the image (in pixels), respectively. > > Ph and Pv do not limit the size of the image defined by the sixel data. However, Ph and Pv let you omit background sixel data from the image definition and still have a color background. They also provide a concise way for the application or terminal to encode the size of an image. > > NOTE: The VT300 uses Ph and Pv to erase the background when P2 is set to 0 or 2. The above raises a couple of questions: What does “pixel positions specified as 0“ mean? : Each sixel character defines a 1x6 (1 column, 6 rows) pixel stripe, where a 1-bit in the character code means use the currently selected sixel color register for the corresponding pixel. You can print multiple sixels on top of each other, to set different pixels in the stripe to different colors. Pixel positions specified as 0 probably means pixels where **no** sixels contained a 1-bit. What does “current background color” mean? : Sixels don’t _have_ background colors. There are sixel color registers, that are used to paint the pixels corresponding to `1`-bits in sixels. I have found no mentioning of some registers being used for something else. “remain at their current color“ _sounds_ like transparency... : I mean, what else could it mean? If 0-pixels are set to the current background color when `P2=0`, why are they explicitly mentioning `Ph/Pv` being used to erase the background? Can’t you just emit empty sixels? : Perhaps it’s as simple as: it allows you to erase a large area without emitting any sixel characters at all. I.e. it saves bandwidth. My understand of the above is this: * Using “Set Raster Attributes” doesn’t _change_ how the background is painted, _except_ that you can use it to erase a much larger area than your final sixel will cover. * `P2=0|2` means use the current ANSI background color, as set with `\E[48m`. * P2=1` means transparency. Unfortunately, both XTerm and mlterm disagree. However, they also disagree with each other... XTerm: behaves as if `P2=1` if no raster attributes is used. I.e. empty pixels are transparent. For `P2=0` (when we _do_ use raster attributes), it does **not** use the current ANSI background color, but uses sixel color register `#0`. mlterm: behaves the same regardless of `P2` and the presence of raster attributes; it fills empty pixels with the current ANSI background color. See attached screenshots. So, mlterm supports the claim that we should be using the current ANSI background color for `P2=0|2`. XTerm supports the claim that `P2=1` means transparent. I haven’t been able to find any support for XTerms choice of using sixel color register `#0` instead of the current ANSI background color. I also do not understand why XTerm treats `P2=0|2` as `P2=1` when the raster attributes are omitted. Given the above, this is what I think foot should do: `P2=0|2`, no raster attributes : Use current ANSI background color for empty pixels. Final image size is determined solely by the emitted sixels. `P2=0|2`, with raster attributes : Use current ANSI background color for empty pixels. Final image size is whatever is largest; the area specified by `Ph` and `Pv`, or the emitted sixels. If case `Ph` and `Pv` is larger, the extra space is also filled with the current ANSI background color. `P2=1`, no raster attributes : Empty pixels are transparent. Final image size is determined solely by the emitted sixels. `P2=1`, with raster attributes : Empty pixels are transparent. Final image size is whatever is largest; the area specified by `Ph` and `Pv`, or the emitted sixels. If case `Ph` and `Pv` is larger, the extra space is transparent. Current behavior: foot ignores `P2`, and always fills empty pixels with the **default** background color (i.e. not the _current_ background color). Final image size is handled correctly in the not-yet-merged `sixel-optimize` branch. The images in the screenshots are from a hand-crafted raw "text" file. Simply `cat` it. Make sure the terminal window is large enough - if the output scrolls the cursor movements will be correct. The size of the X-rectangles aren't calculated, you may have to adjust the font size for it to look good (I used a 12px font). To avoid corrupting the terminal content, you may want to do something like: ```sh echo -e '\e[?1049h' && cat sixel-p2-vs-raster-attributes.txt && sleep 10 ; echo -e '\e[?1049l' ``` `cat`:ing the file will write a green-on-red rectangle of X's (as background, to be able to see transparency), and then write a sixel image on top of it. Before writing the sixel, we set the background color to blue (with `\E[48:2::0:0:255m`).
dnkl added this to the 1.7.0 milestone 8 months ago
dnkl added a new dependency 8 months ago
dnkl added the
enhancement
bug
labels 8 months ago
dnkl added a new dependency 8 months ago
Poster
Owner

Added #388 as a dependendy, because it implements trimming of the final image height, when the last sixel row(s) have all-empty pixel rows.

And because that PR touches nearly all code in sixel.c related to sixel decoding.

Added #388 as a dependendy, because it implements trimming of the final image height, when the last sixel row(s) have all-empty pixel rows. And because that PR touches nearly all code in `sixel.c` related to sixel decoding.
dnkl closed this issue 8 months ago

So if I write a chunk of transparent sixel (ideally just P2==1 and geometry), does that clear any existing sixel output within that region, showing the underlying text? Or would this be a no-op?

So if I write a chunk of transparent sixel (ideally *just* P2==1 and geometry), does that clear any existing sixel output within that region, showing the underlying text? Or would this be a no-op?
Poster
Owner

That would be a no-op. At least on XTerm and foot (though a bug in foot's current implementation means it isn't quite a no-op).

First, emitting a geometry with P2=1 is a no-op (in terms of erasing anything - foot also uses the geometry hint to resize its internal backing buffer, but that's irrelevant here).

Second, XTerm, and foot, tracks the last sixel row containing non-empty pixels, and trims the output image based on this. I.e. if all you do is emit empty pixels (i.e. the ? sixel character), then the image will be trimmed to nothing. Likewise, if all you do is emit the geometry hint, but no sixel characters at all, the image will also be trimmed to nothing. Meaning, the pre-existing image is left untouched.

(I hacked the sixel-p2-vs-raster-attributes.txt (uploaded file not updated!) to test this)

That would be a no-op. At least on XTerm and foot (though a bug in foot's current implementation means it isn't _quite_ a no-op). First, emitting a geometry with `P2=1` is a no-op (in terms of erasing anything - foot _also_ uses the geometry hint to resize its internal backing buffer, but that's irrelevant here). Second, XTerm, and foot, tracks the last sixel row containing non-empty pixels, and trims the output image based on this. I.e. if all you do is emit empty pixels (i.e. the `?` sixel character), then the image will be trimmed to nothing. Likewise, if all you do is emit the geometry hint, but no sixel characters at all, the image will also be trimmed to nothing. Meaning, the pre-existing image is left untouched. (I hacked the [`sixel-p2-vs-raster-attributes.txt`](https://codeberg.org/attachments/b62785da-b9bb-4a14-b455-36877b27eb5c) (uploaded file **not** updated!) to test this)
Poster
Owner

Also worth mentioning; at least mlterm, and foot, erases sixels on a per-cell basis. I.e. printing a small sixel (smaller than a cell) on top of another sixel will erase the whole cell from the original sixel image.

I'm still investigating whether there's a sane way to update foot to match XTerm's behavior.

Anyway, just thought I'd mention it; not sure how e.g. Alacritty's not-yet-merged-PR handles this case.

Also worth mentioning; at least mlterm, and foot, erases sixels on a per-cell basis. I.e. printing a small sixel (smaller than a cell) on top of another sixel will erase the whole cell from the original sixel image. I'm still investigating whether there's a sane way to update foot to match XTerm's behavior. Anyway, just thought I'd mention it; not sure how e.g. Alacritty's not-yet-merged-PR handles this case.

I see you've played around with mlterm; in my experience, it only prints sixels at the origin of the viewable area.

Interesting that a partial cell bitmap write blows away a cell worth of sixel. So there's no way to "stack" sixels in foot via successive output? That's alright; I can easily compose bitmaps (glyphs are much more difficult).

I think you can call this closed; thanks for the reply (I never got a notification via mail, or at least missed it) and sorry for the delay in getting back to you! This confirms a lot of things I've learned elsewhere.

If 0-pixels are set to the current background color when P2=0, why are they explicitly mentioning Ph/Pv being used to erase the background? Can’t you just emit empty sixels?
Perhaps it’s as simple as: it allows you to erase a large area without emitting any sixel characters at all. I.e. it saves bandwidth.

hot shit, i need to be using this!

I see you've played around with mlterm; in my experience, it only prints sixels at the origin of the viewable area. Interesting that a partial cell bitmap write blows away a cell worth of sixel. So there's no way to "stack" sixels in foot via successive output? That's alright; I can easily compose bitmaps (glyphs are much more difficult). I think you can call this closed; thanks for the reply (I never got a notification via mail, or at least missed it) and sorry for the delay in getting back to you! This confirms a lot of things I've learned elsewhere. > If 0-pixels are set to the current background color when P2=0, why are they explicitly mentioning Ph/Pv being used to erase the background? Can’t you just emit empty sixels? Perhaps it’s as simple as: it allows you to erase a large area without emitting any sixel characters at all. I.e. it saves bandwidth. hot shit, i need to be using this!
Poster
Owner

Interesting that a partial cell bitmap write blows away a cell worth of sixel.

Yeah, foot's sixel-overwrite logic (both sixel overwriting sixel, and ascii overwriting sixel) is currently cell based. Or rather, a sixel is always anchored to a cell's upper left corner, rather than a specific window pixel coordinate. This makes it possible to keep sixels around when resizing/reflowing the window. Not in all cases, but some at least.

So there's no way to "stack" sixels in foot via successive output?

You can, in the sense that writing a sixel on top of another sixel doesn't blow away the first sixel competely; only the cells touched by the second sixel. But yeah, no stacking at pixel level.

Solving this is tightly coupled with the first issue, with the main issue being that sixels are anchored to cells.

sorry for the delay in getting back to you!

No worries; I'm tracking notcurses pretty closely anyway. In fact, I'm putting together a small ls-for-images utility based on notcurses :)

> Interesting that a partial cell bitmap write blows away a cell worth of sixel. Yeah, foot's sixel-overwrite logic (both sixel overwriting sixel, and ascii overwriting sixel) is currently cell based. Or rather, a sixel is always anchored to a cell's upper left corner, rather than a specific window pixel coordinate. This makes it possible to keep sixels around when resizing/reflowing the window. Not in all cases, but some at least. > So there's no way to "stack" sixels in foot via successive output? You can, in the sense that writing a sixel on top of another sixel doesn't blow away the first sixel competely; only the **cells** touched by the second sixel. But yeah, no stacking at pixel level. Solving this is tightly coupled with the first issue, with the main issue being that sixels are anchored to cells. > sorry for the delay in getting back to you! No worries; I'm tracking notcurses pretty closely anyway. In fact, I'm putting together a small ls-for-images utility based on notcurses :)
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Blocks
#286 1.7.0
dnkl/foot
Depends on
Loading…
There is no content yet.