Overlapping transparent sixels #562

Closed
opened 5 months ago by dnkl · 6 comments
dnkl commented 5 months ago
Owner

Writing a sixel on top of another sixel erases the first sixel on a per-cell basis. That is, all cells touched by the new sixel are cleared from the old sixel.

This means one cannot print multiple overlapping, transparent sixels on top of each other and get the expected result.

XTerm handles this correctly.

I have no idea how we're supposed to implement this, but I'm sure there's a way.

@dankamongmen

Writing a sixel on top of another sixel erases the first sixel on a per-cell basis. That is, all **cells** touched by the new sixel are cleared from the old sixel. This means one cannot print multiple overlapping, transparent sixels on top of each other and get the expected result. XTerm handles this correctly. I have no idea _how_ we're supposed to implement this, but I'm sure there's a way. @dankamongmen
dnkl added the
enhancement
label 5 months ago

outstanding, glad to hear you're intending to support this.

outstanding, glad to hear you're intending to support this.
Poster
Owner

Each sixel currently lives in it's own pixman image buffer. Whenever a sixel is printed on top of another sixel (or we print a character to a cell covered by a sixel), the already existing sixel is split up into four pieces: top, left, right and bottom tiles.

We could technically skip this step when a sixel is printed on top of another sixel. As long as we render the sixels from bottom (oldest) to top (newest), we'll get the correct result.

However, this breaks down when playing animations, where lots of frames (sixels) are printed in the same location. We need to figure out when to not keep the underlying sixel, but instead revert back to the current behavior; splitting it up into four tiles. Or in case it's completely overwritten, remove it.

Obviously, if a sixel is opaque (P2=0|2), then we know what to - remove/split the underlying sixel. This'll be the case for mpv, and most animations I think.

But what about the, for example, xray demo in notcurses? It animates transparent sixels, if I remember correctly.

There are two things we need to solve in this bug:

  • Determine when to keep the underlying sixel(s), and when to split/remove it
  • Render overlapping sixels in the correct order. Sixels are sorted by their row number. So, if the first sixel starts at a higher row number than the second sixel, the second sixel will be rendered before the first sixel. Not good.
Each sixel currently lives in it's own pixman image buffer. Whenever a sixel is printed on top of another sixel (or we print a character to a cell covered by a sixel), the already existing sixel is split up into four pieces: top, left, right and bottom tiles. We could technically skip this step when a sixel is printed on top of another sixel. As long as we render the sixels from bottom (oldest) to top (newest), we'll get the correct result. However, this breaks down when playing animations, where **lots** of frames (sixels) are printed in the same location. We need to figure out when to _not_ keep the underlying sixel, but instead revert back to the current behavior; splitting it up into four tiles. Or in case it's completely overwritten, remove it. Obviously, if a sixel is opaque (`P2=0|2`), then we know what to - remove/split the underlying sixel. This'll be the case for mpv, and most animations I think. But what about the, for example, xray demo in notcurses? It animates transparent sixels, if I remember correctly. There are two things we need to solve in this bug: * Determine when to keep the underlying sixel(s), and when to split/remove it * Render overlapping sixels in the correct order. Sixels are sorted by their row number. So, if the first sixel starts at a higher row number than the second sixel, the second sixel will be rendered **before** the first sixel. Not good.
Poster
Owner

The plan is to keep doing what we're doing; split up the sixel being overwritten into four tiles. But instead of throwing away the part that's being overwritten, blend it with the new sixel.

This way, we don't need to worry about "leaking" sixel images. We don't need to worry about transparent vs. opaque sixels (though an obvious optimization is to skip blending when the new sixel is fully opaque), and since there's no longer multiple layers of sixels, rendering order is no longer an issue.

If done correctly, this will also solve another problem we're currently having; a sixel that isn't a multiple of the cell width and/or height wipes whole cells from the sixel below.

The plan is to keep doing what we're doing; split up the sixel being overwritten into four tiles. But instead of throwing away the part that's being overwritten, blend it with the new sixel. This way, we don't need to worry about "leaking" sixel images. We don't need to worry about transparent vs. opaque sixels (though an obvious optimization is to skip blending when the new sixel is fully opaque), and since there's no longer multiple layers of sixels, rendering order is no longer an issue. If done correctly, this will also solve another problem we're currently having; a sixel that isn't a multiple of the cell width and/or height wipes whole cells from the sixel below.
Poster
Owner

#576 should solve this one. As expected, xray from the notcurses-demo now looks like this:

https://codeberg.org/dnkl/foot/pulls/576 should solve this one. As expected, `xray` from the notcurses-demo now looks like this:
7.7 MiB
dnkl referenced this issue from a commit 5 months ago
dnkl referenced this issue from a commit 5 months ago
dnkl referenced this issue from a commit 5 months ago
Poster
Owner

xray screenshot:
https://codeberg.org/attachments/7a6cd3b1-e970-4607-a600-cf0ee4127db8

(white line to the right is my window border...)

`xray` screenshot: ![https://codeberg.org/attachments/7a6cd3b1-e970-4607-a600-cf0ee4127db8](https://codeberg.org/attachments/7a6cd3b1-e970-4607-a600-cf0ee4127db8) (white line to the right is my window border...)
dnkl added this to the 1.8.0 milestone 5 months ago

nice work. i need to fix up xray to work properly on such implementations, but this is definitely the way to go.

nice work. i need to fix up xray to work properly on such implementations, but this is definitely the way to go.
dnkl referenced this issue from a commit 5 months ago
dnkl referenced this issue from a commit 5 months ago
dnkl closed this issue 5 months ago
dnkl referenced this issue from a commit 5 months ago
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.