Sixel high cpu usage #658

Closed
opened 4 months ago by timsofteng · 8 comments

Hello. When I open some gif with chafa in foot I notice very high cpu usage of my laptop compair to the ordinary gui image viewer. Is it knowing issue with sixel in terminal?
Cant it be improved to be equal to gui?

Hello. When I open some gif with `chafa` in foot I notice very high cpu usage of my laptop compair to the ordinary gui image viewer. Is it knowing issue with sixel in terminal? Cant it be improved to be equal to gui?
Owner

Unfortunately not much can be done to improve this. Foot's sixel parser is already heavily optimized; for example, the same chafa command would peg xterm at 100% while foot is "only" at ~60-70%.

Sixel is very compute intense. The image has to first be palettized (i.e. converted from 24-bit "truecolor" to, typically, a 256-color palette), and then sixel encoded. Often, the same pixel sequences can be encoded in multiple ways, where some alternatives will be more expensive (much more, in some cases) to decode. Unfortunately, it often requires more CPU time on the producer side (i.e. chafa) to generate the more compact encodings. I.e. there may not be a net win.

In any case, I compared chafa with both img2sixel, and mpv1. Both of the latter two use libsixel, which produces very compact sixels.

In my tests, the same GIF, played at rougly the same size, used ~70% CPU with chafa. With mpv, foot used ~15-20%, and with img2sixel ~10%.

However, one should also take into account the CPU usage of the player. Chafa used ~115% the first loop iteration of the GIF, and ~10% after that. I.e. it's obvious that chafa caches the generated sixels. It's also pretty obvious that chafa isn't capable of playing the GIF at full speed the first iteration (and foot uses less CPU).

mpv uses ~200% CPU and does not appear to cache any sixels. img2sixel is limited to a single core and is actually unable to play the GIF at full speed. It also does not cache the sixels.

Sixel is CPU intense. Not much that can be done here, besides adding support for another image protocol.


  1. mpv can be built with sixel support, but e.g. arch does not enable this in their official build ↩︎

Unfortunately not much can be done to improve this. Foot's sixel parser is already heavily optimized; for example, the same chafa command would peg xterm at 100% while foot is "only" at ~60-70%. Sixel **is** very compute intense. The image has to first be palettized (i.e. converted from 24-bit "truecolor" to, typically, a 256-color palette), and then sixel encoded. Often, the same pixel sequences can be encoded in multiple ways, where some alternatives will be more expensive (much more, in some cases) to decode. Unfortunately, it often requires more CPU time on the producer side (i.e. chafa) to generate the more compact encodings. I.e. there may not be a net win. In any case, I compared chafa with both img2sixel, and mpv[^1]. Both of the latter two use [libsixel](https://github.com/libsixel/libsixel), which produces very compact sixels. In my tests, the same GIF, played at rougly the same size, used ~70% CPU with chafa. With mpv, foot used ~15-20%, and with img2sixel ~10%. **However**, one should also take into account the CPU usage of the player. Chafa used ~115% the **first** loop iteration of the GIF, and ~10% after that. I.e. it's obvious that chafa caches the generated sixels. It's also pretty obvious that chafa isn't capable of playing the GIF at full speed the first iteration (and foot uses less CPU). mpv uses ~200% CPU and does **not** appear to cache any sixels. img2sixel is limited to a single core and is actually unable to play the GIF at full speed. It also does **not** cache the sixels. Sixel **is** CPU intense. Not much that can be done here, besides adding support for another image protocol. [^1]: mpv _can_ be built with sixel support, but e.g. arch does **not** enable this in their official build
Owner

Sixel is usually good enough for static images. But for animations/movies/videos, it really isn't a good fit. Most players will struggle to keep up and tend to drop frames unless the video is played at a very small size.

Sixel is usually good enough for static images. But for animations/movies/videos, it really isn't a good fit. Most players will struggle to keep up and tend to drop frames unless the video is played at a very small size.
Poster

@dnkl Got it!
Thanks a lot for this clear answer!

So now world haven't good protocol to render images and videos in terminal. We have sixel which tmux and fzf doensn't suppot and which is not very efficient but better than nothing.
Did you thnik about implementation of your own protocol?
You designed very good wayland terminal from scratch and you are good enouth for this kind of architecture design.
New protocol which will be more efficient and well integrated to popular tools like tmux and fzf.

@dnkl Got it! Thanks a lot for this clear answer! So now world haven't good protocol to render images and videos in terminal. We have sixel which tmux and fzf doensn't suppot and which is not very efficient but better than nothing. Did you thnik about implementation of your own protocol? You designed very good wayland terminal from scratch and you are good enouth for this kind of architecture design. New protocol which will be more efficient and well integrated to popular tools like tmux and fzf.
Owner

There's kitty's graphics protocol, which is pretty good. I've considered adding support for it, but it's a bit too complex imho.

Not sure it would be that much better performance wise. In my experiments with notcurses (a TUI library that supports both sixel and kitty's graphics protocol), foot+sixel was faster (but barely) than kitty.

It is a much more modern protocol however, better suited for "modern" applications, with proper truecolor (with our without alpha channel) support. But again, it's complex, with too many features imho, and as such may be too much for a lightweight terminal like foot. But perhaps it would be possible to implement a subset of the protocol.

There's kitty's graphics protocol, which is pretty good. I've considered adding support for it, but it's a bit too complex imho. Not sure it would be that much better performance wise. In my experiments with [notcurses](https://github.com/dankamongmen/notcurses) (a TUI library that supports both sixel and kitty's graphics protocol), foot+sixel was faster (but barely) than kitty. It is a much more modern protocol however, better suited for "modern" applications, with proper truecolor (with our without alpha channel) support. But again, it's complex, with too many features imho, and as such may be too much for a lightweight terminal like foot. But perhaps it would be possible to implement a subset of the protocol.
Owner

Closing as a dup of #481

Closing as a dup of https://codeberg.org/dnkl/foot/issues/481
dnkl closed this issue 4 months ago
Poster

@dnkl
updated
I've tried kitty with viu image viewer and it works pretty nice with playing gifs.
CPU usage still very low.

@dnkl updated I've tried `kitty` with `viu` image viewer and it works pretty nice with playing gifs. CPU usage still very low.
Owner

Well, one of its features is "animation" support. So sure, I imagine that results in much lower CPU usage when the application supports it.

Well, one of its features is "animation" support. So sure, I imagine that results in much lower CPU usage when the application supports it.
Poster

Well, one of its features is "animation" support. So sure, I imagine that results in much lower CPU usage when the application supports it.

For me it's quite enough in terms of cpu usage efficient. Please check it. If it is due to protocol so maybe kitty's protocol better than sixel... I don't know. But I know that I really wanna see images and video in terminal:)

It's sad that that we still have no one common good protocol for images in terminal.
Sixel, kitty's protocol, iterm2 protocol...
Tmux and fzf support nothing.

> Well, one of its features is "animation" support. So sure, I imagine that results in much lower CPU usage when the application supports it. For me it's quite enough in terms of cpu usage efficient. Please check it. If it is due to protocol so maybe kitty's protocol better than sixel... I don't know. But I know that I really wanna see images and video in terminal:) It's sad that that we still have no one common good protocol for images in terminal. Sixel, kitty's protocol, iterm2 protocol... Tmux and fzf support nothing.
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.