New Issue

Some Shift+key not detected in neovim (non-stable) with non-qwerty keyboard layout #1009

Closed

opened by cljoly · 20 comments

cljoly commented

foot version:

foot --version
foot version: 1.11.0 +pgo +ime +graphemes -assertions

./foot --version
foot version: 1.11.0-97-gf2870bf5 (Apr 05 2022, branch 'master') -pgo +ime +graphemes +assertions

I’m using a French dvorak-style keyboard layout (bepo) and I recently noticed that my neovim binding on ; (which I enter with Shift+,, , being on the physical key g on a qwerty keyboard) stopped working.

This behavior is only exhibited by neovim on some yet unreleased master commits. Compiling neovim and runnig it like so (i.e. without any custom configuration and defining the mappings on the fly):

./build/bin/nvim -u NONE +'nmap ; :echo ";"<CR>'  +'nmap , :echo ","<CR>'

allows to check this behavior on a locally built neovim from the git repository. On the latest neovim stable (v0.6.1), entering ; results in ; being displayed, but not on those latest master commits. Using git bisect, I’ve found this small commit to be the first with the ; mapping not working, while , works.

Meanwhile, the ; mapping still works with Alacritty, for all neovim commits I tested. That suggests that foot might be sending the wrong sequence here.

foot version: ``` foot --version foot version: 1.11.0 +pgo +ime +graphemes -assertions ``` or ``` ./foot --version foot version: 1.11.0-97-gf2870bf5 (Apr 05 2022, branch 'master') -pgo +ime +graphemes +assertions ``` ************************************** I’m using a French dvorak-style keyboard layout ([bepo][]) and I recently noticed that my [neovim][] binding on `;` (which I enter with `Shift+,`, `,` being on the physical key `g` on a qwerty keyboard) stopped working. This behavior is only exhibited by neovim on some yet unreleased `master` commits. Compiling neovim and runnig it like so (i.e. without any custom configuration and defining the mappings on the fly): ``` ./build/bin/nvim -u NONE +'nmap ; :echo ";"<CR>' +'nmap , :echo ","<CR>' ``` allows to check this behavior on a locally built neovim from the git repository. On the latest neovim stable (`v0.6.1`), entering `;` results in `;` being displayed, but not on those latest `master` commits. Using `git bisect`, I’ve found [this small commit](https://github.com/neovim/neovim/commit/b81c310d4) to be the first with the `;` mapping not working, while `,` works. Meanwhile, the `;` mapping still works with Alacritty, for all neovim commits I tested. That suggests that *foot* might be sending the wrong sequence here. [bepo]: https://bepo.fr/wiki/Accueil [neovim]: https://neovim.io/

cljoly commented

Poster

Just tested with Kitty, couldn’t reproduce the bug either.

cljoly changed title from ~~Some Shift+key not detected in neovim (non-stable) with non-qwerty keyboard~~ to Some Shift+key not detected in neovim (non-stable) with non-qwerty keyboard layout

gpanders commented

Alacritty does not recognize the modifyOtherKeys sequence (the one Neovim is setting in the commit you linked to), and we explicitly do not send the sequence for kitty since it uses its own encoding. Do you have another terminal emulator you can try to reproduce on?

I am not familiar with foot's codebase, but I believe what's happening is:

Neovim sends CSI > 4 ; 2 m which enables foot's modifyOtherKeys handling
When you press Shift + , this is no longer encoded as ASCII ; but instead CSI 27 ; 2 ; 44 ~ (where 2 is the modifier for Shift and 44 is the decimal representation of ,). Neovim is then not interpreting this sequence properly.

Can you also try reproducing in regular Vim by adding the following to your vimrc?

let &t_TI = "\<Esc>[>4;2m"
let &t_TE = "\<Esc>[>4;0m"

Alacritty does not recognize the `modifyOtherKeys` sequence (the one Neovim is setting in the commit you linked to), and we explicitly do not send the sequence for kitty since it uses its own encoding. Do you have another terminal emulator you can try to reproduce on? I am not familiar with foot's codebase, but I believe what's happening is: 1. Neovim sends `CSI > 4 ; 2 m` which enables foot's `modifyOtherKeys` handling 2. When you press `Shift + ,` this is no longer encoded as ASCII `;` but instead `CSI 27 ; 2 ; 44 ~` (where `2` is the modifier for Shift and `44` is the decimal representation of `,`). Neovim is then not interpreting this sequence properly. Can you also try reproducing in regular Vim by adding the following to your vimrc? ```vim let &t_TI = "\<Esc>[>4;2m" let &t_TE = "\<Esc>[>4;0m" ```

dnkl commented

Owner

Could also be worth testing/comparing with xterm, which does support (obviously) modifyOtherKeys.

Also note that foot supports kitty's keyboard protocol as well, so if neovim works better with that, another option is to change neovim to use the kitty kbd protocol (instead of modifyOtherKeys) on foot as well.

I also agree with @gpanders analysis, though I haven't verified it.

Could also be worth testing/comparing with xterm, which does support (obviously) `modifyOtherKeys`. Also note that foot supports kitty's keyboard protocol as well, so if neovim works better with that, another option is to change neovim to use the kitty kbd protocol (instead of modifyOtherKeys) on foot as well. I also agree with @gpanders analysis, though I haven't verified it.

gpanders commented

@cljoly I am not convinced that this is not a bug in Neovim, so could you also open an issue in our issue tracker?

@cljoly I am not convinced that this is *not* a bug in Neovim, so could you also open an issue in our issue tracker?

cljoly commented

Poster

Thanks for your quick feedback on this!

Tomorrow, I will:

try reproducing in regular Vim
try reproducing on xterm
open an issue on neovim issue tracker

Do you have another terminal emulator you can try to reproduce on?

I should be able to test with any terminal emulator installable from Arch’s repositories.

Thanks for your quick feedback on this! Tomorrow, I will: - [x] try reproducing in regular Vim - [x] try reproducing on xterm - [x] open an issue on neovim issue tracker > Do you have another terminal emulator you can try to reproduce on? I should be able to test with any terminal emulator installable from Arch’s repositories.

dnkl commented

Owner

It appears XTerm is somewhat inconsistent here. For regular (a-z) keys, it produces a CSI 27 ; 2 ; N~ escape for shift combinations. But for some, it does not, but instead produces a plain (but shifted) symbol. , vs. ; is one such example. Most (all?) shift+number keys also behave like this (on my Swedish layout).

However, it's not as simple as letter keys vs. non-letter keys; e.g. - produces a CSI 27 escape.

This can be considered to be an issue with foot, in the sense that we're not producing output identical to XTerm.

But it could also be unintended behavior in XTerm.

It appears XTerm is somewhat inconsistent here. For regular (a-z) keys, it produces a `CSI 27 ; 2 ; N~` escape for <kbd>shift</kbd> combinations. But for some, it does not, but instead produces a plain (but shifted) symbol. `,` vs. `;` is one such example. Most (all?) <kbd>shift</kbd>+<kbd>number</kbd> keys also behave like this (on my Swedish layout). However, it's not as simple as letter keys vs. non-letter keys; e.g. `-` produces a `CSI 27` escape. This can be considered to be an issue with foot, in the sense that we're not producing output identical to XTerm. But it _could_ also be unintended behavior in XTerm.

cljoly commented

Poster

Just tried reproducing, XTerm does not have the bug.

cljoly commented

Poster

Trying to reproduce with vim:

~> vim --version --help
VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Mar 30 2022 23:54:07)
Included patches: 1-4651
Compiled by Arch Linux
Huge version without GUI.  Features included (+) or not (-):
+acl               +file_in_path      +mouse_urxvt       -tag_any_white
+arabic            +find_in_path      +mouse_xterm       +tcl/dyn
+autocmd           +float             +multi_byte        +termguicolors
+autochdir         +folding           +multi_lang        +terminal
-autoservername    -footer            -mzscheme          +terminfo
-balloon_eval      +fork()            +netbeans_intg     +termresponse
+balloon_eval_term +gettext           +num64             +textobjects
-browse            -hangul_input      +packages          +textprop
++builtin_terms    +iconv             +path_extra        +timers
+byte_offset       +insert_expand     +perl/dyn          +title
+channel           +ipv6              +persistent_undo   -toolbar
+cindent           +job               +popupwin          +user_commands
-clientserver      +jumplist          +postscript        +vartabs
-clipboard         +keymap            +printer           +vertsplit
+cmdline_compl     +lambda            +profile           +vim9script
+cmdline_hist      +langmap           -python            +viminfo
+cmdline_info      +libcall           +python3/dyn       +virtualedit
+comments          +linebreak         +quickfix          +visual
+conceal           +lispindent        +reltime           +visualextra
+cryptv            +listcmds          +rightleft         +vreplace
+cscope            +localmap          +ruby/dyn          +wildignore
+cursorbind        +lua/dyn           +scrollbind        +wildmenu
+cursorshape       +menu              +signs             +windows
+dialog_con        +mksession         +smartindent       +writebackup
+diff              +modify_fname      -sodium            -X11
+digraphs          +mouse             -sound             -xfontset
-dnd               -mouseshape        +spell             -xim
-ebcdic            +mouse_dec         +startuptime       -xpm
+emacs_tags        +mouse_gpm         +statusline        -xsmp
+eval              -mouse_jsbterm     -sun_workshop      -xterm_clipboard
+ex_extra          +mouse_netterm     +syntax            -xterm_save
+extra_search      +mouse_sgr         +tag_binary        
-farsi             -mouse_sysmouse    -tag_old_static    
   system vimrc file: "/etc/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -flto=auto -D_REENTRANT -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 
Linking: gcc -L. -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,-E -Wl,-rpath,/usr/lib/perl5/5.34/core_perl/CORE -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto -L/usr/local/lib -o vim -lm -ltinfo -lelf -lacl -lattr -lgpm -Wl,-E -Wl,-rpath,/usr/lib/perl5/5.34/core_perl/CORE -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fstack-protector-strong -L/usr/local/lib -L/usr/lib/perl5/5.34/core_perl/CORE -lperl -lpthread -ldl -lm -lcrypt -lutil -lc -L/usr/lib -ltclstub8.6 -ldl -lz -lpthread -lm

Using foot 1.11.0, with this command:

vim -u NONE +'let &t_TE = "\<Esc>[>4;0m"' +'let &t_TI = "\<Esc>[>4;2m"' +'nmap ; :echo ";"<CR>'  +'nmap , :echo ","<CR>'

vim gets the mapping for both ; and ,. (athough it does not understand the <CR> somehow, not a big deal, I get the partial :echo command and that’s enough to test).

EDIT: Tested with vim -u DEFAULTS +'let &t_TE = "\<Esc>[>4;0m"' +'let &t_TI = "\<Esc>[>4;2m"' +'nmap ; :echo ";"<CR>' +'nmap , :echo ","<CR>' as well, and both mappings work.

Trying to reproduce with vim: ``` ~> vim --version --help VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Mar 30 2022 23:54:07) Included patches: 1-4651 Compiled by Arch Linux Huge version without GUI. Features included (+) or not (-): +acl +file_in_path +mouse_urxvt -tag_any_white +arabic +find_in_path +mouse_xterm +tcl/dyn +autocmd +float +multi_byte +termguicolors +autochdir +folding +multi_lang +terminal -autoservername -footer -mzscheme +terminfo -balloon_eval +fork() +netbeans_intg +termresponse +balloon_eval_term +gettext +num64 +textobjects -browse -hangul_input +packages +textprop ++builtin_terms +iconv +path_extra +timers +byte_offset +insert_expand +perl/dyn +title +channel +ipv6 +persistent_undo -toolbar +cindent +job +popupwin +user_commands -clientserver +jumplist +postscript +vartabs -clipboard +keymap +printer +vertsplit +cmdline_compl +lambda +profile +vim9script +cmdline_hist +langmap -python +viminfo +cmdline_info +libcall +python3/dyn +virtualedit +comments +linebreak +quickfix +visual +conceal +lispindent +reltime +visualextra +cryptv +listcmds +rightleft +vreplace +cscope +localmap +ruby/dyn +wildignore +cursorbind +lua/dyn +scrollbind +wildmenu +cursorshape +menu +signs +windows +dialog_con +mksession +smartindent +writebackup +diff +modify_fname -sodium -X11 +digraphs +mouse -sound -xfontset -dnd -mouseshape +spell -xim -ebcdic +mouse_dec +startuptime -xpm +emacs_tags +mouse_gpm +statusline -xsmp +eval -mouse_jsbterm -sun_workshop -xterm_clipboard +ex_extra +mouse_netterm +syntax -xterm_save +extra_search +mouse_sgr +tag_binary -farsi -mouse_sysmouse -tag_old_static system vimrc file: "/etc/vimrc" user vimrc file: "$HOME/.vimrc" 2nd user vimrc file: "~/.vim/vimrc" user exrc file: "$HOME/.exrc" defaults file: "$VIMRUNTIME/defaults.vim" fall-back for $VIM: "/usr/share/vim" Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -flto=auto -D_REENTRANT -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 Linking: gcc -L. -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,-E -Wl,-rpath,/usr/lib/perl5/5.34/core_perl/CORE -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto -L/usr/local/lib -o vim -lm -ltinfo -lelf -lacl -lattr -lgpm -Wl,-E -Wl,-rpath,/usr/lib/perl5/5.34/core_perl/CORE -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fstack-protector-strong -L/usr/local/lib -L/usr/lib/perl5/5.34/core_perl/CORE -lperl -lpthread -ldl -lm -lcrypt -lutil -lc -L/usr/lib -ltclstub8.6 -ldl -lz -lpthread -lm ``` Using foot 1.11.0, with this command: ``` vim -u NONE +'let &t_TE = "\<Esc>[>4;0m"' +'let &t_TI = "\<Esc>[>4;2m"' +'nmap ; :echo ";"<CR>' +'nmap , :echo ","<CR>' ``` vim gets the mapping for both `;` and `,`. (athough it does not understand the `<CR>` somehow, not a big deal, I get the partial `:echo` command and that’s enough to test). **EDIT**: Tested with `vim -u DEFAULTS +'let &t_TE = "\<Esc>[>4;0m"' +'let &t_TI = "\<Esc>[>4;2m"' +'nmap ; :echo ";"<CR>' +'nmap , :echo ","<CR>'` as well, and both mappings work.

cljoly commented

Poster

For the corresponding GitHub issue: https://github.com/neovim/neovim/issues/18010 @gpanders

cljoly commented

Poster

However, it's not as simple as non-letter keys; e.g. - produces a CSI 27 escape.

I’ve tried to Google how to get the escape sequence produced as one enters the the keys in the terminal, without success. Would you have any pointers on how you do that please?

> However, it's not as simple as non-letter keys; e.g. `-` produces a `CSI 27` escape. I’ve tried to Google how to get the escape sequence produced as one enters the the keys in the terminal, without success. Would you have any pointers on how you do that please?

dnkl commented

Owner

In this case, you can do

echo -e '\e[>4;2m' && cat -

But note that you'll probably find the terminal unusable after this...

For "normal" escapes, where you don't have to enable a specific terminal mode, just pressing ctrl+v before your key combo is usually good enough (you'll have to press ctrl+v before each combo).

In this case, you can do ```sh echo -e '\e[>4;2m' && cat - ``` But note that you'll probably find the terminal unusable after this... For "normal" escapes, where you don't have to enable a specific terminal mode, just pressing <kbd>ctrl</kbd>+<kbd>v</kbd> before your key combo is usually good enough (you'll have to press <kbd>ctrl</kbd>+<kbd>v</kbd> before **each** combo).

dnkl commented

Owner

Another option is showkey -a

Another option is `showkey -a`

cljoly commented

Poster

Thanks!

cljoly commented

Poster

So in a newly opened terminal, I’m switching to the CSI > 4 ; 2 m mode and I enter , then ;

echo -e '\e[>4;2m' && cat -

,^[[27;2;59~

so ; is sent as CSI 27 ; 2 ; 59 ~, if I’m not mistaken.

When you press Shift + , this is no longer encoded as ASCII ; but instead CSI 27 ; 2 ; 44 ~ (where 2 is the modifier for Shift and 44 is the decimal representation of ,). Neovim is then not interpreting this sequence properly.

That seems inconsistent with our initial hypothesis

So in a newly opened terminal, I’m switching to the `CSI > 4 ; 2 m` mode and I enter `,` then `;` ``` echo -e '\e[>4;2m' && cat - ,^[[27;2;59~ ``` so `;` is sent as `CSI 27 ; 2 ; 59 ~`, if I’m not mistaken. > When you press Shift + , this is no longer encoded as ASCII ; but instead CSI 27 ; 2 ; 44 ~ (where 2 is the modifier for Shift and 44 is the decimal representation of ,). Neovim is then not interpreting this sequence properly. That seems inconsistent with our [initial hypothesis](https://codeberg.org/dnkl/foot/issues/1009#issuecomment-417523)

dnkl commented

Owner

Correct. 59 is the key code for ;. So it's basically saying, you pressed ;, and shift is the (only) active modifier.

If you do the same experiment in XTerm, you'll get a plain ; instead. But for e.g. shift+a you'll not get a plain A, but a CSI 27 escape.

Correct. 59 is the key code for `;`. So it's basically saying, you pressed `;`, and `shift` is the (only) active modifier. If you do the same experiment in XTerm, you'll get a plain `;` instead. But for e.g. <kbd>shift</kbd>+<kbd>a</kbd> you'll **not** get a plain `A`, but a `CSI 27` escape.

cljoly commented

Poster

If you do the same experiment in XTerm, you'll get a plain ; instead. But for e.g. shift+a you'll not get a plain A, but a CSI 27 escape.

That’s right.

> If you do the same experiment in XTerm, you'll get a plain `;` instead. But for e.g. <kbd>shift</kbd>+<kbd>a</kbd> you'll **not** get a plain `A`, but a `CSI 27` escape. That’s right.

gpanders commented

You said that to enter ; you press Shift + ,, right? But the escape sequence is reporting Shift + ;. I guess that might be the problem.

You said that to enter `;` you press `Shift + ,`, right? But the escape sequence is reporting `Shift + ;`. I guess that might be the problem.

dnkl commented

Owner

It's no different from XTerm's CSI sequences.

For example, shift+a results in CSI 27 ; 2 ; 65~, where 65 is the decimal representation of upper case A.

It's no different from XTerm's CSI sequences. For example, shift+a results in `CSI 27 ; 2 ; 65~`, where 65 is the decimal representation of upper case `A`.

dnkl commented

Owner

What I think XTerm is doing, is checking if the produced symbol (A) is the upper case version of the key's base symbol (a). And if it is, it emits a CSI sequence.

If it's not (for example, ; is not the upper case version of ,), then it emits a plain symbol instead of a CSI symbol.

I'll try to put together a POC for foot, to see if that improves our compatibility with XTerm.

What I think XTerm is doing, is checking if the produced symbol (`A`) is the upper case version of the key's base symbol (`a`). And if it is, it emits a CSI sequence. If it's not (for example, `;` is not the upper case version of `,`), then it emits a plain symbol instead of a CSI symbol. I'll try to put together a POC for foot, to see if that improves our compatibility with XTerm.

👍 1

cljoly commented

Poster

You said that to enter ; you press Shift + ,, right? But the escape sequence is reporting Shift + ;. I guess that might be the problem.

Yes, that’s right.

> You said that to enter `;` you press `Shift + ,`, right? But the escape sequence is reporting `Shift + ;`. I guess that might be the problem. Yes, that’s right.