New Issue

Some Shift+key not detected in neovim (non-stable) with non-qwerty keyboard layout #1009

Closed

opened 2 months ago by cljoly · 20 comments

cljoly commented 2 months ago

foot version:

foot --version
foot version: 1.11.0 +pgo +ime +graphemes -assertions

./foot --version
foot version: 1.11.0-97-gf2870bf5 (Apr 05 2022, branch 'master') -pgo +ime +graphemes +assertions

I’m using a French dvorak-style keyboard layout (bepo) and I recently noticed that my neovim binding on ; (which I enter with Shift+,, , being on the physical key g on a qwerty keyboard) stopped working.

This behavior is only exhibited by neovim on some yet unreleased master commits. Compiling neovim and runnig it like so (i.e. without any custom configuration and defining the mappings on the fly):

./build/bin/nvim -u NONE +'nmap ; :echo ";"<CR>'  +'nmap , :echo ","<CR>'

allows to check this behavior on a locally built neovim from the git repository. On the latest neovim stable (v0.6.1), entering ; results in ; being displayed, but not on those latest master commits. Using git bisect, I’ve found this small commit to be the first with the ; mapping not working, while , works.

Meanwhile, the ; mapping still works with Alacritty, for all neovim commits I tested. That suggests that foot might be sending the wrong sequence here.

cljoly commented 2 months ago

Poster

Just tested with Kitty, couldn’t reproduce the bug either.

cljoly changed title from ~~Some Shift+key not detected in neovim (non-stable) with non-qwerty keyboard~~ to Some Shift+key not detected in neovim (non-stable) with non-qwerty keyboard layout 2 months ago

gpanders commented 2 months ago

Alacritty does not recognize the modifyOtherKeys sequence (the one Neovim is setting in the commit you linked to), and we explicitly do not send the sequence for kitty since it uses its own encoding. Do you have another terminal emulator you can try to reproduce on?

I am not familiar with foot's codebase, but I believe what's happening is:

Neovim sends CSI > 4 ; 2 m which enables foot's modifyOtherKeys handling
When you press Shift + , this is no longer encoded as ASCII ; but instead CSI 27 ; 2 ; 44 ~ (where 2 is the modifier for Shift and 44 is the decimal representation of ,). Neovim is then not interpreting this sequence properly.

Can you also try reproducing in regular Vim by adding the following to your vimrc?

let &t_TI = "\<Esc>[>4;2m"
let &t_TE = "\<Esc>[>4;0m"

dnkl commented 2 months ago

Owner

Could also be worth testing/comparing with xterm, which does support (obviously) modifyOtherKeys.

Also note that foot supports kitty's keyboard protocol as well, so if neovim works better with that, another option is to change neovim to use the kitty kbd protocol (instead of modifyOtherKeys) on foot as well.

I also agree with @gpanders analysis, though I haven't verified it.

gpanders commented 2 months ago

@cljoly I am not convinced that this is not a bug in Neovim, so could you also open an issue in our issue tracker?

cljoly commented 2 months ago

Poster

Thanks for your quick feedback on this!

Tomorrow, I will:

try reproducing in regular Vim
try reproducing on xterm
open an issue on neovim issue tracker

Do you have another terminal emulator you can try to reproduce on?

I should be able to test with any terminal emulator installable from Arch’s repositories.

dnkl commented 1 month ago

Owner

It appears XTerm is somewhat inconsistent here. For regular (a-z) keys, it produces a CSI 27 ; 2 ; N~ escape for shift combinations. But for some, it does not, but instead produces a plain (but shifted) symbol. , vs. ; is one such example. Most (all?) shift+number keys also behave like this (on my Swedish layout).

However, it's not as simple as letter keys vs. non-letter keys; e.g. - produces a CSI 27 escape.

This can be considered to be an issue with foot, in the sense that we're not producing output identical to XTerm.

But it could also be unintended behavior in XTerm.

cljoly commented 1 month ago

Poster

Just tried reproducing, XTerm does not have the bug.

cljoly commented 1 month ago

Poster

Trying to reproduce with vim:

~> vim --version --help
VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Mar 30 2022 23:54:07)
Included patches: 1-4651
Compiled by Arch Linux
Huge version without GUI.  Features included (+) or not (-):
+acl               +file_in_path      +mouse_urxvt       -tag_any_white
+arabic            +find_in_path      +mouse_xterm       +tcl/dyn
+autocmd           +float             +multi_byte        +termguicolors
+autochdir         +folding           +multi_lang        +terminal
-autoservername    -footer            -mzscheme          +terminfo
-balloon_eval      +fork()            +netbeans_intg     +termresponse
+balloon_eval_term +gettext           +num64             +textobjects
-browse            -hangul_input      +packages          +textprop
++builtin_terms    +iconv             +path_extra        +timers
+byte_offset       +insert_expand     +perl/dyn          +title
+channel           +ipv6              +persistent_undo   -toolbar
+cindent           +job               +popupwin          +user_commands
-clientserver      +jumplist          +postscript        +vartabs
-clipboard         +keymap            +printer           +vertsplit
+cmdline_compl     +lambda            +profile           +vim9script
+cmdline_hist      +langmap           -python            +viminfo
+cmdline_info      +libcall           +python3/dyn       +virtualedit
+comments          +linebreak         +quickfix          +visual
+conceal           +lispindent        +reltime           +visualextra
+cryptv            +listcmds          +rightleft         +vreplace
+cscope            +localmap          +ruby/dyn          +wildignore
+cursorbind        +lua/dyn           +scrollbind        +wildmenu
+cursorshape       +menu              +signs             +windows
+dialog_con        +mksession         +smartindent       +writebackup
+diff              +modify_fname      -sodium            -X11
+digraphs          +mouse             -sound             -xfontset
-dnd               -mouseshape        +spell             -xim
-ebcdic            +mouse_dec         +startuptime       -xpm
+emacs_tags        +mouse_gpm         +statusline        -xsmp
+eval              -mouse_jsbterm     -sun_workshop      -xterm_clipboard
+ex_extra          +mouse_netterm     +syntax            -xterm_save
+extra_search      +mouse_sgr         +tag_binary        
-farsi             -mouse_sysmouse    -tag_old_static    
   system vimrc file: "/etc/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -flto=auto -D_REENTRANT -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 
Linking: gcc -L. -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,-E -Wl,-rpath,/usr/lib/perl5/5.34/core_perl/CORE -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto -L/usr/local/lib -o vim -lm -ltinfo -lelf -lacl -lattr -lgpm -Wl,-E -Wl,-rpath,/usr/lib/perl5/5.34/core_perl/CORE -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fstack-protector-strong -L/usr/local/lib -L/usr/lib/perl5/5.34/core_perl/CORE -lperl -lpthread -ldl -lm -lcrypt -lutil -lc -L/usr/lib -ltclstub8.6 -ldl -lz -lpthread -lm

Using foot 1.11.0, with this command:

vim -u NONE +'let &t_TE = "\<Esc>[>4;0m"' +'let &t_TI = "\<Esc>[>4;2m"' +'nmap ; :echo ";"<CR>'  +'nmap , :echo ","<CR>'

vim gets the mapping for both ; and ,. (athough it does not understand the <CR> somehow, not a big deal, I get the partial :echo command and that’s enough to test).

EDIT: Tested with vim -u DEFAULTS +'let &t_TE = "\<Esc>[>4;0m"' +'let &t_TI = "\<Esc>[>4;2m"' +'nmap ; :echo ";"<CR>' +'nmap , :echo ","<CR>' as well, and both mappings work.

Trying to reproduce with vim: ``` ~> vim --version --help VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Mar 30 2022 23:54:07) Included patches: 1-4651 Compiled by Arch Linux Huge version without GUI. Features included (+) or not (-): +acl +file_in_path +mouse_urxvt -tag_any_white +arabic +find_in_path +mouse_xterm +tcl/dyn +autocmd +float +multi_byte +termguicolors +autochdir +folding +multi_lang +terminal -autoservername -footer -mzscheme +terminfo -balloon_eval +fork() +netbeans_intg +termresponse +balloon_eval_term +gettext +num64 +textobjects -browse -hangul_input +packages +textprop ++builtin_terms +iconv +path_extra +timers +byte_offset +insert_expand +perl/dyn +title +channel +ipv6 +persistent_undo -toolbar +cindent +job +popupwin +user_commands -clientserver +jumplist +postscript +vartabs -clipboard +keymap +printer +vertsplit +cmdline_compl +lambda +profile +vim9script +cmdline_hist +langmap -python +viminfo +cmdline_info +libcall +python3/dyn +virtualedit +comments +linebreak +quickfix +visual +conceal +lispindent +reltime +visualextra +cryptv +listcmds +rightleft +vreplace +cscope +localmap +ruby/dyn +wildignore +cursorbind +lua/dyn +scrollbind +wildmenu +cursorshape +menu +signs +windows +dialog_con +mksession +smartindent +writebackup +diff +modify_fname -sodium -X11 +digraphs +mouse -sound -xfontset -dnd -mouseshape +spell -xim -ebcdic +mouse_dec +startuptime -xpm +emacs_tags +mouse_gpm +statusline -xsmp +eval -mouse_jsbterm -sun_workshop -xterm_clipboard +ex_extra +mouse_netterm +syntax -xterm_save +extra_search +mouse_sgr +tag_binary -farsi -mouse_sysmouse -tag_old_static system vimrc file: "/etc/vimrc" user vimrc file: "$HOME/.vimrc" 2nd user vimrc file: "~/.vim/vimrc" user exrc file: "$HOME/.exrc" defaults file: "$VIMRUNTIME/defaults.vim" fall-back for $VIM: "/usr/share/vim" Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -flto=auto -D_REENTRANT -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 Linking: gcc -L. -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,-E -Wl,-rpath,/usr/lib/perl5/5.34/core_perl/CORE -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto -L/usr/local/lib -o vim -lm -ltinfo -lelf -lacl -lattr -lgpm -Wl,-E -Wl,-rpath,/usr/lib/perl5/5.34/core_perl/CORE -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fstack-protector-strong -L/usr/local/lib -L/usr/lib/perl5/5.34/core_perl/CORE -lperl -lpthread -ldl -lm -lcrypt -lutil -lc -L/usr/lib -ltclstub8.6 -ldl -lz -lpthread -lm ``` Using foot 1.11.0, with this command: ``` vim -u NONE +'let &t_TE = "\<Esc>[>4;0m"' +'let &t_TI = "\<Esc>[>4;2m"' +'nmap ; :echo ";"<CR>' +'nmap , :echo ","<CR>' ``` vim gets the mapping for both `;` and `,`. (athough it does not understand the `<CR>` somehow, not a big deal, I get the partial `:echo` command and that’s enough to test). **EDIT**: Tested with `vim -u DEFAULTS +'let &t_TE = "\<Esc>[>4;0m"' +'let &t_TI = "\<Esc>[>4;2m"' +'nmap ; :echo ";"<CR>' +'nmap , :echo ","<CR>'` as well, and both mappings work.

cljoly commented 1 month ago

Poster

For the corresponding GitHub issue: https://github.com/neovim/neovim/issues/18010 @gpanders

cljoly commented 1 month ago

Poster

However, it's not as simple as non-letter keys; e.g. - produces a CSI 27 escape.

I’ve tried to Google how to get the escape sequence produced as one enters the the keys in the terminal, without success. Would you have any pointers on how you do that please?

dnkl commented 1 month ago

Owner

In this case, you can do

echo -e '\e[>4;2m' && cat -

But note that you'll probably find the terminal unusable after this...

For "normal" escapes, where you don't have to enable a specific terminal mode, just pressing ctrl+v before your key combo is usually good enough (you'll have to press ctrl+v before each combo).

dnkl commented 1 month ago

Owner

Another option is showkey -a

cljoly commented 1 month ago

Poster

Thanks!

cljoly commented 1 month ago

Poster

So in a newly opened terminal, I’m switching to the CSI > 4 ; 2 m mode and I enter , then ;

echo -e '\e[>4;2m' && cat -

,^[[27;2;59~

so ; is sent as CSI 27 ; 2 ; 59 ~, if I’m not mistaken.

When you press Shift + , this is no longer encoded as ASCII ; but instead CSI 27 ; 2 ; 44 ~ (where 2 is the modifier for Shift and 44 is the decimal representation of ,). Neovim is then not interpreting this sequence properly.

That seems inconsistent with our initial hypothesis

dnkl commented 1 month ago

Owner

Correct. 59 is the key code for ;. So it's basically saying, you pressed ;, and shift is the (only) active modifier.

If you do the same experiment in XTerm, you'll get a plain ; instead. But for e.g. shift+a you'll not get a plain A, but a CSI 27 escape.

cljoly commented 1 month ago

Poster

If you do the same experiment in XTerm, you'll get a plain ; instead. But for e.g. shift+a you'll not get a plain A, but a CSI 27 escape.

That’s right.

gpanders commented 1 month ago

You said that to enter ; you press Shift + ,, right? But the escape sequence is reporting Shift + ;. I guess that might be the problem.

dnkl commented 1 month ago

Owner

It's no different from XTerm's CSI sequences.

For example, shift+a results in CSI 27 ; 2 ; 65~, where 65 is the decimal representation of upper case A.

dnkl commented 1 month ago

Owner

What I think XTerm is doing, is checking if the produced symbol (A) is the upper case version of the key's base symbol (a). And if it is, it emits a CSI sequence.

If it's not (for example, ; is not the upper case version of ,), then it emits a plain symbol instead of a CSI symbol.

I'll try to put together a POC for foot, to see if that improves our compatibility with XTerm.

👍 1

cljoly commented 1 month ago

Poster