An interesting quirk in ksh, the default shell for OpenBSD

22 March 2023

Note: I am new to OpenBSD, so any thoughts or assumptions come from a couple of decades of experience with Linux and macOS. I did spend some time with FreeBSD back in 2008 or so.

Today I discovered a quirk in ksh, the default shell for OpenBSD. Setting EDITOR=vim in my .profile "broke" the way I navigate history in my shell. Up and down arrows no longer worked, nor did ctrl-r—it would simply echo the terminal bell (ding!).

The change in behavior was entirely unexpected. Neither of the two shells I am used to–bash and zsh–exhibit this behavior. I didn't notice it at first either. Fortunately I had two servers to compare, one that had the change and one that didn't. When I finally determined that my EDITOR change was the culprit, I did some manpage diving.

Explanation

When you set EDITOR or VISUAL, this helps ksh determine which command-line editing mode it should use. It uses emacs mode by default (analogous to set -o emacs).

Setting EDITOR=vim causes vi-like command-line editing to be enabled for the shell (analogous to set -o vi).

Relevant man page excerpts

From ksh(1):

EDITOR     If the VISUAL parameter is not set, this parameter controls
           the command-line editing mode for interactive shells.  See the
           VISUAL parameter below for how this works.

           Note: traditionally, EDITOR was used to specify the name of an
           (old-style) line editor, such as ed(1), and VISUAL was used to
           specify a (new-style) screen editor, such as vi(1).  Hence if
           VISUAL is set, it overrides EDITOR.

VISUAL     If set, this parameter controls the command-line editing mode
           for interactive shells.  If the last component of the path
           specified in this parameter contains the string "vi", "emacs",
           or "gmacs", the vi(1), emacs, or gmacs (Gosling emacs) editing
           mode is enabled, respectively.  See also the EDITOR parameter,
           above.

Interactive input line editing
  The shell supports three modes of reading command lines from a tty(4) in
  an interactive session, controlled by the emacs, gmacs, and vi options
  (at most one of these can be set at once).  The default is emacs.
  Editing modes can be set explicitly using the set built-in, or implicitly
  via the EDITOR and VISUAL environment variables.  If none of these
  options are enabled, the shell simply reads lines using the normal tty(4)
  driver.  If the emacs or gmacs option is set, the shell allows emacs-like
  editing of the command; similarly, if the vi option is set, the shell
  allows vi-like editing of the command.  These modes are described in
  detail in the following sections.

Conclusion

The behavior I want is my original expectation: having vim as my default editor without affecting the command-line editing mode (I did play a bit with the vi-mode, but I don't want to relearn decades of muscle memory).

Fortunately, this is easy enough by explicitly setting the desired mode in .profile:

export EDITOR=vim
set -o emacs

Why not just use a shell you're familiar with?

I am exploring OpenBSD as a stranger in a strange land. When in Rome...