ssh and 'warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory'

This post is about an issue that upon googling today seems to be quite widespread, and there's a lot of confusion about it. I think I have built a fairly good understanding about it, so to hopefully take away some of the confusion, I have created this blogpost.

This post is about when you encounter the following dreaded line after applying an update on Mac OSX when logging into a linux system with ssh:

warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory

This is just a message, and it allows you to carry on. However, much that uses "locales" (language specifics) can stop functioning, or generate errors. This is the confusing bit: because it doesn't immediately stop you from your work, any actual crashes (such as perl crashing) might seem random.

I did encounter did for the first time when I was doing patching at a client. Patching meant things go down, and therefore are time-sensitive. So that was really frustrating. I did find the root cause, and provided this with the vendor which software I was patching, who then put it in its non-public issue database, and the support engineer who didn't do anything took credits for it.

So, what is happening here? This is happening because of ssh (secure shell) mechanisms allowing lots of flexibility about how a session is created on a remote server, and some settings which gets reverted on patching OSX. Let's look at the message again:

warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory

LC_CTYPE makes linux try to set something. Let's query the environment for LC_CTYPE:

$ env | grep LC_CTYPE
LC_CTYPE=UTF-8

So that is the reason. But how come this seems to be set all of a sudden?

The reason is the ssh client, and a ssh setting called 'SendEnv', which allows you to send environment settings along with logging on to a remote ssh server.

So why does this pop up all of a sudden? The reason is there is a global ssh client settings file on mac at /etc/ssh/ssh_config. Every time OSX is updated, this file gets rewritten. Well, to be very precise: I do not make changes to this file, at least the global host declaration (Host *) is changed to send the environment variables LANG and all LC ones: SendEnv LANG LC_* upon OSX update. Every single update.

How to solve it? The only viable solution I have found so far is to simply remove LC_*, so the line in /etc/ssh/ssh_config becomes: SendEnv LANG.

There is a user-specific ssh_config file to in ~/.ssh/config, but using that file it only seems to be possible to add environment variables to SendEnv, not remove them.

Please mind there also is a file called /etc/ssh/sshd_config, that is the sshd (daemon) configuration, so the configuration of a local OSX ssh server, and thus NOT the settings for the client.