Go's UTF-8 Support: An Interesting Limitation

I've been exploring Go's UTF-8 support lately, and was curious about how well it handles non-Latin scripts in code.

Go and UTF-8

Go source files are UTF-8 encoded by default. This means you can, in theory, use Unicode characters in your variable names, function names and more.

For example, in the official Go playground boilerplate code, you might come across code like this:

package main

import "fmt"

func main() {
    消息 := "Hello, World!"
    fmt.Println(消息)
}

Here, 消息 is Chinese for "message." Go handles this without any issues, thanks to its Unicode support. This capability is one reason why Go has gained popularity in countries like China and Japan—developers can write code using identifiers meaningful in their own languages. You won’t believe it, but there’s a huge popularity in China, for writing code in their native language and I love it.

---

Attempting to Use Tamil Identifiers

Naturally, I wanted to try this out with Tamil, my mother tongue.

Here's a simple example I wrote:

package main

import "fmt"

func main() {
    எண்ணிக்கை := 42 // "எண்ணிக்கை" means "number"
    fmt.Println("Value:", எண்ணிக்கை)
}

At first glance, this seems straightforward that can run without any errors.

But, when I tried to compile the code, I ran into errors

./prog.go:6:11: invalid character U+0BCD '்' in identifier
./prog.go:6:17: invalid character U+0BBF 'ி' in identifier
./prog.go:6:23: invalid character U+0BCD '்' in identifier
./prog.go:6:29: invalid character U+0BC8 'ை' in identifier
./prog.go:7:33: invalid character U+0BCD '்' in identifier
./prog.go:7:39: invalid character U+0BBF 'ி' in identifier
./prog.go:7:45: invalid character U+0BCD '்' in identifier
./prog.go:7:51: invalid character U+0BC8 'ை' in identifier

Understanding the Issue with Tamil Combining Marks

To understand what's going on, it's essential to know a bit about how Tamil script works.

Tamil is an abugida—a writing system where each consonant-vowel sequence is written as a unit. In Unicode, this often involves combining a base consonant character with one or more combining marks that represent vowels or other modifiers.

For example:

• The Tamil letter க (U+0B95) represents the consonant sound "ka"

• To represent "ki" you'd combine க with the vowel sign ி (U+0BBF), resulting in கி.

• The vowel sign ி is a combining mark, specifically classified as a "Non-Spacing Mark" in Unicode.

Here's where the problem arises.

Go's language specification allows Unicode letters in identifiers but excludes combining marks. Specifically, identifiers can include characters that are classified as "Letter" (categories Lu, Ll, Lt, Lm, Lo, or Nl) and digits, but not combining marks (categories Mn, Mc, Me).

---

Examples of Combining Marks in Tamil

Let's look at how Tamil characters are formed:

• Standalone Consonant: க (U+0B95) - Allowed in Go identifiers.

• Consonant + Vowel Sign: கா (U+0B95 U+0BBE) - Not allowed because ா (U+0BBE) is a combining mark (Mc).

• Consonant + Vowel Sign: கி (U+0B95 U+0BBF) - Not allowed because ி (U+0BBF) is a combining mark (Mn).

• Consonant + Vowel Sign: கூ (U+0B95 U+0BC2) - Not allowed because ூ (U+0BC2) is a combining mark (Mc).

In the identifier எண்ணிக்கை ("number"), the characters include combining marks:

• எ (U+0B8E) - Letter, allowed.

• ண் (U+0BA3 U+0BCD) - Formed by ண (U+0BA3) and the virama ் (U+0BCD), a combining mark (Mn).

• ண (U+0BA3) - Letter, allowed.

• ிக்கை - Contains combining marks like ி (U+0BBF) and ை (U+0BC8).

Because these combining marks are not allowed in Go identifiers, the compiler throws errors when it encounters them.

---

Why Chinese Characters Work but Tamil Doesn't

Chinese characters are generally classified under the "Letter, Other" (Lo) category in Unicode. They are standalone symbols that don't require combining marks to form complete characters. This is why identifiers like 消息 work perfectly in Go.

Practical Implications

The inability to use combining marks in identifiers has significant implications for scripts like Tamil:

• Limited Expressiveness: Without combining marks, it's nearly impossible to write meaningful identifiers in Tamil.

• Educational Barriers: Using native scripts can make learning to code more accessible, but these limitations hinder that possibility, particular for languages that follow abugida-based writing system.

• Inclusivity Challenges: While Go aims for inclusivity with its UTF-8 support, the restrictions on combining marks exclude many languages that rely on them.

Wrapping Up

Go's UTF-8 support is a great step toward making programming more inclusive. However, the exclusion of combining marks in identifiers creates barriers for languages like Tamil, Hindi and Arabic, where combining marks are integral to the script.

As a developer from Tamilnadu, working primarily in Go, this discovery was both exciting and a bit disappointing. It highlights the complexities of true internationalization in programming languages.

Who codes in native languages for building Software Products !?!!!!

Absolutely! Not so much apart from the East-asian region, where ‘abugida’ based writing system isn’t followed.

And, obviously, creators of Go would not have intended the UTF-8 compliance for ‘native-language coding’ at first place. The reason was more towards providing better ASCII processing, alignment with modern web standards, consistent string handling and a step towards interoperability.

This attempt was just my curiosity in understanding how far we can take the UTF-8 Compliance in Go. As someone who works on building scalable, distributed fintech systems in Go, I found it essential to be aware of these nuances.

---

Thats about it. Thanks for reading along.

Happy coding :) May the code be with you.

Comments

[Ashwin Gopalsamy]

I started posting my writings in Reddit and in my second post, I received a recommendation (more of a suggestion) by Reddit's r/golang moderator jerf to write somewhere else and share the link on Reddit, as Reddit isn't that suitable for blogs.

So, I've decided to write here on dev.to/ashwingopalsamy & share them elsewhere.