Full-width and half-width characters: cleaning up mixed Japanese text

Open almost any Japanese document that several people have worked on and you will find the same word typed in two widths. ＰＣ in one paragraph, PC in the next. A price written as １２０円 in the heading and 120円 in the table. A katakana word that looks slightly squashed because it is half-width: ｻｰﾊﾞｰ instead of サーバー. Spaces that are too wide, spaces that are too narrow, and spaces where Japanese never wants one.

None of this stops a reader from understanding the text. All of it makes the text look unchecked, and some of it breaks things: searches that miss half the instances, sorting that puts the same word in two places, forms that reject a postcode typed in full-width digits. This guide explains what the two widths are, how to set sensible rules, and how to find and fix the mixture without creating new problems.

Where the two widths come from

The terms are 全角 (zenkaku, "full width") and 半角 (hankaku, "half width"). They describe how much horizontal room a character takes on a fixed-pitch screen: a full-width character fills the same square cell as a kanji, a half-width one fills half of it.

The split goes back to the Japanese Industrial Standards for character sets. JIS X 0201 is a single-byte set: Latin letters, digits and symbols much like ASCII (with the yen sign ¥ in the place where ASCII has the backslash), plus a basic set of katakana. JIS X 0208 is a two-byte set that holds the kanji, hiragana and katakana, and also its own copies of the Latin letters, digits and many symbols. On the screens of the time, a one-byte character was drawn half as wide as a two-byte one. So the same letter A existed twice, in two widths, and the katakana existed twice too.

Unicode kept both versions so that old text could be converted without loss. The full-width copies of the ASCII characters sit at U+FF01 to U+FF5E, from ！ to ～, and each one is exactly 0xFEE0 above its ordinary counterpart (Ａ is U+FF21, A is U+0041). Half-width katakana and a few half-width Japanese punctuation marks sit at U+FF61 to U+FF9F. The full-width space, called the ideographic space, is U+3000, in a different block from the rest. The Unicode chart for the Halfwidth and Fullwidth Forms block shows them all.

Why half-width katakana deserve special treatment

Half-width katakana are the one case where nearly every modern style guide agrees: do not use them in running text. They came from a small single-byte set with no room for voiced syllables as separate characters, so a voiced kana is two characters: ｶﾞ is ｶ followed by a separate voicing mark ﾞ, while ガ is one character. That breaks searches (a search for ガイド will not find ｶﾞｲﾄﾞ), throws off character counts and line breaking, and makes the word look cramped next to normal text. The set also lacks some characters entirely, such as ヰ, ヱ and the small ヮ. Older email standards for Japanese did not allow half-width katakana at all, which is part of why they gained a reputation for turning into garbled text.

You still meet them in receipts, bank statement exports, some database fields and text pasted from old systems. When they appear in a document meant for people to read, convert them to full-width. There is rarely a reason to keep them.

Setting house rules for everything else

Letters, digits and spaces are a matter of choice, and different publications choose differently. The important thing is to choose once and write it down. A common set of rules for web and business documents looks like this:

• Katakana: always full-width.
• Latin letters and digits: half-width. (Some publications use full-width for single digits or letters, especially in vertical text, where full-width characters stand upright in the column. If your text is set vertically, decide this separately.)
• Japanese punctuation 、。「」・: full-width, always.
• Brackets in Japanese sentences: full-width （）. Half-width () only inside English or code.
• Question and exclamation marks after Japanese text: full-width ？ and ！.
• Spaces between Japanese words: none.
• Spaces between Japanese and Latin text or digits: pick one, either none (Windows 10を使う) or a half-width space (Windows 10 を使う), and use it everywhere.
• The ideographic space: only where the layout needs it, if at all. In printed text and manuscripts, a paragraph traditionally begins with one full-width space as an indent. On the web this is usually done with styling instead.
• The wave dash for ranges: choose 〜 (U+301C) or ～ (U+FF5E) and stick to it. They look almost the same, and text converted from Shift_JIS through different systems can contain either one for the same mark.

If you keep a style sheet for a long document, put these rules at the top. The piece on keeping spellings consistent covers how to build one and what else belongs in it.

Finding the mixture

You cannot clean what you cannot see. In a proportional font, a full-width Ａ and a half-width A can look similar, and the two spaces differ by only a few pixels. Three ways to make the problem visible:

• Switch your editor to a fixed-pitch font for the check. Full-width characters then take exactly two columns and stand out.
• Turn on the display of invisible characters. Many text editors made in Japan can show the ideographic space as a visible mark, often a small square, and keep doing so while you work.
• Run the text through the checker on the front page, which reports half-width katakana and mixed full-width and half-width letters and digits line by line. The legend of issue types explains each warning.

For a targeted cleanup, search with regular expressions. Most editors that support them accept the \uXXXX form for a code point (some use \x{XXXX} instead). These patterns cover the usual cases:

[Ａ-Ｚａ-ｚ０-９]          full-width letters and digits
[\uFF01-\uFF5E]            any full-width ASCII copy, punctuation included
[\uFF61-\uFF9F]            half-width katakana and half-width Japanese punctuation
\u3000                     every ideographic space
[^\n]\u3000                ideographic spaces that are not at the start of a line
[0-9][０-９]|[０-９][0-9]  numbers typed partly in each width
[ァ-ヶー]-                 a hyphen after katakana, typed instead of ー

The last one catches a common slip: typing サ-バ- with the IME switched to direct input, so the long vowel mark becomes an ordinary hyphen. It is worth a search on any document with many loanwords.

One trap in the katakana ranges: the long vowel mark ー (U+30FC) is outside the range ァ to ヶ, so a pattern like [ァ-ヶ]+ stops at the first ー. Add ー to the class, as above, when you want whole katakana words.

To find stray ordinary spaces between Japanese characters, an editor or script that supports lookbehind and Unicode scripts can use:

(?<=[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}]) (?=[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}])

Review the hits before you delete anything. A space between two Japanese words is usually a slip, but in names (山田 太郎) or in lists it can be deliberate.

Converting without making things worse

Bulk conversion tools are where most cleanups go wrong, because they convert more than you asked for.

Spreadsheet functions

Excel has two functions for this: ASC converts full-width characters to half-width, and JIS converts half-width to full-width. ASC is the one people reach for to fix digits, and it also converts full-width katakana to half-width. Run it over a column of product names and every サーバー becomes ｻｰﾊﾞｰ. Use it only on columns that hold nothing but codes and numbers.

Word processors

Word processors with Japanese support usually have a change-width command for the selection. It converts everything selected, katakana included, depending on the option you pick. Select precisely, or use find and replace with one of the patterns above.

Unicode normalization

NFKC normalization is tempting because it does two right things at once: it turns full-width letters and digits into half-width, and it turns half-width katakana into full-width, joining ｶﾞ into ガ. It also does several things you may not want. It turns the ideographic space into an ordinary space, ～ into a half-width tilde, ① into 1, ㈱ into (株) and ㌔ into キロ. On a whole document that is a lot of silent change. Some of it may be welcome (machine-dependent characters such as ① and ㈱ are a separate issue the checker also flags), but it should be a decision, not a side effect.

If you write a small script, convert each class on its own. In JavaScript, for example:

// full-width letters and digits only, to half-width
text = text.replace(/[Ａ-Ｚａ-ｚ０-９]/g,
  c => String.fromCharCode(c.charCodeAt(0) - 0xFEE0));

// half-width katakana and punctuation only, to full-width
text = text.replace(/[\uFF61-\uFF9F]+/g, s => s.normalize('NFKC'));

The first line leaves full-width brackets, question marks and ～ alone, which is usually what Japanese prose wants. Widen the class to [\uFF01-\uFF5E] only if your house rules really want every symbol half-width. The second line applies normalization only to runs of half-width katakana, so nothing else in the text is touched.

A cleanup pass, in order

The order matters a little, because some steps create work for others.

1. Write down the house rules before touching the text.
2. Convert half-width katakana and half-width Japanese punctuation to full-width. This is almost always safe.
3. Convert full-width letters and digits to half-width, or the reverse, according to the rules. Check product names and proper nouns that may have an official form.
4. Fix hyphens typed in place of ー.
5. Go through every ideographic space. Keep paragraph indents if your layout uses them, remove the rest or replace them per the rules.
6. Look for ordinary spaces between Japanese characters and remove the accidental ones.
7. Make brackets, question marks, exclamation marks and the wave dash consistent.
8. Run the whole text through a checker once more, and read the headings and tables by eye: those are the places where widths most often slip back in.

None of these steps fixes a wrong word. A text can have perfect widths and still say 意外 where it means 以外, which is a different kind of error with its own routine, covered in the piece on IME conversion errors.