Find Non-ASCII Character in File

15th September 2016 at 9:33pm

Keywords: Linux

Abstract

You process a file with fixed length character fields and the file gets rejected because a column is too long. When you examine (i.e. eye-ball) the file, you do not see any misalignment that would indicate too many characters. So you suspect a non-Ascii character that looks normal in your editor but really uses more space than is allowed.

The following are two possible approaches to finding the characters.

Find non-Ascii character in EMACS buffer

Use the occur command with the argument [^[:ascii:]]

(occur "[^[:ascii:]]"))

Find non-Ascii characters using AWK

awk "/[^\\x00-\\x7F]/{print NR, \$0;}"  filename