org.gbif.utils.file

Class CharsetDetection

java.lang.Object

org.gbif.utils.file.CharsetDetection

public class CharsetDetection
extends Object

Utility class to guess the encoding of a given file or byte array. The guess is unfortunately not 100% sure. Especially for 8-bit charsets. It's not possible to know which 8-bit charset is used. Except through statistical analysis.

On the other hand, unicode files encoded in UTF-16 (low or big endian) or UTF-8 files with a Byte Order Marker are easy to find. For UTF-8 files with no BOM, if the buffer is wide enough, it's easy to guess.

To determine whether mostly-English text is UTF-8 or ISO-8859-1, a fairly large buffer may be necessary to find an instance of é, ° etc.

This class is a heavily modified version of the original written by Guillaume LAFORGE: com.glaforge.i18n.io.CharsetToolkit taken from http://glaforge.free.fr/wiki/index.php?wiki=GuessEncoding

Author:

Guillaume LAFORGE, Markus Döring

Method Summary

Modifier and Type	Method and Description
Charset	detectEncoding() Guess the encoding of the provided buffer.
static Charset	detectEncoding(File file)
static Charset	detectEncoding(File file, int bufferLength)
static Charset	getDefaultSystemCharset() Retrieve the default charset of the system.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

detectEncoding

public static Charset detectEncoding(File file)
                              throws IOException

Throws:

IOException

detectEncoding

public static Charset detectEncoding(File file,
                                     int bufferLength)
                              throws IOException

Parameters:

bufferLength - number of bytes to read in for the detection. Needs to be long enough to encounter non-ASCII characters, which could be unusual in English text.

Throws:

IOException

getDefaultSystemCharset

public static Charset getDefaultSystemCharset()

Retrieve the default charset of the system.

Returns:

the default Charset.

detectEncoding

public Charset detectEncoding()

Guess the encoding of the provided buffer.

If Byte Order Markers are encountered at the beginning of the buffer, we immediately return the charset implied by this BOM. Otherwise, the file would not be a human readable text file.

If there is no BOM, this method tries to discern whether the file is UTF-8 or not. If it is not UTF-8, we assume the encoding is the default system encoding (of course, it might be any 8-bit charset, but usually, an 8-bit charset is the default one).

It is possible to discern UTF-8 thanks to the pattern of characters with a multi-byte sequence.

 UCS-4 range (hex.)        UTF-8 octet sequence (binary)
 0000 0000-0000 007F       0xxxxxxx
 0000 0080-0000 07FF       110xxxxx 10xxxxxx
 0000 0800-0000 FFFF       1110xxxx 10xxxxxx 10xxxxxx
 0001 0000-001F FFFF       11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
 0020 0000-03FF FFFF       111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
 0400 0000-7FFF FFFF       1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
 

With UTF-8, 0xFE and 0xFF never appear.

Returns:

the Charset recognized or the system default.