How to use UTF-8 in resource properties with ResourceBundle
JavaGoogle App-EngineUtf 8InternationalizationResourcebundleJava Problem Overview
I need to use UTF-8 in my resource properties using Java's ResourceBundle. When I enter the text directly into the properties file, it displays as mojibake.
My app runs on Google App Engine.
Can anyone give me an example? I can't get this work.
Java Solutions
Solution 1 - Java
Java 9 and newer
From Java 9 onwards property files are encoded as UTF-8 by default, and using characters outside of ISO-8859-1 should work out of the box.
Java 8 and older
The ResourceBundle#getBundle() uses under the covers PropertyResourceBundle when a .properties file is specified. This in turn uses by default Properties#load(InputStream) to load those properties files. As per the javadoc, they are by default read as ISO-8859-1.
> public void load(InputStream inStream) throws IOException
> Reads a property list (key and element pairs) from the input byte stream. The input stream is in a simple line-oriented format as specified in load(Reader) and is assumed to use the ISO 8859-1 character encoding; that is each byte is one Latin1 character. Characters not in Latin1, and certain special characters, are represented in keys and elements using Unicode escapes as defined in section 3.3 of The Java™ Language Specification.
So, you'd need to save them as ISO-8859-1. If you have any characters beyond ISO-8859-1 range and you can't use \uXXXX off top of head and you're thus forced to save the file as UTF-8, then you'd need to use the native2ascii tool to convert an UTF-8 saved properties file to an ISO-8859-1 saved properties file wherein all uncovered characters are converted into \uXXXX format. The below example converts a UTF-8 encoded properties file text_utf8.properties to a valid ISO-8859-1 encoded properties file text.properties.
native2ascii -encoding UTF-8 text_utf8.properties text.properties
When using a sane IDE such as Eclipse, this is already automatically done when you create a .properties file in a Java based project and use Eclipse's own editor. Eclipse will transparently convert the characters beyond ISO-8859-1 range to \uXXXX format. See also below screenshots (note the "Properties" and "Source" tabs on bottom, click for large):
Alternatively, you could also create a custom ResourceBundle.Control implementation wherein you explicitly read the properties files as UTF-8 using InputStreamReader, so that you can just save them as UTF-8 without the need to hassle with native2ascii. Here's a kickoff example:
public class UTF8Control extends Control {
public ResourceBundle newBundle
(String baseName, Locale locale, String format, ClassLoader loader, boolean reload)
throws IllegalAccessException, InstantiationException, IOException
{
// The below is a copy of the default implementation.
String bundleName = toBundleName(baseName, locale);
String resourceName = toResourceName(bundleName, "properties");
ResourceBundle bundle = null;
InputStream stream = null;
if (reload) {
URL url = loader.getResource(resourceName);
if (url != null) {
URLConnection connection = url.openConnection();
if (connection != null) {
connection.setUseCaches(false);
stream = connection.getInputStream();
}
}
} else {
stream = loader.getResourceAsStream(resourceName);
}
if (stream != null) {
try {
// Only this line is changed to make it to read properties files as UTF-8.
bundle = new PropertyResourceBundle(new InputStreamReader(stream, "UTF-8"));
} finally {
stream.close();
}
}
return bundle;
}
}
This can be used as follows:
ResourceBundle bundle = ResourceBundle.getBundle("com.example.i18n.text", new UTF8Control());
See also:
Solution 2 - Java
Given that you have an instance of ResourceBundle and you can get String by:
String val = bundle.getString(key);
I solved my Japanese display problem by:
return new String(val.getBytes("ISO-8859-1"), "UTF-8");
Solution 3 - Java
look at this : http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(java.io.Reader)
the properties accept an Reader object as arguments, which you can create from an InputStream.
at the create time, you can specify the encoding of the Reader:
InputStreamReader isr = new InputStreamReader(stream, "UTF-8");
then apply this Reader to the load method :
prop.load(isr);
BTW: get the stream from .properties file :
InputStream stream = this.class.getClassLoader().getResourceAsStream("a.properties");
BTW: get resource bundle from InputStreamReader:
ResourceBundle rb = new PropertyResourceBundle(isr);
hope this can help you !
Solution 4 - Java
This problem has finally been fixed in Java 9: https://docs.oracle.com/javase/9/intl/internationalization-enhancements-jdk-9
Default encoding for properties files is now UTF-8.
> Most existing properties files should not be affected: UTF-8 and > ISO-8859-1 have the same encoding for ASCII characters, and > human-readable non-ASCII ISO-8859-1 encoding is not valid UTF-8. If an > invalid UTF-8 byte sequence is detected, the Java runtime > automatically rereads the file in ISO-8859-1.
Solution 5 - Java
ResourceBundle.Control with UTF-8 and new String methods don't work, if the properties file uses cp1251 charset, for example.
So I recomended using a common method: write in unicode symbols. For this:
IDEA -- has a special "Transparent native-to-ASCII conversion" option (Settings > File Encoding).
Eclipse -- has a plugin "Properties Editor". It can work as separate application.
Solution 6 - Java
package com.varaneckas.utils;
import java.io.UnsupportedEncodingException;
import java.util.Enumeration;
import java.util.PropertyResourceBundle;
import java.util.ResourceBundle;
/**
* UTF-8 friendly ResourceBundle support
*
* Utility that allows having multi-byte characters inside java .property files.
* It removes the need for Sun's native2ascii application, you can simply have
* UTF-8 encoded editable .property files.
*
* Use:
* ResourceBundle bundle = Utf8ResourceBundle.getBundle("bundle_name");
*
* @author Tomas Varaneckas <[email protected]>
*/
public abstract class Utf8ResourceBundle {
/**
* Gets the unicode friendly resource bundle
*
* @param baseName
* @see ResourceBundle#getBundle(String)
* @return Unicode friendly resource bundle
*/
public static final ResourceBundle getBundle(final String baseName) {
return createUtf8PropertyResourceBundle(
ResourceBundle.getBundle(baseName));
}
/**
* Creates unicode friendly {@link PropertyResourceBundle} if possible.
*
* @param bundle
* @return Unicode friendly property resource bundle
*/
private static ResourceBundle createUtf8PropertyResourceBundle(
final ResourceBundle bundle) {
if (!(bundle instanceof PropertyResourceBundle)) {
return bundle;
}
return new Utf8PropertyResourceBundle((PropertyResourceBundle) bundle);
}
/**
* Resource Bundle that does the hard work
*/
private static class Utf8PropertyResourceBundle extends ResourceBundle {
/**
* Bundle with unicode data
*/
private final PropertyResourceBundle bundle;
/**
* Initializing constructor
*
* @param bundle
*/
private Utf8PropertyResourceBundle(final PropertyResourceBundle bundle) {
this.bundle = bundle;
}
@Override
@SuppressWarnings("unchecked")
public Enumeration getKeys() {
return bundle.getKeys();
}
@Override
protected Object handleGetObject(final String key) {
final String value = bundle.getString(key);
if (value == null)
return null;
try {
return new String(value.getBytes("ISO-8859-1"), "UTF-8");
} catch (final UnsupportedEncodingException e) {
throw new RuntimeException("Encoding not supported", e);
}
}
}
}
Solution 7 - Java
We create a resources.utf8 file that contains the resources in UTF-8 and have a rule to run the following:
native2ascii -encoding utf8 resources.utf8 resources.properties
Solution 8 - Java
Attention: In Java <= 8 java property files should be encoded in ISO 8859-1!
> ISO 8859-1 character encoding. > Characters that cannot be directly > represented in this encoding can be > written using Unicode escapes ; only > a single 'u' character is allowed in > an escape sequence.
@see Properties Java Doc
If you still really want to do this: have a look at: https://stackoverflow.com/questions/863838/problem-with-java-properties-utf8-encoding-in-eclipse -- there are some code samples
Since Java 9: property files are encoded in UTF-8, so there should be no problem/doubt > In Java SE 9, properties files are loaded in UTF-8 encoding. In previous releases, ISO-8859-1 encoding was used for loading property resource bundles.
Solution 9 - Java
http://sourceforge.net/projects/eclipse-rbe/
as already stated property files should be encoded in ISO 8859-1
You can use the above plugin for eclipse IDE to make the Unicode conversion for you.
Solution 10 - Java
Here's a Java 7 solution that uses Guava's excellent support library and the try-with-resources construct. It reads and writes properties files using UTF-8 for the simplest overall experience.
To read a properties file as UTF-8:
File file = new File("/path/to/example.properties");
// Create an empty set of properties
Properties properties = new Properties();
if (file.exists()) {
// Use a UTF-8 reader from Guava
try (Reader reader = Files.newReader(file, Charsets.UTF_8)) {
properties.load(reader);
} catch (IOException e) {
// Do something
}
}
To write a properties file as UTF-8:
File file = new File("/path/to/example.properties");
// Use a UTF-8 writer from Guava
try (Writer writer = Files.newWriter(file, Charsets.UTF_8)) {
properties.store(writer, "Your title here");
writer.flush();
} catch (IOException e) {
// Do something
}
Solution 11 - Java
As one suggested, i went through implementation of resource bundle.. but that did not help.. as the bundle was always called under en_US locale... i tried to set my default locale to a different language and still my implementation of resource bundle control was being called with en_US... i tried to put log messages and do a step through debug and see if a different local call was being made after i change locale at run time through xhtml and JSF calls... that did not happend... then i tried to do a system set default to a utf8 for reading files by my server (tomcat server).. but that caused pronlem as all my class libraries were not compiled under utf8 and tomcat started to read then in utf8 format and server was not running properly... then i ended up with implementing a method in my java controller to be called from xhtml files.. in that method i did the following:
public String message(String key, boolean toUTF8) throws Throwable{
String result = "";
try{
FacesContext context = FacesContext.getCurrentInstance();
String message = context.getApplication().getResourceBundle(context, "messages").getString(key);
result = message==null ? "" : toUTF8 ? new String(message.getBytes("iso8859-1"), "utf-8") : message;
}catch(Throwable t){}
return result;
}
I was particularly nervous as this could slow down performance of my application... however, after implementing this, it looks like as if my application is faster now.. i think it is because, i am now directly accessing the properties instead of letting JSF parse its way into accessing properties... i specifically pass Boolean argument in this call because i know some of the properties would not be translated and do not need to be in utf8 format...
Now I have saved my properties file in UTF8 format and it is working fine as each user in my application has a referent locale preference.
Solution 12 - Java
For what it's worth my issue was that the files themselves were in the wrong encoding. Using iconv worked for me
iconv -f ISO-8859-15 -t UTF-8 messages_nl.properties > messages_nl.properties.new
Solution 13 - Java
I tried to use the approach provided by Rod, but taking into consideration BalusC concern about not repeating the same work-around in all the application and came with this class:
import java.io.UnsupportedEncodingException;
import java.util.Locale;
import java.util.ResourceBundle;
public class MyResourceBundle {
// feature variables
private ResourceBundle bundle;
private String fileEncoding;
public MyResourceBundle(Locale locale, String fileEncoding){
this.bundle = ResourceBundle.getBundle("com.app.Bundle", locale);
this.fileEncoding = fileEncoding;
}
public MyResourceBundle(Locale locale){
this(locale, "UTF-8");
}
public String getString(String key){
String value = bundle.getString(key);
try {
return new String(value.getBytes("ISO-8859-1"), fileEncoding);
} catch (UnsupportedEncodingException e) {
return value;
}
}
}
The way to use this would be very similar than the regular ResourceBundle usage:
private MyResourceBundle labels = new MyResourceBundle("es", "UTF-8");
String label = labels.getString(key)
Or you can use the alternate constructor which uses UTF-8 by default:
private MyResourceBundle labels = new MyResourceBundle("es");
Solution 14 - Java
Properties prop = new Properties();
String fileName = "./src/test/resources/predefined.properties";
FileInputStream inputStream = new FileInputStream(fileName);
InputStreamReader reader = new InputStreamReader(inputStream,"UTF-8");
Solution 15 - Java
Open the Settings / Preferences dialog (Ctrl + Alt + S), then click Editor and File Encodings.
Then, on the bottom, you will fing default encodings for properties files. Choose your encoding type.
Alternatively you can use unicode symbols instead of text in your resource bundle (for example "ів" equals \u0456\u0432)
Solution 16 - Java
From Java 9, the default to load properties file has been changed to UTF-8. https://docs.oracle.com/javase/9/intl/internationalization-enhancements-jdk-9.htm
Solution 17 - Java
Speaking for current (2021-2) Java versions there is still the old ISO-8859-1 function utils.Properties#load.
- If you use Properties.load you must use ISO-8859-1.
- If you use ResourceBundle than UTF-8 should be fine.
Allow me to quote from the official doc.
PropertyResourceBundle
>PropertyResourceBundle can be constructed either from an InputStream or a Reader, which represents a property file. Constructing a PropertyResourceBundle instance from an InputStream requires that the input stream be encoded in UTF-8. By default, if a MalformedInputException or an UnmappableCharacterException occurs on reading the input stream, then the PropertyResourceBundle instance resets to the state before the exception, re-reads the input stream in ISO-8859-1, and continues reading. If the system property java.util.PropertyResourceBundle.encoding is set to either "ISO-8859-1" or "UTF-8", the input stream is solely read in that encoding, and throws the exception if it encounters an invalid sequence. If "ISO-8859-1" is specified, characters that cannot be represented in ISO-8859-1 encoding must be represented by Unicode Escapes as defined in section 3.3 of The Java™ Language Specification whereas the other constructor which takes a Reader does not have that limitation. Other encoding values are ignored for this system property. The system property is read and evaluated when initializing this class. Changing or removing the property has no effect after the initialization.
https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/util/PropertyResourceBundle.html
Properties#load > Reads a property list (key and element pairs) from the input byte stream. The input stream is in a simple line-oriented format as specified in load(Reader) and is assumed to use the ISO 8859-1 character encoding; that is each byte is one Latin1 character. Characters not in Latin1, and certain special characters, are represented in keys and elements using Unicode escapes as defined in section 3.3 of The Java™ Language Specification.