'Malformed UTF-8 characters, possibly incorrectly encoded' in Laravel
PhpJsonLaravel 4Utf 8Php Problem Overview
I'm using Laravel (a PHP framework) to write a service for mobile and have the data returned in JSON format. In the data result there are some fields encoded in UTF-8.
The following statement
return JsonResponse::create($data);
returns the error below
InvalidArgumentException
HELP
Malformed UTF-8 characters, possibly incorrectly encoded
Open: /var/www/html/vendor/symfony/http-foundation/Symfony/Component/HttpFoundation/JsonResponse.php
} catch (\Exception $exception) {
restore_error_handler();
throw $exception;
}
if (JSON_ERROR_NONE !== json_last_error()) {
throw new \InvalidArgumentException($this->transformJsonError());
}
I've changed:
return JsonResponse::create($data);
to
return JsonResponse::create($data, 200, array('Content-Type'=>'application/json; charset=utf-8' ));
but it still isn't working.
How can I fix it?
Php Solutions
Solution 1 - Php
I wrote this method to handle UTF8 arrays and JSON problems. It works fine with array (simple and multidimensional).
/**
* Encode array from latin1 to utf8 recursively
* @param $dat
* @return array|string
*/
public static function convert_from_latin1_to_utf8_recursively($dat)
{
if (is_string($dat)) {
return utf8_encode($dat);
} elseif (is_array($dat)) {
$ret = [];
foreach ($dat as $i => $d) $ret[ $i ] = self::convert_from_latin1_to_utf8_recursively($d);
return $ret;
} elseif (is_object($dat)) {
foreach ($dat as $i => $d) $dat->$i = self::convert_from_latin1_to_utf8_recursively($d);
return $dat;
} else {
return $dat;
}
}
// Sample use
// Just pass your array or string and the UTF8 encode will be fixed
$data = convert_from_latin1_to_utf8_recursively($data);
Solution 2 - Php
I found the answer to this problem here
Just do
mb_convert_encoding($data['name'], 'UTF-8', 'UTF-8');
Solution 3 - Php
In my case I had a ucfirst on the asian letters string. This was not possible and produced a non utf8 string.
Solution 4 - Php
In Laravel 7.x, this helped me to get rid of this error.
$newString = mb_convert_encoding($arr, "UTF-8", "auto");
return response()->json($newString);
Solution 5 - Php
In my case, this causes error:
return response->json(["message" => "Model status successfully updated!", "data" => $model], 200);
but this not:
return response->json(["message" => "Model status successfully updated!", "data" => $model->toJson()], 200);
Solution 6 - Php
I've experienced the same problem. The thing is that I forgot to start the apache and mysql in xampp... :S
Solution 7 - Php
I know it's already an old question, but i had the same error today. For me setting the connection variable on model did the work.
/**
* Table properties
*/
protected $connection = 'mysql-utf8';
protected $table = 'notification';
protected $primaryKey = 'id';
I don't know if the issue was with the database (probably), but the texts fields with special chars (like ~, ´ e etc) were all messed up.
---- Editing
That $connection var is used to select wich db connection your model will use. Sometimes it happens that in database.php (under /config folder) you have multiples connections and the default one is not using UTF-8 charset.
In any case, be sure to properly use charset and collation into your connection.
'connections' => [
'mysql' => [
'driver' => 'mysql',
'host' => env('DB_HOST', '127.0.0.1'),
'port' => env('DB_PORT', '3306'),
'database' => env('DB_DATABASE', 'your_database'),
'username' => env('DB_USERNAME', 'root'),
'password' => env('DB_PASSWORD', 'database_password'),
'unix_socket' => env('DB_SOCKET', ''),
'prefix' => '',
'strict' => false,
'engine' => null
],
'mysql-utf8' => [
'driver' => 'mysql',
'host' => env('DB_HOST', '127.0.0.1'),
'port' => env('DB_PORT', '3306'),
'database' => env('DB_DATABASE', 'your_database'),
'username' => env('DB_USERNAME', 'root'),
'password' => env('DB_PASSWORD', 'database_password'),
'unix_socket' => env('DB_SOCKET', ''),
'charset' => 'utf8',
'collation' => 'utf8_unicode_ci',
'prefix' => '',
'strict' => false,
'engine' => null
],
Solution 8 - Php
For more solution i have completed the solution of (the great) Tiago Gouvêa exposed before only for strings and arrays
i used md_convert_encoding() function instead of utf8_encode(), it works for me : (12 hours loosed ...)
// this object return me a big array who have multiple arrays imbricated
$get_days = program::get_days($ARR, $client);
// and i use this function for well parsing what the server return *
function convert_to_utf8_recursively($dat){
if( is_string($dat) ){
return mb_convert_encoding($dat, 'UTF-8', 'UTF-8');
}
elseif( is_array($dat) ){
$ret = [];
foreach($dat as $i => $d){
$ret[$i] = convert_to_utf8_recursively($d);
}
return $ret;
}
else{
return $dat;
}
}
// use
$data = convert_to_utf8_recursively($get_days);
- what the server return * i talk to that because i test the same code in two different servers the first return me a well formatted json , without any function, as we usually do BUT the second server does not send me back anything if I do not apply this function ...
Solution 9 - Php
In my case, it happened twice:
- I forgot to migrate again. I roll back (
php artisan rollback) the latest migration operation because I want to change something on my database schema but I forgot to execute again the migrate command:php artisan migrate.
More info on migrations: https://laravel.com/docs/9.x/migrations
- It also happened to me when the API is returning JSON responses with attributes of type
blob(column on the database that is created with thebinary()method). For my use case, I hide that attribute by adding the attribute's name to the model's array property$hidden. It works fine after that.
More info on hiding attributes on JSON: https://laravel.com/docs/9.x/eloquent-serialization#hiding-attributes-from-json
Solution 10 - Php
I got this error and i fixed the issue with iconv function like following:
iconv('latin5', 'utf-8', $data['index']);
Solution 11 - Php
Set the charset at after you made the connection to db like
// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
if (!$conn->set_charset("utf8")) {
printf("Error loading character set utf8: %s\n", $conn->error);
exit();
} else {
printf("Current character set: %s\n", $conn->character_set_name());
}
Solution 12 - Php
First thing Correctly specify all settings in the .ENV file for Mail.
I LOST 6 HOURS FOR THIS
Example
MAIL_DRIVER=smtp
MAIL_MAILER=smtp
MAIL_HOST=smtp.yandex.ru
MAIL_PORT=465
MAIL_USERNAME[email protected]
MAIL_PASSWORD=password //Create password for Apps in settings. NOT PASTE YOUR REAL MAIL PASSWORD
MAIL_ENCRYPTION=SSL
MAIL_FROM_ADDRESS[email protected]
MAIL_FROM_NAME="${APP_NAME}"
Solution 13 - Php
I experienced the same issue. However, in my case I had the column in the table specified as varchar(255) and my content was larger when I was doing inserts. So it used to truncate the part which could not fit in the varchar column size.
To fix this, I updated my column type to longtext so larger text can be stored. Also, added a frontend validation to not have text larger than the column. (I was using this to store draft js escaped HTML)
Solution 14 - Php
Found an answer here.
The solution turned out to be in my application code, it has nothing to do with Laravel itself.
return mb_strtoupper(mb_substr($this->first_name, 0, 1) . mb_substr($this->last_name, 0, 1));
Solution 15 - Php
This was my problem and this is how I solved it, hope it works for someone else.
Make sure the encoding of the file is UTF-8, in vscode is in the right bottom part.
I had this particular problem when I modified a file that already was in a development server, somehow the encoding when I edited that file in that server was changed to ANSI, so, make sure too that your file is encoded in UTF-8 in the server where you are doing your deployments.
Solution 16 - Php
In my case, the error was similar to what @dev1234 mentioned here (https://stackoverflow.com/a/71279140/7880201) as part of the suggested answers.
But the problem was as a result of laravel outputing the entire error which contained blob(s). I would advise you do not return just the error message in the case of
try {
} catch (\Throwable $th) {
// $code = $th->getCode();
$msg = $th->getMessage();
// dd($msg);
abort(404, 'THERE WAS AN ERROR: ' . $msg);
}
So to rectify this issue, just show a little friendly error even if you want to show it to yourself alone as a dev.
Check all blobs to ensure that laravel is not just outputing them from the ajax response and you will be fine.
I hope it saves you today. If not, there are still good answers to try. :)
Solution 17 - Php
This happened to me when Laravel tried to return an error in Json format (while developing API). After solving the bug, this error disappeared.