[Solved] Detect file encoding in PHP

I have a script which combines a number of files into one, and it breaks when one of the files has UTF8 encoding. I figure that I should be using the utf8_decode() function when reading the files, but I don’t know how to tell which need decoding.

My code is basically:

$output = '';
foreach ($files as $filename) {
    $output .= file_get_contents($filename) . "n";
}
file_put_contents('combined.txt', $output);

Currently, at the start of a UTF8 file, it adds these characters in the output: 

Enquirer: nickf

||

Solution #1:

Try using the mb_detect_encoding function. This function will examine your string and attempt to “guess” what its encoding is. You can then convert it as desired. As brulak suggested, however, you’re probably better off converting to UTF-8 rather than from, to preserve the data you’re transmitting.

Respondent: Ben Blank

Solution #2:

To make sure that the output is UTF-8, no matter what kind of input it was, I use this check:

if(!mb_check_encoding($output, 'UTF-8')
    OR !($output === mb_convert_encoding(mb_convert_encoding($output, 'UTF-32', 'UTF-8' ), 'UTF-8', 'UTF-32'))) {

    $output = mb_convert_encoding($content, 'UTF-8', 'pass'); 
}

// $output is now safely converted to UTF-8!
Respondent: powtac

Solution #3:

mb_detect_encoding function should be your last choice. That could return the wrong encoding. Linux command file -i /path/myfile.txt is working great. In PHP you could use:

function _detectFileEncoding($filepath) {
    // VALIDATE $filepath !!!
    $output = array();
    exec('file -i ' . $filepath, $output);
    if (isset($output[0])){
        $ex = explode('charset=", $output[0]);
        return isset($ex[1]) ? $ex[1] : null;
    }
    return null;
}
yanek1988m

Solution #4:

This is my solution which worked like a charm:

//check string strict for encoding out of list of supported encodings
$enc = mb_detect_encoding($str, mb_list_encodings(), true);

if ($enc===false){
    //could not detect encoding
}
else if ($enc!=="UTF-8"){
    $str = mb_convert_encoding($str, "UTF-8", $enc);
}
else {
    //UTF-8 detected
}
Respondent: PapaKai

Solution #5:

For Linux servers, I use this command:

$file = 'your/file.ext'
exec( "from=`file -bi $file | awk -F'=' '{print $2 }'` && iconv -f $from -t utf-8 $file -o $file" );
Respondent: jgpATs2w

Solution #6:

Scans all file, finds any kind of encoding from mb_list_encodings, good performance..

    function detectFileEncoding($filePath){

    $fopen=fopen($filePath,'r');

    $row = fgets($fopen);
    $encodings = mb_list_encodings();
    $encoding = mb_detect_encoding( $row, "UTF-8, ASCII, Windows-1252, Windows-1254" );//these are my favorite encodings

    if($encoding !== false) {
        $key = array_search($encoding, $encodings) !== false;
        if ($key !== false)
            unset($encodings[$key]);
        $encodings = array_values($encodings);
    }

    $encKey = 0;
    while ($row = fgets($fopen)) {
        if($encoding == false){
            $encoding = $encodings[$encKey++];
        }

        if(!mb_check_encoding($row, $encoding)){
            $encoding =false;
            rewind($fopen);
        }

    }

    return $encoding;
}
Respondent: akakargul

Solution #7:

How are you going to handle the non-ASCII characters from the UTF-8 or 16 or 32 file?

I ask because I think you may have a design issue here.

I would convert your output file into UTF-8 (or 16 or 32) instead of the other way around.

Then you won’t have this problem.

Have you also considered the security issues that may arise from converting an escaped UTF-8 code? See this comment:

Detecting multi-byte encoding

Figure out what encoding your source file is in, then convert it to UTF-8, and you should be good to go.

Respondent: cbrulak

Solution #8:

I recently encountered this issue and the mb_convert_encoding() function output was UTF-8.

After taking a look at the response headers, there wasn’t anything mentioning the encoding type, so I found Set HTTP header to UTF-8 using PHP, which proposes the following:

<?php
header('Content-Type: text/html; charset=utf-8');

After adding that to the top of the PHP file, all of the funky characters went away and it rendered as it should. I am not sure if that’s the issue the original poster was seeking for, but I found this in trying to solve the issue myself and figured I’d share.

Respondent: Amereservant

Solution #9:

<?php

$file = 'myfile.csv';

function detect_encoding($file){
    return mb_detect_encoding(file_get_contents($file), mb_list_encodings());
}

if ( detect_encoding($file) == 'ISO-8859-1' ) {
    echo "ISO-8859-1 detected";
}
Respondent: Arnoldas

The answers/resolutions are collected from stackoverflow, are licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0 .

Leave a Reply

Your email address will not be published.