Each Answer to this Q is separated by one/two green lines.
There are two ways to open a text file in Python:
f = open(filename)
import codecs f = codecs.open(filename, encoding="utf-8")
codecs.open preferable to
Since Python 2.6, a good practice is to use
io.open(), which also takes an
encoding argument, like the now obsolete
codecs.open(). In Python 3,
io.open is an alias for the
open() built-in. So
io.open() works in Python 2.6 and all later versions, including Python 3.4. See docs: http://docs.python.org/3.4/library/io.html
Now, for the original question: when reading text (including “plain text”, HTML, XML and JSON) in Python 2 you should always use
io.open() with an explicit encoding, or
open() with an explicit encoding in Python 3. Doing so means you get correctly decoded Unicode, or get an error right off the bat, making it much easier to debug.
Pure ASCII “plain text” is a myth from the distant past. Proper English text uses curly quotes, em-dashes, bullets, € (euro signs) and even diaeresis (¨). Don’t be naïve! (And let’s not forget the Façade design pattern!)
Because pure ASCII is not a real option,
open() without an explicit encoding is only useful to read binary files.
Personally, I always use
codecs.open unless there’s a clear identified need to use
open**. The reason is that there’s been so many times when I’ve been bitten by having utf-8 input sneak into my programs. “Oh, I just know it’ll always be ascii” tends to be an assumption that gets broken often.
Assuming ‘utf-8’ as the default encoding tends to be a safer default choice in my experience, since ASCII can be treated as UTF-8, but the converse is not true. And in those cases when I truly do know that the input is ASCII, then I still do
codecs.open as I’m a firm believer in “explicit is better than implicit”.
** – in Python 2.x, as the comment on the question states in Python 3
In Python 2 there are unicode strings and bytestrings. If you just use bytestrings, you can read/write to a file opened with
open() just fine. After all, the strings are just bytes.
The problem comes when, say, you have a unicode string and you do the following:
>>> example = u'??? ?????? ????????' >>> open('sample.txt', 'w').write(example) Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)
So here obviously you either explicitly encode your unicode string in utf-8 or you use
codecs.open to do it for you transparently.
If you’re only ever using bytestrings then no problems:
>>> example="??? ?????? ????????" >>> open('sample.txt', 'w').write(example) >>>
It gets more involved than this because when you concatenate a unicode and bytestring string with the
+ operator you get a unicode string. Easy to get bitten by that one.
codecs.open doesn’t like bytestrings with non-ASCII chars being passed in:
codecs.open('test', 'w', encoding='utf-8').write('??? ??????') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.7/codecs.py", line 691, in write return self.writer.write(data) File "/usr/lib/python2.7/codecs.py", line 351, in write data, consumed = self.encode(object, self.errors) UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 0: ordinal not in range(128)
The advice about strings for input/ouput is normally “convert to unicode as early as possible and back to bytestrings as late as possible”. Using
codecs.open allows you to do the latter very easily.
Just be careful that you are giving it unicode strings and not bytestrings that may have non-ASCII characters.
When you need to open a file that has a certain encoding, you would use the
codecs.open, i suppose, is just a remnant from the
Python 2 days when the built-in open had a much simpler interface and fewer capabilities. In Python 2, built-in
open doesn’t take an encoding argument, so if you want to use something other than binary mode or the default encoding, codecs.open was supposed to be used.
Python 2.6, the io module came to the aid to make things a bit simpler.
According to the official documentation
New in version 2.6. The io module provides the Python interfaces to stream handling. Under Python 2.x, this is proposed as an alternative to the built-in file object, but in Python 3.x it is the default interface to access files and streams.
Having said that, the only use i can think of
codecs.open in the current scenario is for the backward compatibility. In all other scenarios (unless you are using Python < 2.6) it is preferable to use
io.open. Also in
io.open is the same as
There is a syntactical difference between
io.open as well.
open(filename, mode="rb", encoding=None, errors="strict", buffering=1)
open(file, mode="r", buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None)
When you want to load a binary file, use
f = io.open(filename, 'b').
For opening a text file, always use
f = io.open(filename, encoding='utf-8')with explicit encoding.
In python 3 however
open does the same thing as
io.open and can be used instead.
codecs.openis planned to become deprecated and replaced by
io.openafter its introduction in python 2.6. I would only use it if code needs to be compatible with earlier python versions. For more information on codecs and unicode in python see the Unicode HOWTO.
When you’re working with text files and want transparent encoding and decoding into Unicode objects.