Python 3000, Files and Text Encodings - The Diigo Meta page

• 17 Apr 09
  

  Marcin Kasperski

  imported python unicode encoding article programming text

• 26 Jun 08
  

  Stefan Kueppers

  python unicode encoding codecs

• 17 Apr 07
  

  Moe Mauch

  python unicode

  • <span class="pytext">encoding_matrix</span> <span class="pyoperator">=</span> <span class="pyoperator">{</span><br><br/>&nbsp;&nbsp; <span class="pytext">codecs</span><span class="pyoperator">.</span><span class="pytext">BOM_UTF8</span><span class="pyoperator">:</span> <span class="pystring">'utf_8'</span><span class="pyoperator">,</span><br><br/>&nbsp;&nbsp; <span class="pytext">codecs</span><span class="pyoperator">.</span><span class="pytext">BOM_UTF16</span><span class="pyoperator">:</span> <span class="pystring">'utf_16'</span><span class="pyoperator">,</span><br><br/>&nbsp;&nbsp; <span class="pytext">codecs</span><span class="pyoperator">.</span><span class="pytext">BOM_UTF16BE</span><span class="pyoperator">:</span> <span class="pystring">'utf16_be'</span><span class="pyoperator">,</span><br><br/>&nbsp;&nbsp; <span class="pytext">codecs</span><span class="pyoperator">.</span><span class="pytext">BOM_UTF16LE</span><span class="pyoperator">:</span> <span class="pystring">'utf16_le'</span><span class="pyoperator">,</span><br><br/><span class="pyoperator">}</span><br><br/><span class="pykeyword">def</span> <span class="pytext">guess_encoding</span><span class="pyoperator">(</span><span class="pytext">data</span><span class="pyoperator">)</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pystring">"""<br><br/>&nbsp;&nbsp;&nbsp;&nbsp;Given a byte string, guess the encoding.<br><br/><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;First it tries for UTF8/UTF16 BOM.<br><br/><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;Next it tries the standard 'UTF8', 'ISO-8859-1', and 'cp1252' encodings,<br><br/>&nbsp;&nbsp;&nbsp;&nbsp;Plus several gathered from locale information.<br><br/><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;The calling program *must* first call<br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;locale.setlocale(locale.LC_ALL, '')<br><br/><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;If successful it returns<br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(decoded_unicode, successful_encoding)<br><br/>&nbsp;&nbsp;&nbsp;&nbsp;If unsuccessful it raises a ``UnicodeError``.<br><br/>&nbsp;&nbsp;&nbsp;&nbsp;"""</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">for</span> <span class="pytext">bom</span><span class="pyoperator">,</span> <span class="pytext">enc</span> <span class="pykeyword">in</span> <span class="pytext">encoding_matrix</span><span class="pyoperator">.</span><span class="pytext">items</span><span class="pyoperator">(</span><span class="pyoperator">)</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">if</span> <span class="pytext">data</span><span class="pyoperator">.</span><span class="pytext">startswith</span><span class="pyoperator">(</span><span class="pytext">bom</span><span class="pyoperator">)</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">return</span> <span class="pytext">data</span><span class="pyoperator">.</span><span class="pytext">decode</span><span class="pyoperator">(</span><span class="pytext">enc</span><span class="pyoperator">)</span><span class="pyoperator">,</span> <span class="pytext">enc</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pytext">encodings</span> <span class="pyoperator">=</span> <span class="pyoperator">[</span><span class="pystring">'ascii'</span><span class="pyoperator">,</span> <span class="pystring">'UTF-8'</span><span class="pyoperator">]</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pytext">successful_encoding</span> <span class="pyoperator">=</span> <span class="pytext">None</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">try</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pytext">encodings</span><span class="pyoperator">.</span><span class="pytext">append</span><span class="pyoperator">(</span><span class="pytext">locale</span><span class="pyoperator">.</span><span class="pytext">nl_langinfo</span><span class="pyoperator">(</span><span class="pytext">locale</span><span class="pyoperator">.</span><span class="pytext">CODESET</span><span class="pyoperator">)</span><span class="pyoperator">)</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">except</span> <span class="pytext">AttributeError</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">pass</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">try</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pytext">encodings</span><span class="pyoperator">.</span><span class="pytext">append</span><span class="pyoperator">(</span><span class="pytext">locale</span><span class="pyoperator">.</span><span class="pytext">getlocale</span><span class="pyoperator">(</span><span class="pyoperator">)</span><span class="pyoperator">[</span><span class="pynumber">1</span><span class="pyoperator">]</span><span class="pyoperator">)</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">except</span> <span class="pyoperator">(</span><span class="pytext">AttributeError</span><span class="pyoperator">,</span> <span class="pytext">IndexError</span><span class="pyoperator">)</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">pass</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">try</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pytext">encodings</span><span class="pyoperator">.</span><span class="pytext">append</span><span class="pyoperator">(</span><span class="pytext">locale</span><span class="pyoperator">.</span><span class="pytext">getdefaultlocale</span><span class="pyoperator">(</span><span class="pyoperator">)</span><span class="pyoperator">[</span><span class="pynumber">1</span><span class="pyoperator">]</span><span class="pyoperator">)</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">except</span> <span class="pyoperator">(</span><span class="pytext">AttributeError</span><span class="pyoperator">,</span> <span class="pytext">IndexError</span><span class="pyoperator">)</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">pass</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pycomment"># latin-1<br><br/></span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pytext">encodings</span><span class="pyoperator">.</span><span class="pytext">append</span><span class="pyoperator">(</span><span class="pystring">'ISO8859-1'</span><span class="pyoperator">)</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pytext">encodings</span><span class="pyoperator">.</span><span class="pytext">append</span><span class="pyoperator">(</span><span class="pystring">'cp1252'</span><span class="pyoperator">)</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">for</span> <span class="pytext">enc</span> <span class="pykeyword">in</span> <span class="pytext">encodings</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">if</span> <span class="pykeyword">not</span> <span class="pytext">enc</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">continue</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">try</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pytext">decoded</span> <span class="pyoperator">=</span> <span class="pytext">unicode</span><span class="pyoperator">(</span><span class="pytext">data</span><span class="pyoperator">,</span> <span class="pytext">enc</span><span class="pyoperator">)</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pytext">successful_encoding</span> <span class="pyoperator">=</span> <span class="pytext">enc</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">break</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">except</span> <span class="pyoperator">(</span><span class="pytext">UnicodeError</span><span class="pyoperator">,</span> <span class="pytext">LookupError</span><span class="pyoperator">)</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">pass</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">if</span> <span class="pytext">successful_encoding</span> <span class="pykeyword">is</span> <span class="pytext">None</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <span class="pykeyword">raise</span> <span class="pytext">UnicodeError</span><span class="pyoperator">(</span><span class="pystring">'Unable to decode input data. Tried the'</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pystring">' following encodings: %s.'</span> <span class="pyoperator">%</span> <span class="pystring">', '</span><span class="pyoperator">.</span><span class="pytext">join</span><span class="pyoperator">(</span><span class="pyoperator">[</span><span class="pytext">repr</span><span class="pyoperator">(</span><span class="pytext">enc</span><span class="pyoperator">)</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">for</span> <span class="pytext">enc</span> <span class="pykeyword">in</span> <span class="pytext">encodings</span> <span class="pykeyword">if</span> <span class="pytext">enc</span><span class="pyoperator">]</span><span class="pyoperator">)</span><span class="pyoperator">)</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">else</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">if</span> <span class="pytext">successful_encoding</span> <span class="pyoperator">==</span> <span class="pystring">'ascii'</span><span class="pyoperator">:</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pycomment"># our default ascii encoding<br><br/></span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pytext">successful_encoding</span> <span class="pyoperator">=</span> <span class="pystring">'ISO8859-1'</span><br><br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="pykeyword">return</span> <span class="pyoperator">(</span><span class="pytext">decoded</span><span class="pyoperator">,</span> <span class="pytext">successful_encoding</span><span class="pyoperator">)</span>
• 28 Feb 07
  

  Carlos Santos

  python unicode text encoding programming article

Would you like to comment?

Join Diigo for a free account, or sign in if you are already a member.