UTF-8: The Secret of Character Encoding - HTML Purifier - The Diigo Meta page

• 22 Jun 17
  

  Lawrence Lagerlof

  pocket backup public

• 01 Jul 16
  

  amoicity
  • A character encoding tells the computer how to  interpret raw zeroes and ones into real characters. It  usually does this by pairing numbers with characters.
  • The first step of our journey is to find out what the encoding of your website is. The most reliable way is to ask your browser:
  • Mozilla Firefox

     

    Tools > Page Info: Encoding

     

    Internet Explorer

     

    View > Encoding: bulleted item is unofficial name
  • For all those skeptics out there, there is a very good reason  why the character encoding should be explicitly stated. When the  browser isn't told what the character encoding of a text is, it  has to guess: and sometimes the guess is wrong. Hackers can manipulate  this guess in order to slip XSS past filters and then fool the  browser into executing it as active code. A great example of this  is the Google UTF-7  exploit.
  • In fact, the META tag is designed as a substitute for the HTTP header for contexts where sending headers is impossible (such as locally stored files without a webserver). Thus the name http-equiv (HTTP equivalent).

  • If your website:

      

     

    ...only uses ASCII characters,

     

    Either way is fine, but I recommend switching both to  UTF-8 (more on this later).

     

    ...uses special characters, and they display  properly,

     

    Change the embedded encoding to the server encoding.

     

    ...uses special characters, but users often complain that  they come out garbled,

     

    Change the server encoding to the embedded encoding.
  • Thus, a chicken-egg problem: a character encoding is necessary to interpret the text of a document. A META tag is in the text of a document. The META tag gives the character encoding. How can we determine the contents of a META tag, inside the text, if we don't know it's character encoding? And how do we figure out the character encoding, if we don't know the contents of the META tag?
  • Fortunately for us, the characters we need to write the META are in ASCII, which is pretty much universal over every character encoding that is in common use today. So, all the web-browser has to do is parse all the way down until it gets to the Content-Type tag, extract the character encoding tag, then re-parse the document according to this new information.
  • Obviously this is complicated, so browsers prefer the simpler and more efficient solution: get the character encoding from a somewhere other than the document itself, i.e. the HTTP headers, much to the chagrin of HTML authors who can't set these headers.

  • Many software projects, at one point or another, suddenly realize that they should be supporting more than one language. Even regular usage in one language sometimes requires the occasional special character that, without surprise, is not available in your character set. Sometimes developers get around this by adding support for multiple encodings: when using Chinese, use Big5, when using Japanese, use Shift-JIS, when using Greek, etc. Other times, they use character references with great zeal.

      

    UTF-8, however, obviates the need for any of these complicated measures. After getting the system to use UTF-8 and adjusting for sources that are outside the hand of the browser (more on this later), UTF-8 just works. You can use it for any language, even many languages at once, you don't have to worry about managing multiple encodings, you don't have to use those user-unfriendly entities.
  • Wikipedia is a great case study for an application that originally used ISO-8859-1 but switched to UTF-8 when it became far to cumbersome to support foreign languages.
  • application/x-www-form-urlencoded which is used for GET and by default for POST, and multipart/form-data which may be used by POST, and is required when you want to upload files.

  10 more annotations...

• 10 Mar 16
  

  makeller63

  "Character encoding and character sets are not that difficult to understand, but so many people blithely stumble through the worlds of programming without knowing what to actually do about it, or say "Ah, it's a job for those internationalization experts." No, it is not! This document will walk you through determining the encoding of your system and how you should handle this information. It will stay away from excessive discussion on the internals of character encoding."

  unicode utf-8 blackbird-training

• 17 Feb 16
  

  hafley66

  encoding UTF-8 Unicode utf8 html i18n

• 27 Nov 15
  

  Jochen Fromm

  If you don't hate character encodings, you're not a real programmer.

  encoding UTF-8 utf8 html web programming

• 10 Apr 15
  

  adrianomelgaco

  mysql mysql php efestas melhores práticas

• 16 Sep 14
  

  jeubeda

  encoding utf-8 unicode charset

• 03 Jun 14
  

  Pali Madra

  UTF-8 encoding Unicode utf8

• 27 Feb 14
  

  danthedane

  unicode character sets encoding

• 23 Sep 13
  

  Alan Levine

  Character encoding and character sets are not that difficult to understand, but so many people blithely stumble through the worlds of programming without knowing what to actually do about it, or say "Ah, it's a job for those internationalization experts."

  webdesign

• 19 Sep 13
  

  steven durow
• 23 May 13
  

  Vag Kos

  UTF-8

  • UTF-8: The Secret of Character Encoding
• 14 Feb 13
  

  Stephen Calnan

  utf8 character encoding web development i18n

• 06 Feb 13
  

  Magdalena Tubis

  utf-8 programming

• 25 Nov 12
  

  Saul Haro

  Pequeña explicación sobre el manejo del encoding y como afecta a nuestros proyectos Web. Recomendable porque vemos aspectos de seguridad que seguramente no habíamos contemplado antes.

  recomendación text análisis programación seguridad web

• 23 Nov 12
  

  Giancarlo Terrassan

  Code html Tutorial WEB CSS reference programming utf8 Webdesign

• 20 Nov 12
  

  mchinko

  utf8 UTF-8 encoding Unicode

• 16 Jun 12
  

  rfessler

  "UTF-8: The Secret of Character Encoding"

  UTF-8 encoding meta tag

• 13 May 12
  

  spiewok

  unicode programming utf-8 encoding utf8

• 02 May 12
  

  Jeff Roush

  claimed

  encoding

• 22 Apr 12
  

  Celia Palacios

  character encoding i18n utf-8 utf8 development webdev webmaster Web Design HTML

• 14 Mar 12
  

  Eko Gunawan

  Tips for implementing character encoding

  UTF-8 encoding Unicode

• 25 Jan 12
  

  Leila Menouar

  UTF-8 encoding Unicode html utf8 web programming code

• 24 Jan 12
  

  Heinz Lang

  Embedded encoding disagrees

  UTF-8 encoding web

• 

  biiber

  k UTF-8 encoding Unicode html

• 13 Dec 11
  

  Manimaran vadivel

  UTF-8 Unicode

• 08 Dec 11
  

  lepuck

  dev

• 15 Nov 11
  

  Hector Carcedo

  encoding UTF-8 utf8

• 25 Oct 11
  

  Mem Phisto

  encoding UTF-8 Unicode html utf8 i18n web programming

• 22 Oct 11
  

  dingodingo

  encoding UTF-8 webdev troubleshooting howto text xhtml code html programming international

• 28 Sep 11
  

  html

• 27 Sep 11
  

  François Georgy

  encoding utf-8 unicode utf8

• 02 Sep 11
  

  curly "smart" quotes from Microsoft

  encoding utf-8 utf8 typography webdesign xhtml html

• 05 Aug 11
  

  Javier Gómez

  encoding UTF-8

• 26 Jul 11
  

  Jeff Gima

  encoding imported-from-Delicious

• 18 Jun 11
  

  eric_kelly

  encoding html

• 01 Jun 11
  

  Fausto Carrera

  encoding utf-8 unicode html web php programming character i18n utf8

• 16 May 11
  

  Phil G

  UTF-8: The Secret of Character Encoding - HTML Purifier

• 24 Apr 11
  

  UTF-8 Unicode encoding

• 17 Mar 11
  

  garlo01

  description of hwo to program with appropriate character encoding

  SWdev

• 12 Mar 11
  

  Luca Filipozzi

  encoding unicode html utf8 i18n web programming utf-8

• 22 Feb 11
  
• 12 Feb 11
  

  Daniel Riaño
• 05 Jan 11
  

  Bernard Boulanger

  utf-8 encoding

  html php programming webdev web delicious

• 

  bboulanger84

  utf-8 encoding

  html php programming webdev web del

• 16 Dec 10
  

  Paolo Da Ponte Becher

  utf8 charset

• 13 Dec 10
  

  Nawid Tabibi

  encoding utf-8 unicode html programming php

• 12 Dec 10
  

  Ali Nabavi

  UTF-8 Unicode encoding import-delicious

• 24 Nov 10
  

  David Barker

  charset encoding

• 11 Nov 10
  

  Alexander K

  utf-8 utf8 encoding header programming development 2read

• 29 Sep 10
  

  ava tarantella

  utf-8 encoding webDevelopment

• 08 Sep 10
  

  D C

  utf-8 utf8 charset encoding webdev html unicode tutorial tips

• 27 Aug 10
  

  Demetris K

  08 2010 htmlpurifier.org utf8 unicode php web webdev development programming articles charset tips encoding

• 16 Jul 10
  

  Rob S

  encoding UTF-8 html web Unicode web development

• 21 Jun 10
  

  rinaldi fonseca

  charset ut8

• 29 Apr 10
  

  Bruce Hill

  CharacterSet Encoding Web Programming UTF8 ISO

• 23 Apr 10
  

  J M
• 10 Apr 10
  

  formosapearl

  unicode html utf-8 encoding programming delicious

• 10 Mar 10
  

  François Parmentier

  encoding web HTML

• 06 Mar 10
  

  bhuvidya

  utf-8 character-encoding charset i18n utf8

• 04 Mar 10
  

  Sean O'Halpin

  Handy overview of character encoding

  utf8 encoding utf-8 character-encoding charset i18n

• 24 Feb 10
  

  Marcelo Akira

  encoding utf-8 unicode html utf8 i18n web php

• 22 Feb 10
  

  Birger Eriksson

  utf-8 utf8

• 21 Jan 10
  

  Sub Proletarian

  utf-8 encoding unicode html

• 02 Nov 09
  

  islegmar

  enconding

• 27 Oct 09
  

  Steeve Cayla

  utf8

• 14 Aug 09
  

  Fernando Caldas

  utf-8

• 11 Aug 09
  

  shudson310

  encoding utf-8 Unicode i18n

• 05 Aug 09
  

  Eduardo Moreno

  html web programación tutorial utf-8 unicode encoding utf8 caracteres

• 24 Jul 09
  

  Zoltan Szasz

  php character encoding utf8 i18n i10n

• 10 Jun 09
  

  Snooky Snook

  From Safari

• 27 May 09
  

  jowouters

  utf-8 characterencoding utf8 delicious

• 22 Apr 09
  

  elle m

  Character encoding and character sets are not that difficult to understand, but so many people blithely stumble through the worlds of programming without knowing what to actually do about it, or say "Ah, it's a job for those internationalization experts."

  utf-8 unicode characters encoding i18n reference charset

• 11 Mar 09
  

  sreekumar kv

  php web apache programming tutorial utf-8 encoding unicode utf charset character utf8 delicious

• 05 Mar 09
  

  anycolorilike

  tutorials utf8 unicode web programming i18n

• 18 Feb 09
  

  Dustin Burke

  web programming php webdev html unicode character encoding utf8 utf-8 i18n

• 18 Dec 08
  

  Mu Kinger

  Configuring your database

  utf-8 i18n

• 14 Dec 08
  

  Doug Foster

  web_standards

• 19 Nov 08
  

  Tina McGuire

  utf8 web webdev utf-8 unicode tutorials resources

• 17 Nov 08
  

  Jerry Wong

  utf8 encoding

• 

  Yeray Darias

  Character encoding and character sets are not that difficult to understand, but so many people blithely stumble through the worlds of programming without knowing what to actually do about it, or say "Ah, it's a job for those internationalization experts."

  xml utf8 utf-8 unicode utf encoding

• 14 Nov 08
  

  Bradley Dilger

  Good explanations, HTML, PHP, server stuff too

  code web software programming standards how-to

• 13 Nov 08
  

  James Dempster

  utf8 utf-8 unicode webdev

• 10 Nov 08
  

  Thomas Kiser

  utf8 unicode delicious

• 09 Nov 08
  

  crazylion lee

  encoding webdev

• 06 Nov 08
  

  pblmnop

  programming character-encoding utf-8 i18n

• 

  Yoram Blumenberg

  Character encoding and character sets are not that difficult to understand, but so many people blithely stumble through the worlds of programming without knowing what to actually do about it, or say "Ah, it's a job for those internationalization experts."

  !check !read.on utf-8 html xhtml character encoding article howto resource

• 

  webdesign tutorial tips utf8 unicode

• 

  Sergio EH

  utf-8 codificación

• 05 Nov 08
  

  Andy Brockie

  utf8 utf-8 utf unicode encoding html_css

• 

  Vipin Chandran

  typography utf8 utf-8 unicode

• 

  jason_rastovski

  utf8 utf-8 unicode utf

• 

  Chu Yeow Cheah

  text

• 

  inayaili

  utf utf-8 utf8 unicode reference character encoding

• 

  siddartha B

  Determine encoding of your system.

  utf8 character encoding utf-8 unicode html i18n web IN

• 

  Hrvoje Blažeković

  utf-8 utf8 character-encoding howto resource

• 

  Eric Lemoine

  encoding utf

• 

  Pedro Trindade

  Describes the rationale for using UTF-8, the ramifications otherwise, and how to make the switch.

  xml webdev webdesign utf8 web utf-8 utf unicode typography

• 

  Stuart H

  unicode utf-8 charset encoding reference typography

• 

  pero palotes

  utf-8

Would you like to comment?

Join Diigo for a free account, or sign in if you are already a member.