We Keep Coding

Handle non-ASCII filenames in XHR uploading

I have pretty standard javascript/XHR drag-and-drop file upload code, and just came across an unfortunate real-world snag. I have a file on my (Win7) desktop called "TEST-é-TEST.txt". In Chrome (30.0.1599.69), it arrives at the server with filename in UTF-8, which works out fine. In Firefox (24.0), the filename seems mangled when it arrives at the server.
I didn't trust what Firebug/Chrome might be telling me about the encoding, so I examined the hex of the request packet. Everything else is the same except the non-ASCII character is indeed being encoded differently in the two browsers:
Chrome: C3 A9 (this is the expected UTF-8 for that character)
Firefox: EF BF BD (UTF-8 "replacement character"?!)
Is this a Firefox bug? I tried renaming the file, replacing the é with ó, and the Firefox hex was the same... so such a mangle really seems like a browser bug. (If Firefox were confusedly sending along ISO-8859-1, for example, without touching it, I'd see an E9 byte, and I could handle that on the server side, but it shouldn't mangle it!)
Regardless of the reason, is there something I can do on either the client or server sides to correct for this? If a replacement character is indeed being sent to the server, then it would seem unrecoverable there, so I almost certainly need to do it on the client side.
And yes, the page on which this code exists has charset=utf-8, and Firefox confirms that it perceives the page as UTF-8 under View>Character Encoding.
Furthermore, if I dump the filename to console.log, it appears fine there--I guess it's just getting mangled in/after setRequestHeader("X-File-Name",file.name).
Finally, it would seem that the value passed to setRequestHeader() should be able to have code points up to U+00FF, so U+00E9 (é) and U+00F3 (ó) shouldn't cause a problem, though higher codes could trigger a SyntaxError: http://www.w3.org/TR/XMLHttpRequest2/#the-setrequestheader-method

Thanks so much for Boris's help. Here's a summary of what I discovered through our interactions in comments:
1) The core issue is that HTTP Request headers are supposed to be ISO-8859-1. Prior versions of Chrome and Firefox both passed along UTF-8 strings unchanged in setRequestHeader() calls. This changed in FF24.0 (and apparently will be changing in Chrome soon too), such that FF drops high bytes and passes along only the low byte for each character. In the example I gave in the question, this was recoverable, but characters with higher codes could be mangled irretrievably.
2) One workaround would be to encode on the client side, e.g.:
setRequestHeader('X-File-Name',encodeURIComponent(filename))
and then decode on the server side, e.g. in PHP:
$filename=rawurldecode($_SERVER['HTTP_X_FILE_NAME'])
3) Note that this is only problematic because my ajax file upload approach is to send the raw file data in the request body, so I need to send the filename via a custom request header (as shown in many tutorials online). If I used FormData instead, I wouldn't have to worry about this. I believe if you want solid, standards-based unicode filename support, you should use FormData and not the request header approach.

Login with yahoo using OpenID for Multilingual site

I am integrating yahoo open ID for my site. My site is running for different languages( en, jp, cn etc.). When I am trying to logged in from english site with yahoo opend ID then its fine but when the same this I tried from japanese or chinese site then it's not redirecting me to yahoo open id login.
Each time I am getting the below error from javascript
Error: The character encoding of the plain text document was not
declared. The document will render with garbled text in some browser
configurations if the document contains characters from outside the
US-ASCII range. The character encoding of the file needs to be
declared in the transfer protocol or file needs to use a byte order
mark as an encoding signature. Source File:
http://uatstorefrontjpcr.mobi-book.com/ReturnFromSocial/LogOnYahoo
Line: 0
Can anyone suggest what to do.
I have used SocialAuth-net.dll for this purpose. I have set all required wrapper in web.config. Same coding is okay with Google and facebook open ID.

Your web server is probably sending back a response with a Content-Type of text/plain. When a web browser receives a response with that content type, it doesn't know what encoding should be used to decode it; since you haven't told it how to decode it, different browsers might choose different ways.
The solution is to provide an explicit encoding. For example, if you know that the text is UTF-8 encoded, then you could provide it in a header like so:
Content-Type: text/plain; charset=UTF-8
According to an informational page from the W3C, you can get ASP.Net to include that bit in the header using Response.ContentEncoding. Again using UTF-8 as an example, you can set it like so:
Response.ContentEncoding = Encoding.UTF8;

What is the purpose of charset attribute of script element in HTML?

In HTML, script element has optional charset attribute.
What is the purpose of it?
When is it useful?

If your javascript files are encoded with a different encoding than the page is using, you can use the charset attribute to tell the browser how to interpret it.
For example, if the page is using Latin1, but the JS file was created with UTF-8.

The purpose of the charset parameter is to specify the encoding of the external script in cases where the encoding is not specified at the HTTP protocol level. It is not meant to override encoding information in HTTP headers, and it does not do that.
This is useful when the author cannot control HTTP headers and the headers do not specify character encoding. It is also useful for offline files, such as in a local copy of a web page accessed directly, not via an HTTP server, so that no HTTP headers exist.
In practice, it is not very useful. If you need to use non-Ascii characters in a JavaScript file, you can use UTF-8 encoding. If you use UTF-8 with a leading BOM, the BOM acts as a useful indicator that lets browsers infer the encoding. But it does not hurt to additionally use charset=utf-8.

Each JavaScript file is a separate element from page, after all you can even load JS from some remote author's server that otherwise have no relations to your page at all. Just as with any other external element, you can manually specify "charset" if remote server returns wrong charset for some reason or just to be sure.
Also, if you have write access to this JS file yourself, you may want to replace all non-ASCII with Unicode position escapes - this will guarantee that symbols will always be interpreted correctly, no matter what encoding is specified in headers. Some JS minifiers, like Google Closure Compiler, can do it for you automatically.

Javascript Charset problem

I want to read a file from my server with javascript and display it's content in a html page.
The file is in ANSI charset, and it has romanian characters.. I want to display those characters in the way they are :D not in different black symbols..
So I think my problem is the charset.. I have a get request that takes the content of the file, like this:
function IO(U, V) {//LA MOD String Version. A tiny ajax library. by, DanDavis
var X = !window.XMLHttpRequest ? new ActiveXObject('Microsoft.XMLHTTP') : new XMLHttpRequest();
X.open(V ? 'PUT' : 'GET', U, false );
X.setRequestHeader('Content-Type', 'Charset=UTF-8');
X.send(V ? V : '');return X.responseText;}
As far as I know the romanian characters are included in UTF-8 charset so I set the charset of the request header to utf-8.. the file is in utf-8 format and I have the meta tag that tells the browser that the page has utf-8 content..
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
So if I query the server the direct file, the browser shows me the romanian characters but if I display the content of the page through this script, I see only symbols instead of characters..
So what I am doing wrong?
Thank you!
PS: I want this to work on Firefox at least not necessarily in all browsers..

While my initial assumption was the same as T.J. Crowder's, a quick chat established that the OP uses some hosting service and cannot easily change the Content-Type headers.
The files were sent as text/plain or text/html without any Charset paramter, hence the browser interprets them as UTF-8 (which is the default).
So saving the files in UTF-8 (instead of ANSI/Windows-1252) did the trick.

You need to ensure that the HTTP response returning the file data has the correct charset identified on it. You have to do that server-side, I don't think you can force it from the client. (When you set the content type in the request header, you're setting the content type of the request, not the response.) So for instance, the response header from the server would be along the lines of:
Content-Type: text/plain; charset=windows-1252
...if by "ANSI" you mean the Windows-1252 charset. That should tell the browser what it needs to do to decode the response text correctly before handing it to the JavaScript layer.
One problem, though: As far as I can tell, Windows-1252 doesn't have the full Romanian alphabet. So if you're seeing characters like Ș, ș, Ţ, ţ, etc., that suggests the source text is not in Windows-1252. Now, perhaps it's okay to drop the diacriticals on those in Romanian (I wouldn't know) and so if your source text just uses S and T instead of Ș and Ţ, etc., it could still be in Windows-1252. Or it may be ISO-8859 or ISO-8859-2 (both of which drop some diacriticals) or possibly ISO-8859-16 (which has full Romanian support). Details here.
So the first thing to do is determine what character set the source text is actually in.

Detect the file size of a link's href using JavaScript

Would like to write a script to detect the file size of the target of a link on a web page.
Right now I have a function that finds all links to PDF files (i.e. the href ends with '.pdf') and appends the string '[pdf]' to the innerText. I would like to extend it so that I can also append some text advising the user that the target is a large file (e.g. greater than 1MB).
Thanks

Some web servers may give you a Content-Length header in response to a HEAD request. You could potentially use an XmlHttpRequest to send the HEAD request and see what you get.
Here's what one of my IIS servers says about a PDF file:
HTTP/1.1 200 OK
Content-Length: 127791
Content-Type: application/pdf
...
However, anything that's not delivered directly by the web server (a file served by PHP or ASP.net, for example) won't work unless the script specifically handles HEAD requests.

You should be able to do a HEAD request using XMLHttpRequest, assuming the files are under the same domain.
This is however something that should really be done on the server side. Doing it with extra requests has no benefit whatsoever.

You can't do this, or at least - not in any practical-cross-browser way.
If you know the filesize beforehand, for example when generating the document linking to the files you could hard-code the sizes into the HTML document.
large_file.pdf