We Keep Coding

Javascript Fetch - Interpret header as UTF-8? [duplicate]

I'm using a "fun" HTML special-character (✰)(see http://html5boilerplate.com/ for more info) for a Server HTTP-header and am wondering if it is "allowed" per spec.
Using the Network Tab in the dev tools in Chrome on Windows Xp Pro SP 3 I see the ✰ just fine.
In IE8 the ✰ is not rendered correctly.
The w3.org HTML validator does not render it correctly (displays "â°" instead).
Now, I'm not too keen on character encodings ... and frankly I don't really care too much about them; I just blindly use UTF-8 cus I'm told to. :-)
Is the disparity caused by bugs in the different parsers/browses/engines/(whatever-they-are-called)?
Is there a spec for this or maybe a list of allowed characters for an HTTP-header "value"?

In short: Only ASCII is guaranteed to work. Some non-ASCII bytes are allowed for backwards compatibility, but are not supposed to be displayable.
HTTPbis gave up and specified that in the headers there is no useful encoding besides ASCII:
Historically, HTTP has allowed field content with text in the
ISO-8859-1 charset [ISO-8859-1], supporting other charsets only
through use of [RFC2047] encoding. In practice, most HTTP header
field values use only a subset of the US-ASCII charset [USASCII].
Newly defined header fields SHOULD limit their field values to
US-ASCII octets. A recipient SHOULD treat other octets in field
content (obs-text) as opaque data.
Previously, RFC 2616 from 1999 defined this:
Words of *TEXT MAY contain characters from character sets other than ISO-
8859-1 [22] only when encoded according to the rules of RFC 2047 [14].
and RFC 2047 is the MIME encoding, so it'd be:
=?UTF-8?Q?=E2=9C=B0?=
but I don't think that many (if any) clients support it.

Please read comments first, this answer likely draws wrong conclusions from the right sources, needs edit.
You can use any printable ASCII chars, and no special chars like ✰ (Which is not ASCII)
Tip: you can encode anything in JSON.
Edit: may not be obvious at first, the character encoding defined in the header only applies for the response body, not for the header itself. (As it would cause a chicken-&-egg problem.)
I'd like to sum up all the relevant definitions as per the spec linked by Penchant.
message-header = field-name ":" [ field-value ]
field-name = token
field-value = *( field-content | LWS )
So, we are after field-value.
LWS = [CRLF] 1*( SP | HT )
CRLF = CR LF
CR = <US-ASCII CR, carriage return (13)>
LF = <US-ASCII LF, linefeed (10)>
SP = <US-ASCII SP, space (32)>
HT = <US-ASCII HT, horizontal-tab (9)>
LWS stands for Linear White Space. Essentially, LWS is Space or Tab, but you can break your field-value into multiple lines by starting a new line before a Space or Tab.
Let's simplify it to this:
field-value = <any field-content or Space or Tab>
Now we are after field-content.
field-content = <the OCTETs making up the field-value
and consisting of either *TEXT or combinations
of token, separators, and quoted-string>
OCTET = <any 8-bit sequence of data>
TEXT = <any OCTET except CTLs,
but including LWS>
CTL = <any US-ASCII control character
(octets 0 - 31) and DEL (127)>
token = 1*<any CHAR except CTLs or separators>
separators = "(" | ")" | "<" | ">" | "#"
| "," | ";" | ":" | "\" | <">
| "/" | "[" | "]" | "?" | "="
| "{" | "}" | SP | HT
TEXT is the most general and includes all the rest -so forget about the rest-.
Here is the US-ASCII charset (= ASCII)
As you can see, all printable ASCII chars are allowed.

Converting multiple emojis into a single character [duplicate]

I'm not sure what this is called so I'm having trouble searching for it. How can I decode a string with unicode from http\u00253A\u00252F\u00252Fexample.com to http://example.com with JavaScript? I tried unescape, decodeURI, and decodeURIComponent so I guess the only thing left is string replace.
EDIT: The string is not typed, but rather a substring from another piece of code. So to solve the problem you have to start with something like this:
var s = 'http\\u00253A\\u00252F\\u00252Fexample.com';
I hope that shows why unescape() doesn't work.

Edit (2017-10-12):
#MechaLynx and #Kevin-Weber note that unescape() is deprecated from non-browser environments and does not exist in TypeScript. decodeURIComponent is a drop-in replacement. For broader compatibility, use the below instead:
decodeURIComponent(JSON.parse('"http\\u00253A\\u00252F\\u00252Fexample.com"'));
> 'http://example.com'
Original answer:
unescape(JSON.parse('"http\\u00253A\\u00252F\\u00252Fexample.com"'));
> 'http://example.com'
You can offload all the work to JSON.parse

UPDATE: Please note that this is a solution that should apply to older browsers or non-browser platforms, and is kept alive for instructional purposes. Please refer to #radicand 's answer below for a more up to date answer.
This is a unicode, escaped string. First the string was escaped, then encoded with unicode. To convert back to normal:
var x = "http\\u00253A\\u00252F\\u00252Fexample.com";
var r = /\\u([\d\w]{4})/gi;
x = x.replace(r, function (match, grp) {
return String.fromCharCode(parseInt(grp, 16)); } );
console.log(x); // http%3A%2F%2Fexample.com
x = unescape(x);
console.log(x); // http://example.com
To explain: I use a regular expression to look for \u0025. However, since I need only a part of this string for my replace operation, I use parentheses to isolate the part I'm going to reuse, 0025. This isolated part is called a group.
The gi part at the end of the expression denotes it should match all instances in the string, not just the first one, and that the matching should be case insensitive. This might look unnecessary given the example, but it adds versatility.
Now, to convert from one string to the next, I need to execute some steps on each group of each match, and I can't do that by simply transforming the string. Helpfully, the String.replace operation can accept a function, which will be executed for each match. The return of that function will replace the match itself in the string.
I use the second parameter this function accepts, which is the group I need to use, and transform it to the equivalent utf-8 sequence, then use the built - in unescape function to decode the string to its proper form.

Note that the use of unescape() is deprecated and doesn't work with the TypeScript compiler, for example.
Based on radicand's answer and the comments section below, here's an updated solution:
var string = "http\\u00253A\\u00252F\\u00252Fexample.com";
decodeURIComponent(JSON.parse('"' + string.replace(/\"/g, '\\"') + '"'));
http://example.com

Using JSON.decode for this comes with significant drawbacks that you must be aware of:
You must wrap the string in double quotes
Many characters are not supported and must be escaped themselves. For example, passing any of the following to JSON.decode (after wrapping them in double quotes) will error even though these are all valid: \\n, \n, \\0, a"a
It does not support hexadecimal escapes: \\x45
It does not support Unicode code point sequences: \\u{045}
There are other caveats as well. Essentially, using JSON.decode for this purpose is a hack and doesn't work the way you might always expect. You should stick with using the JSON library to handle JSON, not for string operations.
I recently ran into this issue myself and wanted a robust decoder, so I ended up writing one myself. It's complete and thoroughly tested and is available here: https://github.com/iansan5653/unraw. It mimics the JavaScript standard as closely as possible.
Explanation:
The source is about 250 lines so I won't include it all here, but essentially it uses the following Regex to find all escape sequences and then parses them using parseInt(string, 16) to decode the base-16 numbers and then String.fromCodePoint(number) to get the corresponding character:
/\\(?:(\\)|x([\s\S]{0,2})|u(\{[^}]*\}?)|u([\s\S]{4})\\u([^{][\s\S]{0,3})|u([\s\S]{0,4})|([0-3]?[0-7]{1,2})|([\s\S])|$)/g
Commented (NOTE: This regex matches all escape sequences, including invalid ones. If the string would throw an error in JS, it throws an error in my library [ie, '\x!!' will error]):
/
\\ # All escape sequences start with a backslash
(?: # Starts a group of 'or' statements
(\\) # If a second backslash is encountered, stop there (it's an escaped slash)
| # or
x([\s\S]{0,2}) # Match valid hexadecimal sequences
| # or
u(\{[^}]*\}?) # Match valid code point sequences
| # or
u([\s\S]{4})\\u([^{][\s\S]{0,3}) # Match surrogate code points which get parsed together
| # or
u([\s\S]{0,4}) # Match non-surrogate Unicode sequences
| # or
([0-3]?[0-7]{1,2}) # Match deprecated octal sequences
| # or
([\s\S]) # Match anything else ('.' doesn't match newlines)
| # or
$ # Match the end of the string
) # End the group of 'or' statements
/g # Match as many instances as there are
Example
Using that library:
import unraw from "unraw";
let step1 = unraw('http\\u00253A\\u00252F\\u00252Fexample.com');
// yields "http%3A%2F%2Fexample.com"
// Then you can use decodeURIComponent to further decode it:
let step2 = decodeURIComponent(step1);
// yields http://example.com

I don't have enough rep to put this under comments to the existing answers:
unescape is only deprecated for working with URIs (or any encoded utf-8) which is probably the case for most people's needs. encodeURIComponent converts a js string to escaped UTF-8 and decodeURIComponent only works on escaped UTF-8 bytes. It throws an error for something like decodeURIComponent('%a9'); // error because extended ascii isn't valid utf-8 (even though that's still a unicode value), whereas unescape('%a9'); // © So you need to know your data when using decodeURIComponent.
decodeURIComponent won't work on "%C2" or any lone byte over 0x7f because in utf-8 that indicates part of a surrogate. However decodeURIComponent("%C2%A9") //gives you © Unescape wouldn't work properly on that // © AND it wouldn't throw an error, so unescape can lead to buggy code if you don't know your data.

This is not an answer to this exact question, but for those who are hitting this page via a search result and who are trying to (like I was) construct a single Unicode character given a sequence of escaped codepoints, note that you can pass multiple arguments to String.fromCodePoint() like so:
String.fromCodePoint(parseInt("1F469", 16), parseInt("200D", 16), parseInt("1F4BC", 16)) // 👩‍💼
You can of course parse your string to extract the hex codepoint strings and then do something like:
let codePoints = hexCodePointStrings.map(s => parseInt(s, 16));
let str = String.fromCodePoint(...codePoints);

In my case, I was trying to unescape HTML file sth like
"\u003Cdiv id=\u0022app\u0022\u003E\r\n \u003Cdiv data-v-269b6c0d\u003E\r\n \u003Cdiv data-v-269b6c0d class=\u0022menu\u0022\u003E\r\n \u003Cdiv data-v-269b6c0d class=\u0022faux_column\u0022\u003E\r\n \u003Cdiv data-v-269b6c0d class=\u0022row\u0022\u003E\r\n \u003Cdiv data-v-269b6c0d class=\u0022col-md-12\u0022\u003E\r\n"
to
<div id="app">
<div data-v-269b6c0d>
<div data-v-269b6c0d class="menu">
<div data-v-269b6c0d class="faux_column">
<div data-v-269b6c0d class="row">
<div data-v-269b6c0d class="col-md-12">
Here below works in my case:
const jsEscape = (str: string) => {
return str.replace(new RegExp("'", 'g'),"\\'");
}
export const decodeUnicodeEntities = (data: any) => {
return unescape(jsEscape(data));
}
// Use it
const data = ".....";
const unescaped = decodeUnicodeEntities(data); // Unescaped html

How do I match URLs with regular expressions?

We want to check if a URL matches mail.google.com or mail.yahoo.com (also a subdomain of them is accepted) but not a URL which contains this string after a question mark. We also want the strings "mail.google.com" and "mail.yahoo.com" to come before the third slash of the URL, for example https://mail.google.com/ is accepted, https://www.facebook.com/mail.google.com/ is not accepted, and https://www.facebook.com/?mail=https://mail.google.com/ is also not accepted. https://mail.google.com.au/ is also not accepted. Is it possible to do it with regular expressions?
var possibleURLs = /^[^\?]*(mail\.google\.com|mail\.yahoo\.com)\//gi;
var url;
// assign a value to var url.
if (url.match(possibleURLs) !== null) {
// Do something...
}
Currently this will match both https://mail.google.com/ and https://www.facebook.com/mail.google.com/ , but we don't want to match https://www.facebook.com/mail.google.com/.
Edit: I want to match any protocol (any string which doesn't contain "?" and "/") followed by a slash "/" twice (the string and the slash can both be twice), then any string which doesn't contain "?" and "/" (if it's not empty, it must end with a dot "."), and then (mail\.google\.com|mail\.yahoo\.com)\/. Case insensitive.

Not being funny - but why must it be a regular expression?
Is there are reason why you couldn't simplify the process using URL (or webkitURL in Chrome and Safari) - the URL constructor simply takes a string and then contains properties for each part of the URL. Whether it supports all the host types that you want to support, I don't know.
Granted, you might still need a regex after that (although really you'd just be checking that the hostname ends with either yahoo.com or google.com), but you would just be running it against the hostname of the URL object rather than the whole URI.
The API is not ubiquitous, but seems reasonably well supported and, anyway, if this is client-side validation then I hope you're checking it on the server, too, because sidestepping javascript validation is easy.

How about
^[a-z]+:\/\/([^.\/]+\.)*mail\.(google|yahoo).com\/
Regex Example Link
^ Anchors the regex at the start of the string
[a-z]+ Matches the protocol. If you want a specific set of protocols, then (https?|ftp) may do the work
([^.\/]+\.)* matches the subdomin part

^([-a-z]+://|^cid:|^//)([^/\?]+\.)?mail\.(google|yahoo)\.com/
Should do the trick
The first ^ means "match beginning of line", the second negates the allowed characters, thus making a slash / not allowed.
Nb. You still have to escape the slashes, or use it as a string in new RegExp(string):
new RegExp('^([-a-z]+://|^cid:|^//)([^/\?]+\.)?mail\.(google|yahoo)\.com/')

OK, I found that it works with:
var possibleURLs = /^([^\/\?]*\/){2}([^\.\/\?]+\.)*(mail\.google\.com|mail\.yahoo\.com)\//gi;

Regex to detect urls without www and http

Could you update my regex to match with next requirements
Must match urls without www and http
If query contains - match too
Url ends when space or comma(,) or string end meet
match only with TopLevelDomains from list
var srg = new RegExp(/(^|[\s])([\w\.]+\.(com|cc|net))/ig);
For sample, must match:
jsfiddle.net
jmitty.cc:8080/test3s.html
www.ru,sample.com,google.com/?l=en
very.secure.dotster.com/i?ewe
As result i need
<a>jsfiddle.net</a>
<a>jmitty.cc:8080/test3s.html</a>
<a>www.ru</a>,<a>sample.com</a>,<a>google.com/?l=en</a>
<a>very.secure.dotster.com/i?ewe</a>
Fiddle http://jsfiddle.net/tYnU7/

Well, I guess you can change some little things in your regex:
([\w\.]+\.(?:com|cc|net|ru)[^,\s]*)
Replace by:
$1
I'm not sure why you were having (^|[\s]) at the beginning and it didn't seem useful to me, so I removed it. If you had your reasons, you can put it back.
I added ru to the extensions to match www.ru as you required and added [^,\s]* to continue matching until a comma or space is encountered.
Your updated fiddle is here.

This is a very complex problem with no perfect answer, but if you don't need perfection, check out Jeff Roberson's Linkify page and this post by Van Goyvaerts discussing Jeff Atwood's blog post, "The Problem with URLs".

/
(?:^|\b) # match word boundary or beginning of line
( # begin cpature
[\w.]+ # domain part
\.[a-z]{2,3} # domain suffix
(?:\:[0-9]{1,5})? # optional port
(?:\/.*)? # path details
) # end capture
(?:[,\s]|$) # comma, space or eol
/ig
Some details:
[\w.]+ may need more work depending on what you classify as acceptable domain characters (I've heard they're accepting unicode characters now?)
You can change [a-z]{2,3} in to a list of acceptable top-level domains (e.g. (?:com|org|net|info|edu). In your example you only list com, cc & net, but your result shows www.ru as captured.
(?:\/.*)? is greedy by default, but should be okay since you want query information.
And the fiddle
Oh, and if you want your links clickable (because those without a protocol don't work):
var r = t.replace(srg, function(match,b,m,e){
return b + '' + m + '' + e;
});
Which is demonstrated here

My regex does not match numbers

I am new at making regular expressions, and so this might just be a stupid oversight, but my regex (that aims to match URL's) is not working. My goal was to have it match any urls like:
http://www.somewhere.com
somewhere.com
https://ww3.some_where-hi.com
www.goop.go/herp/derp.lol
The regex i built is below, however, it does not match a URL like http://t.co/GZhtBh6c, it stops matching at the number 6 (As determined by www.regexpal.com).
((http|https)://)?([a-z0-9]+\.)?[a-z0-9\-_]+.[a-z]+(/[a-z0-9\-_]*)*([a-z0-9\-_]*\.[a-z]+){0,1}
Can anyone tell me why this is not working? Also, I'm sure this is not the best solution. If you have a more elegant regex for this, I would love to see it.
P.S. This regex will be used with javascript.

Validate if a string holds a URL as specified in RFC 3986. Both absolute and relative URLs are supported.
This matches your provide sample and more. It also lets you extract the different parts of the url
^
(# Scheme
[a-z][a-z0-9+\-.]*:
(# Authority & path
//
([a-z0-9\-._~%!$&'()*+,;=]+#)? # User
([a-z0-9\-._~%]+ # Named host
|\[[a-f0-9:.]+\] # IPv6 host
|\[v[a-f0-9][a-z0-9\-._~%!$&'()*+,;=:]+\]) # IPvFuture host
(:[0-9]+)? # Port
(/[a-z0-9\-._~%!$&'()*+,;=:#]+)*/? # Path
|# Path without authority
(/?[a-z0-9\-._~%!$&'()*+,;=:#]+(/[a-z0-9\-._~%!$&'()*+,;=:#]+)*/?)?
)
|# Relative URL (no scheme or authority)
([a-z0-9\-._~%!$&'()*+,;=#]+(/[a-z0-9\-._~%!$&'()*+,;=:#]+)*/? # Relative path
|(/[a-z0-9\-._~%!$&'()*+,;=:#]+)+/?) # Absolute path
)
# Query
(\?[a-z0-9\-._~%!$&'()*+,;=:#/?]*)?
# Fragment
(\#[a-z0-9\-._~%!$&'()*+,;=:#/?]*)?
$
In javascript this becomes
if (/^([a-z][a-z0-9+\-.]*:(\/\/([a-z0-9\-._~%!$&'()*+,;=]+#)?([a-z0-9\-._~%]+|\[[a-f0-9:.]+\]|\[v[a-f0-9][a-z0-9\-._~%!$&'()*+,;=:]+\])(:[0-9]+)?(\/[a-z0-9\-._~%!$&'()*+,;=:#]+)*\/?|(\/?[a-z0-9\-._~%!$&'()*+,;=:#]+(\/[a-z0-9\-._~%!$&'()*+,;=:#]+)*\/?)?)|([a-z0-9\-._~%!$&'()*+,;=#]+(\/[a-z0-9\-._~%!$&'()*+,;=:#]+)*\/?|(\/[a-z0-9\-._~%!$&'()*+,;=:#]+)+\/?))(\?[a-z0-9\-._~%!$&'()*+,;=:#\/?]*)?(#[a-z0-9\-._~%!$&'()*+,;=:#\/?]*)?$/im.test(subject)) {
// Successful match
} else {
// Match attempt failed
}

use a [A-z] instead of [a-z]
your little a-z is only matching lowercase letters.