Here is an example bit of Javascript to reproduce the issue:
let a = "qNqrdXUrk5S7dCM1MAYH3qSVDXznb-6prQoGqiACR10=";
console.log(atob(a));
In Safari we get:
Invalid Character Error: the string contains an invalid character.
In Chrome we get:
Uncaught DOMException: Failed to execute 'atob' on 'Window': The string to be decoded is not correctly encoded.
I can decode the above string on base64decode.org. By the way the value is intended to be a random sequence of bytes, not any sort of human readable string. atob ought to handle that.
From the "Design" section of that same page:
MIME's Base64 implementation uses A-Z, a-z, and 0-9 for the first 62 values, as well as "+" and "/" for the last two. Other variations, usually derived from Base64, share this property but differ in the symbols chosen for the last two values; an example is the URL and filename safe "RFC 4648 / Base64URL" variant, which uses "-" and "_".
JS uses the MIME-based encoding, and therefore the - in your initial string is invalid. It likely works on the decoder on the site by detecting which characters are in the input string and choosing the proper decoding format versus blindly using atob().
I've run into an issue of regex match not evaluating in Internet Explorer and in Firefox. It works fine in Chrome and Opera. I know Chrome is generally much more tolerant of mistakes so I suspect I've dropped the ball somewhere along the way - yet none of the online evaluation tools seem to find any errors in my expression. I'm sorry that it's such a convoluted expression but hopefully something will be easily obvious as the culprit. The expression is as follows:
keyData = data.match(/\w+\u0009\w+\u0009[\u0009]?\w+\u0009([-]?\w+|%%)[#]?\u0009([-]?\w+|%%)[#]?\u0009([-]?\w+|%%)[#]?(\u0009([-]?\w+|%%)[#]?)?(\u0009([-]?\w+|%%)[#]?)?(\u0009([-]?\w+|%%)[#]?)?\u0009\u0009\/\//g);
'data' is a text file which I am parsing with no errors. I wont post the whole file here but what I am hoping to match is something such as the following:
10 Q 1 0439 0419 -1 // CYRILLIC SMALL LETTER SHORT I, CYRILLIC CAPITAL LETTER SHORT I, <none>
I believe that when I post the string here it removes the 'u0009' characters so if you'd like to see one of the full files, I've linked one here. If there is anything more I can clarify, please let me know!
Edit:
My goal in this post is understanding not only why this is failing, but also if this expression well-formatted. After further review, it seems that it's an issue with how Internet Explorer and Firefox parse the text file. They seem to strip out the tabs and replace them with spaces. I tried to update the expression and it matches with no problems in an online validator but it still fails in IE/FF.
Edit 2
I have since updated my expression to a clearer form taking into account feedback. The issue still is persisting in IE and Firefox. It seems to be an issue with the string itself. IE won't let me match more than a single character, no matter what my expression is. For example, if the character string of the file is KEYBOARD and I try to match with /\w+/, it will just return K.
/[0-9](\w)?(\t+|\s+)\w+(\t+|\s+)[0-9](\t+|\s+)(-1|\w+#?|%%)(\t+|\s+)(-1|\w+#?|%%)(\t+|\s+)(-1|\w+#?|%%)((\t+|\s+)(-1|\w+#?|%%))?((\t+|\s+)(-1|\w+#?|%%))?((\t+|\s+)(-1|\w+#?|%%))?(\t+|\s+)\/\//g
After poking around with my regex for a while, I suspected something was wrong with the way IE was actually reading the text file as compared to Chrome. Specifically, if I had the string KEYBOARDwithin the text file and I tried to match it using /\w+/, it would simply return K in IE but in Chrome it would match the whole string KEYBOARD. I suspected IE was inserting some dead space between characters so I stepped through the first few characters of the file and printed their unicode equivalent.
for (i = 0; i < 30; i++) {
console.log(data.charCodeAt(i) + ' ' + data[i]);
}
This confirmed my suspicion and I saw u0000 pop up between each character. I'm not sure why there are NULL characters between each character but to resolve my issue I simply performed:
data = data.replace(/\u0000+/g, '');
This completely resolved my issue and I was able to parse my string like normal using the expression:
keyData = data.match(/[0-9](\w)?(\t+|\s+)\w+(\t+|\s+)[0-9](\t+|\s+)(-1|\w+#?|%%)(\t+|\s+)(-1|\w+#?|%%)(\t+|\s+)(-1|\w+#?|%%)((\t+|\s+)(-1|\w+#?|%%))?((\t+|\s+)(-1|\w+#?|%%))?((\t+|\s+)(-1|\w+#?|%%))?(\t+|\s+)\/\//g);
What are the allowed characters in both cookie name and value? Are they same as URL or some common subset?
Reason I'm asking is that I've recently hit some strange behavior with cookies that have - in their name and I'm just wondering if it's something browser specific or if my code is faulty.
According to the ancient Netscape cookie_spec the entire NAME=VALUE string is:
a sequence of characters excluding semi-colon, comma and white space.
So - should work, and it does seem to be OK in browsers I've got here; where are you having trouble with it?
By implication of the above:
= is legal to include, but potentially ambiguous. Browsers always split the name and value on the first = symbol in the string, so in practice you can put an = symbol in the VALUE but not the NAME.
What isn't mentioned, because Netscape were terrible at writing specs, but seems to be consistently supported by browsers:
either the NAME or the VALUE may be empty strings
if there is no = symbol in the string at all, browsers treat it as the cookie with the empty-string name, ie Set-Cookie: foo is the same as Set-Cookie: =foo.
when browsers output a cookie with an empty name, they omit the equals sign. So Set-Cookie: =bar begets Cookie: bar.
commas and spaces in names and values do actually seem to work, though spaces around the equals sign are trimmed
control characters (\x00 to \x1F plus \x7F) aren't allowed
What isn't mentioned and browsers are totally inconsistent about, is non-ASCII (Unicode) characters:
in Opera and Google Chrome, they are encoded to Cookie headers with UTF-8;
in IE, the machine's default code page is used (locale-specific and never UTF-8);
Firefox (and other Mozilla-based browsers) use the low byte of each UTF-16 code point on its own (so ISO-8859-1 is OK but anything else is mangled);
Safari simply refuses to send any cookie containing non-ASCII characters.
so in practice you cannot use non-ASCII characters in cookies at all. If you want to use Unicode, control codes or other arbitrary byte sequences, the cookie_spec demands you use an ad-hoc encoding scheme of your own choosing and suggest URL-encoding (as produced by JavaScript's encodeURIComponent) as a reasonable choice.
In terms of actual standards, there have been a few attempts to codify cookie behaviour but none thus far actually reflect the real world.
RFC 2109 was an attempt to codify and fix the original Netscape cookie_spec. In this standard many more special characters are disallowed, as it uses RFC 2616 tokens (a - is still allowed there), and only the value may be specified in a quoted-string with other characters. No browser ever implemented the limitations, the special handling of quoted strings and escaping, or the new features in this spec.
RFC 2965 was another go at it, tidying up 2109 and adding more features under a ‘version 2 cookies’ scheme. Nobody ever implemented any of that either. This spec has the same token-and-quoted-string limitations as the earlier version and it's just as much a load of nonsense.
RFC 6265 is an HTML5-era attempt to clear up the historical mess. It still doesn't match reality exactly but it's much better then the earlier attempts—it is at least a proper subset of what browsers support, not introducing any syntax that is supposed to work but doesn't (like the previous quoted-string).
In 6265 the cookie name is still specified as an RFC 2616 token, which means you can pick from the alphanums plus:
!#$%&'*+-.^_`|~
In the cookie value it formally bans the (filtered by browsers) control characters and (inconsistently-implemented) non-ASCII characters. It retains cookie_spec's prohibition on space, comma and semicolon, plus for compatibility with any poor idiots who actually implemented the earlier RFCs it also banned backslash and quotes, other than quotes wrapping the whole value (but in that case the quotes are still considered part of the value, not an encoding scheme). So that leaves you with the alphanums plus:
!#$%&'()*+-./:<=>?#[]^_`{|}~
In the real world we are still using the original-and-worst Netscape cookie_spec, so code that consumes cookies should be prepared to encounter pretty much anything, but for code that produces cookies it is advisable to stick with the subset in RFC 6265.
In ASP.Net you can use System.Web.HttpUtility to safely encode the cookie value before writing to the cookie and convert it back to its original form on reading it out.
// Encode
HttpUtility.UrlEncode(cookieData);
// Decode
HttpUtility.UrlDecode(encodedCookieData);
This will stop ampersands and equals signs spliting a value into a bunch of name/value pairs as it is written to a cookie.
I think it's generally browser specific. To be on the safe side, base64 encode a JSON object, and store everything in that. That way you just have to decode it and parse the JSON. All the characters used in base64 should play fine with most, if not all browsers.
Here it is, in as few words as possible. Focus on characters that need no escaping:
For cookies:
abdefghijklmnqrstuvxyzABDEFGHIJKLMNQRSTUVXYZ0123456789!#$%&'()*+-./:<>?#[]^_`{|}~
For urls
abdefghijklmnqrstuvxyzABDEFGHIJKLMNQRSTUVXYZ0123456789.-_~!$&'()*+,;=:#
For cookies and urls ( intersection )
abdefghijklmnqrstuvxyzABDEFGHIJKLMNQRSTUVXYZ0123456789!$&'()*+-.:#_~
That's how you answer.
Note that for cookies, the = has been removed because it is
usually used to set the cookie value.
For urls this the = was kept. The intersection is obviously without.
var chars = "abdefghijklmnqrstuvxyz"; chars += chars.toUpperCase() + "0123456789" + "!$&'()*+-.:#_~";
Turns out escaping still occuring and unexpected happening, especially in a Java cookie environment where the cookie is wrapped with double quotes if it encounters the last characters.
So to be safe, just use A-Za-z1-9. That's what I am going to do.
Newer rfc6265 published in April 2011:
cookie-header = "Cookie:" OWS cookie-string OWS
cookie-string = cookie-pair *( ";" SP cookie-pair )
cookie-pair = cookie-name "=" cookie-value
cookie-value = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )
cookie-octet = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
; US-ASCII characters excluding CTLs,
; whitespace DQUOTE, comma, semicolon,
; and backslash
If you look to #bobince answer you see that newer restrictions are more strict.
you can not put ";" in the value field of a cookie, the name that will be set is the string until the ";" in most browsers...
that's simple:
A <cookie-name> can be any US-ASCII characters except control
characters (CTLs), spaces, or tabs. It also must not contain a
separator character like the following: ( ) < > # , ; : \ " / [ ] ? =
{ }.
A <cookie-value> can optionally be set in double quotes and any
US-ASCII characters excluding CTLs, whitespace, double quotes, comma,
semicolon, and backslash are allowed. Encoding: Many implementations
perform URL encoding on cookie values, however it is not required per
the RFC specification. It does help satisfying the requirements about
which characters are allowed for though.
Link: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie#Directives
There are 2 versions of cookies specifications
1. Version 0 cookies aka Netscape cookies,
2. Version 1 aka RFC 2965 cookies
In version 0 The name and value part of cookies are sequences of characters, excluding the semicolon, comma, equals sign, and whitespace, if not used with double quotes
version 1 is a lot more complicated you can check it here
In this version specs for name value part is almost same except name can not start with $ sign
There is another interesting issue with IE and Edge. Cookies that have names with more than 1 period seem to be silently dropped.
So
This works:
cookie_name_a=valuea
while this will get dropped
cookie.name.a=valuea
One more consideration. I recently implemented a scheme in which some sensitive data posted to a PHP script needed to convert and return it as an encrypted cookie, that used all base64 values I thought were guaranteed 'safe". So I dutifully encrypted the data items using RC4, ran the output through base64_encode, and happily returned the cookie to the site. Testing seemed to go well until a base64 encoded string contained a "+" symbol. The string was written to the page cookie with no trouble. Using the browser diagnostics I could also verify the cookies was written unchanged. Then when a subsequent page called my PHP and obtained the cookie via the $_COOKIE array, I was stammered to find the string was now missing the "+" sign. Every occurrence of that character was replaced with an ASCII space.
Considering how many similar unresolved complaints I've read describing this scenario since then, often siting numerous references to using base64 to "safely" store arbitrary data in cookies, I thought I'd point out the problem and offer my admittedly kludgy solution.
After you've done whatever encryption you want to do on a piece of data, and then used base64_encode to make it "cookie-safe", run the output string through this...
// from browser to PHP. substitute troublesome chars with
// other cookie safe chars, or vis-versa.
function fix64($inp) {
$out =$inp;
for($i = 0; $i < strlen($inp); $i++) {
$c = $inp[$i];
switch ($c) {
case '+': $c = '*'; break; // definitly won't transfer!
case '*': $c = '+'; break;
case '=': $c = ':'; break; // = symbol seems like a bad idea
case ':': $c = '='; break;
default: continue;
}
$out[$i] = $c;
}
return $out;
}
Here I'm simply substituting "+" (and I decided "=" as well) with other "cookie safe" characters, before returning the encoded value to the page, for use as a cookie. Note that the length of the string being processed doesn't change. When the same (or another page on the site) runs my PHP script again, I'll be able to recover this cookie without missing characters. I just have to remember to pass the cookie back through the same fix64() call I created, and from there I can decode it with the usual base64_decode(), followed by whatever other decryption in your scheme.
There may be some setting I could make in PHP that allows base64 strings used in cookies to be transferred back to to PHP without corruption. In the mean time this works. The "+" may be a "legal" cookie value, but if you have any desire to be able to transmit such a string back to PHP (in my case via the $_COOKIE array), I'm suggesting re-processing to remove offending characters, and restore them after recovery. There are plenty of other "cookie safe" characters to choose from.
If you are using the variables later, you'll find that stuff like path actually will let accented characters through, but it won't actually match the browser path. For that you need to URIEncode them. So i.e. like this:
const encodedPath = encodeURI(myPath);
document.cookie = `use_pwa=true; domain=${location.host}; path=${encodedPath};`
So the "allowed" chars, might be more than what's in the spec. But you should stay within the spec, and use URI-encoded strings to be safe.
Years ago MSIE 5 or 5.5 (and probably both) had some serious issue with a "-" in the HTML block if you can believe it. Alhough it's not directly related, ever since we've stored an MD5 hash (containing letters and numbers only) in the cookie to look up everything else in server-side database.
I ended up using
cookie_value = encodeURIComponent(my_string);
and
my_string = decodeURIComponent(cookie_value);
That seems to work for all kinds of characters. I had weird issues otherwise, even with characters that weren't semicolons or commas.
As expected I need to encode a URI component before I call an API using it but when it hits our server somewhere along the line our backend framework (tapestry) converts spaces too early: Java URLEncoding / Decoding URL Spaces with dashes
I figured out that if I changed the URI %20 to $0020 it works. So the code below works in Chrome and Firefox and converts the % to a $00.
function furtherEncode(uriComp) {
var nonSafe = encodeURIComponent(uriComp);
return nonSafe.replace(/%/g, "$00");
}
In Internet Explorer 11 (and IE10) it doesn't do the replacement.
I have tried /\x25/g and /%/g as well as "$00" and '$00' but to no avail.
Any help would be greatly appreciated.
You need to have two dollar signs like here (tested in IE and Chrome):
"%".replace(/%/g, "$$00") /// returns "$00"
See docs here: docs
Relevant parts are defined under "Specifying a string as a parameter":
$$ | Inserts a "$".
$n | Where n or nn are decimal digits, inserts the nth parenthesized submatch string, provided the first argument was a RegExp object
Here is the sample path, assuming the format is fixed
For the 4th image, the path is 'demo/medium/Web081112_P004_medium.jpg'
so if the image is the 100 th ,then
'demo/medium/Web081112_P100_medium.jpg '
Is there any way to get the number in this format, I have thought about using reg exp? But it may cause a problem because I may retrieve 004 instead of 4, how to fix this? Thanks
This answer is based on the message in the original question
sample path, assuming the format is fixed
If the string is not going to change then you can use split:
var num = parseInt(path.split("_P")[1],10);
Note remember the parseInt needs the radix 10 to remove the leading 0s which would otherwise make the number OCTAL
Also note that splitting on _P will (as mentioned by Cerbrus) get the the string 004_medium.jpg which still works with parseInt because parseInt will ignore trailing non-numeric characters
Yes, thats definitelyprobably regular expression work.
/.*?\/.*?\/.*?_P(\d+).*?/.exec( imagesource )[ 1 ];
That way, you can have a "strict" pattern definition where you only parse that number if the input string has a certain format. You can write it more sloppy, like suggested in the comments
/.*?_P(\d+)/.exec( imagesource )[ 1 ];
disclaimer: I'm not that regexp genius, so please comment any optimizations :P
You don't need a regex, this will get a int from between the first and second occurrence of "_".
var path = 'demo/medium/Web081112_P100_medium.jpg'
var id = ~~path.split('_')[1].substr(1);
Remove the .substr(1) in case the "P" prefix is removed.
I'm using the ~~ as a quick-and dirty way to convert the output string to a int (Double bitwise not), since you're probably not going to have negative id's for your images.
Also, this appears to be faster than the parseInt function suggested in another answer, on Google Chrome.
Not on IE / Firefox, however, but they've lost the lead position as most-used browsers already.