We Keep Coding

Correct way to pass base64 info in URL as query string?

I am sending some parameters in query string by converting them in base64 using method btoa(). I just want to know, do I still need to wrap the result into the encodeURI method to pass on the correct information to the server in URL.
For example:
http://example.com/{name} - For this, which one is correct from below?
"http://example.com/" + btoa("some-name")
"http://example.com/" + encodeURI(btoa("some-name"))

You only need to transform your string to Base 64. See this SO answer for C#:
var plainTextBytes = System.Text.Encoding.UTF8.GetBytes(plainText);
return System.Convert.ToBase64String(plainTextBytes);
You can also read the reason why we use Base 64.
No need to use a method such as encodeURI after, it would be useless because all the characters for Base 64 (A-Z, a-z, 0-9, =, +) are not escaped:
encodeURI escapes all characters except:
A-Z a-z 0-9 ; , / ? : # & = + $ - _ . ! ~ * ' ( ) #
By the way, encodeURI and btoa are about JavaScript, not asp.net as you mention in tags.

Replace All Occurrences Of a String in JavaScript

I have a string as follows
var company = "Microst+Apple+Google";
And I want replace all the + signs with %2B
But when I use this code. It returns 0
var company = company.replace(/+/g, "%2B");
I think JavaScript thinks that + is an arithmetic operation. Is there a special symbol to be used? or can user a variable except directly using the + sign?
If so please mention. Any Idea how to do this ?

You need escape it :
var company = company.replace(/\+/g, "%2B");
It is because + is special symbol used to indicate that preceding character should match 1 or more times.
You can read more about regular expressions syntax here: https://developer.mozilla.org/en/docs/Web/JavaScript/Guide/Regular_Expressions

No, JavaScript doesn't think it's an arithmetic operation but + is a quantifier in regular expressions and the regular expression parser doesn't understand yours.
You must escape the + :
var company = company.replace(/\+/g, "%2B");

You can use this :
var company = company.replace(/\+/g, "%2B");
Or an easier way :
var company = encodeURIComponent(company);
which will do the same operation as the regex one. Also, it encodes all URI chars like &, " (quotes), % etc... if there is one like it in the string given.
In both cases, the output is :
Microst%2BApple%2BGoogle

Javascript RegEx non-capturing prefix

I am trying to do some string replacement with RegEx in Javascript. The scenario is a single line string containing long comma-delimited list of numbers, in which duplicates are possible.
An example string is: 272,2725,2726,272,2727,297,272 (The end may or may not end in a comma)
In this example, I am trying to match each occurrence of the whole number 272. (3 matches expected)
The example regex I'm trying to use is: (?:^|,)272(?=$|,)
The problem I am having is that the second and third matches are including the leading comma, which I do not want. I am confused because I thought (?:^|,) would match, but not capture. Can someone shed light on this for me? An interesting bit is that the trailing comma is excluded from the result, which is what I want.
For what it is worth, if I were using C# there is syntax for prefix matching that does what I want: (?<=^|,)
However, it appears to be unsupported in JavaScript.
Lastly, I know I could workaround it using string splitting, array manipulation and rejoining, but I want to learn.

Use word boundaries instead:
\b272\b
ensures that only 272 matches, but not 2725.
(?:...) matches and doesn't capture - but whatever it matches will be part of the overall match.
A lookaround assertion like (?=...) is different: It only checks if it is possible (or impossible) to match the enclosed regex at the current point, but it doesn't add to the overall match.

Here is a way to create a JavaScript look behind that has worked in all cases I needed.
This is an example. One can do many more complex and flexible things.
The main point here is that in some cases,
it is possible to create a RegExp non-capturing prefix
(look behind) construct in JavaScript .
This example is designed to extract all fields that are surrounded by braces '{...}'.
The braces are not returned with the field.
This is just an example to show the idea at work not necessarily a prelude to an application.
function testGetSingleRepeatedCharacterInBraces()
{
var leadingHtmlSpaces = ' ' ;
// The '(?:\b|\B(?={))' acts as a prefix non-capturing group.
// That is, this works (?:\b|\B(?=WhateverYouLike))
var regex = /(?:\b|\B(?={))(([0-9a-zA-Z_])\2{4})(?=})/g ;
var string = '' ;
string = 'Message has no fields' ;
document.write( 'String => "' + string
+ '"<br>' + leadingHtmlSpaces + 'fields => '
+ getMatchingFields( string, regex )
+ '<br>' ) ;
string = '{LLLLL}Message {11111}{22222} {ffffff}abc def{EEEEE} {_____} {4444} {666666} {55555}' ;
document.write( 'String => "' + string
+ '"<br>' + leadingHtmlSpaces + 'fields => '
+ getMatchingFields( string, regex )
+ '<br>' ) ;
} ;
function getMatchingFields( stringToSearch, regex )
{
var matches = stringToSearch.match( regex ) ;
return matches ? matches : [] ;
} ;
Output:
String => "Message has no fields"
fields =>
String => "{LLLLL}Message {11111}{22222} {ffffff}abc def{EEEEE} {_____} {4444} {666666} {55555}"
fields => LLLLL,11111,22222,EEEEE,_____,55555

Equivalent JavaScript functions for Python's urllib.parse.quote() and urllib.parse.unquote()

Are there any equivalent JavaScript functions for Python's urllib.parse.quote() and urllib.parse.unquote()?
The closest I've come across are encodeURI()/encodeURIComponent() and escape() (and their corresponding un-encoding functions), but they don't encode/decode the same set of special characters as far as I can tell.

JavaScript | Python
-----------------------------------
encodeURI(str) | urllib.parse.quote(str, safe='~##$&()*!+=:;,?/\'');
-----------------------------------
encodeURIComponent(str) | urllib.parse.quote(str, safe='~()*!\'')
On Python 3.7+ you can remove ~ from safe=.

OK, I think I'm going to go with a hybrid custom set of functions:
Encode: Use encodeURIComponent(), then put slashes back in.
Decode: Decode any %hex values found.
Here's a more complete variant of what I ended up using (it handles Unicode properly, too):
function quoteUrl(url, safe) {
if (typeof(safe) !== 'string') {
safe = '/'; // Don't escape slashes by default
}
url = encodeURIComponent(url);
// Unescape characters that were in the safe list
toUnencode = [ ];
for (var i = safe.length - 1; i >= 0; --i) {
var encoded = encodeURIComponent(safe[i]);
if (encoded !== safe.charAt(i)) { // Ignore safe char if it wasn't escaped
toUnencode.push(encoded);
}
}
url = url.replace(new RegExp(toUnencode.join('|'), 'ig'), decodeURIComponent);
return url;
}
var unquoteUrl = decodeURIComponent; // Make alias to have symmetric function names
Note that if you don't need "safe" characters when encoding ('/' by default in Python), then you can just use the built-in encodeURIComponent() and decodeURIComponent() functions directly.
Also, if there are Unicode characters (i.e. characters with codepoint >= 128) in the string, then to maintain compatibility with JavaScript's encodeURIComponent(), the Python quote_url() would have to be:
def quote_url(url, safe):
"""URL-encodes a string (either str (i.e. ASCII) or unicode);
uses de-facto UTF-8 encoding to handle Unicode codepoints in given string.
"""
return urllib.quote(unicode(url).encode('utf-8'), safe)
And unquote_url() would be:
def unquote_url(url):
"""Decodes a URL that was encoded using quote_url.
Returns a unicode instance.
"""
return urllib.unquote(url).decode('utf-8')

The requests library is a bit more popular if you don't mind the extra dependency
from requests.utils import quote
quote(str)

Python: urllib.quote
Javascript:unescape
I haven't done extensive testing but for my purposes it works most of the time. I guess you have some specific characters that don't work. Maybe if I use some Asian text or something it will break :)
This came up when I googled so I put this in for all the others, if not specifically for the original question.

Here are implementations based on a implementation on github repo purescript-python:
import urllib.parse as urllp
def encodeURI(s): return urllp.quote(s, safe="~##$&()*!+=:;,.?/'")
def decodeURI(s): return urllp.unquote(s, errors="strict")
def encodeURIComponent(s): return urllp.quote(s, safe="~()*!.'")
def decodeURIComponent(s): return urllp.unquote(s, errors="strict")

Try a regex. Something like this:
mystring.replace(/[\xFF-\xFFFF]/g, "%" + "$&".charCodeAt(0));
That will replace any character above ordinal 255 with its corresponding %HEX representation.

decodeURIComponent() is similar to unquote
const unquote = decodeURIComponent
const unquote_plus = (s) => decodeURIComponent(s.replace(/\+/g, ' '))
except that Python is much more forgiving. If one of the two characters after a % is not a hex digit (or there's not two characters after a %), JavaScript will throw a URIError: URI malformed error, whereas Python will just leave the % as is.
encodeURIComponent() is not quite the same as quote, you need to percent encode a few more characters and un-escape /:
const quoteChar = (c) => '%' + c.charCodeAt(0).toString(16).padStart(2, '0').toUpperCase()
const quote = (s) => encodeURIComponent(s).replace(/[()*!']/g, quoteChar).replace(/%2F/g, '/')
const quote_plus = (s) => quote(s).replace(/%20/g, '+')
The characters that Python's quote doesn't escape is documented here and is listed as (on Python 3.7+) "Letters, digits, and the characters '_.-~' are never quoted. By default, this function is intended for quoting the path section of a URL. The optional safe parameter specifies additional ASCII characters that should not be quoted — its default value is '/'"
The characters that JavaScript's encodeURIComponent doesn't encode is documented here and is listed as uriAlpha (upper and lowercase ASCII letters), DecimalDigit and uriMark, which are - _ . ! ~ * ' ( ).

When are you supposed to use escape instead of encodeURI / encodeURIComponent?

When encoding a query string to be sent to a web server - when do you use escape() and when do you use encodeURI() or encodeURIComponent():
Use escape:
escape("% +&=");
OR
use encodeURI() / encodeURIComponent()
encodeURI("http://www.google.com?var1=value1&var2=value2");
encodeURIComponent("var1=value1&var2=value2");

escape()
Don't use it!
escape() is defined in section B.2.1.1 escape and the introduction text of Annex B says:
... All of the language features and behaviours specified in this annex have one or more undesirable characteristics and in the absence of legacy usage would be removed from this specification. ...
... Programmers should not use or assume the existence of these features and behaviours when writing new ECMAScript code....
Behaviour:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/escape
Special characters are encoded with the exception of: #*_+-./
The hexadecimal form for characters, whose code unit value is 0xFF or less, is a two-digit escape sequence: %xx.
For characters with a greater code unit, the four-digit format %uxxxx is used. This is not allowed within a query string (as defined in RFC3986):
query = *( pchar / "/" / "?" )
pchar = unreserved / pct-encoded / sub-delims / ":" / "#"
unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"
pct-encoded = "%" HEXDIG HEXDIG
sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
/ "*" / "+" / "," / ";" / "="
A percent sign is only allowed if it is directly followed by two hexdigits, percent followed by u is not allowed.
encodeURI()
Use encodeURI when you want a working URL. Make this call:
encodeURI("http://www.example.org/a file with spaces.html")
to get:
http://www.example.org/a%20file%20with%20spaces.html
Don't call encodeURIComponent since it would destroy the URL and return
http%3A%2F%2Fwww.example.org%2Fa%20file%20with%20spaces.html
Note that encodeURI, like encodeURIComponent, does not escape the ' character.
encodeURIComponent()
Use encodeURIComponent when you want to encode the value of a URL parameter.
var p1 = encodeURIComponent("http://example.org/?a=12&b=55")
Then you may create the URL you need:
var url = "http://example.net/?param1=" + p1 + "&param2=99";
And you will get this complete URL:
http://example.net/?param1=http%3A%2F%2Fexample.org%2F%Ffa%3D12%26b%3D55&param2=99
Note that encodeURIComponent does not escape the ' character. A common bug is to use it to create html attributes such as href='MyUrl', which could suffer an injection bug. If you are constructing html from strings, either use " instead of ' for attribute quotes, or add an extra layer of encoding (' can be encoded as %27).
For more information on this type of encoding you can check: http://en.wikipedia.org/wiki/Percent-encoding

The difference between encodeURI() and encodeURIComponent() are exactly 11 characters encoded by encodeURIComponent but not by encodeURI:
I generated this table easily with console.table in Google Chrome with this code:
var arr = [];
for(var i=0;i<256;i++) {
var char=String.fromCharCode(i);
if(encodeURI(char)!==encodeURIComponent(char)) {
arr.push({
character:char,
encodeURI:encodeURI(char),
encodeURIComponent:encodeURIComponent(char)
});
}
}
console.table(arr);

I found this article enlightening :
Javascript Madness: Query String Parsing
I found it when I was trying to undersand why decodeURIComponent was not decoding '+' correctly. Here is an extract:
String: "A + B"
Expected Query String Encoding: "A+%2B+B"
escape("A + B") = "A%20+%20B" Wrong!
encodeURI("A + B") = "A%20+%20B" Wrong!
encodeURIComponent("A + B") = "A%20%2B%20B" Acceptable, but strange
Encoded String: "A+%2B+B"
Expected Decoding: "A + B"
unescape("A+%2B+B") = "A+++B" Wrong!
decodeURI("A+%2B+B") = "A+++B" Wrong!
decodeURIComponent("A+%2B+B") = "A+++B" Wrong!

encodeURIComponent doesn't encode -_.!~*'(), causing problem in posting data to php in xml string.
For example:
<xml><text x="100" y="150" value="It's a value with single quote" />
</xml>
General escape with encodeURI
%3Cxml%3E%3Ctext%20x=%22100%22%20y=%22150%22%20value=%22It's%20a%20value%20with%20single%20quote%22%20/%3E%20%3C/xml%3E
You can see, single quote is not encoded.
To resolve issue I created two functions to solve issue in my project, for Encoding URL:
function encodeData(s:String):String{
return encodeURIComponent(s).replace(/\-/g, "%2D").replace(/\_/g, "%5F").replace(/\./g, "%2E").replace(/\!/g, "%21").replace(/\~/g, "%7E").replace(/\*/g, "%2A").replace(/\'/g, "%27").replace(/\(/g, "%28").replace(/\)/g, "%29");
}
For Decoding URL:
function decodeData(s:String):String{
try{
return decodeURIComponent(s.replace(/\%2D/g, "-").replace(/\%5F/g, "_").replace(/\%2E/g, ".").replace(/\%21/g, "!").replace(/\%7E/g, "~").replace(/\%2A/g, "*").replace(/\%27/g, "'").replace(/\%28/g, "(").replace(/\%29/g, ")"));
}catch (e:Error) {
}
return "";
}

encodeURI() - the escape() function is for javascript escaping, not HTTP.

Small comparison table Java vs. JavaScript vs. PHP.
1. Java URLEncoder.encode (using UTF8 charset)
2. JavaScript encodeURIComponent
3. JavaScript escape
4. PHP urlencode
5. PHP rawurlencode
char JAVA JavaScript --PHP---
[ ] + %20 %20 + %20
[!] %21 ! %21 %21 %21
[*] * * * %2A %2A
['] %27 ' %27 %27 %27
[(] %28 ( %28 %28 %28
[)] %29 ) %29 %29 %29
[;] %3B %3B %3B %3B %3B
[:] %3A %3A %3A %3A %3A
[#] %40 %40 # %40 %40
[&] %26 %26 %26 %26 %26
[=] %3D %3D %3D %3D %3D
[+] %2B %2B + %2B %2B
[$] %24 %24 %24 %24 %24
[,] %2C %2C %2C %2C %2C
[/] %2F %2F / %2F %2F
[?] %3F %3F %3F %3F %3F
[#] %23 %23 %23 %23 %23
[[] %5B %5B %5B %5B %5B
[]] %5D %5D %5D %5D %5D
----------------------------------------
[~] %7E ~ %7E %7E ~
[-] - - - - -
[_] _ _ _ _ _
[%] %25 %25 %25 %25 %25
[\] %5C %5C %5C %5C %5C
----------------------------------------
char -JAVA- --JavaScript-- -----PHP------
[ä] %C3%A4 %C3%A4 %E4 %C3%A4 %C3%A4
[ф] %D1%84 %D1%84 %u0444 %D1%84 %D1%84

I recommend not to use one of those methods as is. Write your own function which does the right thing.
MDN has given a good example on url encoding shown below.
var fileName = 'my file(2).txt';
var header = "Content-Disposition: attachment; filename*=UTF-8''" + encodeRFC5987ValueChars(fileName);
console.log(header);
// logs "Content-Disposition: attachment; filename*=UTF-8''my%20file%282%29.txt"
function encodeRFC5987ValueChars (str) {
return encodeURIComponent(str).
// Note that although RFC3986 reserves "!", RFC5987 does not,
// so we do not need to escape it
replace(/['()]/g, escape). // i.e., %27 %28 %29
replace(/\*/g, '%2A').
// The following are not required for percent-encoding per RFC5987,
// so we can allow for a little better readability over the wire: |`^
replace(/%(?:7C|60|5E)/g, unescape);
}
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent

For the purpose of encoding javascript has given three inbuilt functions -
escape() - does not encode #*/+
This method is deprecated after the ECMA 3 so it should be avoided.
encodeURI() - does not encode ~!##$&*()=:/,;?+'
It assumes that the URI is a complete URI, so does not encode reserved characters that have special meaning in the URI.
This method is used when the intent is to convert the complete URL instead of some special segment of URL.
Example - encodeURI('http://stackoverflow.com');
will give - http://stackoverflow.com
encodeURIComponent() - does not encode - _ . ! ~ * ' ( )
This function encodes a Uniform Resource Identifier (URI) component by replacing each instance of certain characters by one, two, three, or four escape sequences representing the UTF-8 encoding of the character. This method should be used to convert a component of URL. For instance some user input needs to be appended
Example - encodeURIComponent('http://stackoverflow.com');
will give - http%3A%2F%2Fstackoverflow.com
All this encoding is performed in UTF 8 i.e the characters will be converted in UTF-8 format.
encodeURIComponent differ from encodeURI in that it encode reserved characters and Number sign # of encodeURI

Also remember that they all encode different sets of characters, and select the one you need appropriately. encodeURI() encodes fewer characters than encodeURIComponent(), which encodes fewer (and also different, to dannyp's point) characters than escape().

Just try encodeURI() and encodeURIComponent() yourself...
console.log(encodeURIComponent('##$%^&*'));
Input: ##$%^&*. Output: %40%23%24%25%5E%26*. So, wait, what happened to *? Why wasn't this converted? It could definitely cause problems if you tried to do linux command "$string". TLDR: You actually want fixedEncodeURIComponent() and fixedEncodeURI(). Long-story...
When to use encodeURI()? Never. encodeURI() fails to adhere to RFC3986 with regard to bracket-encoding. Use fixedEncodeURI(), as defined and further explained at the MDN encodeURI() Documentation...
function fixedEncodeURI(str) {
return encodeURI(str).replace(/%5B/g, '[').replace(/%5D/g, ']');
}
When to use encodeURIComponent()? Never. encodeURIComponent() fails to adhere to RFC3986 with regard to encoding: !'()*. Use fixedEncodeURIComponent(), as defined and further explained at the MDN encodeURIComponent() Documentation...
function fixedEncodeURIComponent(str) {
return encodeURIComponent(str).replace(/[!'()*]/g, function(c) {
return '%' + c.charCodeAt(0).toString(16);
});
}
Then you can use fixedEncodeURI() to encode a single URL piece, whereas fixedEncodeURIComponent() will encode URL pieces and connectors; or, simply, fixedEncodeURI() will not encode +#?=:#;,$& (as & and + are common URL operators), but fixedEncodeURIComponent() will.

I've found that experimenting with the various methods is a good sanity check even after having a good handle of what their various uses and capabilities are.
Towards that end I have found this website extremely useful to confirm my suspicions that I am doing something appropriately. It has also proven useful for decoding an encodeURIComponent'ed string which can be rather challenging to interpret. A great bookmark to have:
http://www.the-art-of-web.com/javascript/escape/

The accepted answer is good.
To extend on the last part:
Note that encodeURIComponent does not escape the ' character. A common
bug is to use it to create html attributes such as href='MyUrl', which
could suffer an injection bug. If you are constructing html from
strings, either use " instead of ' for attribute quotes, or add an
extra layer of encoding (' can be encoded as %27).
If you want to be on the safe side, percent encoding unreserved characters should be encoded as well.
You can use this method to escape them (source Mozilla)
function fixedEncodeURIComponent(str) {
return encodeURIComponent(str).replace(/[!'()*]/g, function(c) {
return '%' + c.charCodeAt(0).toString(16);
});
}
// fixedEncodeURIComponent("'") --> "%27"

Inspired by Johann's table, I've decided to extend the table. I wanted to see which ASCII characters get encoded.
var ascii = " !\"#$%&'()*+,-./0123456789:;<=>?#ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~";
var encoded = [];
ascii.split("").forEach(function (char) {
var obj = { char };
if (char != encodeURI(char))
obj.encodeURI = encodeURI(char);
if (char != encodeURIComponent(char))
obj.encodeURIComponent = encodeURIComponent(char);
if (obj.encodeURI || obj.encodeURIComponent)
encoded.push(obj);
});
console.table(encoded);
Table shows only the encoded characters. Empty cells mean that the original and the encoded characters are the same.
Just to be extra, I'm adding another table for urlencode() vs rawurlencode(). The only difference seems to be the encoding of space character.
<script>
<?php
$ascii = str_split(" !\"#$%&'()*+,-./0123456789:;<=>?#ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~", 1);
$encoded = [];
foreach ($ascii as $char) {
$obj = ["char" => $char];
if ($char != urlencode($char))
$obj["urlencode"] = urlencode($char);
if ($char != rawurlencode($char))
$obj["rawurlencode"] = rawurlencode($char);
if (isset($obj["rawurlencode"]) || isset($obj["rawurlencode"]))
$encoded[] = $obj;
}
echo "var encoded = " . json_encode($encoded) . ";";
?>
console.table(encoded);
</script>

Modern rewrite of #johann-echavarria's answer:
console.log(
Array(256)
.fill()
.map((ignore, i) => String.fromCharCode(i))
.filter(
(char) =>
encodeURI(char) !== encodeURIComponent(char)
? {
character: char,
encodeURI: encodeURI(char),
encodeURIComponent: encodeURIComponent(char)
}
: false
)
)
Or if you can use a table, replace console.log with console.table (for the prettier output).

I have this function...
var escapeURIparam = function(url) {
if (encodeURIComponent) url = encodeURIComponent(url);
else if (encodeURI) url = encodeURI(url);
else url = escape(url);
url = url.replace(/\+/g, '%2B'); // Force the replacement of "+"
return url;
};