We Keep Coding

What is the right way to safely and accurately insert user-provided URL data into an HTML5 document?

Given an arbitrary customer input in a web form for a URL, I want to generate a new HTML document containing that URL within an href. My question is how am I supposed to protect that URL within my HTML.
What should be rendered into the HTML for the following URLs that are entered by an unknown end user:
http://example.com/?file=some_19%affordable.txt
http://example.com/url?source=web&last="f o o"&bar=<
https://www.google.com/url?source=web&sqi=2&url=https%3A%2F%2Ftwitter.com%2F%3Flang%3Den&last=%22foo%22
If we assume that the URLs are already uri-encoded, which I think is reasonable if they are copying it from a URL bar, then simply passing it to attr() produces a valid URL and document that passes the Nu HTML checker at validator.w3.org/nu.
To see it in action, we set up a JS fiddle at https://jsfiddle.net/kamelkev/w8ygpcsz/2/ where replacing the URLs in there with the examples above can show what is happening.
For future reference, this consists of an HTML snippet
<a>My Link</a>
and this JS:
$(document).ready(function() {
$('a').attr('href', 'http://example.com/request.html?data=>');
$('a').attr('href2', 'http://example.com/request.html?data=<');
alert($('a').get(0).outerHTML);
});
So with URL 1, it is not possible to tell if it is URI encoded or not by looking at it mechanically. You can surmise based on your human knowledge that it is not, and is referring to a file named some_19%affordable.txt. When run through the fiddle, it produces
My Link
Which passes the HTML5 validator no problem. It likely is not what the user intended though.
The second URL is clearly not URI encoded. The question becomes what is the right thing to put into the HTML to prevent HTML parsing problems.
Running it thru the fiddle, Safari 10 produces this:
My Link
and pretty much every other browser produces this:
My Link
Neither of these passes the validator. Three complaints are possible: the literal double quote (from un-escaping HTML), the spaces, or the trailing < character (also from un-escaping HTML). It just shows you the first of these it finds. This is clearly not valid HTML.
Two ways to try to fix this are a) html-escape the URL before giving it to attr(). This however results in every & becoming & and the entities such as & and < become double-escaped by attr(), and the URL in the document is entirely inaccurate. It looks like this:
My Link
The other is to URI-encode it before passing to attr(), which does result in a proper validating URL which actually clicks to the intended destination. It looks like this:
My Link
Finally, for the third URL, which is properly URI encoded, the proper HTML that validates does come out.
My Link
and it does what the user would expect to happen when clicked.
Based on this, the algorithm should be:
if url is encoded then
pass as-is to attr()
else
pass encodeURI(url) to attr()
however, the "is encoded" test seems to be impossible to detect in the affirmative based on these two prior discussions (indeed, see example URL 1):
How to find out if string has already been URL encoded?
How to know if a URL is decoded/encoded?
If we bypass the attr() method and forcibly insert the HTML-escaped version of example URL 2 into the document structure, it would look like this:
My Link
Which seemingly looks like valid HTML, yet fails the HTML5 validator because it unescapes to have invalid URL characters. The browsers, however, don't seem to mind it. Unfortunately, if you do any other manipulation of the object, the browser will re-escape all the &'s anyway.
As you can see, this is all very confusing. This is the first time we're using the browser itself to generate the HTML, and we are not sure if we are getting it right. Previously, we did it server side using templates, and only did the HTML-escape filter.
What is the right way to safely and accurately insert user-provided
URL data into an HTML5 document (using JavaScript)?

If you can assume the URL is either encoded or not encoded, you may be able to get away with something along the lines of this. Try to decode the URL, treat an error as the URL not being encoded and you should be left with a decoded URL.
<script>
var inputurl = 'http://example.com/?file=some_19%affordable.txt';
var myurl;
try {
myurl = decodeURI(inputurl);
}
catch(error) {
myurl = inputurl;
}
console.log(myurl);
</script>

Javascript regex to replace ampersand in all links href on a page

I've been going through and trying to find an answer to this question that fits my need but either I'm too noob to make other use cases work, or their not specific enough for my case.
Basically I want to use javascript/jQuery to replace any and all ampersands (&) on a web page that may occur in a links href with just the word "and". I've tried a couple different versions of this with no luck
var link = $("a").attr('href');
link.replace(/&/g, "and");
Thank you

Your current code replaces the text of the element within the jQuery object, but does not update the element(s) in the DOM.
You can instead achieve what you need by providing a function to attr() which will be executed against all elements in the matched set. Try this:
$("a").attr('href', function(i, value) {
return value.replace(/&/g, "and");
});
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
link
link

Sometimes when replacing &, I've found that even though I replaced &, I still have amp;. There is a fix to this:
var newUrl = "#Model.UrlToRedirect".replace(/&/gi, '%').replace(/%amp;/gi, '&');
With this solution you replace & twice and it will work. In my particular problem in an MVC app, window.location.href = #Model.UrlToRedirect, the url was already partially encoded and had a query string. I tried encoding/decoding, using Uri as the C# class, escape(), everything before coming up with this solution. The problem with using my above logic is other things could blow up the query string later. One solution is to put a hidden field or input on the form like this:
<input type="hidden" value="#Model.UrlToRedirect" id="url-redirect" />
then in your javascript:
window.location.href = document.getElementById("url-redirect").value;
in this way, javascript won't take the c# string and change it.

Passing variables to Javascript from "pretty" URL

My problem is, I would like to create "pretty" URLs for visitors that look like this:
http://domain.com/Name
I have users that often send friends to my service, and I have been created customized pages for each one with the person's First Name in the headline. E.g., "John, here's an easy way to fix this widget"
I then save the page as an index.html file in a custom folder so the link structure for the custom page is domain/Name with Name being their First Name.
This is getting tedious and I would love to use Javascript to automate the process. However, the only documentation I can find on passing variables to Javascript involves "ugly" domains such as domain/jspass2.html?FirstName=John&LastName=Smith
Is there a way to beautify these domains and still pass the variables to a javascript code that inputs their name into the html code? I don't want to "cloak" an ugly domain (using a href, for example)
Thanks for the help!

Well, you could make it "prettier" by making the querystring cleaner.
example:
http://www.domain.com/?John,Smith
The javascript in your index file can read that.
var getQueryString = function() {
queryString = window.location.search;
queryStringCleaned = queryString.substring(queryString.indexOf('?') + 1 );
return queryStringCleaned;
};

if "http://domain.com/Name" is your domain, variable firstName will have the value "Name".
var firstName = window.location.pathname.match(/([\w-]+)\/?.*/)[1];

You could just take the whole URL in JS, and parse it "by hand". Use this regex (for example) to find the parameters passed.

In addition to Paul, I wrote you something that extracts the first name field from the url you provided. If the format is consistent, and you can obtain the url in javascript, you can use this. You may possibly have to create the page first, then redirect the user because javascript is a client side language and the page will already be rendered.
var url = "domain/jspass2.html?FirstName=John&LastName=Smith";
url = url.slice(url.indexOf("FirstName=") + 10, url.length);
url = url.slice(0, url.indexOf("&"));

Extract data from url with JavaScript

EDIT_2: I forgot to specify its for Android app, so i dont think this is any use, i made a new post instead :( Added Android TAG..
EDIT: Im making an Android App
I need help to extract a number from an url, generated by JavaScript!
Site is:
http://www.oddsportal.com/sure-bets/
And the path looks like this:
<span class="logos l60"> </span>
<div class="odds-nowrp" xodd="xzoxfxzox">2.62</div> // <- 2.62 is the numer i need
For full path see this screenshot:
What library would do this best? (I know Jsoup cant do it) I have searched a few like:
HtmlUnit
Java Script Engine
Apache Commons BSF
Rhino
But i cant really make sense of it or find any examples for android which look like my problem

or find any examples for android which look like my problem
You need it for android?
Pretty much any library allowing DOM traversing will allow you to do this providing you know how to find your value.
is this value exactly at the same position in DOM every time?
is it wrapped by an easy to identify element? i.e. with a static ID
are there any other value that look alike in the DOM that you don't want?
Based on that, using JQuery for example, you could select it like this :
$('.table-main td.center > a[href^="/bookmaker"] + div[xodd]')
or this:
$('.table-main tr:nth-child(3) div.odds-nowrp[xodd]')

Use Jquery:
var number = $(".odds-nowrp").text();

you can just use regex if you have the url already in escaped string format
reg = /[A-z\"\>\<=?()0-9 \/]*(\d+.\d+)[A-z\"\>\<=?()0-9 \/]*/
reg.exec(url)[1] // this will return your number
if it's already rendered and the xodd value doesn't change, you could do something like this
document.querySelectorAll('.odds-nowrp[xodd=xzoxfxzox]')[0].innerText

Making a URL W3C valid AND work in Ajax Request

I have a generic function that returns URLs. (It's a plugin function that returns URLs to resources [images, stylesheets] within a plugin).
I use GET parameters in those URLs.
If I want to use these URLs within a HTML page, to pass W3C validation, I need to mask ampersands as &
/plugin.php?plugin=xyz&resource=stylesheet&....
but, if I want to use the URL as the "url" parameter for a AJAX call, the ampersand is not interpreted correctly, screwing up my calls.
Can I do something get & work in AJAX calls?
I would very much like to avoid adding parameters to th URL generating function (intendedUse="ajax" or whatever) or manipulating the URL in Javascript, as this plugin model will be re-used many times (and possibly by many people) and I want it as simple as possible.

It seems to me that you're running into the problem of having one piece of your application cross multiple layers. In this case it's the plugin.
A URL as specified by RFC 1738 states that a URL should use a & token to separate key/value pairs from one another. However ampersand is a reserved token in HTML and therefore should be escaped into &. Since escaping the ampersands is an artifact of HTML, your plugin should probably not be escaping them directly. Instead you should have a function or something that escapes a canonical URL so that it can be embedded in HTML markup.

The only place that this is likely to actually happen is if you are:
Using XHTML
Serving it as text/html
Using inline <script>
This is not a happy combination, and the solution is in the spec.
Use external scripts if your script
uses < or & or ]]> or --.
The XHTML media types note includes the same advice, but also provides a workaround if you choose to ignore it.

Try returning JSON instead of just a string, that way your Javascript can read the URL value as an object, and you shouldn't have that issue. Other than that, try simply HTML decoding the string, using something like:
function escapeHTML (str)
{
var div = document.createElement('div');
var text = document.createTextNode(str);
div.appendChild(text);
return div.innerHTML;
};
Obviously you'll want to make sure you remove any reference to DOM elements you might create (which I've not done here to simplify the example).
I use this technique in the AJAX sites I create at my work and have used it many times to solve this problem.

When you have markup of the form:
<a href="?a=1&b=2">
Then the value of the href attribute is ?a=1&b=2. The & is only an escape sequence in HTML/XML and doesn't affect the value of the attribute. This is similar to:
<a href="<>">
Where the value of the attribute is <>.
If, instead, you have code of the form:
<script>
var s = "?a=1&b=2";
</script>
Then you can use a JavaScript function:
<script>
var amp = String.fromCharCode(38);
var s = "?a=1"+amp+"b=2";
</script>
This allows code that would otherwise only be valid HTML or only valid XHTML to be valid in both. (See Dorwald's comments for more info.)