We Keep Coding

Parse UTF-8 XML in javascript

I'm trying to load and parse a simple utf-8-encoded XML file in javascript using node and the xpath and xmldom packages. There are no XML namespaces used and the same XML parsed when converted to ASCII. I can see in the debugger in VS Code that the string has embedded spaces in between each character (surely due to loading the utf-8 file incorrectly) but I can't find a way to properly load and parse the utf-8 file.
Code:
var xpath = require('xpath')
, dom = require('xmldom').DOMParser;
const fs = require('fs');
var myXml = "path_to_my_file.xml";
var xmlContents = fs.readFileSync(myXml, 'utf8').toString();
// this line causes errors parsing every single tag as the tag names have spaces in them from improper utf-8 decoding
var doc = new dom().parseFromString(xmlContents, 'application/xml');
var cvNode = xpath.select1("//MyTag", doc);
console.log(cvNode.textContent);
The code works fine if the file is ASCII (textContent has the proper data), but if it is UTF-8 then there are a number of parsing errors and cvNode is undefined.
Is there a proper way to parse UTF-8 XML in node/javascript? I can't for the life of me find a decent example.

When you see additional white spaces between each letter, this suggests that the file isn't actually encoded using utf-8 but uses a 16 bit unicode encoding.
Try 'utf16le'.
For a list of supported encodings see Buffers and Character Encodings.

C# version of the javascript function btoa

I need to recode something from js to c# that utilises the btoa method in js on a string of unicode chars to convert them to base64. However, as far as I know the encoding used by javascrpt is different to all those available in c#. I need the encoding to be exactly the same and not return different values across these languages. I have tried setting up a nodejs server and making get requests to it, in order to run the encoding that way, but this is far too slow and unstable. I am under the impression I would need to make my own encoding table but I have no idea where to start or how to implement this. If anyone could help that would be greatly appreciated.
tl;dr: javascript's btoa returns different value than base 64 encoding in c# does. I need it to be the same values.
code and outputs:
c#:
String fromChar = new
String(247,71,186,8,125,72,2,1.0078125,0.003936767578125);
var plainTextBytes = System.Text.Encoding.UTF8.GetBytes(fromChar);
Console.WriteLine(System.Convert.ToBase64String(plainTextBytes));
output = w7dHwroIfUgCAQA=
javascript:
var x = btoa(String.fromCharCode(247,71,186,8,125,72,2,1.0078125,0.003936767578125);
console.log(x)
output =
90e6CH1IAgEA
I am aware the former example uses utf8 encoding which js does not, the problem is there is no encoding in .net that matches the javascript encoding.
Edit: Tried to compare the byte arrays of both c# and javascript but the problem is the btoa function uses an unnamed encoding, so I can't actually get the bytes to print the byte array for it without assuming it is something like utf8 which it is not.

Worked it out. For anyone wondering the encoding used is iso encoding. The btoa function in javascript can be replicated by using the following c# method:
public string encoding(string toEncode)
{
byte[] bytes= Encoding.GetEncoding(28591).GetBytes(toEncode);
string toReturn = System.Convert.ToBase64String(bytes);
return toReturn;
}

The decoding will be the following:
string base64EncodedString = "6Q==";
Encoding.GetEncoding(28591).GetString(Convert.FromBase64String(base64EncodedString));
// "é"

Getting MD5 hash in JavaScript and converting it to base 64

I’d like to use a set of REST API through JavaScript and I’m reading the documentation explaining how to implement authentication.
The following instructions are illustrated in pseudocode but I have some issue on understanding how to implement it in JavaScript (my JS level is quite basic).
This is the unclear part:
= FromBytesToBase64String(MD5Hash("{\n \"data\": {\n \"type\": \"company\",\n \"id\": \"879f2dfc-57ea-4dbb-96c7-c546f8812f1e\",\n \"external_1_value\": \"Updated value\"\n }\n}"))
Basically I should calculate MD5 hash of the string in question and then I should encode it in base 64 string If I understood well.
The documentation shows the flow broken in sub-steps:
= FromBytesToBase64String(b'eC\xcda\xa3\xa7\xaf\xa53\x93\xb4.\xa2\xb1\xe3]')
And then the final result:
"ZUPNYaOnr6Uzk7QuorHjXQ=="
I tried to do the same by using crypto.js library and I get a MD5 hash string but then how can I get this value "ZUPNYaOnr6Uzk7QuorHjXQ==" ?
Any idea on how I could do it?
Thanks so much for helping!

That final result is the base64 encoded string. The function FromBytesToBase64String is what produces it, but this is not a standard function in JavaScript.
Instead, try using one of the built-in functions documented here. Specifically:
window.btoa(MD5Hash("Your input string"));

window.btoa(MD5Hash("Your input string")); does not work because btoa takes the md5 string and converts that character by character, hence you need to feed it an byte array.I ended up combining ArrayBuffer to base64 encoded string
with https://github.com/pvorb/node-md5/issues/25
into :
function md5ToBase64(md5String,boolTrimLast){
var strRet = arrayBufferToBase64(hexByteStringToByteArray(md5String));
return boolTrimLast?strRet.slice(0,22):strRet;
}

use the btoa() function to get a base64 encoded string.

Use WindowBase64.btoa():
var encodedData = window.btoa(md5Hash);

Produce same result as CryptoJS.enc.Base64 using PHP

I have a javascript function that I'm trying to convert to PHP, It uses CryptoJS library, speciafically components/enc-base64-min.js and rollups/md5.js. They can be found here.
In it is this bit of code
// Let's say str = 'hello';
var md5 = CryptoJS.MD5(str);
md5 = md5.toString(CryptoJS.enc.Base64);
// md5 outputs "XUFAKrxLKna5cZ2REBfFkg=="
I assumed the str variable is hashed using md5 then encoded to Base64, so I tried this simple code
$md5 = md5($str);
$md5 = base64_encode($md5);
// md5 outputs "MmZjMGE0MzNiMjg4MDNlNWI5NzkwNzgyZTRkNzdmMjI="
Then I tried validating both the outputs, looks like the JS output isnt even a valid Base64 string.
To understand further I tried to look for toString() parameter from W3Schools, but it doesnt make sense to me, as per the reference the parameter is supposed to be an integer (2, 8 or 16), then why is CryptoJS.enc.Base64 used instead?
My goal here isn't to produce a valid base64 encoded string using JS but rather to produce the same output using PHP.

php's md5() with a single parameter returns the md5 hash as a hex string.
Instead you want the raw bytes to be encoded into Base64 so you have to pass the optional parameter $raw_output too to md5() (set to true)
$md5 = md5($str, true);
http://php.net/manual/it/function.md5.php

correct way to exclude certain characters from crypto.randomBytes

i have the following code, based on http://nodejs.org/docs/v0.6.9/api/crypto.html#randomBytes
crypto.randomBytes 32, (ex, buf) ->
user.tokenString = buf.toString("hex")
user.tokenExpires = Date.now() + TOKEN_TIME
next()
i am using this to generate a tokenString to use for a node.js/express user validation.
in some cases the tokenString generated includes '/' forward slash character, and this breaks my routes, for example, tokenString if the tokenString is like '$2a$10$OYJn2r/Ts.guyWqx7iJTwO8cij80m.uIQV9nJgTt18nqu8lT8OqPe' it can't find /user/activate/$2a$10$OYJn2r and i get an 404 error
is there a more direct way to exclude certain characters from being included when generating the crypto.randomBytes?

Crypto.randomBytes generates random bytes . That has nothing to do with characters, characters are determined by the way we look at the bytes.
For example:
user.tokenString = buf.toString("hex")
Would convert the buffer to a string (where two characters represent each byte), in the character range 0-9a-f
Another (might be more suiting approach is to use a more compact encoding. Base64Url is an encoding that provides string encoding that is URL/Filename safe
user.tokenString = base64url(buf)
Here is an NPM package you can use for it.
Other than that, your code seems fine. If you were to call .toString() without specifying "hex" or specifying something like "ascii" for example, it would break just like in your question description.