We Keep Coding

utf8 encoding issue with discord node.js and mysql for 4-byte characters

I have a case where I am trying to get the name from a discord channel. I use:
message.channel.name
and this works just fine, except for 4-byte encoded names, typically things like some Asian language characters, and some emojis. as an example:
🏀sports (fails)
¡§™sports (works)
Here is an example from my Windows PowerShell(v7) output of my sql insert error:
INSERT INTO serverdata (channel_name, isChannelActive) VALUES ('🏀sports',1) ON DUPLICATE KEY UPDATE channel_name='🏀sports', isChannelActive = 1
throw err; // Rethrow non-MySQL errors
^
Error: ER_TRUNCATED_WRONG_VALUE_FOR_FIELD: Incorrect string value: '\xF0\x9F\x8F\x80sp...' for column 'channel_name' at row 1
Powershell puts up the "?" for the basketball, but I think this is just a console issue and not related to my execution flow.
I have chased through much of the internet searches...all my files are encoded for UTF8(mb4 where possible to specify), I use VS Code to edit.
I believe I have no place where UTF8MB3 is being used. I also can take the exact SQL string, put it in the MYSQL workbench, and execute a script directly, and the syntax works fine. This leads me to believe it is some kind of interaction between discord and node.js?
Any advice? TIA!
Code source in JS that shows the issue
var tempBuf = message.channel.name;
// var tempBuf = Buffer.from(message.channel.name);
// var tempBufLen = tempBuf.write(message.channel.name);
// var tempBufLen = tempBuf.length + 1;
// tempBuf = tempBuf.toString('utf8',0,tempBufLen)
// tempBuf = tempBuf.toLocaleString();
// console.log("buffer : " + `${tempBuf}`);
// console.log("buffer length : " + tempBufLen);
// console.log("buffer (len) character : " + tempBuf.toString('utf8', 0, tempBufLen));
// insert into the server database
let sql = (`INSERT INTO serverdata (channel_name, isChannelActive) VALUES ('${tempBuf}',${onOrOff}) ON DUPLICATE KEY UPDATE channel_name='${ tempBuf}', isChannelActive = ${onOrOff}`);
console.log(sql)
// execute the insert statment
connection.query(sql, function (err) { if (err) throw err });

If you require emoji support, ensure each MySQL table that includes them is configured with CHARSET=utf8mb4. You will also need to ensure your connection config in the node app includes the related option with matching value.
const mysql = require('mysql');
const connection = mysql.createPool({
connectionLimit: 10,
host: process.env.DB_HOST || '127.0.0.1',
user: process.env.DB_USER || 'local_user',
password: process.env.DB_PASSWORD || 'local_password',
database: process.env.DB_NAME || 'local_database',
charset: 'utf8mb4' // necessary for emoji character support
});

Working with memory to fetch string yields incorrect result

I am following the solutions from here:
How can I return a JavaScript string from a WebAssembly function
and here:
How to return a string (or similar) from Rust in WebAssembly?
However, when reading from memory I am not getting the desired results.
AssemblyScript file, helloWorldModule.ts:
export function getMessageLocation(): string {
return "Hello World";
}
index.html:
<script>
fetch("helloWorldModule.wasm").then(response =>
response.arrayBuffer()
).then(bytes =>
WebAssembly.instantiate(bytes, {imports: {}})
).then(results => {
var linearMemory = results.instance.exports.memory;
var offset = results.instance.exports.getMessageLocation();
var stringBuffer = new Uint8Array(linearMemory.buffer, offset, 11);
let str = '';
for (let i=0; i<stringBuffer.length; i++) {
str += String.fromCharCode(stringBuffer[i]);
}
debugger;
});
</script>
This returns an offset of 32. And finally yields a string that starts too early and has spaces between each letter of "Hello World":
However, if I change the array to an Int16Array, and add 8 to the offset (which was 32), to make an offset of 40. Like so:
<script>
fetch("helloWorldModule.wasm").then(response =>
response.arrayBuffer()
).then(bytes =>
WebAssembly.instantiate(bytes, {imports: {}})
).then(results => {
var linearMemory = results.instance.exports.memory;
var offset = results.instance.exports.getMessageLocation();
var stringBuffer = new Int16Array(linearMemory.buffer, offset+8, 11);
let str = '';
for (let i=0; i<stringBuffer.length; i++) {
str += String.fromCharCode(stringBuffer[i]);
}
debugger;
});
</script>
Then we get the correct result:
Why does the first set of code not work like its supposed to in the links I provided? Why do I need to change it to work with Int16Array to get rid of the space between "H" and "e" for example? Why do I need to add 8 bytes to the offset?
In summary, what on earth is going on here?
Edit: Another clue, is if I use a TextDecoder on the UInt8 array, decoding as UTF-16 looks more correct than decoding as UTF-8:

AssemblyScript uses utf-16: https://github.com/AssemblyScript/assemblyscript/issues/43
Additionally AssemblyScript stores the length of the string in the first 32 or 64 bits.
That's why my code behaves differently. The examples in the links at the top of this post were for C++ and Rust, which do string encoding differently

Hmac-SHA256 not returning expected hash

I realize there are a ton of questions on here about this, but after looking through a good portion of them I haven't really seen anything addressing my issue.
Using SHA256 on the following input I get the correct output:
var canonString = 'GET\n'+
'/\n'+
'Action=ListUsers&Version=2010-05-08\n'+
'content-type:application/x-www-form-urlencoded; charset=utf-8\n'+
'host:iam.amazonaws.com\n'+
'x-amz-date:20150830T123600Z\n'+
'\n'+
'content-type;host;x-amz-date\n'+
'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855';
console.log(CryptoJS.SHA256(canonString).toString()); //returns the expected value of f536975d06c0309214f805bb90ccff089219ecd68b2577efef23edd43b7e1a59
So SHA256 is working properly on that. Similarly, using the Hmac-SHA256 on the following input I get the correct response:
var kDate = CryptoJS.HmacSHA256("20150830", "AWS4wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY");
var kRegion = CryptoJS.HmacSHA256('us-east-1', kDate);
var kService = CryptoJS.HmacSHA256('iam', kRegion);
var kSigning = CryptoJS.HmacSHA256("aws4_request", kService);
console.log(kSigning.toString()); //returns the expected value of c4afb1cc5771d871763a393e44b703571b55cc28424d1a5e86da6ed3c154a4b9
So this Hmac-SHA256 function works correctly on this input. However, on the following input, Hmac-SHA256 DOES NOT return the expected output.
var stringToSign = 'AWS4-HMAC-SHA256\n'+
'20150830T123600Z\n'+
'20150830/us-east-1/iam/aws4_request\n'+
CryptoJS.SHA256(canonString).toString();
CryptoJS.HmacSHA256(kSigning.toString(), stringToSign); //Returns 8a96b6691875490d30d05731cc9aa26be1fd64cf611ed929753b6498075aa886
//Expected value is 5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7
//Trying in opposite order just in case
CryptoJS.HmacSHA256(stringToSign, kSigning.toString()); //Returns fe52b221b5173b501c9863cec59554224072ca34c1c827ec5fb8a257f97637b1
//Still not expected value which is 5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7
So, something is clearly going wrong with my stringToSign, and I don't know what it is. I was thinking that the newline character is being interpreted as two different characters and not just a single character. However, escaping it like '\\n' did not fix it either! I am at a loss here. Here are the two docs I've been following (doc1 doc2). Does anyone know why I can't get the expected output?

Remember that the sha256 digest is a byte sequence: it is not a "normal string". It looks like CryptoJS is converting the true sha256 digest to something else for convenience, so make it not do that and you're good to go.
Using Node's crypto library (which is a built-in API) rather than CryptoJS (which has absolutely terrible documentation, so using it is kind of questionable):
const crypto = require("crypto");
function HMAC(key, text) {
return crypto.createHmac("sha256", key).update(text).digest();
}
And then we form the canonical hash:
const canonString = [
'GET',
'/',
'Action=ListUsers&Version=2010-05-08',
'content-type:application/x-www-form-urlencoded; charset=utf-8',
'host:iam.amazonaws.com',
'x-amz-date:20150830T123600Z',
'',
'content-type;host;x-amz-date',
'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
].join('\n');
// note: plain hash, not a secret-key-seeded hash
const canonHash = crypto.createHash("sha256").update(canonString).digest();
console.log("Canonical hash is :", canonHash.toString('hex'));
This yields f536975d06c0309214f805bb90ccff089219ecd68b2577efef23edd43b7e1a59, but only because we logged it as hexadecimal string using .toString('hex'): the real value is still a byte sequence.
We then continue:
const kSecret = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
const kDate = HMAC("AWS4" + kSecret,"20150830");
const kRegion = HMAC(kDate,"us-east-1");
const kService = HMAC(kRegion,"iam");
const kSigning = HMAC(kService,"aws4_request");
console.log("kSigning hash is :", kSigning.toString('hex'));
Which yields c4afb1cc5771d871763a393e44b703571b55cc28424d1a5e86da6ed3c154a4b9: note that again this is only after toString('hex') for console logging purposes. The sha256 byte digest kSigning itself is not a hex string.
Then finally:
const stringToSign = [
'AWS4-HMAC-SHA256',
'20150830T123600Z',
'20150830/us-east-1/iam/aws4_request',
canonHash.toString('hex')
].join('\n');
const signed = HMAC(kSigning, stringToSign);
console.log("Final signed hash is:", signed.toString('hex'));
Which yields 5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7, and note that we had to turn the canonical hash into a hexadecimal string for signing purposes, not just logging, as per the instructions in the pages you link to. But, we still do not touch the kSigning digest, that stays a real sha256 byte sequence.

PHP Pack/Unpack implementation in Javascript Mismatch

As per this question's related answer, I'm attempting to put together a pack/unpack solution resembling this PHP process, however in Nodejs (Javascript) using md5 and bufferpack
Here's the PHP approach (adapted from DaloRADIUS:
$challenge = 'c731395aca5dcf45446c0ae83db5319e';
$uamsecret = 'secret';
$password = 'password';
$hexchal = pack ("H32", $challenge);
$newchal = pack ("H*", md5($hexchal . $uamsecret));
$response = md5("\0" . $password . $newchal);
$newpwd = pack("a32", $password);
$pappassword = implode ("", unpack("H32", ($newpwd ^ $newchal)));
echo "Response: ---> ", $response, "\n";
echo "New Password: ---> ", $newpwd, "\n";
echo "Pap Password: ---> ", $pappassword, "\n";
The above echos these:
Above in plaintext:
Response: ---> 2d4bd27184f5eb032641137f728c6043
New Password: ---> password
Pap Password: ---> 356a1fb08f909fc400dfe448fc483ce3
In Javascript, here's what I'm doing now:
var challenge = 'c731395aca5dcf45446c0ae83db5319e';
var uamsecret = 'secret';
var password = 'password';
var hexchal = pack.pack("H32", challenge);
var newchal = pack.pack("H*", md5(hexchal + uamsecret));
var response = md5("\0" + password + newchal);
var newpwd = pack.pack("a32", password);
var pappassword = pack.unpack("H32", (newpwd ^ newchal)).join("");
console.log("Response: --> ", response);
console.log("New Password: -->", newpwd);
console.log("Pap Password: --->", pappassword);
Which gives the result:
In JSON:
In plaintext:
Response: --> e8a54a55cbcd81dbc2bdfd9b197d62af
New Password: --> <Buffer >
Pap Password: ---> NaN
All the above snippets are available here: RadiusNES
My understanding in this whole process isn't the best, and will appreciate insights and where I'm going wrong.
Why is there a mismatch?

The translation does not work because the PHP Pack function uses different format strings and returns strings, whilst the Javascript bufferpack module returns arrays. Also you cannot xor strings in Javascript.
Whilst there may be modules to do what you want, I have my own functions for parsing hex strings. Also I like modifying prototypes which not everyone agrees with, but these could be converted to regular functions.
String.prototype.pad = function( length ,padding ) {
var padding = typeof padding === 'string' && padding.length > 0 ? padding[0] : '\x00'
,length = isNaN( length ) ? 0 : ~~length;
return this.length < length ? this + Array( length - this.length + 1 ).join( padding ) : this;
}
String.prototype.packHex = function() {
var source = this.length % 2 ? this + '0' : this
,result = '';
for( var i = 0; i < source.length; i = i + 2 ) {
result += String.fromCharCode( parseInt( source.substr( i , 2 ) ,16 ) );
}
return result;
}
var challenge = 'c731395aca5dcf45446c0ae83db5319e'
,uamsecret = 'secret'
,password = 'password';
var hexchal = challenge.packHex();
var newchal = md5( hexchal + uamsecret ).packHex();
var response = md5( '\0' + password + newchal );
var newpwd = password.pad( 32 );
var pappassword = '';
for( var i = 0; i < newchal.length; i++ ) {
pappassword += ( newpwd.charCodeAt( i ) ^ newchal.charCodeAt( i ) ).toString( 16 );
}
console.log("Response: --> ", response);
console.log("New Password: -->", newpwd);
console.log("Pap Password: --->", pappassword);
Two functions are defined in the String prototype to replace the use of the pack function:
.pad( 32, string ) is used to pad out a string with nulls to give the same results as pack( 'a32', string ). Although not needed here it also takes a second parameter if wanting to pad the string ith a character other than nulls.
.packHex is the equivalent of pack( 'H*' ,string ) and translating the code of each pair of hex characters into a character. The function ideally needs more validation to test the string is a valid hex one if is to be used.
After the inputs have been defined, the next four lines instead set variables using these functions rather than pack.
Because Javascript cannot natively xor strings, you then need to use a loop to extract each character, convert it to a numeric, xor those values, then convert the result back into a character to create the pappassword string.
That will return, for me:
Response: --> – "fbfd42ffde05fcf8dbdd02b7e8ae2d90"
New Password: --> – "password������������������������"
Pap Password: ---> – "dcbdacb03f5d38ca33c128b931c272a"
Which is a result, but unfortunately a different on from the PHP code.
This is because my installation of PHP is configured to use ISO-8859-1 encoding internally, whilst Javascript natively uses UTF-16.
This is not a problem in normal use, but it means the respective md5 functions will be seeing different values and therefore return a different hash.
Assuming you are writing an authentication routine using a PHP backend you will obviously need consistent results. There may be modules available to convert the encoding of the Javscript values for compatibility, but it is much easier to make changes to the PHP code.
Because we know the hex strings will be one byte, Javascript is effectively using UTF-8, so PHP can do the same by using the utf8_encode() function to convert the packed hex strings before md5ing them.
Originally I thought that Javascript was internally converting the encoded hex characters into their unicode equivalents because of this, but this was not the case. Instead it was the md5 module being used in Javascript that was performing a UTF-8 conversion on the input.
This leaves two possible options.
1. Use UTF-8 PHP
If possible you can reconfigure your PHP server to use UTF-8 encoding. Or you can change your script to use the utf8_encode() function to mirror the same process as is happening in the Javascript, and convert the hex packed strings to UTF-8 before passing them to md5()
$challenge = 'c731395aca5dcf45446c0ae83db5319e';
$uamsecret = 'secret';
$password = 'password';
$hexchal = pack ("H32", $challenge);
$newchal = pack ("H*", md5(utf8_encode($hexchal) . $uamsecret));
$response = md5("\0" . $password . utf8_encode($newchal));
$newpwd = pack("a32", $password);
$pappassword = implode ("", unpack("H32", ($newpwd ^ $newchal)));
echo "Response: ---> ", $response, "\n";
echo "New Password: ---> ", $newpwd, "\n";
echo "Pap Password: ---> ", $pappassword, "\n";
This then returns the same results as the Javscript:
Response: ---> fbfd42ffde05fcf8dbdd02b7e8ae2d90
New Password: ---> password
Pap Password: ---> dcbdacb03f5d38ca33c128b9310c272a
2. Change the md5 module in Javascript
I am assuming you are using the bluimp JavaScript-MD5 module, as this is what it used by DaloRADIUS routine you linked. You can patch this to bypass the UTF-8 conversion.
There are various ways you can patch this, but on line 259 is the definition of the md5() function itself. This is simply a set of if statements to parse the input options and call the appropriate internal function.
However, the functions called by this block simply provide the UTF-8 input conversion, through a function called str2rstrUTF8() before then call the appropriate functions to provide the actual hashing. You may therefore want to patch the md5() to accept a third parameter to indicate whether the UTF-8 conversion should be applied and then call other functions as appropriate.
However to simply remove the conversion completely the easier way is to change str2rstrUTF8() to return the input unchanged. This function can be found on line 239, changing it to just read as follows will stop the conversion:
function str2rstrUTF8 (input) {
return input
}
Alternatively to remove the redundant function call you can instead just remove the references to it. Change the function starting on line 246 to read as follows:
function rawMD5 (s) {
return rstrMD5(s)
}
The rawHMACMD5() function on line 252 also includes calls to the str2rstrUTF8() function which you may also want to patch for consistency but this is not required for the above routine. That function is called instead when a second parameter is passed to provide a key hash, a feature not available in the native PHP md5() function.
After making either of those changes the Javascript routine now returns the same output as your original (ISO-8859-1 using) PHP code:
Response: --> – "2d4bd27184f5eb032641137f728c6043"
New Password: --> – "password������������������������"
Pap Password: ---> – "356a1fb08f909fc40dfe448fc483ce3"

OAuth nonce value

I am working with the FatSecret REST API
Im using the OAuthSimple javascript library to generate the signed url.
Here's the code I have -
params['oauth_timestamp'] = Math.floor(new Date().getTime()/1000);
params['oauth_nonce'] = '1234';
params['oauth_version'] = '1.0';
var paramStr = '';
for(var key in params){
paramStr += key+"="+params[key]+"&";
}
paramStr = paramStr.substring(0,paramStr.length-1);
var oauth = OAuthSimple();
oauth.setAction('GET');
var o = oauth.sign(
{
path:this.requestURL,
parameters: paramStr,
signatures:{
api_key:this.apiKey,
shared_secret:this.sharedSecret,
access_token: this.accessToken,
access_secret: this.accessSecret
}
});
console.log(o.signed_url);
return o.signed_url;
params is an associative array containing all the non oauth related parameters for this call.
When I use this signed url I get an "invalid/used nonce"
The OAuth Testing Tool uses the same OAuthSimple library and if I put in all the same parameters (including the timestamp) it generates exactly the same url.
The only difference is that the url generated by the testing tool works and gives me the full response from the server. The url generated by my code does't.
I tried various nonce values including sending a MD5 of the timestamp but I get the same error. The reason I'm using 1234 right now is that the testing tool uses 1234 by default and that seems to work.
Any help is appreciated. Thanks in advance.

Updating #Saravanan's answer with something that works on current browsers:
function genNonce() {
const charset = '0123456789ABCDEFGHIJKLMNOPQRSTUVXYZabcdefghijklmnopqrstuvwxyz-._~'
const result = [];
window.crypto.getRandomValues(new Uint8Array(32)).forEach(c =>
result.push(charset[c % charset.length]));
return result.join('');
}
console.info(genNonce());

The nonce value as per twitter documentation:
The value for this request was generated by base64 encoding 32 bytes of random data, and stripping out all non-word characters, but any
approach which produces a relatively random alphanumeric string should
be OK here.
Based on the above notes, I use the following javascript code to generate nonce value each time I send a request:
var nonceLen = 32;
return crypto.randomBytes(Math.ceil(nonceLen * 3 / 4))
.toString('base64') // convert to base64 format
.slice(0, nonceLen) // return required number of characters
.replace(/\+/g, '0') // replace '+' with '0'
.replace(/\//g, '0'); // replace '/' with '0'
Try this if it works!

Try this
This works every time
var nonce = Math.random().toString(36).replace(/[^a-z]/, '').substr(2);