We Keep Coding

Solidity and web3 sha3

I try to hash a tokenId with a seed in my smart contract. For simplicity and to avoid other errors I leave the seed out for now. I basically just want to hash a number on my contract and hash the same number on my javascript code and receive the same output.
Code looks something like this on Solidity:
function _tokenURI(uint256 tokenId) internal view returns (string memory) {
string memory currentBaseURI = _baseURI();
bytes32 hashedToken = keccak256(abi.encodePacked(tokenId));
return
bytes(currentBaseURI).length > 0
? string(abi.encodePacked(currentBaseURI, hashedToken, baseExtension))
: "";
}
which also leads to an error on client side invalid codepoint at offset. To tackle this I tried to cast bit32 to string using these functions
function _bytes32ToString(bytes32 _bytes32)
private
pure
returns (string memory)
{
uint8 i = 0;
bytes memory bytesArray = new bytes(64);
for (i = 0; i < bytesArray.length; i++) {
uint8 _f = uint8(_bytes32[i / 2] & 0x0f);
uint8 _l = uint8(_bytes32[i / 2] >> 4);
bytesArray[i] = _toByte(_f);
i = i + 1;
bytesArray[i] = _toByte(_l);
}
return string(bytesArray);
}
function _toByte(uint8 _uint8) private pure returns (bytes1) {
if (_uint8 < 10) {
return bytes1(_uint8 + 48);
} else {
return bytes1(_uint8 + 87);
}
}
though I'm not sure if this is equivalent. Code on the frontend looks like:
const hashed = web3.utils.soliditySha3(
{ type: "uint256", value: tokenId}
);
What do I need to change in order to receive the exact same output? And what does
invalid codepoint at offset
mean?

Maybe issue is that tokenId is not uint256 or Web3, Solidity version? I did few tests with Remix IDE and I recieved the same results.
Solidity code:
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
contract Hash {
function getHashValue_1() public view returns(bytes32){
return keccak256(abi.encodePacked(uint256(234)));
}
// bytes32: 0x61c831beab28d67d1bb40b5ae1a11e2757fa842f031a2d0bc94a7867bc5d26c2
function getHashValue_3() public view returns(bytes32){
return keccak256(abi.encodePacked(uint256(10),string('StringSecretValue')));
}
// bytes32: 0x5938b4caf29ac4903ee34628c3dc1eb5c670a6bd392a006d0cb91f1fc5db3819
}
JS code:
(async () => {
try {
console.log('Web3 version is '+ Web3.version);
// Web3 version is 1.3.0
let theValueYouNeed = web3.utils.soliditySha3("234");
theValueYouNeed = web3.utils.soliditySha3({type: 'uint256', value: '234'});
theValueYouNeed = web3.utils.soliditySha3({t: 'uint256', v: '234'});
// above hashed value is 0x61c831beab28d67d1bb40b5ae1a11e2757fa842f031a2d0bc94a7867bc5d26c2
console.log('Hashed value 1 is '+theValueYouNeed);
theValueYouNeed = web3.utils.soliditySha3({t: 'uint256', v: '10'},{t: 'string', v: 'StringSecretValue'});
console.log('Hashed value 2 is '+theValueYouNeed);
// above hashed value is 0x5938b4caf29ac4903ee34628c3dc1eb5c670a6bd392a006d0cb91f1fc5db3819
} catch (e) {
console.log(e.message)
}
})()
I'm not sure but invalid codepoint at offset should mean that a designated value does not fall within the range or set of allowed values... So maybe there is something wrong with tokenId and you could do some tests with hardcoded values?

You get the invalid codepoint error because you mix string and byte data when you call abi.encodePacked(currentBaseURI, hashedToken, baseExtension)).
When Javascript gets the return value from the contract it expects a UTF8 string, but inside your hashedToken you have byte values that are not valid for a UTF8-encoded string.
This kind of error might be "intermittent". It might happen in just some cases. You're lucky to see it during development and not in production.
How to fix it?
You are on the right track converting the hash result to a string.
There is an alternative way to do it in this answer which uses less gas by using only bitwise operations.
To convert the hex value in Javascript you can use web3.utils.hexToNumberString().

Length of generated RSA key is different on Puttygen and NodeJS

I have been trying to generate a RSA key pair to on an AWS Lambda in order to replace the current manual generation of keys on PuTTYgen. The key generated on PuTTYgen is of RSA type with a length of 2048 which results in a sample key (without comment) as follows -
---- BEGIN SSH2 PUBLIC KEY ----
Comment: ""
AAAAB3NzaC1yc2EAAAABJQAAAQEAjloNCA4mycem+WTb49zUhYK7aRmg1uuorUvD
7GzE97C9EmmhUrVbp4d5dWF8zkT2sh5mRFrAnsSogxEtCzvh59mzbqUj+3Xw+xqJ
DMrHmnT8XKIGep++v3e+SV7RLio06ymp0H7zyHhbxLhZEnpGEKwkXmY53+RSUF7s
wfmvxS5mCo7677lbIZxGvvx65tT5as5m+ng7tKlqDAliuPl2vslyFhQw9B49cvOx
Z+UekK2iHD+DNCMQyxEelOru9YMwRozOwgtWPEyHcLinonAn2fUne28POsT3zXbv
rW10hkGH5JIHzGUoPxP6N7RRCnSN/NgS8rrHs51Skvhl0WzV6w==
---- END SSH2 PUBLIC KEY ----
Now I have been trying to replicate the same on a NodeJS lambda with the following code -
const generation = util.promisify(crypto.generateKeyPair);
const result = await generation('rsa',{
modulusLength: 2048,
publicKeyEncoding: {
type: 'spki',
format: 'pem'
},
privateKeyEncoding: {
type: 'pkcs8',
format: 'pem',
}
});
console.log(result['publicKey']);
console.log(result['privateKey']);
On execution of this lambda, the generated public key looks something like this -
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnF1VDHq0vu5iL0nkbum8
cVzdhxiqmR6XcZbcsilF+Se6tlS9VAbN8QTTLdqwhJ5Dw7DvBGUXpCqUIqyT5IU5
wjQGnWHAWhPmalAgYWDwwdiOxxgd6NnNRR2Q5P4PSruxvFG7BtiSKGXSZpMzTIyZ
sXajEY2vhkf77bMEgzJhpXGAvzZsGEDi9jni8FCabVH6jvXh/svpmoCxwhQY1HHh
9RksscuAfllMwOE4uiQvfq6CpPNJUwU4kWtiaAtgX26nnPvqaUX52xMuYBrWQI2m
vUiXuxynqnrVSAFt/QY/0lMKRgnzwkq6YTIf8PeMQQA6TVQbtGN+j0MFQJDxF2/l
dQIDAQAB
-----END PUBLIC KEY-----
As far as I understand about RSA keys, the first and the last line should not make any difference since they are basically comments. But I see that the PuTTYgen key consists of about 5 and 1/2 lines of content whereas the NodeJS key has 6+ lines of content. Why is there a difference between the two when both of them have a length of 2048?
Thank you.

Thanks for the comments, helped me understand the formats of the keys better. The key generated with the use of the NodeJS code was of PEM format which would need to be converted to OpenSSH format to match the output key from PuTTYgen.
The module sshpk helps in converting the PEM key to a OpenSSH format. The code to do that is as follows -
var sshpk = require('sshpk');
let pemPublicKey = sshpk.parseKey(publicKey, 'pem');
let openSSHPublicKey = pemPublicKey.toString('ssh');
console.log(openSSHPublicKey);
This would result in a key with the following format -
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCqWWfYMm/q1Ee4AzRJMc+BgsqzVIZgwTidGDb6E3V2EQ85+iX09pmDF8E23Fsmd9WoUMB2L/FXUIlWZmofPmoInKk740UcsQGT0MVYQfDAiVBqrbymIR/lMBCirFpFq7hIDzACgWYAmSEFtDBdP4LmFozx6Vi3+Ss5g8EjkXcDLkPiBSM8YjCQO6CraH1dMIC4/hywitL1G2rngFdwgFAuG1HEqgXZKRZAm2043OVY2SDVnMvHcYjarPk94BydZ7mzZRCdaetDY73+8opVL0uRol2GfYwAoediz65iIm183w4p4F3JH09W4xSoT8FeTRfIc2iAlde3wFA6aK/NV0G5 (unnamed)
The (unnamed) at the end can be removed by adding a name when you are using the parseKey() method.
Now, you could format this string as per your requirements as follows -
let openSSHPublicKeyMaterial = openSSHPublicKey.split(" ")[1];
let formattedSSH2KeyArray = []
formattedSSH2KeyArray.push('---- BEGIN SSH2 PUBLIC KEY ----');
formattedSSH2KeyArray.push('Comment: ""');
for (let index = 0; index <= openSSHPublicKeyMaterial.length; index += 64 ) { // 64 corresponds to the number of characters in a single line
let singleLine = openSSHPublicKeyMaterial.substring(index, index + 64);
formattedSSH2KeyArray.push(singleLine);
}
formattedSSH2KeyArray.push('---- END SSH2 PUBLIC KEY ----');
let formattedSSH2Key = formattedSSH2KeyArray.join('\n');
console.log(formattedSSH2Key);
The execution of the above section would give you the key in the required format -
---- BEGIN SSH2 PUBLIC KEY ----
Comment: ""
AAAAB3NzaC1yc2EAAAADAQABAAABAQCqWWfYMm/q1Ee4AzRJMc+BgsqzVIZgwTid
GDb6E3V2EQ85+iX09pmDF8E23Fsmd9WoUMB2L/FXUIlWZmofPmoInKk740UcsQGT
0MVYQfDAiVBqrbymIR/lMBCirFpFq7hIDzACgWYAmSEFtDBdP4LmFozx6Vi3+Ss5
g8EjkXcDLkPiBSM8YjCQO6CraH1dMIC4/hywitL1G2rngFdwgFAuG1HEqgXZKRZA
m2043OVY2SDVnMvHcYjarPk94BydZ7mzZRCdaetDY73+8opVL0uRol2GfYwAoedi
z65iIm183w4p4F3JH09W4xSoT8FeTRfIc2iAlde3wFA6aK/NV0G5
---- END SSH2 PUBLIC KEY ----

Size of SQS messages

I'm sending messages to SQS using the AWS-SDK for JavaScript. Each message needs to be 256kb in size tops.
Each message is a JSON object that gets decoded on another service.
Option 1: JSON Object as string: Count length and make sure it's less than 262144?
function* getStuff(rows, someConfig) {
let totesPayload = 0
let payload = []
for (const row of rows) {
const singleItemInPayload = rowToPayload(row, someConfig)
if (singleItemInPayload.length + totesPayload < 262144 - (enclosingObjectSize())) {
payload.push(singleItemInPayload)
totesPayload += singleItemInPayload.length
} else {
yield({ payload })
payload = []
}
}
Option 2: Buffer.from(JSON Object as string): Count length of JSON Object and make sure it's less than 262144?
Most of the data is text, so I'm not sure I'm going to get any good help from putting it in a byte array.
Is option 2 necessary?

SQS uses UTF-8 for strings, so if any part of your message can contain non-ASCII characters, then you will need to measure the size by converting to bytes because UTF-8 is a variable width encoding which uses 1 to 4 bytes for a single character.

Trying to make hmac-sha256 work with Powershell for Canvas API

I was appointed the task of making a process in which a PowerShell script needs to make a call to Canvas servers in order to get data out of it for other uses that are outside the scope of this question.
The first thing I did was research how the Canvas API actually works. I eventually found this post holds everything I think I should know about the API. The API requires an HMAC SHA 256 hash.
I have decided to reverse engineer his the writer's code that makes the hash in order to make the same script in PowerShell.
Here is my slightly edited code (node.js)
var crypto = require('crypto')
var url = require('url')
var HMAC_ALG = 'sha256'
var apiAuth = module.exports = {
buildMessage: function(secret, timestamp, uri) {
var urlInfo = url.parse(uri, false);
var query = urlInfo.query ? urlInfo.query.split('&').sort().join('&') : '';
var parts = [ 'GET', urlInfo.host, '', '', urlInfo.pathname, query, timestamp, secret ]
console.log(parts);
return parts.join('\n');
},
buildHmacSig: function(secret, timestamp, reqOpts,message) {
//var message = apiAuth.buildMessage(secret, timestamp, reqOpts);
var hmac = crypto.createHmac(HMAC_ALG, new Buffer(secret))
hmac.update(message)
Console.log(message);
return hmac.digest('base64')
}
}
Here are the parameters that I put in the node js application
var canvas = require('[filepath]/new_canvas');
var secret = 'mycrazysecret';
var today = new Date();
var timestamp= today.toUTCString();
var regOpts = 'mycrazymessage';
var message = canvas.buildMessage(secret, timestamp, regOpts)
var hmac = canvas.buildHmacSig(secret, timestamp, regOpts,message);
the final code it this
'Oexq8/ulAGxSIQXGDVqoXyqk5x+n9cMrc3avcTW9aZk='
Here is my PowerShell file:
function buffer
{
param ($string)
$c=#()
Foreach ($element in $string.toCharArray()) {$c+= [System.Convert]::ToSByte($element)}
return $c
}
$message = 'GET\n\n\n\nmycrazymessage\n\nFri, 18 Nov 2016 15:29:52 GMT\nmycrazysecret'
$secret = 'mycrazysecret'
$hmacsha = New-Object System.Security.Cryptography.HMACSHA256
$hmacsha.key = buffer -string $secret
$signature = $hmacsha.ComputeHash([Text.Encoding]::UTF8.GetBytes($message))
$signature = [Convert]::ToBase64String($signature)
echo $signature
The final result is 'pF92zam81wclnnb8csDsscsSYNQ7it9qLrcJkRTi5rM='
I do not know getting the to produce the same results is even possible, but the question I am asking why aren't they producing to different results? (the keys are the same as well)

In PowerShell, the default escape sequence uses backticks ` rather than backslash \.
In order for the parser to recognize the escape sequence as not just a backtick character literal and the letter n, use an expandable string (" instead of '):
$message = "GET`n`n`n`nmycrazymessage`n`nFri, 18 Nov 2016 15:29:52 GMT`nmycrazysecret"
Other than that, your HMAC signature procedure is correct (it correctly outputs Oexq8/ulAGxSIQXGDVqoXyqk5x+n9cMrc3avcTW9aZk= after changing the $message value)

Can I 'match' values in 2 arrays and then use the subsequent 'matched' value?

I'm trying to achieve the following though with my intermediate JavaScript skills I'm not sure if this is possible.
This is related in part to this question.
Now I have 2 arrays
a) Has the various language in (e.g. "en-GB", "fr", "de" etc)
b) Has a suffix of a URL based on the browser language above (e.g. "fr/","de/","uk/")
What I am trying to achieve is:
1) User hits a page, browser detects which browser it is using from the array (a)
2) Depending on what the browser is based on (a), it then searches through (b) and if they match, e.g. if the language is "fr" it will use the suffix "fr/" from the array in (b).
3) It will then add this suffix to a top level domain (which is always constant)
Is this even possible to achieve (I'm sure it is)? Can it be done purely via JavaScript (or JQuery)? How would I go about doing this?
Here's some of the code I have so far:
var IAB_Array = new Array("de-at","nl-be","fr-be","da","de","hu","en-ie","ga","es","fr","it","nl","no","pl","en","en-GB","en-US","en-gb","en-us"); //language array
var IAB_suffix = new Array("at/","be-nl/","be-fr","den/","de/","hu/","ie/","es/","fr/","it/","nl/","nor/","pl/","uk/"); //URL suffix Array
var IAB_lang = "en-GB"; //default language
var IAB_link = "http://www.mysitegoeshere/";
if(navigator.browserLanguage) IAB_lang = navigator.browserLanguage; //change lang if detection supported
if(window.navigator.language) IAB_lang = window.navigator.language; //change lang if detection supported
function IAB_Lang_detect () { //execute search
for (var i=0;i<IAB_Array.length;i++) {
if(IAB_Array[i]==IAB_lang) {
document.write(IAB_Array[i]); //output matched array value
}
}
return false;
}
var IAB_URL = ""+IAB_link+IAB_suffix[1]+""; //this is the resulting URL
document.write(IAB_URL);
IAB_Lang_detect ();
I hope someone can help as I'm a little confused! It's more so the matching the values from the 2 arrays and then subsequently selecting the correct suffix that I'm having trouble with.
Thanks

(function () {
"use strict";
var lang_map = {
"de-at": "at/",
"nl-be": "be-nl/",
"fr-be": "be-fr",
"da": "den/",
"de": "de/",
"hu": "hu/",
"en-ie": "ie/",
"ga": "ie/",
"es": "es/",
"fr": "fr/",
"it": "it/",
"nl": "nl/",
"no": "nor/",
"pl": "pl/",
"en": "uk/",
"en-GB": "uk/",
"en-US": "uk/",
"en-gb": "uk/",
"en-us": "uk/"
},
lang = (navigator && navigator.browserLanguage) || (window.navigator && window.navigator.language) || "en-GB";
window.location = "http://www.mysitegoeshere/" + lang_map[lang];
}());

I'd do it differently and use an object:
var IAB_Object = { "it-It": "it/", "en-Gb": "en/" ....}
if(IAB_Object.hasOwnProperty(IAB_lang)){
//you have a match, the suffix is
var suffix = IAB_Object[IAB_lang];
}else{
//you don't have a match use a standard language
}

I probably wouldn't use arrays for this at all. You can use an object:
var IABInfo = {
"de-at": "at/",
"ln-be": "be-nl/",
// ...and so on
};
Then index directly into that object:
var value = IABInfo[IABLang]; // Where IABLang contains a string, like "de-at"
So:
var suffix = IABInfo[IABLang];
if (suffix) { // Did we have it?
document.write(suffix);
}
This works because all JavaScript objects are free-form key/value maps. Here's a simpler example:
var lifeTheUniverseAndEverything = {
answer: 42,
question: "?"
};
You can look up a property either using dotted notation with a literal, or by using square bracket ([]) notation with a string. So all four of these output exactly the same thing:
// 1. Dotted notation with a literal:
console.log("The answer is " + lifeTheUniverseAndEverything.answer);
// 2. Bracketed notation with a string
console.log("The answer is " + lifeTheUniverseAndEverything["answer"]);
// 3. The string needn't be a literal, it can come from a variable...
var name = "answer";
console.log("The answer is " + lifeTheUniverseAndEverything[name]);
// 4. ...or indeed any expression:
console.log("The answer is " + lifeTheUniverseAndEverything["a" + "n" + "swer"]);
So by making your IAB info a map in an object literal, you can make it much easier to look things up: Just use bracketed notation with the desired language code.