How to make SJCL and Python hashlib generate identical pdkdf2 output - javascript

I have a JavaScript app and a Python app that communicate using a key derived from a password using pbdkf2. The problem is, the generated keys don't match. I've produced a minimal test case for each.
Python
import hashlib, binascii
bytes = hashlib.pbkdf2_hmac('sha256', "password".encode(), b'', 100000)
print(binascii.hexlify(bytes).decode())
Generates: 64a868d4b23af696d3734d0b814d04cdd1ac280128e97653a05f32b49c13a29a
JavaScript
<script src="lib/sjcl.js"></script>
<script>
var hmacSHA256 = function (key) {
var hasher = new sjcl.misc.hmac(key, sjcl.hash.sha256);
this.encrypt = function () {
return hasher.encrypt.apply(hasher, arguments);
};
};
hash = sjcl.misc.pbkdf2("password", [0], 100000, 256, hmacSHA256);
console.log(sjcl.codec.hex.fromBits(hash));
</script>
Generates: 41c04f824d843d5be0ae66b3f621d3f05db7d47e7c46ee0e9171b5cbff7f3631
I'm scratching my head a lot now. I think b'' and [0] are equivalent salts, but I'm not sure. I think they both use utf-8 to encode the password, but I'm not sure. And I'm not convinced the JavaScript hmacSHA256 function exactly matches what Python is doing. Or it could be something else still.

Off the top of my head, have you checked if
hash = sjcl.misc.pbkdf2("password", "", 100000, 256);
gives the correct result?
As far as I can tell from the docs, SJCL's PBKDF2 implementation defaults to HMAC-SHA256 if you don't explicitly give it a PRF. If making that change fixes the bug, then there's probably something wrong with your hmacSHA256 wrapper.
Also, I'm not sure if specifying an empty salt as [0] really works (or is guaranteed to work in future versions, given that the format of SJCL's bitArrays is explicitly subject to change), but "" definitely should work.

This is sharing for the rest based on my recent experience.
My objective:
To generate PBKDF2 password using Python. The client will be Android (Java), and the back end will be on Flask (Python).
Issue:
While testing, I discovered that both versions (Java vs Python) produced different hashing output (all other parameters were equal - SHA256, 1000 iterations, similar SALT)
What I have found out:
Using PBKDF2 generation tools available on the internet, the Android result was an exact match, while the Python was not. So there is a good chance the Python result was somehow skewed .....
Problem solved:
While looking for possible explanation in SO, I discovered that the way I converted String to bytes in Python was somehow not entirely correct:
Original code:
dk = hashlib.pbkdf2_hmac('sha256', b'base64_message', b'salt', 1000)
Working code:
dk = hashlib.pbkdf2_hmac('sha256', base64_message.encode(), salt.encode(), 1000)
This is probably due to my lack of experience in Python. Hope this note will be of use to others, especially those who are new to Python!

Related

JavaScript RSA encryption to PHP

I need to authenticate myself via PHP script on remote website, and website uses JS-based RSA encryption for passwords. Here's the code from website:
function rsa_encrypt(strPlainText) {
var strModulus = "some_random_string";
var strExponent = "10001";
var rsa = new RSAKey();
rsa.setPublic(strModulus, strExponent);
var res = rsa.encrypt(strPlainText);
if (res) {
return res;
}
return false;
}
Browsed a lot of topics on this website, and found that the recommended way is to use phpseclib (if there's another one, let me know). However, using basic example from http://phpseclib.sourceforge.net/rsa/examples.html#encrypt,enc2 I get just an empty page. I entered some_random_string into $rsa->loadKey('...'); - not sure if I did it right? However, I can't see a place to enter strExponent (which is 10001) in this example.
So I tried another solution - Encrypt and Decrypt text with RSA in PHP and modified my code to look the following:
include('Crypt/RSA.php');
$privatekey = "some_random_string";
$rsa = new Crypt_RSA();
$rsa->loadKey($privatekey);
$plaintext = new Math_BigInteger('10001');
echo $rsa->_exponentiate($plaintext)->toBytes();
However, I get this error:
Fatal error: Call to a member function abs() on null in Math\BigInteger.php on line 1675
The solution was posted some time ago, so I guess something got changed in phpseclib library during this time, and I'm just not sure how to re-modify my code.
Popular formats for RSA keys typically contain both the exponent and the modulus within them. See, for example, my answer to I understand the mathematics of RSA encryption: How are the files in ~/.ssh related to the theory? for a more detailed discussion of one particular type of key format.
If you have the exponent and modulo as distinct values try doing this:
$rsa->loadKey([
'e' => new Math_BigInteger('10001', 16),
'n' => new Math_BigInteger('some_random_string', 16);
]);
Note the , 16 bit. 65537 (10001 in hex) is a common RSA exponent. Math_BigInteger assumes, by default, that the number being passed to it is in base-10, unless you specifically tell it otherwise. One requirement of RSA is that e be coprime to either phi(n) or lcm(n). 65537 is trivially coprime because it is prime. 10001 is not prime. It can be factored into 73*137.

RNCryptor JS simple case fails

I have used RNCryptor successfully in Objective-C, and now need to encrypt data for my iOS app from Javascript. But, this simple test fails...
<script type="text/javascript" src="js/utils/sjcl.js"></script>
<script type="text/javascript" src="js/utils/rncryptor.js"></script>
function testEncodeEncrypt_RN(plaintext) {
var secret = "rosebud";
var encrypted = RNCryptor.Encrypt(secret, plaintext);
var decrypted = RNCryptor.Decrypt(secret, encrypted);
console.log("decrypted to " + decrypted);
}
...with the error "Uncaught CORRUPT: pkcs#5 padding corrupt" thrown by sjcl.js.
I've tried various options objects, but I figure that no options ought to work for both Encrypt and Decrypt. Varying the input string and secret do no good, either. Any ideas?
/*
Takes password string and plaintext bitArray
options:
iv
encryption_salt
html_salt
Returns ciphertext bitArray
*/
RNCryptor.Encrypt = function(password, plaintext, options) {
Is plaintext a bitArray? If you're passing a UTF-8 string, you'll need to convert it with sjcl.codec.utf8String.toBits. There are also codecs for hex and base64 encoding. See the SJCL docs.
The latest versions of SJCL will accept strings and autoconvert them to bitArrays, but I probably won't touch the JS implementation myself again until I finish work on the v4 format (should be before the end of 2015). I am happy to accept pull requests.
Note that RNCryptor-js is not fully compatible with any of the other implementations (including the ObjC implementation). JavaScript is too slow to handle the 10,000 PBKDF2 iterations that are required for the v3 format, so it defaults to using 1,000. That means you have to modify the code on the other side to match (or configure JS to use 10,000, but it'll take 10x as long to deal with passwords). Look for the two .rounds configuration settings in RNCryptor.h.
One of the major goals of the v4 format is to make iteration count configurable to make JavaScript capable of interoperating (unfortunately by dramatically reducing the security of the encryption, but it's all JavaScript can handle today).

Differing SHA1 hashes for identical values on the server and the client

On the client I'm using Rusha, which I've put into a wrapper:
function cSHA1(m){
return (new Rusha).digest(m);
}
On the server I'm using Node's native crypto module,
function sSHA1(m){
var h = crypto.createHash('sha1');
h.update(m);
return h.digest('hex');
}
Let's try it:
cSHA1('foo')
"0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33"
sSHA1('foo')
'0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33'
cSHA1('bar')
"62cdb7020ff920e5aa642c3d4066950dd1f01f4d"
sSHA1('bar')
'62cdb7020ff920e5aa642c3d4066950dd1f01f4d'
So far, so good.
Now let's throw them a curveball...
cSHA1(String.fromCharCode(10047))
"5bab61eb53176449e25c2c82f172b82cb13ffb9d"
sSHA1(String.fromCharCode(10047))
'5bab61eb53176449e25c2c82f172b82cb13ffb9d'
Ok, fine.
I have a string, and it shouldn't be important how I got it, and it's a long story, anyway, but:
s.split('').map(function(c){
return c.charCodeAt();
})
yields the exact same result in both places:
[58, 34, 10047, 32, 79]
Now, let's hash it:
s
":"✿ O"
cSHA1(s)
"a199372c8471f35d14955d6abfae4ab12cacf4fb"
s
':"? O'
sSHA1(s)
'fc67b1e4ceb3e57e5d9f601ef4ef10c347eb62e6'
This has caused me a fair bit of grief; what the hell?
I have run into the same problem with the German Umlaut character when comparing SHA1 hashes of PHPs sha1 and Rusha.
The reason is simple: some stoned fool decided Javascript strings are UTF16 - and PHP doesn't give a sh*t about encoding, it just takes what is there. So, if you supply PHP a json_decode("\u00e4"), it will turn this into a 2-byte string 0xc3 0xa4 (UTF8).
JS instead will make a single UTF16 byte out of this (0xE4) - and Rusha's manual explicitly says all codepoints must be below 256.
To help yourself, use the UTF18-to-8 library at http://www.onicos.com/staff/iz/amuse/javascript/expert/utf.txt like sha.digest(utf16to8("\u00e4")). This will feed rusha correct codepoints.

is there any builtin javascript string hash function in newest browsers?

Everytime a new versions of browsers show up I hear about new stuff being added, like say webGL and other technologies that no one really knows if they catch up.
But I wonder if someone ever thought about such basic stuff in JS like hashing functions (MD5,SHA1 and the like).
By newest browsers I mean today's development versions too like Opera 12, Chrome 17 or Firefox 10.
Looking now for solution I found this comment on another thread here: https://stackoverflow.com/questions/7204097/short-hashing-function-for-javascript (Do you know that javascript objects already are hashtables ?). So what are these 'hashtables' ? Does it mean that I can make any string into a hash, but not an established one like md5 or sha1 but some JS build in specific ?
basically what I need to do is:
var txt="Hello world!";
var hash = txt.toSha1();
For anybody still looking for this information. There is a WebCrypto API which appears to have been finalised at the beginning of 2017.
To use it in a browser, you can find it at window.crypto.subtle which contains methods for encryption, digests etc. Documentation on the available functions here.
Paul Johnston has implemented the following algorithms in javascript
MD5, RIPEMD-160, SHA-1, SHA-256 and sha-512
You can find the source code and some examples here:
http://pajhome.org.uk/crypt/md5/
I hope this is what you were looking for.
async function sha256(source) {
const sourceBytes = new TextEncoder().encode(source);
const digest = await crypto.subtle.digest("SHA-256", sourceBytes);
const resultBytes = [...new Uint8Array(digest)];
return resultBytes.map(x => x.toString(16).padStart(2, '0')).join("");
}
Note: This answer was written in 2014 when the Web Cryptography API was not available. Do not use this in the context where cryptographic security is needed. This may be useful when you need a simple reversible encryption with "builtin" support.
When I need simple client side hashing without external libraries I use the browsers' built in atob() and btoa() functions.
window.btoa() creates a base-64 encoded ASCII string from a "string" of binary data.
function utf8_to_b64( str ) {
return window.btoa(encodeURIComponent( escape( str )));
}
The window.atob() function decodes a string of data which has been encoded using base-64 encoding.
function b64_to_utf8( str ) {
return unescape(decodeURIComponent(window.atob( str )));
}
http://caniuse.com/#search=btoa and http://caniuse.com/#search=atob shows it is hugely supported by the modern browsers
Example taken from https://developer.mozilla.org/en-US/docs/Web/API/window.btoa

Encrypting data with ruby decrypting with node

I want to encrypt some data in a ruby app and then decode it in a nodejs app. I have been trying to get this to work and now I am just trying to encrypt the same piece of data in both languages to get the same result but I can't seem to do it.
//js
var crypto = require('crypto');
var key = crypto.createHash('sha1').update('key').digest('hex');
console.log(key); // a62f2225bf70bfaccbc7f1ef2a397836717377de
var encrypted = "";
var cipher = crypto.createCipher('bf-cbc', key);
encrypted += cipher.update('text');
encrypted += cipher.final('hex');
console.log(encrypted); //outputs 4eafd5542875bd3c
So it looks like I get a hexadecimal string from the encoding.
#ruby
require 'openssl'
require 'digest/sha1'
c = OpenSSL::Cipher::Cipher.new("bf-cbc")
c.encrypt
# your pass is what is used to encrypt/decrypt
c.key = key = Digest::SHA1.hexdigest("key")
p key # a62f2225bf70bfaccbc7f1ef2a397836717377de
e = c.update("text")
e << c.final
p e # 皋?;??
Is there some sort of encoding issue that I am missing. I tried to base64 decode e but that didn't produce the same result as the node app. Any pointers?
UPDATE: So this is as close as a friend and I can get: https://gist.github.com/a880ea13d3b65a21a99d. Sheesh, I just want to encrypt something in ruby and decrypt it in node.
UPDATE2: Alright, the code in this issue gets me a lot of the way there: https://github.com/joyent/node/issues/1395
There are several subtle things that make this fail. The most important one - you are not specifying an IV in your code, so a random value will be generated for you. You would notice that you couldn't even decrypt your ciphertext within the same programming language this way.
So you need to provide an explicit IV to both implementations. But before I show you the code, some advice:
Key generation:
Blowfish operates on 64 bit blocks, its key size varies, but OpenSSL (which currently powers both Ruby's and node.js' cipher implementation) uses 128 bit by default, that is 16 bytes.
So your key violates two principles - the first: it's simply too long. It's the hex representation of a SHA-1 hash, which is 20 bytes * 2 = 40 bytes instead of 16. Most of the time this is fine, because the implementation truncates the values appropriately, but that is something you should not depend on.
The second mistake, much more severe, is that you use the hex representation instead of the raw bytes: big security issue! Hex characters are not random at all, so in effect you reduce the entropy of your input to half the length (because the underlying bytes were random).
A secure way to generate random keys is using OpenSSL::Random
key = OpenSSL::Random.random_bytes(cipher_key_len)
A third mistake is to keep your key hard-coded in the sources. It's a bad idea. The least you should do is to store it elsewhere on the file system, where access is tightly restricted. See also my answer to another question. The key should be stored out-of-band and only loaded dynamically within the application.
Cipher:
Blowfish grows old. It's still considered unbroken in the sense that brute-forcing it is the only way to break it. But a search space of 2^64 is not out of reach for resourceful attackers. So you should indeed move on to AES.
Padding:
OpenSSL pads using PKCS5Padding (also known as PKCS7Padding) by default. Ruby profits from this and my bet is node.js utilizes this, too - so you should be safe on this.
Now to the working solution. We need to generate an IV, Blowfish requires it to be 64 bit - 8 bytes. You will need rbytes to get secure random numbers in node. The IV may be hardcoded in your sources (it's public information, no security impact) - but it must be the same on both sides. You should pregenerate a value and use it for both node.js and Ruby.
/*node.js*/
var rbytes = require('rbytes');
var iv = rbytes.randomBytes(8);
/*see advice above - this should be out-of-band*/
var key = rbytes.randomBytes(16);
var encrypted = "";
var cipher = crypto.createCipheriv('bf-cbc', key, iv);
encrypted += cipher.update('text');
encrypted += cipher.final('hex');
Now the Ruby part:
require 'openssl'
c = OpenSSL::Cipher::Cipher.new("bf-cbc")
c.encrypt
# should be out-of-band again
c.key = OpenSSL::Random.random_bytes(16)
# may be public but has to be the same for Ruby and node
iv = OpenSSL::Random.random_bytes(8)
c.iv = iv
e = c.update("text")
e << c.final
puts e.unpack('H*')[0]
Your cyphertext will be some random looking bytes. Those bytes can be expressed as hex, Base64 or in other ways. It looks as if your ruby code is outputting the raw bytes. I suggest that you convert those raw bytes to hex to make your comparison.
Looking at your code, you should also change from Blowfish ("bf") to AES. Blowfish has a 64-bit block size and is now obsolete.
You would do well to explicitly specify padding, PKCS7 is common
OK. I want to thank everyone for helping me out. Basically this thread here answers my question: https://github.com/joyent/node/issues/1395. I am going to go ahead and post the two programs in case anyone else has to go through this rigamarole. Keep in mind this isn't mean to be hardcore secure, this is a stepping stone for ruby encrypting data and node decrypting it. You will have to take more steps to make sure higher security measures are taken.
The code is located at this gist: https://gist.github.com/799d6021890f34734470
These were run on ruby 1.9.2p290 and node 0.4.10

Categories