An API that I am trying to use is returning base64 encoded responses. The response is first compressed using GZip with 4 bits of offset and then base64 encoded. I tried parsing the response using JavaScript (pako and zlib) and in both cases, it failed. The API has an example of C# code on how should the response decompression work, but I don't really know how can I transform that into JavaScript. So can anyone give me a hand transforming this function into JavaScript or just give me some tips on how to handle the 4 bytes offset? I didn't find anything relevant in the libraries' documentation.
public string Decompress(string value)
{
byte[] gzBuffer = Convert.FromBase64String(value);
using (MemoryStream ms = new MemoryStream())
{
int msgLength = BitConverter.ToInt32(gzBuffer, 0);
ms.Write(gzBuffer, 4, gzBuffer.Length - 4);
byte[] buffer = new byte[msgLength];
ms.Position = 0;
using (System.IO.Compression.GZipStream zip = new System.IO.Compression.GZipStream(ms, System.IO.Compression.CompressionMode.Decompress))
{
zip.Read(buffer, 0, buffer.Length);
}
return System.Text.Encoding.Unicode.GetString(buffer, 0, buffer.Length);
}
}
I'm going to use fflate (disclaimer, I'm the author) to do this. If you want to translate that function line for line:
// This is ES6 code; if you want better browser compatibility
// use the ES5 variant.
import { gunzipSync, strToU8, strFromU8 } from 'fflate';
const decompress = str => {
// atob converts Base64 to Latin-1
// strToU8(str, true) converts Latin-1 to binary
const bytes = strToU8(atob(str), true);
// subarray creates a new view on the same memory buffer
// gunzipSync synchronously decompresses
// strFromU8 converts decompressed binary to UTF-8
return strFromU8(gunzipSync(bytes.subarray(4)));
}
If you don't know what ES6 is:
In your HTML file:
<script src="https://cdn.jsdelivr.net/npm/fflate/umd/index.js"></script>
In your JS:
var decompress = function(str) {
var bytes = fflate.strToU8(atob(str), true);
return fflate.strFromU8(fflate.gunzipSync(bytes.subarray(4)));
}
I'd like to mention that streams are almost totally useless if you're going to accumulate into a string at the end, so the C# code is suboptimal. At the same time, since you're using the standard library, it is the only option.
In addition, I'd highly recommend using the callback variant (i.e. gunzip instead of gunzipSync) if possible because that runs on a separate thread to avoid causing the browser to freeze.
Related
I am relatively new to JavaScript and I want to get the hash of a file, and would like to better understand the mechanism and code behind the process.
So, what I need: An MD5 or SHA-256 hash of an uploaded file to my website.
My understanding of how this works: A file is uploaded via an HTML input tag of type 'file', after which it is converted to a binary string, which is consequently hashed.
What I have so far: I have managed to get the hash of an input of type 'text', and also, somehow, the hash of an uploaded file, although the hash did not match with websites I looked at online, so I'm guessing it hashed some other details of the file, instead of the binary string.
Question 1: Am I correct in my understanding of how a file is hashed? Meaning, is it the binary string that gets hashed?
Question 2: What should my code look like to upload a file, hash it, and display the output?
Thank you in advance.
Basically yes, that's how it works.
But, to generate such hash, you don't need to do the conversion to string yourself. Instead, let the SubtleCrypto API handle it itself, and just pass an ArrayBuffer of your file.
async function getHash(blob, algo = "SHA-256") {
// convert your Blob to an ArrayBuffer
// could also use a FileRedaer for this for browsers that don't support Response API
const buf = await new Response(blob).arrayBuffer();
const hash = await crypto.subtle.digest(algo, buf);
let result = '';
const view = new DataView(hash);
for (let i = 0; i < hash.byteLength; i += 4) {
result += view.getUint32(i).toString(16).padStart(2, '0');
}
return result;
}
inp.onchange = e => {
getHash(inp.files[0]).then(console.log);
};
<input id="inp" type="file">
I have a Pentaho process that is is base64 encoding a binary (pdf) via java - with this command:
Step: Load File Content In Memory
Outputs: pdf_content
Then
Step: Javascript
Outputs: encString
var encString = new Packages.java.lang.String( Packages.org.apache.commons.codec.binary.Base64.encodeBase64( pdf_content.getBytes() ) );
Then
Step: Rest Client (posts data)
On the NodeJs side
const binary = new Buffer(base64Encoded, 'base64');
The problem is that the binary file on the node side differs from that on the source (i have access to and can copy the files around).
Further confusing the matter I introduced an intermediate step saving the base64Encoded string out to disk prior to decoding it. I also then opened the original file (from the Pentaho Side) and encoded that using the
Buffer(fs.readFileSync(originalPath)).toString('base64')
and compared the base 64 encoded versions of each. I was hoping to confirm that the algorithm is different (although there is no guarantee). The files themselves were the same length, and started and finished with no differences. Scattered about the file were bunch of minor differences. 1 byte here 1 byte there.
Other bits: Apache.Commons...Base64 uses "rfc 2045" Buffer on the Node Js side uses "rfc 4648" (I may be misreading the description). Questions,
Is there a known way to communicate base64 between a Java encoding and Node Js decoding?
Are there other recommendation on base64 encoding in Pentaho?
Here is how you encode it on the Java side:
in Java 7 using Google guava's com.google.common.io.BaseEncoding and com.google.common.io.Files:
byte[] data = BaseEncoding.base64().encode(bytes).getBytes();
Files.write(data, new File(outputFile));
in Java 8 using java.util.Base64 and java.nio.file.Files/java.nio.file.Paths:
byte[] data = Base64.getEncoder().encode(bytes);
Files.write(Paths.get(outputFile), data);
and decode on the node side (synchronous version):
var fs = require("fs");
var buf fs.readFileSync("fromJava.dat");
var res = new Buffer(buf.toString(), "base64").toString("utf-8");
And here are the corresponding Java-side tests (for Java8 native java.util.Base64 and for Guava's com.google.common.io.BaseEncoding) invoking node.js process and proving correctness:
#Test
public void testJ8() throws IOException {
String input = "This is some UTF8 test data, which also includes some characters "
+ "outside the 0..255 range: ❊ ✓ ❦. Let's see how we get them to node and back";
byte[] data = Base64.getEncoder().encode(input.getBytes());
Files.write(Paths.get("fromJava.dat"), data);
Process p = Runtime.getRuntime().exec(new String[]{"node", "-e",
"console.log(new Buffer(require('fs').readFileSync('fromJava.dat').toString(), 'base64').toString('utf-8'))"});
assertEquals(input, new Scanner(p.getInputStream()).useDelimiter("\n").next());
}
Test run:
Process finished with exit code 0
#Test
public void testJ7() throws IOException {
String input = "This is some UTF8 test data, which also includes some characters "
+ "outside the 0..255 range: ❊ ✓ ❦. Let's see how we get them to node and back";
byte[] data = BaseEncoding.base64().encode(input.getBytes()).getBytes();
Files.write(data, new File("fromJava.dat"));
Process p = Runtime.getRuntime().exec(new String[]{"node", "-e",
"console.log(new Buffer(require('fs').readFileSync('fromJava.dat').toString(), 'base64').toString('utf-8'))"});
assertEquals(input, new Scanner(p.getInputStream()).useDelimiter("\n").next());
}
Test run:
Process finished with exit code 0
Both tests executed with Java 8 on OSX/unix but Guava 19 used here is fully compatible with Java 7 and if the node executable is on the path in Windows then there is no reason why the tests would not run there (provided it can also evaluate a script after the -e argument, no idea).
Our application is built in Angularjs, In one scenario We are sending good amount of JSON data to client side. So it take much time. So What we have done now, we make that JSON data as GZIP string.
public static string Compress(string s)
{
var bytes = Encoding.Unicode.GetBytes(s);
using (var msi = new MemoryStream(bytes))
using (var mso = new MemoryStream())
{
using (var gs = new GZipStream(mso, CompressionMode.Compress))
{
msi.CopyTo(gs);
}
return Convert.ToBase64String(mso.ToArray());
}
}
After using above code our string size reduced to good extent.
But now our problem is that we are unable to decompress that GZIP in client side.
I have tried following library,
GZIP Library ZLIB
but still As we are using TypeScript, we dont get method accessible in page,
var gunzip = new Zlib.Gunzip(bytes);
var plain = gunzip.decompress();
In above line Zlib is not available. May be due to TypeScript definition not available for same.
So can anyone help me to decompress above Gzip string.
Just declare Zlib at a global level so the transpailer doesn't complain:
declare var Zlib : any;
Other option is to import the type definitions for Zlib so the transpiler recognize the symbols:
http://definitelytyped.org/docs/from--from/index.html
http://definitelytyped.org/docs/from--from/modules/zlib.html
I've been trying to decrypt an ArrayBuffer object using CryptoJS, but so far it always returns a blank WordArray. The files (images) are encrypted in an iOS and Android app, sent to a server, and downloaded in this web app to be decrypted and displayed. The iOS and Android apps are able to decrypt the files without problems, so there's nothing wrong with the encryption process.
The files are downloaded with an XMLHttpRequest with responseType set to arraybuffer. Here's my code so far:
// Decrypt a Base64 encrypted string (this works perfectly)
String.prototype.aesDecrypt = function(key) {
var nkey = CryptoJS.enc.Hex.parse(key.sha256());
return CryptoJS.AES.decrypt(this.toString(), nkey, {
iv: CryptoJS.enc.Hex.parse('00000000000000000000000000000000'),
mode: CryptoJS.mode.CBC,
padding: CryptoJS.pad.Pkcs7
}).toString(CryptoJS.enc.Utf8);
}
// Decrypt a plain encrypted ArrayBuffer (this is the problem, it always outputs an empty WordArray)
ArrayBuffer.prototype.aesDecrypt = function(key) {
// Get key
if (!key) return null;
var nkey = CryptoJS.enc.Hex.parse(key.sha256());
// Get input (if I pass the ArrayBuffer directly to the create function, it returns
// a WordList with sigBytes set to NaN)
//var input = CryptoJS.lib.WordArray.create(this);
var input = CryptoJS.lib.WordArray.create(new Uint8Array(this));
// Decrypt
var output = CryptoJS.AES.decrypt(input, nkey, {
iv: CryptoJS.enc.Hex.parse('00000000000000000000000000000000'),
mode: CryptoJS.mode.CBC,
padding: CryptoJS.pad.Pkcs7
});
// Output is an empty WordList
console.log("Output: ", output);
}
Another question I have is how do you convert a WordArray to an ArrayBuffer?
The conversion of ArrayBuffer -> WordArray has been discussed in CryptoJS's issue 46. For that reason a TypedWordArraywhere you can also pass an ArrayBuffer has been added.
To use that additionally include the following script:
<script src="http://crypto-js.googlecode.com/svn/tags/3.1/build/components/lib-typedarrays.js"></script>
Then you can simply do:
var wordArray = CryptoJS.lib.WordArray.create(arrayBuffer);
/* perform decryption of `wordArray` */
To reconvert the resulting decryptedWordArray to an ArrayBuffer, the simplest approach would probably be, to first convert it to a Base64-String (as discussed here) and then decode that String to the desired ArrayBuffer (see here). The whole procedure would look something like this:
dcWordArray = ... // your decrypted WordArray
dcBase64String = dcWordArray.toString(CryptoJS.enc.Base64); // to Base64-String
dcArrayBuffer = base64DecToArr(dcBase64String).buffer; // to ArrayBuffer
Edit:
For a more efficient conversion (no intermediate Base64String necessary) check out Aletheios answer to that question (the function wordToByteArray(wordArray) and then do .buffer).
you can use the web crypto API to directly decrypt arrayBuffer.
No need to convert binary data to a string, which is so inefficient, and it could cause memory and CPU rises.
Context: JavaScript, as part of a SDK (can be on node.js or browser).
Start point: I have a base64 string that's actually a base64 encoded PNG image (I got it from selenium webdriver - takeScreenshot).
Question: How do I crop it?
The techniques involving the canvas seem irrelevant (or am I wrong?). My code runs as part of tests - probably on node.js. The canvas approach doesn't seem to fit here and might also cause additional noise in the image.
All the libraries I found either deal with streams (maybe I should convert the string to stream somehow?) or deal directly with the UI by adding a control (irrelevant for me).
Isn't there something like (promises and callbacks omitted for brevity):
var base64png = driver.takeScreenshot();
var png = new PNG(base64png);
return png.crop(50, 100, 20, 80).toBase64();
?
Thanks!
Considering you wish to start with base64 string and end with cropped base64 string (image), here is the following code:
var Stream = require('stream');
var gm = require('gm');
var base64png = driver.takeScreenshot();
var stream = new Stream();
stream.on('data', function(data) {
print data
});
gm(stream, 'my_image.png').crop(WIDTH, HEIGHT, X, Y).stream(function (err, stdout, stderr) {
var data = '';
stdout.on('readable', function() {
data += stream.read().toString('base64');
});
stream.on('end', function() {
// DO something with your new base64 cropped img
});
});
stream.emit('data', base64png);
Be aware that it is unfinished, and might need some polishing or debugging (I am in no means a node.js guru), but the idea is next:
Convert string into stream
Read stream into GM module
Manipulate the image
Save it into a stream
Convert stream back into 64base string
Adding my previous comment as an answer:
Anyone looking to do this will need to decode the image to get the raw image data using a library such as node-pngjs and manipulate the data yourself (perhaps there is a library for such operations that doesn't rely on the canvas).