We Keep Coding

Decompressing bzip2 data in Javascript

I ultimately have to consume some data from a Javascript file that looks as follows:
Note: The base64 is illustrative only.
function GetTripsDataCompressed() { return 'QlpoOTFBWSZTWdXoWuEDCAgfgBAHf/.....=='; }
GetTripsDataCompressed() returns a base64 string that is derived as an array of objects converted to JSON using JSON.NET and the resulting string then compressed to bzip2 using SharpCompress with the resulting memory stream Base64 encoded.
This is what I have and cannot change it.
I am struggling to find a bzip2 JavaScript implementation that will take the result of:
var rawBzip2Data = atob(GetTripsDataCompressed());
and convert rawBzip2Data back into the string that is the JSON array. I cannot use something like compressjs as I need to support IE 10 and as it uses typed arrays that means IE10 support is out.
So it appears that my best option is https://github.com/antimatter15/bzip2.js however because I have not created an archive and only bzip2 a string it raises an error of Uncaught No magic number found after doing:
var c = GetTripsDataCompressed();
c = atob(c);
var arr = new Uint8Array(c);
var bitstream = bzip2.array(arr);
bzip2.simple(bitstream);
So can anyone help me here to decompress a BZip2, Base64 encoded string from JavaScript using script that is IE 10 compliant? Ultimately I don't care whether it uses https://github.com/antimatter15/bzip2.js or some other native JavaScript implementation.

It seems to me the answer is in the readme:
decompress(bitstream, size[, len]) does the main decompression of a single block. It'll return -1 if it detects that it's the final block, otherwise it returns a string with the decompressed data. If you want to cap the output to a certain number of bytes, set the len argument.
Also, keep in mind the repository doesn't have a license attached. You'll need to reach out to the author if you want to use the code. That might be tricky given that the repository is eight years old.
On the other hand, the Bzip2 algorithm itself is open-source (BSD-like license), so you can just reimplement it yourself in Javascript. It's just a few hundred lines of relatively straight-forward code.

How do I insert a PNG comment block when saving an HTML5 canvas using javascript toDataURL?

I have a compact canvas-to-png download saver function (see code below).
This code works very well and I am satisfied with its output... mostly.
Would a second replace suffice? What would that replace look like?
My only other option is to post-process the file with imagemagick.
Any ideas?
More completely: I want to add metadata from javascript.
I found this link http://dev.exiv2.org/projects/exiv2/wiki/The_Metadata_in_PNG_files
which details the structures, and I may be able to figure it out with sufficient time.
If anyone has experience and can shorten this for me, I would appreciate it.
//------------------------------------------------------------------
function save () // has to be function not var for onclick to work.
//------------------------------------------------------------------
{
var element = document.getElementById("saver");
element.download = savename;
element.href = document.
getElementById(id.figure1a.canvas).
toDataURL("image/png").
replace(/^data:image\/[^;]/,'data:application/octet-stream');
}

The Base-64 representation has little to do with the internal chunks. It's just [any] binary data encoded as string so it can be transferred over string-only protocols (or displayed in a textual context).
It's perhaps a bit broad to create an example, but hopefully showing the main steps will help to achieve what you're looking for:
To add a chunk to a PNG you would first have to convert the data for it into an ArrayBuffer using XHR/fetch in the case of Data-URIs, or FileReader in case you have the PNG as Blob (which I recommend. See toBlob()).
Add a DataView to the ArrayBuffer
Go to position 0x08 in the array which will represent the start of the IHDR chunk, read the length of the chunk (Uint32) (it's very likely it has the same static size for almost any PNG but since it's possible to have changes, and you don't need to remember the chunk size we'll just read it from here). Add length to position (+4 for CRC-32 at the end of the chunk, and +4 if you didn't move the pointer while reading the length), typically this should land you at position 0x21.
You now have the position for the next chunk which we can use to insert our own text chunks
Split that first part into a part-array (a regular array) using a sub-array with the original ArrayBuffer, e.g. new Uint8Array(arraybuffer, 0, position); - you can also use the subarray method.
Produce the new chunk* as typed array and add to part-array
Add the remaining part of the original PNG array without the first part to the part-array, e.g. new Uint8Array(arraybuffer, position, length - position);
Convert the part-array to a Blob using the part-array directly as argument (var newPng = new Blob(partArray, {type: "image/png"});). This will now contain the custom chunk. From there you can use an Object-URL with it to read it back as an image (or make it available for download).
*) Chunk:
For tEXt be aware of it is limited to the Latin-1 charset which means you'll have to whitewash the string you want to use - use iTXt for unicode (UTF-8) content - we'll use tEXt here for simplicity.
The keyword and value is separated by a NUL-byte (0x00) in a tEXt chunk, and the keyword must be exactly typed as defined in the spec.
Build the chunk this way:
get byte-size from string
add 12 bytes (for length, four-cc and crc-32)
format the array this way (you can use a DataView here as well):
Uint32 - length of chunk (data only in number of bytes)
Uint32 - "tEXt" as four-cc
[...] - The data itself (copy byte-wise)
Uint32 - CRC32* which includes the FourCC but not length and itself.
All data in a PNG is big-endian.
To calculate CRC-32 feel free to use this part of my pngtoy solution (the LUT is built this way). Here is one way to format a four-cc:
function makeFourCC(n) { // n = "tEXt" etc., big-endian
var c = n.charCodeAt.bind(n);
return (c(0) & 0x7f) << 24 | (c(1) & 0x7f) << 16 | (c(2) & 0x7f) << 8 | c(3) & 0x7f
}

Encrypting data with ruby decrypting with node

I want to encrypt some data in a ruby app and then decode it in a nodejs app. I have been trying to get this to work and now I am just trying to encrypt the same piece of data in both languages to get the same result but I can't seem to do it.
//js
var crypto = require('crypto');
var key = crypto.createHash('sha1').update('key').digest('hex');
console.log(key); // a62f2225bf70bfaccbc7f1ef2a397836717377de
var encrypted = "";
var cipher = crypto.createCipher('bf-cbc', key);
encrypted += cipher.update('text');
encrypted += cipher.final('hex');
console.log(encrypted); //outputs 4eafd5542875bd3c
So it looks like I get a hexadecimal string from the encoding.
#ruby
require 'openssl'
require 'digest/sha1'
c = OpenSSL::Cipher::Cipher.new("bf-cbc")
c.encrypt
# your pass is what is used to encrypt/decrypt
c.key = key = Digest::SHA1.hexdigest("key")
p key # a62f2225bf70bfaccbc7f1ef2a397836717377de
e = c.update("text")
e << c.final
p e # 皋?;??
Is there some sort of encoding issue that I am missing. I tried to base64 decode e but that didn't produce the same result as the node app. Any pointers?
UPDATE: So this is as close as a friend and I can get: https://gist.github.com/a880ea13d3b65a21a99d. Sheesh, I just want to encrypt something in ruby and decrypt it in node.
UPDATE2: Alright, the code in this issue gets me a lot of the way there: https://github.com/joyent/node/issues/1395

There are several subtle things that make this fail. The most important one - you are not specifying an IV in your code, so a random value will be generated for you. You would notice that you couldn't even decrypt your ciphertext within the same programming language this way.
So you need to provide an explicit IV to both implementations. But before I show you the code, some advice:
Key generation:
Blowfish operates on 64 bit blocks, its key size varies, but OpenSSL (which currently powers both Ruby's and node.js' cipher implementation) uses 128 bit by default, that is 16 bytes.
So your key violates two principles - the first: it's simply too long. It's the hex representation of a SHA-1 hash, which is 20 bytes * 2 = 40 bytes instead of 16. Most of the time this is fine, because the implementation truncates the values appropriately, but that is something you should not depend on.
The second mistake, much more severe, is that you use the hex representation instead of the raw bytes: big security issue! Hex characters are not random at all, so in effect you reduce the entropy of your input to half the length (because the underlying bytes were random).
A secure way to generate random keys is using OpenSSL::Random
key = OpenSSL::Random.random_bytes(cipher_key_len)
A third mistake is to keep your key hard-coded in the sources. It's a bad idea. The least you should do is to store it elsewhere on the file system, where access is tightly restricted. See also my answer to another question. The key should be stored out-of-band and only loaded dynamically within the application.
Cipher:
Blowfish grows old. It's still considered unbroken in the sense that brute-forcing it is the only way to break it. But a search space of 2^64 is not out of reach for resourceful attackers. So you should indeed move on to AES.
Padding:
OpenSSL pads using PKCS5Padding (also known as PKCS7Padding) by default. Ruby profits from this and my bet is node.js utilizes this, too - so you should be safe on this.
Now to the working solution. We need to generate an IV, Blowfish requires it to be 64 bit - 8 bytes. You will need rbytes to get secure random numbers in node. The IV may be hardcoded in your sources (it's public information, no security impact) - but it must be the same on both sides. You should pregenerate a value and use it for both node.js and Ruby.
/*node.js*/
var rbytes = require('rbytes');
var iv = rbytes.randomBytes(8);
/*see advice above - this should be out-of-band*/
var key = rbytes.randomBytes(16);
var encrypted = "";
var cipher = crypto.createCipheriv('bf-cbc', key, iv);
encrypted += cipher.update('text');
encrypted += cipher.final('hex');
Now the Ruby part:
require 'openssl'
c = OpenSSL::Cipher::Cipher.new("bf-cbc")
c.encrypt
# should be out-of-band again
c.key = OpenSSL::Random.random_bytes(16)
# may be public but has to be the same for Ruby and node
iv = OpenSSL::Random.random_bytes(8)
c.iv = iv
e = c.update("text")
e << c.final
puts e.unpack('H*')[0]

Your cyphertext will be some random looking bytes. Those bytes can be expressed as hex, Base64 or in other ways. It looks as if your ruby code is outputting the raw bytes. I suggest that you convert those raw bytes to hex to make your comparison.
Looking at your code, you should also change from Blowfish ("bf") to AES. Blowfish has a 64-bit block size and is now obsolete.
You would do well to explicitly specify padding, PKCS7 is common

OK. I want to thank everyone for helping me out. Basically this thread here answers my question: https://github.com/joyent/node/issues/1395. I am going to go ahead and post the two programs in case anyone else has to go through this rigamarole. Keep in mind this isn't mean to be hardcore secure, this is a stepping stone for ruby encrypting data and node decrypting it. You will have to take more steps to make sure higher security measures are taken.
The code is located at this gist: https://gist.github.com/799d6021890f34734470
These were run on ruby 1.9.2p290 and node 0.4.10

How do you access byte level information in JavaScript?

The generally accepted answer is that you can't. However there is mounting evidence that this is not true based on the existence of projects that read in types of data that are not basic HTML types. Some projects that do this are the JavaScript version of ProtoBuf and Smokescreen.
Smokescreen is a flash interpreter written in JS so if it is not possible to get at the bytes directly how are these projects working around this? The source to Smokescreen can be found here. I have looked it over but with JS not being my primary language right now the solution eludes me.

They both look to be using a String (in this case the responseText of an XMLHttpRequest) directly as a collection of bytes.
data = ... // a binary string
bytes = [];
for ( i = 0; i < data.length; i++ )
{
// This coverts the unicode character to a byte stripping
// off anything past the first 8 bits
bytes[i] = data.charCodeAt( i ) & 0xFF;
}

Protobuf does all its magic on an XMLHttpRequest.requestText field, which is just a DOMString.

Returning a byte string to ExternalInterface.call throws an error

I am working on my open source project Downloadify, and up until now it simply handles returning Strings in response to ExternalInterface.call commands.
I am trying to put together a test case using JSZip and Downloadify together, the end result being that a Zip file is created dynamically in the browser, then saved to the disk using FileReference.save. However, this is my problem:
The JSZip library can return either a base64 encoded string of the Zip, or the raw byte string. The problem is, if I return that byte string in response to the ExternalInterface.call command, I get this error:
Error #1085: The element type "string" must be terminated by the matching end-tag "</string>"
ActionScript 3:
var theData:* = ExternalInterface.call('Downloadify.getTextForSave',queue_name);
Where queue_name is just a string used to identify the correct instance in JS.
JavaScript:
var zip = new JSZip();
zip.add("test.txt", "Hello world!\n");
var content = zip.generate(true);
return content;
If I instead return a normal string instead of the byte string, the call works correctly.I would like to avoid using base64 as I would have to include a base64 decoder in my swf which will increase its size.
Finally: I am not looking for a AS3 Zip generator. It is imperative to my project to have that part run in JavaScript
I am admittedly not a AS3 programmer by trade, so if you need any more detail please let me know.

When data is being returned from javascript calls it's being serialized into an XML string. So if the "raw string" returned by JSZip will include characters which make the XML non-valid, which is what I think is happening here, you'll get errors like that.
What you get as a return is actually:
<string>[your JSZip generated string]</string>
Imagine your return string includes a "<" char - this will make the xml invalid, and it's hard to tell what character codes will a raw byte stream translate too.
You can read more about the external API's XML format on LiveDocs

i think the problem is caused by the fact, that flash expects a utf8 String and you throw some binary stuff at it. i think for example 0x00FF will not turn out to be valid utf8 ...
you can try fiddling around with flash.system::System.setCodePage, but i wouldn't be too optimistic ...
i guess a base64 decoder is probably really the easiest ... i'd rather worry about speed than about file size though ... this rudimentary decoder method uses less than half a K:
public function decodeBase64(source:String):ByteArray {
var ret:ByteArray = new ByteArray();
var map:Object = new Object();
var i:int = 0;
for each (var char:String in "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/".split("")) map[char] = i++;
map["="] = 0;
source = source.split("\n").join("").split("\r").join("");//remove linebreaks
for (i = 0; i < source.length/4; i++) {
var buf:int = 0;
for each (char in source.substr(i * 4, 4).split("")) buf = (buf << 6) + map[char];
ret.writeByte(buf >>> 16);
ret.writeShort(buf);
}
return ret;
}
you could simply shorten function names and take a smaller image ... or use ColorTransform or ConvolutionFilter on one image instead of four ... or compile the image into the SWF for smaller overall size ... or reduce function name length ...
so unless you're planning on working with MBs of data, this is the way to go ...