We Keep Coding

Extract selected data in Bokeh to text file and other questions

I am currently working on a Bokeh script that will display an earthquake catalogue on a map. So far, I have had little trouble, but I have stumbled upon a couple questions.
1) I came upon this stackoverflow Q&A: Get selected data contained within box select tool in Bokeh, which was very helpful in learning how to use CutsomJS to take the selected data and output it. I have, however encountered a bit of oddness. The script only seems to work properly for me in the Jupyter notebook. The CustomJS portion reads as follows:
source.callback = CustomJS(args=dict(p=p), code="""
var inds = cb_obj.get('selected')['1d'].indices;
var d1 = cb_obj.get('data');
console.log(d1)
var kernel = IPython.notebook.kernel;
IPython.notebook.kernel.execute("inds = " + inds);
If I try running it through and html window, the following command has no output:
subset = zip([source.data['xvals'][i] for i in inds],
[source.data['yvals'][i] for i in inds])
subset_array = np.array(subset)
outfile = 'Scripts/subset.xy'
fo = open(outfile,'w')
for i in range(len(subset_array)):
print(subset_array[i,0],subset_array[i,1],file=fo)
fo.close()
I could use some advice on how to get the selected data to output to a variable, and ultimately a text file when using html, as opposed to the Jupyter notebook.
2) I have been trying to create a legend for my plot. The earthquakes are indicated by circles (so it is a circle plot), and the circles vary in size based on their magnitude and in colour based on their depth, which I have gotten to work. What I would like to do is show circles of different sizes in the legend, each reflecting different magnitudes (Mag 1, Mag 2, Mag 3, etc.). Here's the command I use for plotting the data:
p.circle("xvals", "yvals",source=source, size="radius", fill_color=transform("depth", mapper),legend = 'Magnitude')
I've tried messing around with the legend commands, but it's all a bit difficult for me to grok, even though I feel like this should be a pretty simple thing to do.
I am utilizing the latest version of Bokeh (0.12.7) and Python 2.7.12. Thanks for any help, and I appreciate the time!

On question 1):
If you are creating your html file with Bokeh's output_file function, you cannot access any python code from it, since the resulting file is just html + javascript. The file saving functionality could be embedded in the javascript code (a bit tricky since it seems there is no solution that works in every browser). Another option would be to run a bokeh server and implement some mechanism to push the data to the client.
Below is a rough example where the file saving functionality is included in the html file. This should work on non-IE browsers.
from random import random
from bokeh.plotting import figure, output_file, show, ColumnDataSource
from bokeh.models import CustomJS, Button
from bokeh.layouts import row, column
# Create some random data and put it into a ColumnDataSource
x = [random() for x in range(100)]
y = [random() for y in range(100)]
source_data = ColumnDataSource(data=dict(x=x, y=y))
# Create a button that saves the coordinates of selected data points to a file
savebutton = Button(label="Save", button_type="success")
savebutton.callback = CustomJS(args=dict(source_data=source_data), code="""
var inds = source_data.selected['1d'].indices;
var data = source_data.data;
var out = "";
for (i = 0; i < inds.length; i++) {
out += "(" + data['x'][inds[i]] + ", " + data['y'][inds[i]] +") ";
}
var file = new Blob([out], {type: 'text/plain'});
var elem = window.document.createElement('a');
elem.href = window.URL.createObjectURL(file);
elem.download = 'selected-data.txt';
document.body.appendChild(elem);
elem.click();
document.body.removeChild(elem);
""")
# Plot the data and save the html file
p = figure(plot_width=400, plot_height=400, tools="lasso_select, reset")
p.circle(x='x', y='y', source=source_data)
plot = column(p, savebutton)
output_file("test.html")
show(plot)

Generate a designed PDF report from a web application

I want to generate a PDF report from a web application. The PDF should contain charts (pie, bar), tables, different fonts and colors.
The server-side of the application is Java, the client-side is AngularJS (and of course CSS3 and HTML).
Two main options:
The client side will pass some parameters to the server, and the server will generate the PDF report, using a Java package. Then the report will be sent back to the client as a downloaded file.
The client will generate the report, using a JS package that converts HTML and CSS to PDF.
In the Java world, I've found for example iText and JFreeChart, like here. The problem here is that the design of charts look bad in the example, and I don't know if it can be changed to be designed by the style-guide I have (a design that can be done easily with CSS).
In the JS world, I've found for example html2canvas and pdfMake, like here. The problem here is that I'm not sure the conversion from HTML to canvas and then to PDF will work good in an Angular application. And I'm not sure it converts well complicated DOM elements, like charts in svg or canvas elements.
Do you have any experience with these packages? Do you know other recommended packages for this task, client or server?

Want to share my solution... I chose a client-side solution.
I started with jsPDF, but had some problems. For example, it was hard to convert tables with the style I want.
I chose pdfMake for the PDF generation, html2canvas for taking screenshots of complicated designed components, and canvg for conversion of d3js charts (svg charts) to canvas (pdfMake can add canvas as image to the document).
I wrote a function that gets the CSS class of the HTML root of the part I want to convert to PDF (remember it's a single-page application), and also gets a meta data of which HTML nodes (again, by their CSS classes) should be added to the PDF (and what type is the node - table/text/image/svg).
Then, with DOM traversing, I walked through the elements I want to add to the PDF, and handled each one by its type. Part of the code (the traversing and the switch-case by type):
$(htmlRootSelector).contents().each(function processNodes(index, element) {
var classMeta = getMetaByClass(element.className);
if (!classMeta) {
$(element).contents().each(processNodes);
return;
}
var pdfObj = {};
pdfObj.width = classMeta.width || angular.undefined;
pdfObj.height = classMeta.height || angular.undefined;
pdfObj.style = classMeta.style || angular.undefined;
pdfObj.pageBreak = classMeta.pageBreak || angular.undefined;
switch (classMeta.type) {
case 'text':
pdfObj.text = element.innerText;
pdfDefinition.content.push(pdfObj);
break;
case 'table':
var tableArray = [];
var headerArray = [];
var headers = $(element).find('th');
var rows = $(element).find('tr');
$.each(headers, function (i, header) {
headerArray.push({text: header.innerHTML, style: classMeta.style + '-header'});
});
tableArray.push(headerArray);
$.each(rows, function (i, row) {
var rowArray = [];
var cells = $(row).find('td');
if (cells.length) {
$.each(cells, function (j, cell) {
rowArray.push(i % 2 === 1 ? {text: cell.innerText, style: classMeta.style + '-odd-row'} : cell.innerText);
});
tableArray.push(rowArray);
}
});
pdfObj.table = {
widths: $.map(headers, function (d, i) {
return i === 0 ? 80 : '*';
}),
body: tableArray
};
pdfDefinition.content.push(pdfObj);
break;
case 'image':
html2CanvasCount++;
htmlToCanvas(element, pdfObj);
pdfDefinition.content.push(pdfObj);
break;
case 'svg':
svgToCanvas(element, pdfObj);
pdfDefinition.content.push(pdfObj);
break;
default:
break;
}
$(element).contents().each(processNodes);
});
This is the solution in general.
Hope it will help someone.

Using pdf.js to display pdf from raw data

I am just getting started with pdf.js and I am trying to load a pdf file from the raw pdf data. I have seen the code:
PDFJS.getPdf('cwpdf.pdf', function getPdfHelloWorld(data) {
...
}
But I am wondering if there is any way to load a pdf from the raw pdf data instead of from the filename. Is this possible?

I put together some complete code and was able to find the problem with the solution below:
var int8View = new Uint8Array(...); //populate int8View with the raw pdf data
PDFJS.getDocument(int8View).then(function(pdf) {
}
When using this solution I ran into the problem other users have seen (#MurWade and #user94154) - the stream must have data error message. It looks like the problem is in the following line:
var int8View = new Uint8Array(...);
The array containing the data does not get properly created, since the data is not in the expected format. Therefore, this line works for some cases, but it might not work in the general case.
I've put together a complete solution, that seems to work better. It loads a PDF file, and it converts it to a raw PDF stream. This is there just for testing purposes, in a real world example, the PDF stream will probably be received in a different fashion. You can examine the stream in a debugger, and it will show as plain text. Below is the key line of the code to make this sample work. Instead converting the raw PDF stream to an array, convert it to data.
var docInitParams = { data: pdfraw };
Then proceed with loading the data. Below is the complete working sample of how to load a standard raw PDF stream and display it. I used to PDF JS hello world sample as a starting point. Please let me know in the comments if any clarification is necessary on this.
'use strict';
PDFJS.getDocument('helloworld.pdf').then(function(pdf) {
pdf.getData().then(function(arrayBuffer) {
var pdfraw = String.fromCharCode.apply(null, arrayBuffer);
var docInitParams = {
data: pdfraw
};
PDFJS.getDocument(docInitParams).then(function(pdfFromRaw) {
pdfFromRaw.getPage(1).then(function(page) {
var scale = 1.5;
var viewport = page.getViewport(scale);
var canvas = document.getElementById('the-canvas');
var context = canvas.getContext('2d');
canvas.height = viewport.height;
canvas.width = viewport.width;
var renderContext = {
canvasContext: context,
viewport: viewport
};
page.render(renderContext);
});
});
});
});

Well, since no one else has answered I will post my findings. I figured out that yes, it is possible to load a pdf file from the raw data. The way this can be done is by using a UInt8Array populated with data in place of the url to where the pdf file is stored.
Example code to do this is below:
var int8View = new Uint8Array(...); //populate int8View with the raw pdf data
PDFJS.getDocument(int8View).then(function(pdf) {
}

Rapidly updating image with Data URI causes caching, memory leak

I have a webpage that rapidly streams JSON from the server and displays bits of it, about 10 times/second. One part is a base64-encoded PNG image. I've found a few different ways to display the image, but all of them cause unbounded memory usage. It rises from 50mb to 2gb within minutes. Happens with Chrome, Safari, and Firefox. Haven't tried IE.
I discovered the memory usage first by looking at Activity Monitor.app -- the Google Chrome Renderer process continuously eats memory. Then, I looked at Chrome's Resource inspector (View > Developer > Developer Tools, Resources), and I saw that it was caching the images. Every time I changed the img src, or created a new Image() and set its src, Chrome cached it. I can only imagine the other browsers are doing the same.
Is there any way to control this caching? Can I turn it off, or do something sneaky so it never happens?
Edit: I'd like to be able to use the technique in Safari/Mobile Safari. Also, I'm open to other methods of rapidly refreshing an image if anyone has any ideas.
Here are the methods I've tried. Each one resides in a function that gets called on AJAX completion.
Method 1 - Directly set the src attribute on an img tag
Fast. Displays nicely. Leaks like crazy.
$('#placeholder_img').attr('src', 'data:image/png;base64,' + imgString);
Method 2 - Replace img with a canvas, and use drawImage
Displays fine, but still leaks.
var canvas = document.getElementById("placeholder_canvas");
var ctx = canvas.getContext("2d");
var img = new Image();
img.onload = function() {
ctx.drawImage(img, 0, 0);
}
img.src = "data:image/png;base64," + imgString;
Method 3 - Convert to binary and replace canvas contents
I'm doing something wrong here -- the images display small and look like random noise. This method uses a controlled amount of memory (grows to 100mb and stops), but it is slow, especially in Safari (~50% CPU usage there, 17% in Chrome). The idea came from this similar SO question: Data URI leak in Safari (was: Memory Leak with HTML5 canvas)
var img = atob(imgString);
var binimg = [];
for(var i = 0; i < img.length; i++) {
binimg.push(img.charCodeAt(i));
}
var bytearray = new Uint8Array(binimg);
// Grab the existing image from canvas
var ctx = document.getElementById("placeholder_canvas").getContext("2d");
var width = ctx.canvas.width,
height = ctx.canvas.height;
var imgdata = ctx.getImageData(0, 0, width, height);
// Overwrite it with new data
for(var i = 8, len = imgdata.data.length; i < len; i++) {
imgdata.data[i-8] = bytearray[i];
}
// Write it back
ctx.putImageData(imgdata, 0, 0);

I know it's been years since this issue was posted, but the problem still exists in recent versions of Safari Browser. So I have a definitive solution that works in all browsers, and I think this could save jobs or lives!.
Copy the following code somewhere in your html page:
// Methods to address the memory leaks problems in Safari
var BASE64_MARKER = ';base64,';
var temporaryImage;
var objectURL = window.URL || window.webkitURL;
function convertDataURIToBlob(dataURI) {
// Validate input data
if(!dataURI) return;
// Convert image (in base64) to binary data
var base64Index = dataURI.indexOf(BASE64_MARKER) + BASE64_MARKER.length;
var base64 = dataURI.substring(base64Index);
var raw = window.atob(base64);
var rawLength = raw.length;
var array = new Uint8Array(new ArrayBuffer(rawLength));
for(i = 0; i < rawLength; i++) {
array[i] = raw.charCodeAt(i);
}
// Create and return a new blob object using binary data
return new Blob([array], {type: "image/jpeg"});
}
Then when you receive a new frame/image base64Image in base64 format (e.g. data:image/jpeg;base64, LzlqLzRBQ...) and you want to update a html <img /> object imageElement, then use this code:
// Destroy old image
if(temporaryImage) objectURL.revokeObjectURL(temporaryImage);
// Create a new image from binary data
var imageDataBlob = convertDataURIToBlob(base64Image);
// Create a new object URL object
temporaryImage = objectURL.createObjectURL(imageDataBlob);
// Set the new image
imageElement.src = temporaryImage;
Repeat this last code as much as needed and no memory leaks will appear. This solution doesn't require the use of the canvas element, but you can adapt the code to make it work.

Try setting image.src = "" after drawing.
var canvas = document.getElementById("placeholder_canvas");
var ctx = canvas.getContext("2d");
var img = new Image();
img.onload = function() {
ctx.drawImage(img, 0, 0);
//after drawing set src empty
img.src = "";
}
img.src = "data:image/png;base64," + imgString;
This might be helped

I don't think there are any guarantees given about the memory usage of data URLs. If you can figure out a way to get them to behave in one browser, it guarantees little if not nothing about other browsers or versions.
If you put your image data into a blob and then create a blob URL, you can then deallocate that data.
Here's an example which turns a data URI into a blob URL; you may need to change / drop the webkit- & WebKit- prefixes on browsers other than Chrome and possibly future versions of Chrome.
var parts = dataURL.match(/data:([^;]*)(;base64)?,([0-9A-Za-z+/]+)/);
//assume base64 encoding
var binStr = atob(parts[3]);
//might be able to replace the following lines with just
// var view = new Uint8Array(binStr);
//haven't tested.
//convert to binary in ArrayBuffer
var buf = new ArrayBuffer(binStr.length);
var view = new Uint8Array(buf);
for(var i = 0; i < view.length; i++)
view[i] = binStr.charCodeAt(i);
//end of the possibly unnecessary lines
var builder = new WebKitBlobBuilder();
builder.append(buf);
//create blob with mime type, create URL for it
var URL = webkitURL.createObjectURL(builder.getBlob(parts[1]))
return URL;
Deallocating is as easy as :
webkitURL.revokeObjectURL(URL);
And you can use your blob URL as your img's src.
Unfortunately, blob URLs do not appear to be supported in IE prior to v10.
API reference:
http://www.w3.org/TR/FileAPI/#dfn-createObjectURL
http://www.w3.org/TR/FileAPI/#dfn-revokeObjectURL
Compatibility reference:
http://caniuse.com/#search=blob%20url

I had a very similar issue.
Setting img.src to dataUrl Leaks Memory
Long story short, I simply worked around the Image element. I use a javascript decoder to decode and display the image data onto a canvas. Unless the user tries to download the image, they'll never know the difference either. The other downside is that you're going to be limited to modern browsers. The up side is that this method doesn't leak like a sieve :)

patching up ellisbben's answer, since BlobBuilder is obsoleted and https://developer.mozilla.org/en-US/Add-ons/Code_snippets/StringView provides what appears to be a nice quick conversion from base64 to UInt8Array:
in html:
<script src='js/stringview.js'></script>
in js:
window.URL = window.URL ||
window.webkitURL;
function blobify_dataurl(dataURL){
var parts = dataURL.match(/data:([^;]*)(;base64)?,([0-9A-Za-z+/]+)/);
//assume base64 encoding
var binStr = atob(parts[3]);
//convert to binary in StringView
var view = StringView.base64ToBytes(parts[3]);
var blob = new Blob([view], {type: parts[1]}); // pass a useful mime type here
//create blob with mime type, create URL for it
var outURL = URL.createObjectURL(blob);
return outURL;
}
I still don't see it actually updating the image in Safari mobile, but chrome can receive dataurls rapid-fire over websocket and keep up with them far better than having to manually iterate over the string. And if you know you'll always have the same type of dataurl, you could even swap the regex out for a substring (likely quicker...?)
Running some quick memory profiles, it looks like Chrome is even able to keep up with deallocations (if you remember to do them...):
URL.revokeObjectURL(outURL);

I have used different methods to solve this problem, none of them works. It seems that memory leaks when img.src = base64string and those memory can never get released. Here is my solution.
fs.writeFile('img0.jpg', img_data, function (err) {
// console.log("save img!" );
});
document.getElementById("my-img").src = 'img0.jpg?'+img_step;
img_step+=1;
Note that you should convert base64 to jpeg buffer.
My Electron app updating img every 50ms, and memory doesn't leak.
Forget about disk usage. Chrome's memory management piss me off.

Unless Safari or Mobile Safari don't leak data urls, server-side might be the only way to do this on all browsers.
Probably most straightforward would be to make a URL for your image stream, GETting it gives a 302 or 303 response redirecting to a single-use URL that will give the desired image. You will probably have to destroy and re-create the image tags to force a reload of the URL.
You will also be at the mercy of the browser regarding its img caching behavior. And the mercy of my understanding (or lack of understanding) of the HTTP spec. Still, unless server-side operation doesn't fit your requirements, try this first. It adds to the complexity of the server, but this approach uses the browser much more naturally.
But what about using the browser un-naturally? Depending on how browsers implement iframes and handle their associated content, you might be able to get data urls working without leaking the memory. This is kinda Frankenstein shit and is exactly the sort of nonsense that no one should have to do. Upside: it could work. Downside: there are a bazillion ways to try it and uneven, undocumented behavior is exactly what I'd expect.
One idea: embed an iframe containing a page; this page and the page that it is embedded in use cross document messaging (note the GREEN in the compatibility matrix!); embeddee gets the PNG string and passes it along to the embedded page, which then makes an appropriate img tag. When the embeddee needs to display a new message, it destroys the embedded iframe (hopefully releasing the memory of the data url) then creates a new one and passes it the new PNG string.
If you want to be marginally more clever, you could actually embed the source for the embedded frame in the embeddee page as a data url; however, this might leak that data url, which I guess would be poetic justice for trying such a reacharound.
"Something that works in Safari would be better." Browser technology keeps on moving forward, unevenly. When they don't hand the functionality to you on a plate, you gotta get devious.

var inc = 1;
var Bulk = 540;
var tot = 540;
var audtot = 35.90;
var canvas = document.getElementById("myCanvas");
//var imggg = document.getElementById("myimg");
canvas.width = 550;
canvas.height = 400;
var context = canvas.getContext("2d");
var variation = 0.2;
var interval = 65;
function JLoop() {
if (inc < tot) {
if (vid.currentTime < ((audtot * inc) / tot) - variation || (vid.currentTime > ((audtot * inc) / tot) + variation)) {
contflag = 1;
vid.currentTime = ((audtot * inc) / tot);
}
// Draw the animation
try {
context.clearRect(0, 0, canvas.width, canvas.height);
if (arr[inc - 1] != undefined) {
context.drawImage(arr[inc - 1], 0, 0, canvas.width, canvas.height);
arr[inc - 1].src = "";
//document.getElementById("myimg" + inc).style.display = "block";;
// document.getElementById("myimg" + (inc-1)).style.display = "none";
//imggg.src = arr[inc - 1].src;
}
$("#audiofile").val(inc);
// clearInterval(ref);
} catch (e) {
}
inc++;
// interval = 60;
//setTimeout(JLoop, interval);
}
else {
}
}
var ref = setInterval(JLoop, interval);
});
Worked for me on memory leak thanks dude.

Is it possible to capture/find the dimensions of a flash movie as it loads into a webpage?

I have a cms where my client loads banner ads from their clients. They don't know the dimensions or don't want to enter them when they upload the banners. Is it possible to capture the dimensions to set the object attributes as we grab the movies from the db to display in the page?
I read that the movie dimensions are available in the loaderinfo object, but this is a AS3 object right? Are AS3 objects available outside the flash movie? Can I access the loaderobject via javascript?
The Flash movies we're trying to load are not ours, we don't have access to the flas or anything like that so we can't edit anything to do with the movies themselves.

This might help
http://blog.codefidelity.com/?p=14
I think you can even use jquery with more ease
Try something like $("object").height() and $("object").width(); if you have just one object tag for flash .

You can also do something server side, if you want the size to be stored 100% accurately without user input.
This python script, for example, will output the compression type, version, size (compressed or uncompressed) and the dimensions.
#!/usr/bin/env python
import bitstring
import zlib
swf_file = file("test.swf", "rb")
swf = swf_file.read()
swf_file.close()
bits = bitstring.ConstBitStream(filename="test.swf")
type = "".join(map(chr, bits.readlist('uint:8, uint:8, uint:8')))
print "Compression: ", type
print "Version: ", bits.read('uint:8')
swf_length = bits.read('uintle:32')
if type == 'CWS':
print "Compressed Size: ", swf_length
f = zlib.decompress(bits.bytes[8:], zlib.MAX_WBITS)
bits = bitstring.ConstBitStream(bytes=f, length=swf_length * 8 - 64)
else:
print "Uncompressed Size: ", swf_length
nbits = bits.read('uint:5')
xmin, xmax, ymin, ymax = bits.readlist('uint:{size}, uint:{size}, uint:{size}, uint:{size}'.format(size = nbits))
print "Dimensions: {x}x{y}".format(x = (xmax - xmin) / 20, y = (ymax - ymin) / 20)
Example Output:
Compression: CWS
Version: 10
Compressed Size: 166608
Dimensions: 200x100