We Keep Coding

Chrome ERR_INSUFFICIENT_RESOURCES workaround

I have a JS library that is responsible to perform the download of JPEG images for the client. All of this is done asynchronously. In some cases, the count of images is really large... Around 5000 images. In this case, the Chrome browser issues the "ERR_INSUFFICIENT_RESOURCES" error for the ajax request.
Each request must be done individually, there is no option to pack the images on the server-side.
What are my options here? How can I find a workaround for this problem? The download works fine in Firefox...
Attached code of the actual download:
function loadFileAndDecrypt(fileId, key, type, length, callback, obj) {
var step = 100 / length;
eventBus.$emit('updateProgressText', "downloadingFiles");
var req = new dh.crypto.HttpRequest();
req.setAesKey(key);
let dataUrl;
if (type == "study") {
dataUrl = "/v1/images/";
}else {
dataUrl = "/v1/dicoms/";
}
var url = axios.defaults.baseURL + dataUrl + fileId;
req.open("GET", url, true);
req.setRequestHeader("Authorization", authHeader().Authorization+"")
req.setRequestHeader("Accept", "application/octet-stream, application/json, text/plain, */*");
req.responseType = "arraybuffer";
req.onload = function() {
console.log(downloadStep);
downloadStep += step;
eventBus.$emit('updatePb', Math.ceil(downloadStep));
var data = req.response;
obj.push(data);
counter ++;
//last one
if (counter == length) {
callback(obj);
}
};
req.send();
}

The error means your code is overloading your memory (most likely, or the quota of pending requests was exhausted). Instead of sending all the data from the backend, make your frontend request for 5000 individual images instead and control the requests flow. regardless, downloading 5000 images is bad. You should pack them up for downloading. If you mean fetching the images, then loading images from the frontend through static or dynamic links is much more logical ;)

Create a class:
Which accepts the file-Id (image that needs to be downloaded) as an argument
Which can perform the HTTP API request
Which can store the result of the request
Create an array of objects from this class using how many ever file-Ids that needs to be downloaded.
Store the array in a RequestManager which can start and manage the downloads:
can batch the downloads, say it fires 5 requests from the array and waits for them to finish before starting the next batch
can stop the downloads on multiple failures
manipulate batch size depending on the available bandwidth
stops download on auth expiry and resumes on auth refresh
offers to retry the previously failed downloads

How to display base64 image in Javascript

I am trying to stream Raspberry Pi camera, and for security reasons I'm tunneling my server through a secured server even though the client is on the same LAN. However, this resulted in horrible latency. I'm trying now to stream the video over the LAN instead of through the server. I know that browsers only allow HTTP connections over HTTPS servers only on passive elements, such as .
My server is saving the image as a local jpeg file encoded as base64:
camera = cv2.VideoCapture(0)
grabbed, frame = camera.read()
frame = cv2.resize(frame, (320, 240))
buffer = cv2.imencode('.jpg', frame)[1]
buffer = base64.b64encode(buffer)
buffer.decode('utf-8')
with open(ROOT + '/static/image.jpeg', mode='wb+') as image:
image.write(buffer)
my client has an image tag, and a simple script to request the file saved:
<img id='img'>
<script>
setInterval(function() {
var myImageElement = document.getElementById('img');
myImageElement.src = 'data:image/jpg;base64,http://10.0.0.35:8000/static/image.jpg?rand=' + Math.random();
}, 500);
</script>
The result is a constant stream of console errors such as:
data:image/jpg;charset=utf-8;base64,http://10.0.0.35:8000/static/image.jpg?rand=0.7520646586573345:1 GET data:image/jpg;charset=utf-8;base64,http://10.0.0.35:8000/static/image.jpg?rand=0.7520646586573345 net::ERR_INVALID_URL
I have checked that the image is accessible (entered http://10.0.0.35:8000/static/image.jpg in my broswer, and haven't gotten any errors)
I have also checked that the file is a base64 jpeg using an online tool.
I have looked at this question but nothing there worked,
I can't understand why am I getting this error, and how to solve it.
Can someone please direct me in the right direction?

To solve this issue, I used 2 different solutions:
[A]: I used mjpg-streamer to stream my video to LAN
mjpg-streamer is opening a local server to stream or capture images from camera on port 8080. so I've changed the src attribute of the image to:
[B]: myImageElement.src = http://10.0.0.35:8080
And that's it.

How to download file from URL and Alert user that file is downloaded successfully in Javascript? [duplicate]

I have an application that hosts videos, and we recently migrated to Azure.
On our old application we gave the ability for users to either play or download the video. However on Azure it seems like I have to pick between which functionality I want, as the content disposition has to be set on the file and not on the request.
So far I have came up with two very poor solutions.
The first solution is streaming the download through my MVC server.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings["StorageConnectionString"]);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("videos");
string userFileName = service.FirstName + service.LastName + "Video.mp4";
Response.AddHeader("Content-Disposition", "attachment; filename=" + userFileName); // force download
container.GetBlobReference(service.Video.ConvertedFilePath).DownloadToStream(Response.OutputStream);
return new EmptyResult();
This option works okay for smaller videos, but it is very taxing on my server. For larger videos the operation times out.
The second option is hosting every video twice.
This option is obviously bad, as I will have to pay double the storage cost.

However on Azure it seems like I have to pick between which
functionality I want, as the content disposition has to be set on the
file and not on the request.
There's a workaround for that. As you may know there's a Content-Disposition property that you can define on a blob. However when you define a value for this property, it will always be applied on that blob. When you want to selectively apply this property on a blob (say on a per request basis), what you do is create a Shared Access Signature (SAS) on that blob and override this request header there. Then you can serve the blob via SAS URL.
Here's the sample code for this:
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings["StorageConnectionString"]);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("videos");
string userFileName = service.FirstName + service.LastName + "Video.mp4";
CloudBlockBlob blob = container.GetBlockBlobReference(userFileName);
SharedAccessBlobPolicy policy = new SharedAccessBlobPolicy()
{
Permissions = SharedAccessBlobPermissions.Read,
SharedAccessExpiryTime = DateTime.UtcNow.AddHours(1)
};
SharedAccessBlobHeaders blobHeaders = new SharedAccessBlobHeaders()
{
ContentDisposition = "attachment; filename=" + userFileName
};
string sasToken = blob.GetSharedAccessSignature(policy, blobHeaders);
var sasUrl = blob.Uri.AbsoluteUri + sasToken;//This is the URL you will use. It will force the user to download the video.
I wrote a blog post about the same long time ago that you may find useful: http://gauravmantri.com/2013/11/28/new-changes-to-windows-azure-storage-a-perfect-thanksgiving-gift/.

As far as I know, azure blob storage doesn't support add the custom header to the special container.
I suggest you could follow and vote this feedback to push the azure develop team to support this feature.
Here is a workaround, you could compression the video file firstly, then uploaded to the azure blob storage.
It will not be opened by the browser.

Large file upload with WebSocket

I'm trying to upload large files (at least 500MB, preferably up to a few GB) using the WebSocket API. The problem is that I can't figure out how to write "send this slice of the file, release the resources used then repeat". I was hoping I could avoid using something like Flash/Silverlight for this.
Currently, I'm working with something along the lines of:
function FileSlicer(file) {
// randomly picked 1MB slices,
// I don't think this size is important for this experiment
this.sliceSize = 1024*1024;
this.slices = Math.ceil(file.size / this.sliceSize);
this.currentSlice = 0;
this.getNextSlice = function() {
var start = this.currentSlice * this.sliceSize;
var end = Math.min((this.currentSlice+1) * this.sliceSize, file.size);
++this.currentSlice;
return file.slice(start, end);
}
}
Then, I would upload using:
function Uploader(url, file) {
var fs = new FileSlicer(file);
var socket = new WebSocket(url);
socket.onopen = function() {
for(var i = 0; i < fs.slices; ++i) {
socket.send(fs.getNextSlice()); // see below
}
}
}
Basically this returns immediately, bufferedAmount is unchanged (0) and it keeps iterating and adding all the slices to the queue before attempting to send it; there's no socket.afterSend to allow me to queue it properly, which is where I'm stuck.

Use web workers for large files processing instead doing it in main thread and upload chunks of file data using file.slice().
This article helps you to handle large files in workers. change XHR send to Websocket in main thread.
//Messages from worker
function onmessage(blobOrFile) {
ws.send(blobOrFile);
}
//construct file on server side based on blob or chunk information.

I believe the send() method is asynchronous which is why it will return immediately. To make it queue, you'd need the server to send a message back to the client after each slice is uploaded; the client can then decide whether it needs to send the next slice or a "upload complete" message back to the server.
This sort of thing would probably be easier using XMLHttpRequest(2); it has callback support built-in and is also more widely supported than the WebSocket API.

In order to serialize this operation you need the server to send you a signal every time a slice is received & written (or an error occurs), this way you could send the next slice in response to the onmessage event, pretty much like this:
function Uploader(url, file) {
var fs = new FileSlicer(file);
var socket = new WebSocket(url);
socket.onopen = function() {
socket.send(fs.getNextSlice());
}
socket.onmessage = function(ms){
if(ms.data=="ok"){
fs.slices--;
if(fs.slices>0) socket.send(fs.getNextSlice());
}else{
// handle the error code here.
}
}
}

You could use https://github.com/binaryjs/binaryjs or https://github.com/liamks/Delivery.js if you can run node.js on the server.

EDIT : The web world, browsers, firewalls, proxies, changed a lot since this answer was made. Right now, sending files using websockets
can be done efficiently, especially on local area networks.
Websockets are very efficient for bidirectional communication, especially when you're interested in pushing information (preferably small) from the server. They act as bidirectional sockets (hence their name).
Websockets don't look like the right technology to use in this situation. Especially given that using them adds incompatibilities with some proxies, browsers (IE) or even firewalls.
On the other end, uploading a file is simply sending a POST request to a server with the file in the body. Browsers are very good at that and the overhead for a big file is really near nothing. Don't use websockets for that task.

I think this socket.io project has a lot of potential:
https://github.com/sffc/socketio-file-upload
It supports chunked upload, progress tracking and seems fairly easy to use.

WebSocket JavaScript: Sending complex objects

I am using WebSockets as the connection between a Node.js server and my client JS code.
I want to send a number of different media types (Text, Audio, Video, Images) through the socket.
This is not difficult of course. message.data instanceof Blob separates text from media files. The problem is, that I want to include several additional attributes to those media files.
F.e.:
Dimension of an image
Name of the image
. . .
Now I could send one message containing these informations in text form and follow it up with another message containing the blob.
I would very much prefer though, to be able to build an object:
imageObject = {
xDimension : '50px',
yDimension : '50px',
name : 'PinkFlowers.jpg'
imageData : fs.readFileSync(".resources/images/PinkFlowers.jpg")
}
And send this object as it is via socket.send(imageObject).
So far so good, this actually works, but how do I collect the object and make its fields accessible in the client again?
I have been tampering with it for a while now and I would be grateful for any ideas.
Best regards,
Sticks

Well I did get it to work using base64.
On the server side I am running this piece of code:
var imageObject = newMessageObject('img', 'flower.png');
imageObject.image = new Buffer(fs.readFileSync('./resources/images/flower.png'), 'binary').toString('base64');
imageObject.datatype = 'png';
connection.send(JSON.stringify(imageObject));
The new Buffer() is necessary to ensure a valid utf encoding. If used without, Chrome(dont know about Firefox and others) throws an error, that invalid utf8 encoding was detected and shuts down the execution after JSON.parse(message).
Note: newMessageObject is just an object construction method with two fields, type and name which I use.
On the client side its really straight forward:
websocketConnection.onmessage = function(evt) {
var message = JSON.parse(evt.data);
... // Some app specific stuff
var image = new Image();
image.onload = function() {
canvas.getContext("2d").drawImage(image, 0, 0);
}
image.src = "data:image/" + message.datatype + ";base64," + message.image;
}
This draws the image on the canvas.
I am not convinced, that this is practicable for audio or video files, but for images it does the job.
I will probably fall back to simply sending an obfuscated URL instead of audio/video data and read the files directly from the server. I dont like the security implications though.