I have a pretty run-of-the-mill node.js server with expressjs installed locally for development purposes; I store various files and request them via a basic HTTP call that returns the file through express' res.download feature. Most of the time, this works without a hitch. For a very small subset of files, however, the end-user receives a file that is much larger than expected (almost 2x) and is unreadable by any conventional viewer. Out of maybe a hundred files, this has only happened twice, and both were JPG files, but the sample is too small to draw any conclusion. What I know :
The issue is reproductible : if it happens with a file, it always happens;
The issue is not related to the way files are stored : if I swap the problematic file with another one but keep everything else the same (name, location, etc.), it works fine;
Right before the res.download happens, the file is okay : checking its size with fs.stats returns the correct value
The HTTP response encounters no visible problem : no error, 200 response code...
The source file seems to have normal metadata and JPG markers
UPDATE I did some tests, and the issue seems to be somehow related to encoding : the mangled response file is, for reasons unknown, encoded in UTF-8 ; the size discrepancy comes from all non-UTF-8 characters being replaced by EF BF BD (the unknown character symbol) ! I still can't understand why it happens, what makes these few files different from others, and if it can be detected and/or corrected upstream.
UPDATE 2 After some additional tests, I still can't quite pinpoint the cause, but I can add the following info :
Systems-wise, the issue happens during the data streaming in fs.js
The root cause is located somewhere in the EXIF data of the image
For those interested, the source image (source.JPG) and download result (response.JPG) can be found here : http://www.sycomor.fr/test/ ; I also added a similar image that isn't affected by the download and comes out clean. For what it's worth, both pictures were taken minutes apart, with the same camera at the same settings, so I strongly doubt the issue is caused by some external source.
Thanks !
Your issue comes from 'connect-livereload' in your express configuration.
It corrupts binary stream while injecting the reloading script.
Refer to https://github.com/intesso/connect-livereload/issues/39 for details.
Related
My Blazor Server app uses a canvas element to manipulate an image that is loaded into an img element via a file drop zone, it serves as a reference for the canvas and holds the information that is worked on via JS. I'm using Excubo's Canvas wrapper to work within C# as much as possible but I don't think this error is related to it as it also happens with pure JS. Since this is more of a logic problem I don't think sharing code would be much help but here is a crud visual representation of what I mean, perhaps it will be more useful in trying to convey the problem.
The end goal is to invoke a function in JS to load the image src into a secondary work canvas, and using the data present in the main canvas, manipulate pixel information accordingly for both. After parsing the data, I convert it back toDataURL and set the img.src attribute.
The problem happens when returning the base64 data from that function particularly when using large images. The server disconnects immediately and I'm not sure if there is a way to extend the allowed wait time or this is just an implementation problem on my part.
Also worth mentioning that I extended the maximum file size for the dropzone InputFile component to 30MB but this happens with files of 3MB (and maybe even lower but I couldn't troubleshoot the threshold value yet).
Perhaps there's another way to achieve what I'm aiming for but I'm not sure how I would reference the img src attribute in C# without JS. Client-Server side logic isn't a concern because this app is supposed to be run locally on the server machine only so they are essentially the same target.
Found a solution that works for my case. Here is the reference for the fix.
The default MaximumReceiveMessageSize is too low for my needs, I increased it to 30MB and it works without disconnecting from server. Also tried increasing the ApplicationMaxBufferSize and TransportMaxBufferSize on the MapBlazorHub pipeline configurations but it didn't help. Increasing JS interop timeout on the AddServerSideBlazor service options was also unfruitful.
Here is the relevant Program.cs code I'm using now:
var maxBufferSize = 30 * 1024 * 1024;
builder.Services.AddServerSideBlazor().AddHubOptions(opt => { opt.MaximumReceiveMessageSize = maxBufferSize; });
The R rgl package exports an HTML widget with the rglwidget() function, built using the htmlwidgets package. Often the data for this widget is quite large, and Pandoc and webshot2 choke on it.
I would like to try compressing the data when the HTML page is created, and uncompressing it in Javascript before display. I can see that there's a Javascript package pako that appears to do what I want, and it can be "browserified", but I can't see how to make it available to rglwidget(). Can anyone describe what's necessary?
Edited to add some more detail in response to a comment: This needs to happen on the server (or actually, even before the file gets to the server). htmlwidgets produces output in a Markdown document that is converted to HTML by Pandoc, and that step is failing because Pandoc chokes on the large JSON datasets. I'm not sure if it's just the size, or the complexity (I think Pandoc parses the JSON, and appears to blow up to a huge memory footprint before it crashes). I'm hoping that by using a base64 blob Pandoc will handle it better. webshot2 also has problems that may be the same.
2nd Edit: I've got some evidence that it's the size that matters, not the complexity. I used base64 encoding on the JSON to make it simpler (just one long string), but 33% bigger, and things were worse. So I'm back to thinking that compression would help.
$(function(){
$("#testdiv").load("sometext.txt",function(){
alert("Succes");
});
});
I get the alert but I am not able to see anything in the "#testdiv".The file,sometext.txt is in the same folder.I tried giving it absolute path but i get 404 ,url not found on this server.
What am I doing wrong here?
Windows has several "features" that are built to make their operating system more usable for most people. Hiding the file extension is one of those "features" that often do more harm than how much they help.
Though, as you might know, a file doesn't need a file extension, the first few bytes of the file are used to tell the OS what file type it is, the extension just makes it easy.
When using windows to develop anything, you probably want to see the actual name of your files, so make sure to check out one of the guides on how to enable displaying the file extension, like this one.
If you can't GET a file with JS always double, or better even tripple check, that the file can be accessed through the browser first.
I'm trying to save a 2gb variable into a file.This variable comes from webtorrent. I tried to use the blob url , but it crashes as it is bigger than max blob size. I also tried StreamSaver.js and got 2 different problems. By writting periodically while I download the variable, streamSaver fails at some point (no idea why, it writes like 500 mb then crashes). By writting it all at the end it obviously crashes the tab. I even tried to write it at the end but chunks by chunks but nothing seems to get this working. So I was wondering if you guys knew how I could save this variable to a file (any API, tricks, I'll take anything)
When I try to load a very large file using the appropriate loaders provided with the library, the tab my website runs in crashes. I have tried implementing the Worker class, but it doesnt seem to work. Heres what happens:
In the main javascript file I have:
var worker = new Worker('loader.js');
When user selects one of available models I check for the extension and pass the file URL/path to the worker: (in this instance a pcd file)
worker.postMessage({fileType: "pcd", file: file});
Now the loader.js has the appropriate includes that are necessary to make it work:
importScripts('js/libs/three.js/three.js');
importScripts('js/libs/three.js/PCDLoader.js');
and in its onmessage method, it uses the apropriate loader depending on file extension.
var loader = new THREE.PCDLoader();
loader.load(file, function (mesh) {
postMessage({points: mesh.geometry.attributes.position.array, colors: mesh.geometry.attributes.color.array});
});
The data is passed back successfully to the main javascript which adds it to the scene. At least for small files - large ones, like I said, take too long and the browser decides there was an error. Now I thought the worker class was supposed to work asynchronously, so whats the deal here?
Currently Three.js's loaders rely on strings and arrays of strings to parse data from a file. They dont split files into pieces, which leads to excessive memory usage which browsers immediately interrupt. Loading a 64 MB file spikes to over 1 GB memory used during load (which then results in an error).