I am trying to write some HTML/JS code which will facilitate uploading large files (multi-GB) to a remote server. Previously we had been using a flash uploader which uploaded a given file in a single network request. The flash would open a network connection, read a chunk of a file into memory then write that chunk to the network connection then grab the next chunk then write to the network etc. etc. until the entire file is uploaded. It was done this way because most web browsers will attempt to read an entire file into memory before attempting to upload. When dealing with multi-GB files, this essentially crashes the client system because it uses all of the client memory. Now we are having issues with using flash, so it needs to go, we want to replace it without needing to modify the existing server-side code.
A few google searches for jquery uploaders reveals that there are plenty of libraries which support "chunking" but they "chunk" over multiple requests. We do not want to chunk a file over multiple network requests, we merely want the JS to read the file in chunks as it writes the file to a single network connection.
Anybody know a library which can do this out of the box?
We are not opposed to modifying an existing library if need be. Anyone have a snippet that resembles the bellow pseudo-code that I may be able to retrofit into a library?
connection = fopen(...);
fputs("123", connection);
... some unrelated code ...
fputs("456", connection);
fclose(connection);
(excuse my use of C functions in pseudo JS code ... I know that is not how you do it in JS, I am merely demonstrating at a low-level the flow for how I want to write to the network connection before closing it)
NOTE: We are not trying to "modernize" or improve this project extensively -- we are not trying to re-do this project. We have some old code that has sat here for years and we want to make as few changes to the server-side code as possible. I have more important projects to modernize and make more efficient -- this one we just need to work. Please don't advise me to impliment "proper" file chunking on the server side -- that was my suggestion, and if my suggestion were taken then that task would have been assigned to a different developer. Out of my control now, this is a client-side-only fix please!
Thanks, sorry for any headache!
You could try binaryjs. I haven't looked into the internals but I know it supports manually setting the chunk size. Maybe you can even set it to Infinity.
Specifically you could try:
var client = new BinaryClient('example.com', { chunkSize: Number.POSITIVE_INFINITY });
client.send('data...');
Note: binaryjs is a NodeJS server library, and a browser-compatible client library.
Related
I have a node web app that needs to convert a docx file into pdf (using client side resources only and no plugins). I've found a possible solution by converting my docx into HTML using docxjs and then HTML to PDF using jspdf (docx->HTML->PDF).
This solution could make it but I encountered several issues especially with rendering. I know that docxjs doesn't keep the same rendering in HTML as the docx file so it is a problem...
So my question is do you know any free module/solution that could directly do the job without going through HTML (I'm open to odt as a source as well)? If not, what would you advise me to do?
Thanks
As you already know there is no ready-to-use and open libs for this.. You just can't get good results with available variants. My suggesition is:
Use third party API. Like https://market.mashape.com/convertapi/word2pdf-1#!documentation
Create your own service for this purpose. If you have such ability, I suggest to create a small server on node.js (I bet you know how to do this). You can use Libreoffice as a good converter with good render quality like this:
libreoffice -headless -invisible -convert-to pdf {$file_name} -outdir /www-disk/
Don't forget that this is usually takes a lot of time, do not block the request-answer flow: use separate process for each convert operation.
And the last thing. Libreoffice is not very lightweight but it has good quality. You can also find notable unoconv tool.
As of January 2019, there is docx-wasm, which works in node and performs the conversion locally where node is installed. Proprietary but freemium.
It appears that even after three years ncohen had not found an answer. It was also unclear if it had to be a free (as in dollars) solution.
The original requirements were:
using client side resources only and no plugins
Do you mean you don't want server side conversion? Right, I would like my app to be totally autonomous.
Since all the other answers/comments only offered server side component solutions, which the author clearly stated was not what they wanted, here is a proposed answer.
The company I work for has had this solution for a few years now, that can convert DOCX (not odt yet) files to PDF completely in the browser, with no server side component required. This currently uses either asm.js/PNaCl/WASM depending on the exact browser being used.
https://www.pdftron.com/samples/web/samples/viewing/viewing/
Open an office file using the demo above, and you will see no server communication. Everything is done client side. This demo works on mobile browsers also.
After searching around in Google for a while I have not had any luck or guidance in my question.
I want to be able to load up a website using javascript, ajax, in order to reduce the amount of requests needed by the server from the client. My goal is to embed/encode data within an image such that only the client needs to request this image through ajax call, and then be decoded to find the js, css, and other files needed. Then the js, css and other files will be inserted into the DOM.
If I can get the above to work then I could have a lot of flexibility on how my webapp is loaded and be able to notify the user how close the webapp is to being ready for viewing.
Currently my problem is that I cannot find how I would encode the data within an image.
Even if this is not the way to be going about serving up a webapp my curiosity is getting the best of me and I would just really like to do this.
Any guidance or pointers would be greatly appreciated!
Also: I am learning Python so if you know of a python module that I could play with that would be cool. Currently i'm playing with the pypng module to see if this could be done.
To be frank. Don't do that.
The brightest minds on earth use other methods to keep the number of requests and response time down. The most common technique for minimizing the number of requests is called Bundling. In short, you just copy'n paste all js files after each other into one big js file and all the css files into one big css file. This way you need to download two files, one js and one css. Better than that is usually not worth the trouble.
To further keep response times down you usually minify your js and css files. This is a process where all white space, comments, etc are removed and internal variable names are made as short as possible.
Finally you can serve both js and css files as gziped files to further reduce the file size to transfer.
There are many tools out there that does both bundling and minification for you. Google and pick one that suits your other tooling support.
I would like to implement an in-browser Microsoft Word document merge feature that will convert the merged document into PDF and offer it to the user for download. I would like to this process to be supported in Google Chrome and Firefox. Here is how I would like it to work:
Client-side JavaScript obtains the Word template document in docx format, either from a server, or by asking the user for a file upload (which it can then read using the FileReader API)
The JavaScript uses its local data structures (e.g., data lists it has obtained via Ajax) to expand the template into a document. It can do this either directly, by unzipping the docx file and processing its contents, or using DOCx.js. The template expansion is just a matter of substituting template variables with values obtained from the local data structures.
The JavaScript then converts the expanded template into PDF.
The JavaScript offers the PDF file to the user for download, e.g., using Downloadify.
The difficulty I am having is in step 3. My understanding (based on all the Googling I have done so far) is that I have the following options:
Require that the local machine is a Windows machine, and invoke Word on it, to convert to PDF. This can be done using a little bit of scripting using WScript.shell, and it looks doable with Internet Explorer. But based on what I have read, it doesn't look like I can call WScript.shell from within either Chrome or Firefox, because of their security constraints.
I am open to trying Silverlight to do the conversion, but I have not found enough documentation on how to do this. Ideally, if I used Silverlight, I would like to write the Silverlight code in JavaScript, because (a) I don't know much CSharp, and (b) I think it would be much easier in JavaScript.
Create a web service that will convert a given docx file to a pdf file, and invoke that service via Ajax. I would rather not do this, if possible, for a few reasons: (a) I tried using docx4java (I am a reasonably skilled Java programmer) but the conversion process is far too slow, and it does not preserve document content very well; and (b) I would like to avoid a call out to the network, to avoid security issues. It does seem possible to write a little service on a Windows server for doing the conversion, and if there is no other good option, I might go that route.
If I have been unclear about anything, please let me know. I would appreciate your ideas and feedback.
I love command line tools.
Load the doc to your server and use LibreOffice to convert it to PDF via the command line
soffice.exe --headless --convert-to pdf --outdir E:\Docs\Out E:\Docs\In\a.doc
You can display a progress bar to the user and when complete give them the option to download the doc.
More info on LibreOffice's command line parameters go here
Done.
Old old question now, but for anyone who stumbles across this, web assembly (wasm) now makes this sort of approach possible.
We've just released https://www.npmjs.com/package/#nativedocuments/docx-wasm which can perform the conversion locally.
Is it possible with Javascript or jQuery to convert mp3, wav, etc. to m4r format?
Let's assume you had a library that can change the format of files.
Let's also assume you only need the application to work on current browsers that implement FileAPI or FileReference so you can have access to uploaded files (you can't have access to them without FileAPI or FileReference unless you use Flash or Java Applets or equivalent technologies).
You wouldn't be able to write the output file back to the user because JavaScript is not allowed to access the local filesystem.
Your only solution would become sending the converted file to the server and the server sending it back to you with a force download directive so that the user will be prompted to download the results.
Now back to if there were a library that can the conversion (or even native JavaScript)... I haven't heard of any. It's not impossible to build one but it is impractical and wouldn't run very fast.
Edit:
Let's not forget Node.js!
It's a backend server that uses Google Chrome's V8 JavaScript interpretor/compiler. And it runs JavaScript as a backend scripting engine.
You have access to filesystem, databases and everything if you use that (or any other backend system for that matter) and still be using JavaScript. You can use libraries too. Either written in JavaScript or libraries written in other languages that have been linked to interface with Node.js.
Edit 2:
There is a PC emulator written entirely in JavaScript. It runs binary executables if you want it to. It's called JSLinux.
If you're feeling particularly rambunctious you can grab the ffmpeg binary executable (compiled with static linking). And embed it into your application code as an uuencoded string then use JSLinux to execute the commands and grab the results.
Indeed, it is possible doing this on the client using the latest js technologies. A web-worker thread can do the work in the background. At least in Firefox and Chrome it is also possible to read ("upload in memory") and write ("download from memory") files using the new W3C File API, see here.
I managed to read files via drag&drop from and within the client using google's GWT which in the end is plain javascript, so it must also be possible to do it "natively".
Besides that, the conversation algorithm of course has to be implemented in a javascript web worker to avoid blocking the gui. This should be the hardest part, but not impossible, though.
You would need a backend to do this. You may want to look into the PHPExtension of FFmpeg
I have a Rails application that has some JavaScript that needs to parse CSVs and make some AJAX calls based on each record.
I'd like to just load the local CSV directly into browser memory and have the JavaScript parse it and make the required AJAX calls but I haven't found a cross-browser, dependable way to accomplish this (I need to support cruddy old IE6).
I could upload the CSV to my rails application but I plan on hosting the application on Heroku and as I understand it, Heroku doesn't allow you to edit the files system(create files). I could also write the CSVs to a database but these CSVs are large 10mb+ and I imagine I will undoubtedly suffer performance costs in doing this.
Is my best option pushing the CSV to Rails and having Rails respond with a JSON or string version of the CSV? This seems somewhat computationally expensive given the size of these CSVs. I'd prefer to keep it on the client-side. If that's the case can someone point me to an example on how to accomplish this or something similar?
Edit: I don't want users to have to copy and paste these CSVs into a textfield manually.
Edit2: Also, I'm aware of the security restrictions on accessing the local filesystem via JS. A solid flash embed is an acceptable option.