We Keep Coding

How to send binary file to the server using JavaScript

I'm working on file encryption for my messenger and I'm struggling with uploading the file after encryption is done.
The encryption seems fine in terms of performance, but when I try to make an upload, the browser hangs completely. Profiler writes "small GC" events infinitely, and the yellow bar about the hung up script is appearing every 10 seconds.
What I already tried:
Read the file with FileReader to ArrayBuffer, then turn it into a basic Array, encrypt it, then create a FormData object, create a File from the data, append it to FormData and send. It worked fast with original, untouched file around 1.3 Mb in size when I did not do the encryption, but on the encrypted "fake" File object after upload I get file with 4.7 Mb and it was not usable.
Send as a plain POST field (multipart formdata encoding). The data is corrupted after it is saved on PHP this way.
Send as a Base64-encoded POST field. Finally it started working this way after I found a fast converting function from binary array to Base64 string. btoa() gave wrong results after encode/decode. But after I tried a file of 8.5 Mb in size, it hung again.
I tried moving extra data to URL string and send file as Blob as described here. No success, browser still hangs.
I tried passing to Blob constructor a basic Array, a Uint8Array made of it, and finally I tried to send File as suggested in docs, but still the same result, even with small file.
What is wrong with the code? HDD load is 0% when this hang happens. Also the files in question are really very small
Here is what I get on the output from my server script when I emergency terminate the JS script by pressing the button:
Warning: Unknown: POST Content-Length of 22146226 bytes exceeds the limit of 8388608 bytes in Unknown on line 0
Warning: Cannot modify header information - headers already sent in Unknown on line 0
Warning: session_start() [function.session-start]: Cannot send session cache limiter - headers already sent in D:\xmessenger\upload.php on line 2
Array ( )
Here is my JavaScript:
function uploadEncryptedFile(nonce) {
if (typeof window['FormData'] !== 'function' || typeof window['File'] !== 'function') return
var file_input = document.getElementById('attachment')
if (!file_input.files.length) return
var file = file_input.files[0]
var reader = new FileReader();
reader.addEventListener('load', function() {
var data = Array.from(new Uint8Array(reader.result))
var encrypted = encryptFile(data, nonce)
//return //Here it never hangs
var form_data = new FormData()
form_data.append('name', file.name)
form_data.append('type', file.type)
form_data.append('attachment', arrayBufferToBase64(encrypted))
/* form_data.append('attachment', btoa(encrypted)) // Does not help */
form_data.append('nonce', nonce)
var req = getXmlHttp()
req.open('POST', 'upload.php?attachencryptedfile', true)
req.onload = function() {
var data = req.responseText.split(':')
document.getElementById('filelist').lastChild.realName = data[2]
document.getElementById('progress2').style.display = 'none'
document.getElementById('attachment').onclick = null
encryptFilename(data[0], data[1], data[2])
}
req.send(form_data)
/* These lines also fail when the file is larger */
/* req.send(new Blob(encrypted)) */
/* req.send(new Blob(new Uint8Array(encrypted))) */
})
reader.readAsArrayBuffer(file)
}
function arrayBufferToBase64(buffer) {
var binary = '';
var bytes = new Uint8Array(buffer);
var len = bytes.byteLength;
for (var i = 0; i < len; i++) {
binary += String.fromCharCode(bytes[i]);
}
return window.btoa(binary);
}
Here is my PHP handler code:
if (isset($_GET['attachencryptedfile'])) {
$entityBody = file_get_contents('php://input');
if ($entityBody == '') exit(print_r($_POST, true));
else exit($entityBody);
if (!isset($_POST["name"])) exit("Error");
$name = #preg_replace("/[^0-9A-Za-z._-]/", "", $_POST["name"]);
$nonce = #preg_replace("/[^0-9A-Za-z+\\/]/", "", $_POST["nonce"]);
if ($name == ".htaccess") exit();
$data = base64_decode($_POST["attachment"]);
//print_r($_POST);
//exit();
if (strlen($data)>1024*15*1024) exit('<script type="text/javascript">parent.showInfo("Файл слишком большой"); parent.document.getElementById(\'filelist\').removeChild(parent.document.getElementById(\'filelist\').lastChild); parent.document.getElementById(\'progress2\').style.display = \'none\'; parent.document.getElementById(\'attachment\').onclick = null</script>');
$uname = uniqid()."_".str_pad($_SESSION['xm_user_id'], 6, "0", STR_PAD_LEFT).substr($name, strrpos($name, "."));
file_put_contents("upload/".$uname, $data);
mysql_query("ALTER TABLE `attachments` AUTO_INCREMENT=0");
mysql_query("INSERT INTO `attachments` VALUES('0', '".$uname."', '".$name."', '0', '".$nonce."')");
exit(mysql_insert_id().":".$uname.":".$name);
}
HTML form:
<form name="fileForm" id="fileForm" method="post" enctype="multipart/form-data" action="upload.php?attachfile" target="ifr">
<div id="fileButton" title="Прикрепить файл" onclick="document.getElementById('attachment').click()"></div>
<input type="file" name="attachment" id="attachment" title="Прикрепить файл" onchange="addFile()" />
</form>

UPD: the issue is not solved, unfortunately. My answer is only partially correct. Now I made a silly mistake in the code (forgot to update the server side), and I found another cause of possible hang. If I submit a basic POST form (x-www-urlencoded) and code in the PHP script tries to execute this line ($uname is defined, $_FILES is an empty array)
if (!copy($_FILES['attachment']['tmp_name'], "upload/".$uname)) exit("Error");
then the whole thing hangs again. If I terminate the script, the server response is code 200, and the body contents are just fine (I have error output on on my dev machine). I know it is a bad thing - calling copy with the first argument which is undefined at all, but even server error 500 must not hang the browser in such a way (btw, new latest version of Firefox is also affected).
I have Apache 2.4 on Windows 7 x64 and PHP 5.3. Can someone please verify this thing? Maybe a bug should be filed to Apache/Firefox team?
Oh my God. This terrible behavior was caused by... post_max_size = 8M set in php.ini. And files smaller than 8 Mb actually did not hang the browser, as I figured it out.
Last question is - why? Why cannot Apache/PHP (I have Apache 2.4 btw, it is not old) somehow gracefully abort the connection, telling the browser that the limit is exceeded? Or maybe it is a bug in XHR implementation, and is not applicable to basic form submit. Anyway, will be useful for people who stumble upon it.
By the way, I tried it in Chrome with the same POST size limit, and it does not hang there completely like in Firefox (the request is still in some hung up condition with "no response available", but the JS engine and the UI are not blocked ).

Javascript, Typescript, Angular 5 - Open and read file

I am working with Angular 5, I have an application in which I need to read an AMP HTML file as text. This file is contained in a component and should only be accessed from this component.
I would like to be able to open the file in read-only by giving its name.
I'm actually searching for something like this:
let file = open('amp.html');
Is it possible? If not how can I do to achieve this?

If you're writing browserside JS
You can't just simply read a file. The JS is running on your browser, and you need to think about where you're getting that file from.
If the file is on a server, you need to fetch that file from the server first by making a request for it.
If you're reading a file on the user's computer, you're gonna be using the File API on the browser to allow the user to select that file.
If you're writing backend JS
Assuming you're using NodeJS, you can conduct file operations like you would with other programming languages. Check out the fs module

If i understand you correct, you can read it as text like this:
function readFile(file){
var raw = new XMLHttpRequest(); // create a request
raw.open("GET", file, false); // open file
raw.onreadystatechange = function (){ // file is ready to read
if(raw.readyState === 4){
if(raw.status === 200 || raw.status == 0){
var allText = raw.responseText;
alert(allText); // can be also console.logged, of course.
}
}
}
raw.send(null); // return control
}
usage:
readFile('link.html')
I solved this issue thankfully to this question.

check uploaded file format on client side

I am creating a web portal where end user will upload a csv file and I will do some manipulation on that file on the server side (python). There is some latency and lag on the server side so I dont want to send the message from server to client regarding the bad format of uploaded file. Is there any way to do heavy lifting on client side may be using js or jquery to check if the uploaded file is "comma" separated or not etc etc?
I know we can do "accept=.csv" in the html so that file extension has csv format but how to do with contents to be sure.

Accessing local files from Javascript is only possible by using the File API (https://developer.mozilla.org/en-US/docs/Using_files_from_web_applications) - by using this you might be able to check the content whether it matches your expectations or not.

Here's some bits of code I used to display a preview image clientside when a file is selected. You should be able to use this as a starting point to do something else with the file data. Determining whether its csv is up to you.
Obvious caveat:
You still have to check server side. Anyone can modify your clientside javascript to pretend a bad file is good.
Another caveat:
I'm pretty sure that you can have escaped comma characters in a valid csv file. I think the escape character might be different across some implementations too...
// Fired when the user chooses a file in the OS dialog box
// They will have clicked <input id="fileId" type="file">
document.getElementById('fileId').onchange = function (evt) {
if(!evt.target.files || evt.target.files.length === 0){
console.log('No files selected');
return;
}
var uploadTitle = evt2.target.files[0].name;
var uploadSize = evt2.target.files[0].size;
var uploadType = evt2.target.files[0].type;
// To manipulate the file you set a callback for the whole contents:
var FR = new FileReader();
// I've only used this readAsDataURL which will encode the file like data:image/gif;base64,R0lGODl...
// I'm sure there's a similar call for plaintext
FR.readAsDataURL($('#file')[0].files[0]);
FR.onload = function(evt2){
var evtData = {
filesEvent: evt,
}
var uploadData = evt2.result
console.log(uploadTitle, uploadSize, uploadType, uploadData);
}
}

Saving text from website using Firefox extension, wrong characters saved

Sorry about the vague title but I'm a bit lost so it's hard to be specific. I've started playing around with Firefox extensions using the add-on SDK. What I'm trying to to is to watch a page for changes, a Twitch.tv chat window in this case, and save those changes to a file.
I've gotten this to work, every time something changes on the page it gets saved. But, "unusual" characters like for example something in Korean doesn't get saved properly. I think this has to do with encoding of the file/string? I tried saving the same characters by copy-pasting them into notepad, it asked me to save in Unicode and when I did everything worked fine. So I figured, ok, I'll change the encoding of the log file to unicode as well before writing to it. Didn't exactly work... Now all the characters were in some kind of foreign language.
The code I'm using to write to the file is this:
var {Cc, Ci, Cu} = require("chrome");
var {FileUtils} = Cu.import("resource://gre/modules/FileUtils.jsm");
var file = FileUtils.getFile("Desk", ["mylogfile.txt"]);
var stream = FileUtils.openFileOutputStream(file, FileUtils.MODE_WRONLY | FileUtils.MODE_CREATE | FileUtils.MODE_APPEND);
stream.write(data, data.length);
stream.close();
I looked at the description of FileUtils.jsm over at MDN and as far as I can tell there's no way to tell it which encoding I want to use?
If you don't know a fix could you give me some good search terms because I seem to be coming up short on that front. Since I know basically nothing on the subject I'm flailing around in the dark a bit at the moment.
edit:
This is what I ended up with (for now) to get this thing working:
var {Cc, Ci, Cu} = require("chrome");
var {FileUtils} = Cu.import("resource://gre/modules/FileUtils.jsm");
var file = Cc['#mozilla.org/file/local;1']
.createInstance(Ci.nsILocalFile);
file.initWithPath('C:\\temp\\temp.txt');
if(!file.exists()){
file.create(file.NORMAL_FILE_TYPE, 0666);
}
var charset = 'UTF-8';
var fileStream = Cc['#mozilla.org/network/file-output-stream;1']
.createInstance(Ci.nsIFileOutputStream);
fileStream.init(file, FileUtils.MODE_WRONLY | FileUtils.MODE_CREATE | FileUtils.MODE_APPEND, 0x200, false);
var converterStream = Cc['#mozilla.org/intl/converter-output-stream;1']
.createInstance(Ci.nsIConverterOutputStream);
converterStream.init(fileStream, charset, data.length,
Ci.nsIConverterInputStream.DEFAULT_REPLACEMENT_CHARACTER);
converterStream.writeString(data);
converterStream.close();
fileStream.close();

Dumping just the raw bytes (well, raw jschars actually) won't work. You need to first convert the data into some sensible encoding.
See e.g. the File I/O Snippets. Here are the crucial bits of creating a converter output stream wrapper:
var converter = Components.classes["#mozilla.org/intl/converter-output-stream;1"].
createInstance(Components.interfaces.nsIConverterOutputStream);
converter.init(foStream, "UTF-8", 0, 0);
converter.writeString(data);
converter.close(); // this closes foStream
Another way is to use OS.File + TextConverter:
let encoder = new TextEncoder(); // This encoder can be reused for several writes
let array = encoder.encode("This is some text"); // Convert the text to an array
let promise = OS.File.writeAtomic("file.txt", array, // Write the array atomically to "file.txt", using as temporary
{tmpPath: "file.txt.tmp"}); // buffer "file.txt.tmp".
It might be even possible to mix both. OS.File has the benefit that it will write data and access files off the main thread (so it won't block the UI while the file is being written).

Use FileAPI to download big generated data file

The JavaScript process generates a lot of data (200-300MB). I would like to save this data for further analysis but the best I found so far is saving using this example http://jsfiddle.net/c2U2T/ which is not an option for me, because it looks like it requires all the data being available before starting the downloading. But what I need is something like
var saver = new Saver();
saver.save(); // The Save As ... dialog appears
saver.onaccepted = function () { // user accepted saving
for (var i = 0; i < 1000000; i++) {
saver.write(Math.random());
}
};
Of course, instead of the Math.random() will be some meaningful construction.

#dader - I would build upon dader's example.
Use HTML5 FileSystem API - but instead of writing to the file each and every line (more IO than it is worth), you can batch some of the lines in memory in a javascript object/array/string, and only write it to the file when they reach a certain threshold. You are thus appending to a local file as the process chugs (makes it easy to pause/restart/stop etc)
Of note is the following, which is an example of how you can spawn the dialoge to request the amount of data that you would need (it sounds large). Tested in chrome.:
navigator.persistentStorage.queryUsageAndQuota(
function (usage, quota) {
var availableSpace = quota - usage;
var requestingQuota = args.size + usage;
if (availableSpace >= args.size) {
window.requestFileSystem(PERSISTENT, availableSpace, persistentStorageGranted, persistentStorageDenied);
} else {
navigator.persistentStorage.requestQuota(
requestingQuota, function (grantedQuota) {
window.requestFileSystem(PERSISTENT, grantedQuota - usage, persistentStorageGranted, persistentStorageDenied);
}, errorCb
);
}
}, errorCb);
When you are done you can use Javascript to open a new window with the url of that blob object that you saved which you can retrieve via: fileEntry.toURL()
OR - when it is done crunching you can just display that URL in an html link and then they could right click on it and do whatever Save Link As that they want.
But this is something that is new and cool that you can do entirely in the browser without needing to involve a server in any way at all. Side note, 200-300MB of data generated by a Javascript Process sounds absolutely huge... that would be a concern for whether you are storing the "right" data...

What you actually are trying to do is a kind of streaming. I mean FileAPI is not suited for the task. Instead, I could suggest two options :
The first, using XHR facility, ie ajax, by splitting your data into several chunks which will sequencially be sent to the server, each chunk in its own request along with an id ( for identifying the stream ) and a position index ( for identifying the chunk position ). I won't recommend that, since it adds work to break up and reassemble data, and since there's a better solution.
The second way of achieving this is to use Websocket API. It allows you to send data sequentially to the server as it is generated. Following a usual stream API. I think you definitely need this.
This page may be a good place to start at : http://binaryjs.com/
That's all folks !
EDIT considering your comment :
I'm not sure to perfectly get your point though but, what about HTML5's FileSystem API ?
There are a couple examples here : http://www.html5rocks.com/en/tutorials/file/filesystem/ among which this sample that allows you to append data to an existant file. You can also create a new file, etc. :
function onInitFs(fs) {
fs.root.getFile('log.txt', {create: false}, function(fileEntry) {
// Create a FileWriter object for our FileEntry (log.txt).
fileEntry.createWriter(function(fileWriter) {
fileWriter.seek(fileWriter.length); // Start write position at EOF.
// Create a new Blob and write it to log.txt.
var blob = new Blob(['Hello World'], {type: 'text/plain'});
fileWriter.write(blob);
}, errorHandler);
}, errorHandler);
}
EDIT 2 :
What you're trying to do is not possible using javascript as said on SO here. Tha author nonetheless suggest to use Java Applet to achieve needed behaviour.
To put it in a nutshell, HTML5 Filesystem API only provides a sandboxed filesystem, ie located in some hidden directory of the browser. So if you want to access the true filesystem, using java would be just fine considering your use case. I guess there is an interface between java and javascript here.
But if you want to make your data only available from the browser ( constrained by same origin policy ), use FileSystem API.