I'm making a server sistem in which the server uploads files to a client via websockets. The server is corretly sending the file, but I don't know how to write it in the client side.
I've tried receiving the file data using "msg", "msg.data" and many other ways, but it always result in a 1kb file. I also tried converting the file to base64 before sending and converting back in the client side(so the message would be text instead of a binary), but didn't work.
Server:
var ws = require("nodejs-websocket")
ws.setBinaryFragmentation(99999999999)
var fs = require('fs'),
file = process.argv[2],
data = fs.readFileSync("./map.zip");
var server = ws.createServer(function (connection) {
connection.sendBinary(data)
}).listen(1000)
Client:
var connection = new WebSocket("ws://localhost:1000")
connection.onmessage=function(msg){
var fs=require("fs")
fs.writeFileSync("./test.zip",msg.data)
}
I think you should consider sticking to your base64 encoding approach, as binary files are a hell to debug.
Send it as base64 and receive it from the client with
var connection = new WebSocket("ws://localhost:1000")
connection.onmessage=function(msg){
var fs=require("fs")
let buffer = new Buffer(msg.data);
fs.writeFileSync("test.zip", buffer);
}
(This assumes that msg.data is a base64 encoded binary file)
Related
I am running a Python HTTP server that host a captive portal. Basically I'm trying to upload a password protected file to the server.
Right now I am able to upload files to the server with JavaScript and FileReader. This is how I do it:
var file_cli_cert = document.getElementById(id="exa_cli_cert").files[0];
const xmlhttp1 = new XMLHttpRequest();
let name = file_cert.name;
let reader = new FileReader();
reader.readAsText(file_cert);
xmlhttp1.open("POST", '/load/cert');
reader.onload = function(){
xmlhttp1.send(name +"&CERT="+reader.result);
With non-password protected files this works well.
For password protected files my idea is to get the file and the password to access to the data. The problem is that I don't know how to access to password protected files in JS and i think it's not possible. So right now i am wondering how to send the file and the password to the server and access the file data there.
If I send the file object with XMLHttpRequest.send(), on the server side I get the object in string format.
To read the POST message, in the server side, I do:
ctype, pdict = cgi.parse_header(self.headers['content-type'])
content_len = int(self.headers.get('Content-length'))
post_body = self.rfile.read(content_len) #read credentials
self.send_response(201)
self.end_headers()
if self.path.endswith('/load/cert'): #if user loads a certificate
post_body = post_body.decode()
post_body = post_body.split("&CERT=") #split name and file content
name_cert = post_body[0]
file_content = post_body[1]
f = open("./certs/"+name_cert, "w")
f.write(file_content)
f.close()
I'm a newbie at this and I have been looking for a solution for hours. Any help will be appreciated.
No python expert but reading a encrypted file as Text with FileReader could be problematic as it could lead to some data loss when encoding it as text. You should rather be reading it as binary using reader.readAsArrayBuffer()
But there is no need to read the content of the file into memory and allocate memory, just upload the blob/file directly to the server and it will take care of sending the data from the disc to the network without touching the main javascript thread without any text transformation.
const [ file ] = document.querySelector('#exa_cli_cert').files
fetch('/load/cert', {
method: 'POST',
body: file,
headers: {
'x-file-name': file.name
}
})
.then(r => r.arrayBuffer())
I'm practicing in programming with websocket, i'm building a java websocket server and want to send a binary data to websocket client, then client side can get that data through websocket and save it to a local file on disk, but i don't know what way using javascript to write an arraybuffer into a file.
My example code is shown below
var ws = new WebSocket(URL);
ws.binaryType = "arraybuffer";
ws.onmessage = function(evt) {
if (evt.data instanceOf ArrayBuffer) {
// binary message is riecieved from server and i want to save it as a local file
}
};
Thank you for any help
You can create a textual/string representation that can be saved to and read from a file:
// returns String
function getArrayBufferString(arrayBuffer) {
return new Uint8Array(arrayBuffer).toString()
}
// returns ArrayBuffer
function parseArrayBufferString(string) {
return new Uint8Array(string.split(',')).buffer
}
Over my node.js application I decode base64 encoded images using the following line of code:
const fileDataDecoded = Buffer.from(base64EncodedfileData,'base64');
So far I can write a file with the following piece of code:
const fs = require('fs');
....
const fileDataDecoded = Buffer.from(base64EncodedfileData,'base64');
fs.writeFile("/tmp/test.png", fileDataDecoded, function(err) {
//Handle Error
});
Now what I want to achieve is the decoded files via the buffer to get written into a file via streams in order to acheive better efficiency on the executed application.
In other words I want the filedata to get written at the same time that the data is getting base64-decoded, in order to write large files with an efficient way. If is not possible via a Buffer to stream base64 decoded data then I would like to know how is possible to decode base64 data.
You can create a readable stream and push the image buffer. Then pipe it to a writeable stream like this.
var base64 = 'iVBORw0KGgoAAAANSUhEUgAAAJAAAACQCAYAAADnRuK4AAAMcklEQVR42u2deVBUVxbGGxHQBOPEZdxr3KJJcO+INGsDDXGJC8RWBKMGFIhRGVxYLDCgINAoq9J2I2jiGCPDKGClZvKXmaWmZq1U4lRiJslkU2dMTWapmiQVWzlzz+vXLCrQ0G/vc6q+egVF1Tvvfr8+5977mvd0OgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKisEFgE9RUdEwc0uLL0k+oQfohWqgMV4tGs4lTaGoQE/QG8XCxCXX6+czI+LbmqfFdzTOWH7ZOl1+nZFB8l4zjj16gF7055XsgaXSlVhch32bqaPxTaZPmP7HdIfpezkUd+W0YiTTGNzhPUAv3kRvXPC4PFMMPCy5KJbkB0zQpXa75Ipj51W65BiXHr58YLpsNSoCIlcCse32lO4E7Y7YdpvD1Ga/x37fKY0aO9m5VSfMW4rxcXphc/rCPGK/g8g36rbICpGrFCLNLnhYYncwOanUq9qpXJKNW5sd2xrEtDZAZFNZvDxzIn4mr7el+7FP0XWp4TFpDJ6eLV8qiPAYcfbYR1PNhpE9PZWy+vhEt1pTu9sWVRw1VSR2DofxQi2E1BTuQS+xGEi6p8AOvrFttp87L9bmELfiNDrlRQC5rlk8gOyOmEtWMDQcvsq8HG42m32lbF8+c3NSRzGAPuUu1jlhpqqjomqEnmHLDLWVfjlxRcR49FSSNuaqPsssBU+wBL7h+3YnwaM6iHAlCOHNFd8GZWxeiJ5KUoWMRdz8Z/jS8twgLINiAOR17WoQbU3AiTQHUMRZi+Pp9KRl6Kk+XYJ5EA+Qn75w13xceQkLEEHiXjVqFAyg8KYKx5NbEgycp1IAxJ8kYNGBHYuEBMhELUvaJX8PgGalrAlDT4PMZn/JAFq4N22xUAAREDLMjXoANHPT6nDVAkQQyASRzACNEAIgMl9GiB4EaISqACLTZYZIzQCR2QqASK0AkckKgUiNAJG5CoJIbQDRPo/C9onUBRAZqrgdazUBRGYqsJWpBSC6MarQG7CqAUjsT1iLFaKKiiBiXz5E7FWRWL6YN+YvSxVSA0BSfMKMtRUQan4JQterUCzvqMoSZqZNeoiUDpBUJTrKcpQzI2rTbtj5ShVkl55QvDBPzBfzjiguhKjXa6SHSMkASTnvcQG0IjUH/njtOnx+67bihXlivph3eFEhRJ6t5CGySzcfIoB6A/Tcjjx4/5Mv4Pa//qt4YZ6Yb0+AOIjO14i7X6YGgKReZbgAWptxEG599U/4/o5D8cI8Md/7AXJCVC36pisB9BCA1mUehK++/jfcu3dP8cI8Md+HASQFRIoFyEQACQKQq52JNidSLEAybJRpFaDuibU4qzPFASTXTquWARIVIucTTQggrQMkGkTKAqiRABIRIFEg4jw7rQyA+AcmEUAiAiQ4RAyguCsEkFcB1LU6E2LBohSA4pwJEUASASTYPhEPUESzhQDyNoAEgYgA8m6AXBANuZ0pASB8DjEBJB9A3TvWNg8BSiCAvBWgIa/OCCACyCOI5AcokwBSEECDhkjzALU3wsoLVkg8VQfrT/StxOIyWJuRDynZh+HSW2/DL375O8UL88R8Me91BUcg4ZilT62psoDplAgQaR2gVecaYOuhSkjLq/B6bS2ogPiGwU6s7d4N0Maqaq+AI/UgO+Z3/7ydKT23/IG/W19mGfTqrN99Iq0DlFxepXl4tpQeh9U/tcG6nzTAi4UWBk85NGZlw28ytoJlb36vv006UiHsZiMBpH4lnj0Jsb86C6a3z0ByVQ3szD0KH6eZ4cvURPh1xjZ4KbfMI4D6hYgAUr+Sj1fDs281wSq2YNhafAzSWQXq2JkBH27fAE17smFHXrnHAPU5JyKANCA293mRLRSwfbl+l8mqTlZOCQeTpy2s39UZAeRd8hSgByAigAggjyAigAggjyC6bCOACCCP7uJ34uqMACKAhqSIM5Wdxtdr+a+0EkAE0BAAinz1GITUF8v1hTICSAsALasudMw0LyeACCAPAEoggAggAogAIoAIIAKIACKACCACiAB6/nA5bMoqJvFKPFRCAA3qvy1+nK/Oh4eLpMhd+wkgAmjoiniZACKACCACiAAigAggAogAIoAEAsjwfCaEJOyAZeu2q06YN+bvpQClyQ6QITED9Ku2wKLlm2Dhs0mqE+aN+eN1EEAyAPTM6m2qBOd+4XUQQDIAtHjl5i4T9CtTYHVaNiSk71e8ME/M15U7XoesAJmXa+8Zie4AhOXfZUJ0Ugb8/t334ePPbypemCfm2wU/uw5ZAdLiQzbdASgkMR0Wr3B+kmOSMuFP167Dpzf+rnhhnpgvV31Y/ngd8gF0yHsBcq7CMiB4bSrEv5AF7374N/jH1/9RvDBPzBfzxvzlXIXJ+KV6cZ9UP9h9oJVpOfDeXz9TxSsvMU/MVwn7QKGnSgkgAmhoAEWdq4LwpnJtvq2HABIfoOiL9dp9XxgBJC5ArH11xl4+pWGA9h4kgHoCtPuAwP8bX6vtd6Yaj5ZAqHknAYRi4xBZWCBk9YGY1gaNv7W5xQpReQUQlrIbQpNeHlAr0/PgvY9UAhDLE/N157rCUnZBxL5ciLQL/HgXr3hvfJsNos5VQ2STZUCtOX8Crt24oQqAME/M153rimy2CNu6sPr8zOolAPHvOMdPzEADs+aNk+oCiOUr6KTYXYAu1HI+KRYgZyuTHiICyA14XjsOMWzl5X0AuQERATSwjC31XR4pGiBRWtkAEBFAbrzdkM0pVQOQWBDhA7MfBhEBNNDEuaG3P2oASDyIbA9AtPx8Hbz6599C21/eUbwwT8xXstZ1sf5Bb1QDUHujZBCR3GhdagNItPkQQTSwXjvWvepSM0CxIlWh/uZEXi8274luPdm3J+oCSODbHEPcbPQmRbfU9+uH6gASs5URRH3sNmsNIIJIAnj4l6loFiCCSMwVF/cSFfd8UDNAokPkhRNrXK73ueLSIkBiQ+RNqzOsPIOCRysAiQ+R9veJ+Hd/DX7stQIQQeThaqvNNrRx1xJAYu8TaQ4i3CS8WO/ReMsO0IID6UtMHXaHUADRjrW7tyeO97/DPEiAIporHDM2r42QDCCj0TicO9nu1IUMnm+EBUjcG7BqX+IPeqXlBkBh9rJvf7R+VSh6qtfr/UQHSGc2++LJxoUtnhx7yfoF13ra7PeEBEiuL6UpFhzWsrivZAx1vvOwsWWe4VgY6g7deuzpWbPRU52zOIgew6ZOnTqSHccaL9Rf5QG6KwZABBFfde7/MpgwVf5uTGsDLCnd/wfm5Tje02GSADQhfsGjCFBw5cFcXEKa2hpFA6irpbV7F0T4BXhjywlBq053++LeG383vLkCgvZsKUMvxxuDAqUCyEc3e3YAOz4eMP6xJ8KaLJ9xEHXYHGJCJEY1UuLEGtsVLs9jLp0SZxwRHrb4iblkBX1Zzk3/sYFPoZe66dNHcN5KApBO5zdqzuRx7DhtbkZSJlsK8okxiMT4xIi45FfMEh/BYXlw//Qn1tjx8OB1h9QcgplJq7PRQ95LPykBwslW4MgxY6ay45x5WamlodYj+P/WXGnkhBNrNtMXUyZ8rIwAYhW0M+p8TSc+8kRq4VMy8NzRrQ3ijRM3YXb6goAuqy6EuTuSKtE75uE09JL3VBKAdHyvxJI3NmD06FnsGDT7hbXZz1Tk3gxrLINo1rtNYt8w7aXTEHfFM2G+RtY6sBJIIXy4E7cZyJblUowRfrjDTpeDvmTfrVkbnstBz3jvxvJeDtNJGD58yUNyJ/qPGvUkO84fOWaUYW56cu3iV7LeWWrJux1youi7MFvpXYO1xCGV8JFtQ5WhocQRXFXgWFqRJ44s+Y7g4wWOkLoidq4joo9FmK3kruFk8XfBlfm3FzFP5mzfWMc8wj2f+bxnE3kP/aSsPj2rEE6mf8A0hZ+MzfPz81vEtCRwyoSYiSFLN0yKNCRPCtPLpOAhSJ88wcAULKAMvKS+/sjg5AkhSzeiF8wTPXqDHvFeTeG9C5C6+vSsQr58+cNEJusCAmb7Px44z9/ffwH7eQEPE0lmoRfoCXqDHnFeOT0bwXvoo5MpekL0GNN4JpxYz2CJzmVJP+UfyIAiySfmAfNiDueJ05vxvFeyw3M/RHgj7hE+OZyY/ZDvsZN4TSZJKte4T+S9QE9G8x75KwWenhC5QPLj++pIPtlHeQWSJJVr3B/hvQjgvfHt4ZfiwpXYMF6+JEXI5YdiwRkIKJK8oqCgoKCgoKCgoKCgoKCgoKCg8Cj+Dyqrhq0vA9X1AAAAAElFTkSuQmCC';
var fs = require('fs')
var Readable = require('stream').Readable
const imgBuffer = Buffer.from(base64, 'base64')
var s = new Readable()
s.push(imgBuffer)
s.push(null)
s.pipe(fs.createWriteStream("test.png"));
Just another solution with base-64 & utf8 in case we're using react-native which doesn't have Buffer global object:
function (base64Str) {
return utf8.decode(base64.decode(base64Str))
}
Im trying to parse a 300MB xml file into json in a worker nodejs app, so the client makes the request to the main web app and the server performs request to the worker server with the file location, after the worker server finished parsing the xml it saves it to a json file and return its location back to the main server.
Everything works fine with xml files under 130MB, however when it encounters a large file the worker server finished parsing the file and saves it, as soon as the response comes to the main server it receives :
{ [Error: socket hang up] code: 'ECONNRESET' }
I have tried to use timeout in the request options, but it still happening.
request
.post({
url:'http://localhost:6666/parsexml',
formData: {filePath:filePath},
json: true
},function(err,httpResponse,jsonResObj){
// throws error here
})
the xml parser worker return 200 message back to the server, and then it crashes.
Anny suggestion on how can i implement this ?
Maybe you have to use different XML parser, for example xml-stream, which can handle large (500+ MB) files.
How can you install it
Install it using the following command:
npm install -g xml-stream
How you can use it in your app
Require it in your project and pass the ReadFile object to initialize it:
var fs = require('fs');
var XmlStream = require('xml-stream');
/*
* Pass the ReadStream object to xml-stream
*/
var stream=fs.createReadStream('file_name.xml');
var xml = new XmlStream(stream);
/*
*Code continues here
*/
If you want to extract only values of, let's say, id from your XML and print them this is a code to do so:
var fs = require('fs')
var XmlStream = require('xml-stream') ;
var stream=fs.createReadStream('some-large-XML.xml');
var xml = new XmlStream(stream);
xml.preserve('id', true);
xml.collect('subitem');
xml.on('endElement: id', function(item) {
console.log(item);
});
More detailed information can be found here.
var oReq = new XMLHttpRequest();
var reader = new FileReader();
reader.onload = function(e) {
var rawData = reader.result;
oReq.open("POST", '/upload', true);
oReq.send(rawData);
console.log(rawData);
}
reader.readAsBinaryString(postObj);
// postObj is an image who's src is set to a data uri, taken via a webcam.
console.log(rawData) yields ==>
"PNG
IHDRÈÜÇ IDATx^d½idizw"##rϬ}ëîªÞfÃ!%2
ø£üølpµ0ù Q&É©¡,Ú2l²!æSeÃ$Y-ÉÎLïKuí{îáëºófe³§¦*3#Nó¾Ïr?÷³¼úOþÓùÉ tóne¸Øm.»ÙhÐtÃÅQ7<w''³nοg³Y·Èkóãn<Xì§G]·²Ø
ç]·8wÇ\å¨;á5vÁ°Ì»E¾?æ÷ywtÒuó¯ù{>Ïám]ÇëF£Qwt|\ßó5ä³NæÜÇ°^{r¿ùÜÁ|¡;>:êón0âß¼gûã6rm_ëgçõü4[vG¼gaa!?_à5#~³0\èxcß}Â}ú|ÜÓóóÅ!Ï=?â{®ã½ó^í½v'ü×úþÞ/×ñÏ}P®}Â÷ݸ;9fX¨?[àUÇ|?à^|þáb=çòɸ[æ·¬#7q]>giÜqÓ~#oq´èMdY4í3-.ùÈzÿü̺Íy
7Ľֺؿ\Ü6¯õ:Ü ß·5r-|ºv½½ôú¬w¶ìõáþ<{/íuí{ÞÎÚy¯Gì%kÎ}NfÓÚÞã5ݶÏÞÔr×
ø«Õｯk¯õï</ëïçz¬ïso§ÙsƳü5öß»{ÛÝádÌñö ïgÍÿù¿üWÝƹÝøüX7øK?ùó ê¶/!Ë«ÝÂxÔ"
G.²Éxbæ]ñg#úýgé£Þ;¿ThƳ/ùòû(D¿Ã>2Þ-ð³7¯18"P<=ç>GSr±;r\à;³°<6ÎòZNAázndûûtù¼1ÇVGÁ¼^vܼ\3ÚÆu¸9×g]ÉñA V°0óîÍ£TÃY½ïdaM">ÆhÎF9ú¿](3¯¢xeÜõt
¢DÞN¾E¾oÈEy×q®?ÂÀ»z=Bó|,¯JáæzíýÞ¯ÿöï&äM!rϼWóQÛ^ÏÛÖ5÷ܤûåϸÛüûÃkôF©]ß¿óÜ1F¤üNëÍsM¯Qü²úYX:f¯DSßÌõ4D1eTÛûüvÿþlrpÐM'SÇuòµÈïù?øî½>è~â'~¢ü$
How can I interpret this in Python? On the server, this data shows up as:
'...\x7f\xc2\xb4r\xc2\x87\x1c\xc2\xaa\n-\xc2\x9c\xc2\xa6QF\xc2\xac\xc2\xb0tS\xc3\xa4\xc2\xb0;\xc3\x8cisL#\xc3\x98\xc2\x94E\xc3\x94\xc2\xb8Bz\xc3\xb0\xc3\xa9\xc2\xaa#8\xc2\x90\xc3\xbb\xc3\xa5>\xc3\xbaO\xc2\xa8\xc3\x81H\xc3\x91\xc2\xaf:i\xc2\x8a\xc2\x926\xc2\x8b\xc2\x81\xc3\xbc\xc3\xa1Y\xc3\x93\xc2\x9b\xc2\xbat\xc2\x8f\xc3\x9e~\xc2\xa3PH4\x02_\x04\xc2\xbf\xc2\x92\xc2\xb7\xc3\xad\xc2\x8f\xc3\x9e\xc3\xbf\xc2\xb8<\xc2\x91V\xc3\xa0\xc3\x8b\x1f\xc3\x88\xc3\x9f\xc2\xa2>)\x1d\xc3\x94eY=\xc3\x8ct\xc2\xa9+L^7\xc2\xa2I\xc3\x84\xc2\xba\x03\xc3\xb5!1f\xc3\x97\xc3\x81\xc3\xbfD\xc3\x87\xc3\xb7\x06\xc2\xaa\xc3\xafcz\xc3\xad(\xc3\xb5\xc2\xab\xc3\x96\xc3\xb5<\xc3\x8e\xc2\xab\x08\xc3\x81\xc2\x88\x0b\xc3\x8a;\xc3\x8e!v\xc3\x84\xc2\xb1?\xc2\x8bVn\x19t\xc3\x80\xc2\x8bT`:\x1c\xc3\x8b\xc2\x99\xc3\xb2\xc3\x9c\xc3\xbf\x0fCsXi\xc3\xa6z\xc3\xb3l\x00\x00\x00\x00IEND\xc2\xaeB`\xc2\x82'
Writing this to file as a PNG yields an invalid PNG. Any guidance on saving the image would be helpful.
You will need to convert your image's binary representation into Base64 before uploading - this makes it safe to work with when transferring data over HTTP.
When you receive the Base64 encoded image server-side, you can convert it back to binary, and write this to a file.
Client side code:
To convert to Base64, you need to use
fileReader.readAsDataURL( fileObject )
this automatically represents your data in a safe to upload via HTTP format.
Server side code:
import base64
coded_string = '''Q5YACgA...'''
binary = base64.b64decode(coded_string)
# now write binary to file...
open('/path/to/new_file.png', 'wb').write(rawData)
On the browser:
// Client Side:
var xhr = new XMLHttpRequest();
xhr.open('POST', uploadUrl);
xhr.send(base64EncodedItem);
On the server (I am using Flask/Python 2.x)
# Server Side:
import re
import uuid
# Obtain the base64 string
image_string = request.data
image_string = re.search(r'base64,(.*)', image_string).group(1)
# Generate a file name.
# We can assume PNG format in my specific implementation, but you could search
# the image_string variable to detect filetype.
output = open(str(uuid.uuid4()) + '.png', 'wb')
output.write(image_string.decode('base64'))
output.close()
This worked for me. Hope it helps others.