How do I pass an uploaded CSV file, into my function? - javascript

first of all, pardon me if this is not the correct way of asking a question but this is my first post.
I am working on a university project and am new to JavaScript (and subsequently, jQuery and NodeJS). I am trying to make a system that will take a CSV file, process it and load the results into a database ready to be used by the rest of the system.
I have made it so that I can upload the file to a directory and have a function to process the CSV file into a JSON object (we have only just started to cover databases so I will have to modify this to store in the database), but the problem I'm currently facing is this...
How do I pass/read the uploaded file into the function for processing?
I have searched around and have found similar questions, but these are either usually using PHP or cover different aspects of my issue, without actually touching upon my problem.
I'm sorry if this is something very basic that I am missing, but I'm pulling my hair out over this and can't progress to the rest of the tasks until i have the first part in place - upload, process and store the CSV file.
I will take any advice, either directly related to the issue or pointers from experience that you think I may face, and thank you for you time.
Upload
var express = require("express");
var multer = require("multer");
var fs = require("fs");
var path = require("path");
var app = express();
var upload = multer({ dest: './uploads/'});
app.use(express.static('public'));
app.use(upload);
app.post('/api/csvUpload',function(req,res) {
upload(req,res,function(err) {
// console.log(req.body);
// console.log(req.files);
if(err) {
return res.end("Error uploading file");
}
else{
res.send("File is uploaded");
}
});
});
var server = app.listen(8081, function () {
var host = server.address().address;
var port = server.address().port;
console.log("working on port 8081!");
});
<!DOCTYPE html>
<html>
<head>
<title>Home Page</title>
</head>
<body>
<form id="uploadForm" action="/api/csvUpload" method="post" enctype="multipart/form-data">
<input type="file" name="csvFile">
<input type="submit" value="Upload File" />
</form>
</body>
</html>
CSV to JSON
//This will process and convert a CSV file in to a JSON objec
function csvJson(csv) {
var lines = csv.split("\n");
var result = [];
var headers = lines[0].split(",");
for(var i = 1; i < lines.length; i++) {
var obj = {};
var currentLine = lines[i].split(",");
for(var j = 0; j < headers.length; j++) {
obj[headers[j]] = currentLine[j];
}
result.push(obj);
}
return JSON.stringify(result);
};
//Below is for testing purposes
var testCSV = "userName, firstName, lastName, studentID, course\n" +
"username1,first1,last1,123456,Software Engineering\n" +
"username2,first2,last2,234567,Software Engineering\n" +
"username3,first3,last3,345678,Software Engineering\n" +
"username4,first4,last4,456789,Software Engineering";
var processedCSV = csvJson(testCSV);
console.log(processedCSV);

Because it's not completely clear for me how the files are uploaded, I'll start with the functions that gives the ability to load a (CSV)-file.
For loading a file or url you can use XMLHttpRequest, more on this can be found at W3Schools. This is a really powerful object, and I would really advice to read more into this.
/**
* Loads file using XMLHttpRequest
* #param {String} path
* #param {Function} success
* #param {Function} error
*/
function loadFile(path, success, error) {
var xhr = new XMLHttpRequest();
xhr.onreadystatechange = function () {
if (xhr.readyState === XMLHttpRequest.DONE) {
if (xhr.status === 200) {
if (success)
success(csvToJSON(xhr.responseText));
} else {
if (error)
error(xhr);
}
}
};
xhr.open("GET", path, true);
xhr.send();
}
To use this function, simply execute the following code:
loadFile('Link_to_CSV_file.csv', function (data) {
myFunction(data);
}, function (xhr) {
console.error(xhr);
});
As soon the data is succesfully loaded, myFunction() will be executed with the data from loadFile(). This can be used in the function.
Finally, load the CSV-data in a function:
function myFunction(data){
console.log(data);
}
PS: I would advice to convert the CSV file to JSON if you want to handle it in a javascript-function, if you're using Node.js, check out How to convert CSV to JSON in Node.js.

Related

Gcloud API file.save() data format

I'm using the gcloud API on a Nodejs web server to upload files. I'd prefer the files not be uploaded on the client side and instead uploaded on the server. Currently, I am producing a blob on the client side, then converting it to text and passing that to the server through a POST request. All of the information gets successfully passed from the client to the server as expected. This data is also uploaded to gcloud, however, Gcloud does not recognize this as a valid file nor does my computer when I download it.
What is the best way to get the contents of the file to gcloud from the server side? I've tried using dataURIs and reading the orignal file by text and both produce similiar issues. I've also explored piping a readFileStream from the blob on the server end but blobs are not natively supported by node so I have not done so yet.
Client Side
function readSingleFile(e, func, func2){
var file = e.target.files[0];
if(!file){
return; // Add error msg_here
}
var reader = new FileReader();
reader.onload = function(e){
let contents = e.target.result;
let img = document.createElement('img')
let cvs = document.createElement('canvas');
img.onload = ()=>{
cvs.width = img.width;
cvs.height= img.height;
let ctx = cvs.getContext('2d');
ctx.drawImage(img,0,0);
cvs.toBlob((res)=>{res.text().then((text)=>{func2(text)})}, "image/jpeg", 0.92);
}
img.src=contents;
func(contents);
}
reader.readAsDataURL(file);
}
Server Side
function publishPrintjob(dataObj){
try{
var newElemKey = database.ref().child('queue').push().key; // Get random Key
// Create a new blob in the bucket and upload the file data.
const gcloudFile = storage.file('images/' + newElemKey + '.jpg');
gcloudFile.save(dataObj.sockImageFile, function(err) {
if (!err) {
Console.log("File Uploaded!")
}
});
var data = {
date: dataObj.Date,
email: dataObj.email,
design: dataObj.Design,
author: dataObj.Author,
address: dataObj.address,
imageKey: newElemKey,
}
admin.database().ref('queue/' + newElemKey).set(data);
} catch(err){
console.log(err)
}
}
Note: func simply shows the image on the client side, func2 just adds the contents to the POST object.
Uploading a file directly from the computer would be easiest using the storage.bucket(bucketName).upload() function from the cloud storage library. However, this uses location of a file locally and thus will not work unless a file is transferred to the server and saved first. This could be achieved using multi-part form data. Using multipart or uploading locally are better methods for uploading to google storage.
Instead, I solve this by first converting the image to a dataURI, sending the data URI to the server via the body of a GET request, and then converting it to a buffer with a readable stream that can be piped to google storage.
Client
let formData = getFormData('myForm');
var xhttp = new XMLHttpRequest();
xhttp.onreadystatechange = function() {
if (this.readyState == 4 && this.status == 200) {
// Typical action to be performed when the document is ready:
}
};
xhttp.open("POST", "dashboard", true);
xhttp.setRequestHeader('Content-Type', 'application/json');
xhttp.send(JSON.stringify(formData));
xhttp.onload = ()=> {
console.log(JSON.parse(xhttp.response))
// Handle server response here
};
}
Server
// DataObject is the body of the GET request, the property imageFile is the URI from readFileAsURI
function uploadImageOnServer(dataObj){
try{
var newElemKey = database.ref().child('queue').push().key; // Get random Key to use as filename
// Create a new blob in the bucket and upload the file data.
const gcloudFile = storage.file('images/' + newElemKey + '.jpeg');
var fs = require('fs'); // Should be required at the top of the file
var string = dataObj.ImageFile;
var regex = /^data:.+\/(.+);base64,(.*)$/;
var matches = string.match(regex);
var ext = matches[1];
var data = matches[2];
var buffer = Buffer.from(data, 'base64');
// Create the readstream
const readableInstanceStream = new Readable({
read() {
this.push(buffer);
this.push(null);
}
});
readableInstanceStream.pipe(gcloudFile.createWriteStream()) // link to gcloud storage api
.on('error', function(err) {
console.log('error')
})
.on('finish', function() {
console.log('upload complete')
});
} catch(err){
console.log(err)
}
}

Not able to upload file of 1GB from filesystem, (hardcoded file not using input type file) using xhr from Electron application using node.js

I need to upload a file reading from filesystem (not specified by the user) using xhr. Is there a way to send it via Ajax?
I understand that javascript has input type file, which gives javascript file object https://developer.mozilla.org/en-US/docs/Web/API/File.
I tried getting file descriptor using Node fs APIs (https://nodejs.org/docs/latest/api/fs.html) . But am unable to send it via xhr. Here is my code snippet.
Any help will be greatly appreciated.
var req = new XMLHttpRequest();
req.open(method, url);
req.overrideMimeType('text/xml');
var progress = 0;
req.upload.addEventListener('progress', function (event) {
if (event.lengthComputable) {
progress = Math.round(event.loaded * 100 / event.total);
}
}, false);
req.onreadystatechange = function () {
// add logic for each state
};
var fs = require('fs');
if (filename) {
// get the file descriptor and send it via xhr
fs.open(filename, "r", function(error, fd) {
// -- THIS IS THE PART NOT WORKING --
req.send(fd);
});
} else {
console.log('no filename');
}
That's not the way to read a file. This is how you do it:
var fs = require('fs');
fs.readFile(filename, { encoding : 'utf8' }, function(err, data){
//data holds the contents of the file.
req.send(data);
});

Send data to a nodeJs server using Ajax and busboy

I'm having some trouble sending data via Ajax to upload a file, if I send the data from the form directly, it works with the following code:
var fs = require('fs');
var fstream;
var folder = 'public/images/tmp/';
var path;
var images = [];
req.pipe(req.busboy);
req.busboy.on('file', function (fieldname, file, filename) {
if(filename){
if (!fs.existsSync(folder)) {
fs.mkdirSync(folder,0744);
}
path = folder+ filename;
fstream = fs.createWriteStream(path);
file.pipe(fstream);
} else {
path= undefined;
}
images.push(path);
res.send(images);
});
And the files are sent to the temp folder in the server. But I need to return images to my javascript, so with that data I can perform other actions.
Now, the options I am thinking are:
1- Find some way to avoid "data.send()" to redirect my form to the result view
2- Send the data with Ajax using a code similar to:
$.ajax( {
url: '/media',
type: 'POST',
data: new FormData(document.getElementById('#myFormId')),
success: //do things,
error: //do other things
});
but it doesn't work, I don't know wich format I should send my data to the POST function so it can work with busboy or if there is another way to copy the files to the server.
Thanks.

Node.js Server: Image Upload / Corruption Issues

So I'm trying to write a basic file server in Node.js, and all the images I've tried uploading and storing on it are coming back as corrupted. The problem seems to have something to do with the way that Node Buffers handle being converted to UTF-8 and back again (which I have to do in order to get the POST body headers out and away from the binary data).
Here's a simple Node server that shows my current approach and the problems I've been having:
var http = require('http');
var server = http.createServer(function(request, response) {
if (request.method === "GET") {
// on GET request, output a simple web page with a file upload form
var mypage = '<!doctype html><html><head><meta charset="utf-8">' +
'<title>Submit POST Form</title></head>\r\n<body>' +
'<form action="http://127.0.0.1:8008" method="POST" ' +
'enctype="multipart/form-data"> <input name="upload" ' +
'type="file"><p><button type="submit">Submit</button>' +
'</p></form></body></html>\r\n';
response.writeHead(200, {
"Content-Type": "text/html",
"Content-Length": mypage.length
});
response.end(mypage);
} else if (request.method === "POST") {
// if we have a return post request, let's capture it
var upload = new Buffer([]);
// get the data
request.on('data', function(chunk) {
// copy post data
upload = Buffer.concat([upload, chunk]);
});
// when we have all the data
request.on('end', function() {
// convert to UTF8 so we can pull out the post headers
var str = upload.toString('utf8');
// get post headers with a regular expression
var re = /(\S+)\r\nContent-Disposition:\s*form-data;\s*name="\w+";\s*filename="[^"]*"\r\nContent-Type: (\S+)\r\n\r\n/i,
reMatch = str.match(re);
var lengthOfHeaders = reMatch[0].length,
boundary = reMatch[1],
mimeType = reMatch[2];
// slice headers off top of post body
str = str.slice(lengthOfHeaders);
// remove the end boundary
str = str.replace("\r\n" + boundary + "--\r\n", '');
// convert back to buffer
var rawdata = new Buffer(str, 'utf8');
// echo back to client
response.writeHead(200, {
"Content-Type": mimeType
});
response.end(rawdata);
});
}
});
server.listen(8008);
console.log("server running on port 8008");
To test it, run the script in node and go to 127.0.0.1:8008 in your browser. Try uploading an image and submitting the form. The image comes back as corrupt every time -- even though the script should just be directly echoing the image data back to the browser.
So does anyone know what I'm doing wrong here? Is there a better way to handle POST body headers in Node that I haven't figured out yet? (And before anyone says anything, no, I don't want to use Express. I want to figure out and understand this problem.)
The problem seems to have something to do with the way that Node Buffers handle being converted to UTF-8 and back again
I guess you are right about that, convert to UTF-8 is a bad idea, but can do it just to work with the file and get the headers and boundaries positions, but keep the buffer file untouched, and when you have all the positions to get the header and boundary out of the file just copy the buffer to a new buffer like that
originalBuffer.copy(newBuffer,0, positionHeader, positionEndBoundary)
var http = require('http');
var fs = require('fs');
var connections = 0;
var server = http.createServer(function (req, res) {
connections++;
console.log(req.url,"connections: "+connections);
if(req.url == '/'){
res.writeHead(200, { 'content-type': 'text/html' });
res.end(
'<form action="/upload" enctype="multipart/form-data" method="post">' +
'<input type="file" name="upload" multiple="multiple"><br>' +
'<input type="submit" value="Upload">' +
'</form>'
);
}
var body = new Buffer([]);
if (req.url == '/upload') {
req.on('data', function (foo) {
//f.write(foo);
body = Buffer.concat([body,foo]);
if(isImage(body.toString())){
console.log("é imagem do tipo "+isImage(body.toString()));
}
else{
console.log("Não é imagem");
res.end("Não é imagem");
}
console.log(body.length, body.toString().length);
});
req.on('end', function () {
// console.log(req.headers);
//I converted the buffer to "utf 8" but i kept the original buffer
var str = body.toString();
console.log(str.length);
imageType = isImage(body.toString());
//get the index of the last header character
//I'm just using the string to find the postions to cut the headers and boundaries
var index = str.indexOf(imageType)+(imageType+"\r\n\r\n").length;
// var headers= str.slice(0,index).split(';');
// console.log(headers);
//Here comes the trick
/*
*I have to cut the last boundaries, so i use the lastIndexOf to cut the second boundary
* And maybe that is the corruption issues, because, I'm not sure, but I guess
* the UTF-8 format only use 7bits to represent all characters, and the buffer can use 8bits, or two hex,
*So, i need to take the difference here (body.length-str.length)
*/
var indexBoundayToBuffer = str.lastIndexOf('------WebKitFormBoundary')+(body.length-str.length);
console.log(index, indexBoundayToBuffer);
//maybe you can change this to use less memory, whatever
var newBuffer = Buffer.alloc(body.length);
/*
*And now use the index, and the indexBoudayToBuffer and you will have only the binary
*/
body.copy(newBuffer,0,index,indexBoundayToBuffer);
// f.end();
//file type
var type = imageType.substr("image/".length);
console.log("END");
fs.writeFile("nameFile."+type,newBuffer,function(err,ok){
if(err){
console.log(err);
return false;
}
res.end();
});
});
}
});
function isImage(str){
if(str.indexOf('image/png')!=-1) return 'image/png';
else if(str.indexOf('image/jpeg')!=-1) return 'image/jpeg';
else if(str.indexOf('image/bmp'!=-1)) return 'image/bmp';
else if(str.indexOf('image/gif'!=-1)) return 'image/gif';
else false;
}
var port = process.env.PORT || 8080;
server.listen(port, function () {
console.log('Recording connections on port %s', port);
});
You really shouldn't use regular expressions like that to parse multipart payloads as it can easily make trying to parse your image data very unreliable. There are modules on npm that parse forms for you such as busboy, multiparty, or formidable. None of them use regular expressions and they don't require Express.

Problem receiving response from node server using ajax post request

I have written a http server using node js
var sys = require("sys"),
http = require("http"),
url = require("url"),
path = require("path"),
fs = require("fs");
http.createServer(function(request, res) {
var parsed_url = url.parse(request.url);
var uri = parsed_url.pathname;
if(uri === "/test"){
res.writeHead(200, {'Content-Type': 'text/javascript'});
request.addListener('data', function (chunk) {
var data = eval("(" + chunk + ")");
console.log(data[0].id);
})
request.addListener('end', function() {
console.log('end triggered');
res.write("Post data");
res.end();
});
}
}).listen(8080);
and i am trying to send back response of ajax request but i am unable to receive any response. Here is the code for ajax request ,
var myhttp = new XMLHttpRequest();
var url = "http://localhost:8080/test";
var data = [{"a":"1"},{"b":"2"},{"c":"3"}];
var dataJson = JSON.stringify(data);
myhttp.open('POST', url, true);
myhttp.send(dataJson);
myhttp.onreadystatechange = function() {
if ((myhttp.readyState == 4) && (myhttp.status == 200)){
alert(myhttp.responseText);
}
else if ((myhttp.readyState == 4) && (myhttp.status != 200))
{
console.log("Error in Connection");
}
Can anyone help me what i am doing wrong ...
Thanks
Vinay
Your code is almost right but on your code sample you have
console.log(data[0].id)
the data object has no property id so if you only have
console.log(data[0])
there you have a response like
{ a: '1' }
therefore you can access the property a by doing
console.log(data[0].a);
UPDATED Updated with a full example
One more thing is that you are using eval and node comes with JSON.parse bundle with it so the snippet below is how i made it work
File: app.js
var sys = require("sys"),
http = require("http"),
url = require("url"),
path = require("path"),
fs = require("fs");
http.createServer(function(request, res) {
var parsed_url = url.parse(request.url);
var uri = parsed_url.pathname;
if(uri === "/test"){
res.writeHead(200, {'Content-Type': 'text/javascript'});
request.addListener('data', function (chunk) {
// removed this - eval("(" + chunk + ")");
var data = JSON.parse(chunk);
console.log(data[0].a);
})
request.addListener('end', function() {
console.log('end triggered');
res.write("Post data");
res.end();
});
} else if(uri === "/") {
fs.readFile("./index.html",function(err, data){
if(err) throw err;
res.writeHead(200, {'Content-Type': 'text/html'});
res.end(data);
});
}
}).listen(8080);
On the same directory create a file index.html with the following:
<html>
<head>
<script type="text/javascript" charset="utf-8">
var myhttp = new XMLHttpRequest();
var url = "http://localhost:8080/test";
var data = [{"a":"1"},{"b":"2"},{"c":"3"}];
var dataJson = JSON.stringify(data);
myhttp.open('POST', url, true);
myhttp.send(dataJson);
myhttp.onreadystatechange = function() {
if ((myhttp.readyState == 4) && (myhttp.status == 200)){
alert(myhttp.responseText);
}
else if ((myhttp.readyState == 4) && (myhttp.status != 200))
{
console.log("Error in Connection");
}
}
</script>
</head>
<body>
</body>
</html>
That is a complete working example of what you want.
With regards to the same origin policy issues you were having is mainly due to the fact that you cant POST data between 2 different domains via ajax unless you use some tricks with iframes but that is another story.
Also i think is good for anyone to understand the backbone of a technology before moving into frameworks so fair play to you.
good luck
You have to read the data in a different way. Posted data arrives on a node server in chunks (the 'data' event), that have to be collected until the 'end' event fires. Inside this event, you are able to access your payload.
var body = '';
request.addListener('data', function (chunk) {
body += chunk;
});
request.addListener('end', function() {
console.log(body);
res.write('post data: ' + body);
});
Additionaly, there seem to be some issues with your client-side code (especially concerning the status-code checks), but i can't really help you with those as i always work with frameworks like jQuery to manage async requests.
If you want to build reliable node.js servers for web use, i highly recommend the high-performance HTTP-Framework Express. It takes away alot of the pain when developing a web-based server application in node and is maintained actively.

Categories