We Keep Coding

How to send real time response from server to client using Node.js express

on post request server is generating data after few seconds, these are almost 1000 to 10000 entries. Currently i'm saving data into csv file and it's working fine. how can i pas data to client with json array.
Name and Age variable getting data after few seconds
app.post('/', (req, res) => {
// currently passing to results.csv. its working fine i want to send this real time (Name and age) data to client.
const stream = createWriteStream('./results.csv', { flags: 'a', encoding: 'utf8' })
// Append evaluation from response to file
stream.write(`${Name}, ${Age}\n`)
// example data: Patrick, 32
// End stream to avoid accumulation
stream.end()
})
res.send() only send first row but Name and age variable getting update after after each 10 seconds
app.listen(port, () => {
console.log("App is running on port " + port);
}); ```

To parse data into json array, you would have to read the stream before sending the data. Example:
let rows = []
fs.createReadStream('./results.csv')
.pipe(parse())
.on("error", (error) => console.error(error))
.on("data", (row) => rows.push(row))
.on("end", () => res.send(rows))
But if you want to send the csv file data, do something like this:
return res.sendFile("./results.csv", options, (err) => {
if(error) console.error(error)
console.log("File has been sent")
})
NOTE: The parse() method used in the first example is gotten from the fast-csv package.
Also, using a package like fast-csv makes it easy to write data to the csv file while setting headers:true. Trying to read the data, the parse method has to be informed of the headers already set in order to use those values for the json keys.

missing returned data from sails.js file upload

I have written a simple function to handle upload of files in my sails.js app.
let upload = file.upload((err, uploadedFiles) => {
if (err) {
return res.serverError(err);
} else {
return res.send({ data: uploadedFiles });
}
});
When the upload is complete I am redirected to a page displaying raw json, which contains the uploaded file information (including the path).
raw json response
What I am expecting when I console.log(upload) is the same information, however I am getting the writestream instead.
console.log output
This is a problem for me because I would like to be able to extract the file name from the object and use it in another part of my program, but I can't do this because all I am able to access is the writestream.
I have tried using async/await and callbacks and can't seem to fix my issue.
Hopefully someone can help me!
Thanks

A helpful person on the sails Gitter suggested that I use this package, which supports async/await: https://www.npmjs.com/package/sails-hook-uploads
I tested it out with the following code and it works:
let upload = await sails
.uploadOne(file, {
maxBytes: 3000000,
})
.intercept('E_EXCEEDS_UPLOAD_LIMIT', 'tooBig')
.intercept(
(err) => new Error('The photo upload failed: ' + util.inspect(err))
);

How to write correct app.get()? (node.js / express / boby-parser)

I create server (node.js / express / boby-parser)
and I need to get array of objects 'users'.
its part of code from server.js file:
let users = [{
name: 'user1',
}];
app.get('/users/', (req, res) => {
const filePath = path.join(pth.dir, 'build', 'index.html');
res.json(users);
res.sendFile(filePath);
});
Its my code from frontend:
const handleResponse = (response) => {
return response.text().then(text => {
const data = text && JSON.parse(text);
if (!response.ok) {
const error = (data && data.message) || response.statusText;
return Promise.reject(error);
}
return data;
});
};
const getAll = (baseUrl) => {
const requestOptions = {
method: 'GET'
};
return fetch(baseUrl, requestOptions).then(handleResponse);
};
Something wrong with my code at server. (I just dodnt know how to use express server).
when I use getAll function I got JSON text replace my page. Can anyone help? How should I write app.get() in server.js. Or do I need write in server part one app.get() to get page or another app.get() to get JSON data?

Why are you trying to send a file in the response?:
res.sendFile(filePath);
For starters, the response content can either be JSON or a file (or any of a variety of other things of course), but not both. With data like JSON or XML it's possible to combine multiple objects into one larger object for a single response, but this won't work if the content types are entirely different as it is with a file.
Looking at your client-side code, you're not even doing anything with that file. You only read the JSON data:
return response.text().then(text => {
const data = text && JSON.parse(text);
if (!response.ok) {
const error = (data && data.message) || response.statusText;
return Promise.reject(error);
}
return data;
});
So the simplest approach here would just be to not try to send back the file:
app.get('/users/', (req, res) => {
res.json(users);
});
Edit: Based on comments below, you seem to be struggling with the different requests the client makes to the server. The browser loading the page is one request with one response. If that page includes JavaScript that needs to fetch data, that would be a separate AJAX request with its own response containing just that data.
It’s possible to use JSON (or any data) server-side to populate a page template and return a whole page with the data. For that you’d need to use (or build) some kind of templating engine in the server-side code to populate the page before returning it.

The res.json() represents the HTTP response that an Express app sends when it gets an HTTP request. On the other hand, res.sendFile() transfers the file at the given path.
In both cases, the flow is essentially transferred to client who might have made the request.
So no, you cannot use res.sendFile and res.json together.
var options = {
headers: {
'name': 'user1',
}
};
res.sendFile(path.join(__dirname, 'build', 'index.html'), options);
Thats really the closest you can do to achieve the desired task.

How to read file from #google-cloud/storage?

I am retriving a file from my bucket.
I get the file and want to read it's contents but I do not want to download it to my local project
i just want to read the contents, take the data and do other operations with it.
my code:
export const fileManager = async () => {
try {
const source = 'upload/';
const options = { prefix: source, delimiter: '/' };
const remoteFile = st.bucket(bName).file('myData.csv');
const readFileData;
remoteFile
.createReadStream()
.on('error', err => {
console.log('error');
})
.on('response', response => {
readFileData = response;
console.log('success');
// Server connected and responded with the specified status and headers.
})
.on('end', () => {
console.log('end');
// The file is fully downloaded.
});
console.log("data", readFileData)
} catch (error) {
console.log('Error Is', error);
}
};
readFileData is undefined.
Is this possible? Every example I find envolves me downloading the file.

createReadStream is asynchronous and returns immediately. You have to use the callbacks to find out when the download to memory is complete. Right now, your code is always going to print "data undefined" because it's trying to print the response before it's available.
createReadStream is definitely the right way to go, but you will have to understand how node streams work in order to process the results correctly. There is a whole section in the linked documentation for reading streams, which is what you want to do here. The way you deal with the stream is not specific to Cloud Storage - it's the same for all node streams.
You might be helped by the answers to these questions that deal with reading node streams:
Node.js: How to read a stream into a buffer?
Convert stream into buffer?
How do I read the contents of a Node.js stream into a string variable?

writestream and express for json object?

I might be out of depth but I really need something to work. I think a write/read stream will solve both my issues but I dont quite understand the syntax or whats required for it to work.
I read the stream handbook and thought i understood some of the basics but when I try to apply it to my situation, it seems to break down.
Currently I have this as the crux of my information.
function readDataTop (x) {
console.log("Read "+x[6]+" and Sent Cached Top Half");
jf.readFile( "loadedreports/top"+x[6], 'utf8', function (err, data) {
resT = data
});
};
Im using Jsonfile plugin for node which basically shortens the fs.write and makes it easier to write instead of constantly writing catch and try blocks for the fs.write and read.
Anyways, I want to implement a stream here but I am unsure of what would happen to my express end and how the object will be received.
I assume since its a stream express wont do anything to the object until it receives it? Or would I have to write a callback to also make sure when my function is called, the stream is complete before express sends the object off to fullfill the ajax request?
app.get('/:report/top', function(req, res) {
readDataTop(global[req.params.report]);
res.header("Content-Type", "application/json; charset=utf-8");
res.header("Cache-Control", "max-age=3600");
res.json(resT);
resT = 0;
});
I am hoping if I change the read part to a stream it will allievate two problems. The issue of sometimes receiving impartial json files when the browser makes the ajax call due to the read speed of larger json objects. (This might be the callback issue i need to solve but a stream should make it more consistent).
Then secondly when I load this node app, it needs to run 30+ write files while it gets the data from my DB. The goal was to disconnect the browser from the db side so node acts as the db by reading and writing. This due to an old SQL server that is being bombarded by a lot of requests already (stale data isnt an issue).
Any help on the syntax here?
Is there a tutorial I can see in code of someone piping an response into a write stream? (the mssql node I use puts the SQL response into an object and I need in JSON format).
function getDataTop (x) {
var connection = new sql.Connection(config, function(err) {
var request = new sql.Request(connection);
request.query(x[0], function(err, topres) {
jf.writeFile( "loadedreports/top"+x[6], topres, function(err) {
if(err) {
console.log(err);
} else {
console.log(x[6]+" top half was saved!");
}
});
});
});
};

Your problem is that you're not waiting for the file to load before sending the response. Use a callback:
function readDataTop(x, cb) {
console.log('Read ' + x[6] + ' and Sent Cached Top Half');
jf.readFile('loadedreports/top' + x[6], 'utf8', cb);
};
// ...
app.get('/:report/top', function(req, res) {
// you should really avoid using globals like this ...
readDataTop(global[req.params.report], function(err, obj) {
// setting the content-type is automatically done by `res.json()`
// cache the data here in-memory if you need to and check for its existence
// before `readDataTop`
res.header('Cache-Control', 'max-age=3600');
res.json(obj);
});
});