I have over 20k documents in my Realtime Database. I need to stream them, but I'm not even sure how to get started with it. This is kind of what I was trying to go for
sendEmail.get('/:types/:message', cors(), async (req, res, next) => {
console.log(5);
const types = JSON.parse(req.params.types);
console.log('types', types);
let recipients = [];
let mails = [];
if (types.includes('students')) {
console.log(1);
const tmpUsers = await admin.database().ref('Users').orderByChild('student').equalTo(true).once('value').then(r => r.val()).catch(e => console.log(e));
recipients = recipients.concat(tmpUsers);
}
if (types.includes('solvers')) {
console.log(2);
let tmpUsers = await admin.database().ref('Users').orderByChild('userType').equalTo('person').once('value').then(r => r.val()).catch(e => console.log(e));
tmpUsers = tmpUsers.concat(arrayFromObject(await admin.database().ref('Users').orderByChild('userType').equalTo('company').once('value').then(r => r.val()).catch(e => console.log(e))));
recipients = recipients.concat(tmpUsers);
}
});
But this code causes my server to run out of memory. Someone suggested streams in my previous question, but as much as I like the idea, I have no idea how to actually do the streaming. I know it should be something like:
const fs = require('fs');
const readStream = fs.createReadStream('path goes here');
const data = [];
readStream.on('data', chunk => {
data.push(chunk);
})
readStream.on('end', () => {
console.log(data);
res.end(data);
});
But how on earth do I pass a firebase query into the path?
I tried this, but it said Argument type Promise is not assignable to parameter type PathLike, which makes sense, but how do I get around it?
const users = fs.createReadStream(admin.database().ref('News').once('value').then(r => r.val()))
To summarize: How do I stream data from a Firebase realtime database?
Edit:
How is that a duplicate? These are 2 completely different questions. The starting code is the same but the ways of solving I'm asking about are completely different
Realtime Database doesn't have any streaming APIs. When you perform a query, the entire result set is always loaded into memory in one shot.
A possible alternative is to page through the results using limitToFirst() along with some query ordering that allows you to perform a subsequent query after the first batch of items is processed, picking up where the last query left off.
Related
The goal is to call a function from my main script that connects to a database, reads a document from it, stores pieces of that document in a new object, and returns that object to my main script. The problem is I cannot get it all to work together. If I try one thing, I get the results but my program locks up. If I try something else I get undefined results.
Long story short, how do I open a database and retrieve something from it to another script.
The program is a quiz site and I want to return the quiz name and the questions.
const myDb = require('./app.js');
var myData = myDb.fun((myData) => {
console.log(myData.quizName);
});
Here is the script that tries to open the database and find the data
const { MongoClient } = require("mongodb");
const {mongoClient} = require("mongodb");
const uri = connection uri goes here but my name is hard coded into it at the moment so I removed for privacy
const client = new MongoClient(uri);
const fun = async (cback) => {
try {
await client.connect();
const database = client.db('Quiz-Capstone');
const quizzes = database.collection('Quiz');
const query = {quizName: "CIS01"};
const options = {
sort: {},
projection: {}
};
const quiz = await quizzes.findOne(query, options);
var quizObject = {
quizName: quiz.quizName,
quizQuestions: quiz.quizQuestions
}
//console.log(testOb);
} finally {
await client.close();
cback(quizObject);
}
}
fun().catch(console.dir);
module.exports = {
fun: fun
}
UPDATE: Still stuck. I have read several different threads here about asynchronous calls and callbacks but I cannot get my function located in one file to return a value to the caller located in another file.
I am attempting to store the contents of an array in my database. I want to add a new row for each item in the array. Currently I am running:
router.post('/saveVenueMatches', (req, res) => {
const id = req.body.user_id;
const data = req.body.data;
res.setHeader('Access-Control-Allow-Origin', 'http://localhost:3000');
for (const venue of data) {
let sqlText = `INSERT INTO user_matches (user_id, venue_id) VALUES ($1, $2)`;
pool
.query(sqlText, [id, venue.id])
.then((response) => {
res.send(response.rows);
})
.catch((err) => {
console.log('Error saving venue matches', err);
});
}
});
This results in successful posts to my database, however my server returns the following error once for each item in the array:
Error saving venue matches Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
{
code: 'ERR_HTTP_HEADERS_SENT'
}
I have tried removing the line where I set the header but I still get the error. I am not exactly sure what is causing this.
Additionally if there is a better way to make a request like this please let me know.
You're trying to send a result to client in each iteration of a cycle but you can do it only once. Try to send results after a cycle or better turn the cycle into Promise.all call
My suggestion will be to use multiple insert at once, dont try to hit database too many times.
You can use this npm packages for multiple insert at once
const pg = require('pg');
const format = require('pg-format');
you need to just format your data accordingly
let user_matches = [[1, 2], [2, 3]];
let query1 = format('INSERT INTO users (user_id, venue_id) VALUES %L returning id', user_matches);
let {rows} = await client.query(query);
for reference you can visit this website, please
click here
I need to request data from my REST server to populate my UI (frontend). In doing so, I need to request some data from my and other servers. One such request is to get a list of states (provinces), process each one and add them to a select HTML component. I use fetch() and .json() amongst other tools to do this.
Problem:
In calling my REST server for json data, I receive the following data (taken from Chrome console):
{provinces:[Eastern Cape,Mpumalanga,Western Cape,Gauteng,KwaZulu Natal,North West,Northern Cape,Free
State,Limpopo]}
I intend to add each of these as an option to a select. While attempting to acquire the value for the provinces key, I get undefined.
I am making this call using:
fetch("http://localhost:3443/app/location/provinces").then(e => e.json()).then(e => console.log(e.provinces));
Further, since I can directly refer to json keys using the [] operator, I attempt this using
fetch("http://localhost:3443/app/location/provinces").then(e => e.json()).then(e => console.log(e['provinces']));
which as you may have guessed aswel, also returns undefined.
For the record, the full Chrome Console output is
Promise {<pending>}
undefined
Looking over some SO examples, I believe my call(s) may be correct, this one, and this one, and this one all which confirm its validity.
What else have I tried:
This SO post and this one suggested to use the json data response inside of the same then() call e.g.
fetch("http://localhost:3443/app/location/provinces").then(e => {
e.json().then(s => {
console.log(s['provinces']);
});
});
and
fetch("http://localhost:3443/app/location/provinces").then(e => {
e.json().then(s => {
console.log(s.provinces);
});
});
both which return:
Promise {<pending>}
undefined
What am I missing / doing wrong?
Update
Screenshot of Chrome console in order of commands listed above:
Resource file za-province-city.json
NodeJS express code:
const express = require('express');
const router = express.Router();
const fs = require('fs');
const raw = fs.readFileSync("./res/za-province-city.json");
const map = JSON.parse(raw);
const mapProvinceCity = {};
map.forEach(item => {
if (!mapProvinceCity.hasOwnProperty(item.ProvinceName)) {
mapProvinceCity[item.ProvinceName] = [];
}
mapProvinceCity[item.ProvinceName].push(item.City);
});
for (let key in mapProvinceCity) {
mapProvinceCity[key].sort((a, b) => a.toLocaleString().localeCompare(b.toLowerCase()));
}
router.get('/location/provinces', function (req, res, next) {
let strings = Object.keys(mapProvinceCity);
let json = JSON.stringify({provinces: strings}).replace(/"/g, '');
return res.json(json);
});
router.get('/location/:province/cities', function (req, res, next) {
let province = req.param('province');
let cities = mapProvinceCity[province];
let json = JSON.stringify({cities: cities}).replace(/"/g, '');
return res.json(json);
});
module.exports = router;
Note: if you are wondering about the replace(), each time I requested data in postman, I got
I think your issues all stem from a misunderstanding of Express' res.json().
This is basically a shortcut for
res.set("Content-type: application/json")
res.status(200).send(JSON.stringify(data))
I imagine your problems started when you thought you needed to stringify your data. What happens then is that your data is double-encoded / double stringified, hence the extra quotes. Removing the quotes though mangles your data.
console.log() is not a particularly good debugging tool as it obfuscates a lot of information. In your code, s is actually a string
"{provinces:[Eastern Cape,Mpumalanga,...]}"
I suggest you use the actual debugger instead.
The simple solution is to use res.json() as intended
router.get('/location/provinces', function (req, res, next) {
return res.json({ provinces: Object.keys(mapProvinceCity) });
});
with your client-side code looking like
fetch("http://localhost:3443/app/location/provinces")
.then(res => {
if (!res.ok) {
throw res
}
return res.json()
})
.then(data => {
console.log('Provinces:', data.provinces)
})
This goes for all your Express routes. Do not use JSON.stringify().
I'm trying to pull an array of data from a MongoDB database, and while the code is rusty (and I do want some corrections on it if it could be done better or is missing something or is wrong), it should be taking the array, finding the "user" and "description" objects, and then putting them into a discord.js message.
I've tried referencing the objects individually, making them strings, parsing the data, but I still cant find out how to do it. Heres the code I've been using.
module.exports.run = async (bot, message, args) => {
const MongoClient = require('mongodb').MongoClient;
const url = 'mongodb+srv://something:something#something/test?retryWrites=true&w=majority';
const assert = require('assert');
try {
function remindersChecker() {
let mongoClientPromise = MongoClient.connect(url, function (err, client) {
const db = client.db("reminders");
let date = new Date().getTime();
let now = Math.floor(Date.now() / 1000);
db.collection("reminders").find().toArray(function (err, result) {
let data = JSON.toObject();
console.log(data);
let user = data.author
let description = data.description
console.log(user);
user.send("I have a reminder for you! " + description)
})
})
}
remindersChecker()
} catch(err) {
catchError()
}
}}
module.exports.help = {
name: "check"
}
(The command is temporary and will be used on a setTimeout later, hence the function rather than just plain old code.)
Thanks! And I hope I can get help soon.
probably some more information would be great to better understand the problem.
from what i can see here, you are receiving an object from your database and converting it into an array here:
db.collection("reminders").find().toArray(function (err, result) {...
now that array is actually that result obtained from the callback and you are not using it at all, you probably have to iterate on that.
plus i remember that I used to write
...find({})..
to search in the database as for SELECT*FROM in SQL. maybe that can help too.
hope this helps.
File called testing.js
I can do whatever I like with the data in saveWeatherData but cannot call this function and return the data without getting 'undefined'
For example if i tried the below code in saveWeatherData it will print out the summary as expected...
console.log(The summary of the weather today is: ${dataArray[0]});
However I want to use these values within another file such as a server file that when connected to will display weather summary temperature etc.
So I need to return an array with these values in it so that I can call this function and get my data stored in an array for further use.
I know that the reason the array --dataArray is returning undefined is because asynchronous code.
The array is returned before we have gotten the data using the callback.
My question, is there anyway to do what I am trying to do?
I tried my best to explain the problem and what I want to do, hopefully its understandable.
Would I have to use a callback inside of a callback? To callback here to return the data when its been fetched?
I just cant get my head about it and have tried multiple things to try and get the result I am looking for.
My last idea and something i would prefer not to do is the use the 'fs' module to save the data to a text or json file for use in my other files through reading the data from the saved file...
I feel im close but cant get over the last hurdle, so ive decided to ask for a little help, even just point me on the right track and Ill continue to try and figure it out.
Phew...
Thank you for your time!
const request = require("request");
let dataArray = [];
let saveWeatherData = function(weatherData) {
dataArray = weatherData;
return dataArray;
};
let getWeatherData = function(callback) {
request({
url: `https://api.forecast.io/forecast/someexamplekey/1,-1`,
json: true
}, (error, response, body) => {
//Creating array to hold weather data until we can save it using callback...
let array = [];
if (error) {
console.log("Unable to connect with Dark Sky API servers.")
}
else {
console.log(`Successfully connected to Dark Sky API servers!\n`);
array.push(body.currently.summary, body.currently.temperature, body.currently.apparentTemperature, body.currently.windSpeed, body.currently.windBearing);
callback(array);
}
});
};
getWeatherData(saveWeatherData);
module.exports = {
saveWeatherData
};
My Other File...
File called server.js
const http = require("http");
const testing = require("./testing");
function onRequest(request, response){
let data = testing.saveWeatherData();
console.log(`A user made a request: ${request.url}`);
response.writeHead(200, {"context-type": "text/plain"});
response.write("<!DOCTYPE html>");
response.write("<html>");
response.write("<head>");
response.write("<title>Weather</title>");
response.write("</head>");
response.write("<body>");
response.write("Weather summary for today: " + data[0]);
response.write("</body>");
response.write("</html>");
response.end();
}
http.createServer(onRequest).listen(8888);
console.log("Server is now running on port 8888...");
I'm still not sure about what are you trying to do. However, I think you're not exporting what you suppose to be exporting. To avoid the use of so many callbacks you may use async/await.
Change this part of your server.js
async function onRequest(request, response) {
let data = await testing.getWeatherData();
console.log(`A user made a request: ${request.url}`);
response.writeHead(200, { 'context-type': 'text/plain' });
response.write('<!DOCTYPE html>');
response.write('<html>');
response.write('<head>');
response.write('<title>Weather</title>');
response.write('</head>');
response.write('<body>');
response.write('Weather summary for today: ' + data[0]);
response.write('</body>');
response.write('</html>');
response.end();
}
And this of your testing.
let getWeatherData = function() {
return new Promise(resolve =>
request(
{
url: `https://api.darksky.net/forecast/someexamplekey/1,-1`,
json: true
},
(error, response, body) => {
//Creating array to hold weather data until we can save it using callback...
let array = [];
if (error) {
console.log('Unable to connect with Dark Sky API servers.');
} else {
console.log(`Successfully connected to Dark Sky API servers!\n`);
array.push(
body.currently.summary,
body.currently.temperature,
body.currently.apparentTemperature,
body.currently.windSpeed,
body.currently.windBearing
);
resolve(array);
}
}
)
);
};
module.exports = {
getWeatherData
};
It will check for new Weather in each request. If you want to save the result to avoid checking every single time you might need to do something else. But I think for a weather app the important is to keep it updated.