We Keep Coding

Firebase function execution and subscription to list that is being updated by a firebase function

I think a firebase function updating a list that I have in the firebase database is being captured by a subscription that is subscribed to that list. From what the list output looks like on my phone (in the app)...and from what my console output looks like (the way it repeats) it seems like it is capturing the whole list and displaying it each time one is added. So (I looked this up)...I believe this equation represents what is happening:
(N(N + 1))/2
It is how you get the sum of all of the numbers from 1 to N. Doing the math in my case (N = 30 or so), I get around 465 entries...so you can see it is loading a ton, when I only want it to load the first 10.
To show what is happening with the output here is a pastebin https://pastebin.com/B7yitqvD.
In the output pay attention to the array that is above/before length - 1 load. You can see that it is rapidly returning an array with one more entry every time and adding it to the list. I did an extremely rough count of how many items are in my list too, and I got 440...so that roughly matches the 465 number.
The chain of events starts in a page that isn't the page with the list with this function - which initiates the sorting on the firebase functions side:
let a = this.http.get('https://us-central1-mane-4152c.cloudfunctions.net/sortDistance?text='+resp.coords.latitude+':'+resp.coords.longitude+':'+this.username);
this.subscription6 = a.subscribe(res => {
console.log(res + "response from firesbase functions");
loading.dismiss();
}, err => {
console.log(JSON.stringify(err))
loading.dismiss();
})
Here is the function on the page with the list that I think is capturing the entire sort for some reason. The subscription is being repeated as the firebase function sorts, I believe.
loadDistances() {
//return new Promise((resolve, reject) => {
let cacheKey = "distances"
let arr = [];
let mapped;
console.log("IN LOADDISTANCES #$$$$$$$$$$$$$$$$$$$$$");
console.log("IN geo get position #$$$$$$$5354554354$$$$$$$");
this.distancelist = this.af.list('distances/' + this.username, { query: {
orderByChild: 'distance',
limitToFirst: 10
}});
this.subscription6 = this.distancelist.subscribe(items => {
let x = 0;
console.log(JSON.stringify(items) + " length - 1 load");
items.forEach(item => {
let storageRef = firebase.storage().ref().child('/settings/' + item.username + '/profilepicture.png');
storageRef.getDownloadURL().then(url => {
console.log(url + "in download url !!!!!!!!!!!!!!!!!!!!!!!!");
item.picURL = url;
}).catch((e) => {
console.log("in caught url !!!!!!!$$$$$$$!!");
item.picURL = 'assets/blankprof.png';
});
this.distances.push(item);
if(x == items.length - 1) {
this.startAtKey4 = items[x].distance;
}
x++;
})
//this.subscription6.unsubscribe();
})
}
The subscription in loadDistances function works fine as long as I don't update the list from the other page - another indicator that it might be capturing the whole sort and listing it repeatedly as it sorts.
I have tried as as I could think of to unsubscribe from the list after I update...so then I could just load the list of 10 the next time the page with the list enters, instead of right after the update (over and over again). I know that firebase functions is in beta. Could this be a bug on their side? Here is my firebase functions code:
exports.sortDistance = functions.https.onRequest((req, res) => {
// Grab the text parameter.
var array = req.query.text.split(':');
// Push the new message into the Realtime Database using the Firebase Admin SDK.
// Get a database reference to our posts
var db = admin.database();
var ref = db.ref("profiles/stylists");
var promises = [];
// Attach an asynchronous callback to read the data at our posts reference
ref.on("value", function(snapshot) {
//console.log(snapshot.val());
var snap = snapshot.val();
for(const user in snap) {
promises.push(new Promise(function(resolve, reject) {
var snapadd = snap[user].address;
console.log(snapadd + " snap user address (((((((())))))))");
if(snapadd != null || typeof snapadd != undefined) {
googleMapsClient.geocode({
address: snapadd
}).asPromise()
.then(response => {
console.log(response.json.results[0].geometry.location.lat);
console.log(" +++ " + response.json.results[0].geometry.location.lat + ' ' + response.json.results[0].geometry.location.lng + ' ' + array[0] + ' ' + array[1]);
var distanceBetween = distance(response.json.results[0].geometry.location.lat, response.json.results[0].geometry.location.lng, array[0], array[1]);
console.log(distanceBetween + " distance between spots");
var refList = db.ref("distances/"+array[2]);
console.log(snap[user].username + " snap username");
refList.push({
username: snap[user].username,
distance: Math.round(distanceBetween * 100) / 100
})
resolve();
})
.catch(err => { console.log(err); resolve();})
}
else {
resolve();
}
}).catch(err => console.log('error from catch ' + err)));
//console.log(typeof user + 'type of');
}
var p = Promise.all(promises);
console.log(JSON.stringify(p) + " promises logged");
res.status(200).end();
}, function (errorObject) {
console.log("The read failed: " + errorObject.code);
});
});
What is weird is, when I check the firebase functions logs, all of this appears to only run once...but I still think the subscription could be capturing the whole sorting process in some weird way while rapidly returning it. To be as clear as possible with what I think is going on - I think each stage of the sort is being captured in an (N(N + 1))/2...starting at 1 and going to roughly 30...and the sum of the sorting ends up being the length of my list (with 1-10 items repeated over and over again).

I updated to angularfire2 5.0 and angular 5.0...which took a little while, but ended up solving the problem:
this.distanceList = this.af.list('/distances/' + this.username,
ref => ref.orderByChild("distance").limitToFirst(50)).valueChanges();
In my HTML I used an async pipe, which solved the sorting problem:
...
<ion-item *ngFor="let z of (distanceList|async)" no-padding>
...

findSuccess method itemsList is always undefined

I am new to Tizen working on a small application I am not able to figure out what is the problem.
When I am using these lines before it was working fine but now
var audioOnly = new tizen.AttributeFilter('type', 'EXACTLY', 'AUDIO');
tizen.content.find(findSuccess, findError, null, audioOnly);
Here is code for findSuccess which add lines in log
findSuccess(itemsList){
console.log('total items:'+itemsList);
console.log(itemsLis.name+'etc..');
}
In findSuccess method itemsList is always undefined, no object fetched even when there are files in the device. All settings are proper permissions for read and write is set in config.xml file.
This is Tizen webapi code

I tried with below code.
function findSuccess(items) {
for ( var i in items) {
console.log('Item title: ' + items[i].title);
console.log('Item URI: ' + items[i].contentURI);
console.log('Item type: ' + items[i].type);
}
}
function onError(error) {
console.log('Error: ' + error);
}
// Function to get list of all certain media files
function getSelectedMediaList() {
var mediasource = null;
mediasource = tizen.content;
var type = 'AUDIO';
var filter = new tizen.AttributeFilter("type", "EXACTLY", type);
try {
mediasource.find(findSuccess, onError, null, filter);
} catch (exc) {
console.log("findItems exception:" + exc.message);
}
}
getSelectedMediaList();
Don't forget to add privileges in config.xml
<tizen:privilege name="http://tizen.org/privilege/content.write"/>
<tizen:privilege name="http://tizen.org/privilege/content.read"/>

Error: CALL_AND_RETRY_LAST Allocation failed - Javascript heap out of memory

Im currently learning Node JS and Javascript. Im trying to develop an application to read and download Mangas.
First i want to build up a Database. Here is where i encounter the problem.
When i run my program on my server which has 4GB of RAM (to fill my DB) i get the Fatal Error Javascript heap out of memory.
When i run the same program on my local computer with 8GB of RAM, everything works as its supposed to.
Here is the code where i fill up my DB with Manga Chapters.
function insertChapters(callback){
sql_selectAll("Mangas", function (selectError, selectResult) {
if(!selectError){
selectResult.forEach(function (mangaItem, mangaIndex) {
gin.mangafox.chapters(mangaItem.Title)
.then(chapters =>{
chapters.forEach(function (chapterItem) {
var Chapter = {
Title: chapterItem.name,
NR: chapterItem.chap_number,
URL: chapterItem.src,
MangaID: mangaItem.MangaID,
MangaName: mangaItem.Title,
VolumeNR: chapterItem.volume
};
sql_insertInto("Chapters", Chapter, function (insertError, insertResult) {
if(!insertError){
var insertedChapter =
"------------------------------------------------------------------------\n" +
" Added new Chapter: " + Chapter.NR + " For: " + mangaItem.Title + "\n" +
"------------------------------------------------------------------------\n";
callback(null,insertedChapter ,insertResult);
}
else{
if(insertError.code === "ER_DUP_ENTRY") {
var dupEntry = "------------------------------------------------------------------------\n" +
" Duplicate Entry: Chapter: " + Chapter.NR + " For: " + mangaItem.Title + "\n" +
"------------------------------------------------------------------------\n"
callback(null, dupEntry, null);
}
else{
callback(insertError, null, null);
}
}
})
})
})
.catch(fetchChapterError => {
callback(fetchChapterError, null, null);
})
})
}
else{
callback(selectError, null, null);
}
});
}`
I dont really know how to solve this problem, because im not sure what the problem is:
Is the problem simply that i dont have enough RAM in my server?
Do i have a problem with my code? Am i leaking memory somewhere?
Is it possible that my code needs that much memory?
Thank you so much in advance, i appreciate every help i can get.
EDIT:
function sql_selectAll(tableName, callback){
var sql = 'SELECT * FROM `' + tableName + '`';
connection.query(sql, function (err, selectAllResult) {
callback(err, selectAllResult);
})
}
function sql_insertInto(tableName, insertionObject, callback) {
var sql = 'insert into ' + tableName + ' set ?';
connection.query(sql, insertionObject, function (err, insertResult) {
callback(err, insertResult);
});
}

You are calling the mangafox endpoint for every SQL result simultaneously, rather than one at a time or in chunks. You can try using async/await for this. I'm not familiar with the sql API you are using, but assuming that the methods sql_selectAll and sql_insertInto return promises if they aren't given a callback, you can rewrite your function to something like this:
async function insertChapters(callback) {
try {
const mangas = await sql_selectAll("Mangas");
for (const mangaItem of mangas) {
const chapters = await gin.mangafox.chapters(mangaItem.Title);
for (const chapterItem of chapters) {
const Chapter = {
Title: chapterItem.name,
NR: chapterItem.chap_number,
URL: chapterItem.src,
MangaID: mangaItem.MangaID,
MangaName: mangaItem.Title,
VolumeNR: chapterItem.volume
};
try {
const insertResult = await sql_insertInto("Chapters", Chapter);
const insertedChapter =
"------------------------------------------------------------------------\n" +
" Added new Chapter: " + Chapter.NR + " For: " + mangaItem.Title + "\n" +
"------------------------------------------------------------------------\n";
callback(null, insertedChapter, insertResult);
} catch (error) {
if (error.code === "ER_DUP_ENTRY") {
const dupEntry = "------------------------------------------------------------------------\n" +
" Duplicate Entry: Chapter: " + Chapter.NR + " For: " + mangaItem.Title + "\n" +
"------------------------------------------------------------------------\n";
callback(null, dupEntry, null);
} else {
throw error;
}
}
}
}
} catch (error) {
callback(error, null, null);
}
}
Notice the await keywords - these are allowed in async functions and essentially tell the JS engine to pause execution of the async function and do something else until the awaited promise resolves.
Also notice that you can use regular old fashioned try/catch blocks when handling promises using the await keyword! I didn't do anything special to handle the fetchChapterError because it will be handled by the outermost catch block! Also, for the insertion errors, if it's not a duplicate entry, we can just rethrow the error and let that get caught by the outermost catch block as well.
If your SQL functions do not return promises, then on Node version 8 (latest) you can use util.promisify:
const util = require('util');
sql_selectAll = util.promisify(sql_selectAll);
If you are not using Node 8, then you can use another promisify implementation (for example, check out bluebird), or you can write your own pretty easily (read the MDN article on promises).

[stompit STOMP client]Failover not working properly with STOMP producer

I am using stompit STOMP client. github - https://github.com/gdaws/node-stomp.
I am using ConnectFailover API for reconnect management. I have below code:
var stompit = require('stompit')
var reconnectOptions = {
'maxReconnects': 100,
'randomize' : false
};
var connManager = new stompit.ConnectFailover("failover:(stomp://mqbroker.nyc:61613,stomp://failovermqbroker.nyc:61613)", reconnectOptions);
connManager.on('error', function(error) {
var connectArgs = error.connectArgs;
var address = connectArgs.host + ':' + connectArgs.port;
console.error('Could not connect to ' + address + ' : ' + error.message);
});
connManager.on('connecting', function(connector) {
var address = connector.serverProperties.remoteAddress.transportPath;
console.log('Connecting to ' + address);
});
var totalMsgs = 50;
var count = 0;
var delayMs = 10000;
connManager.connect(function(error, client, reconnect) {
if (error) {
console.log("terminal error, given up reconnecting: " + error);
return;
}
client.on('error', function(error) {
// destroy the current client
client.destroy(error);
// calling reconnect is optional and you may not want to reconnect if the
// same error will be repeated.
reconnect();
});
var sendParams = {
'destination' : '/queue/myqueue',
'persistent' : 'true'
}
function sendMsg (){
setTimeout( function () {
console.log ('sending message ' + (count));
client.send(sendParams).end('Hello number ' + (count));
if (count++ < totalMsgs) {
sendMsg(count);
}
else {
client.send(sendParams).end('DISCONNECT');
client.disconnect();
console.log("Done.");
}
}, delayMs);
}
sendMsg();
});
The problem is that When the client gets disconnected from message broker, The producer keeps executing the sendMsg code and this causes loss of 2-3 messages in between. I want the client to stop executing when in disconnected state and resume when it is connected to failover instance.
Am I using the API incorrectly? What will be correct way to achieve this?
Have hacked at it for some time but this API lacks little documentation on how to use the features. Appreciate all the help.
Thanks,
xabhi

There is no problem with API but the with setTimeout code. I should clear the timeout when the client sees a connection failure.

Long GET requests are repeating in AngularJS & Node App

So I have an AngularJS(1) app which essentially runs a server side script when a user clicks a button.
client side controller Angular calls this method when a button is clicked:
$scope.executeDeployScript = function (demo) {
console.log("Starting "+ demo.instanceName);
$scope.clearDeploymentStatus();
$scope.processStarted = true;
$http.get('/api/demoSelect/startDemo', {
params: {
name: demo.instanceName,
path: demo.scriptPath
}
}).success(function (data) {
if (data === "Done") {
$http.get("/api/demoSelect/startDemo/ip", {
params: {
name: demo.instanceName
}
}).success(function (data) {
demo.url = data.replace(/\s/g, '') + ":" + demo.port
$scope.deploymentStatus = data.replace(/\s/g, '') + ":" + demo.port;
$scope.deploymentSuccess = true;
});
} else {
$scope.deploymentStatus = "Demo deployed failed. Try again : " + data;
$scope.deploymentSuccess = false;
}
});
server side route:
app.get("/api/demoSelect/startDemo", function (req, res, next) {
var child;
var demoName = req.query.name;
var demoPath = req.query.path;
var response = "";
console.log("Running boot script for "+ demoName);
child = exec(demoPath + " " + demoName,
function (error, stdout, stderr) {
if (error !== null) {
console.log('exec error!: ' + stderr);
res.send(stderr);
} else {
console.log(stdout);
res.send("Done");
}
});
});
The script takes a long time to complete (a couple minutes) and I noticed that the GET request is being repeated after a couple of minutes which in turn interrupts the currently running script and starts it again. The repeat is only happening server side since none of the angular client side code is being executed again. I am noticing the get request being repeated in the gulp debug output
Is there some time of time-out value I need to adjust for long get requests to prevent them from being repeated? If so, is that something that's handled in Node or Angular?
Any help is appreciated