Let's consider a social network to be built by NodeJS and MongoDB.
So, if a user creates a new post, it should be saved to his/her followers feed.
The straightforward implementation of this operation as follow (simplified):
var newPost="New Post";
//get list of followers of user 1
var listOfFollowers = followersCollection.find({u:"1"});
for(var i=0;i<listOfFollowers.length;i++){
var followerId = listOfFollowers[i]._id;
//insert new post of user 1 to every follower feed
feedsCollection.insertOne(
{ownerId:followerId,authorId:"1",content:newPost}
);
}
This, of course, has very bad performance in case of big numbers in followers count. How can do this with a single fast performing MongoDB query?
MongoDB provides bulk document insert functionality, check out this link - https://docs.mongodb.com/manual/reference/method/db.collection.initializeUnorderedBulkOp/
db.collection.initializeUnorderedBulkOp() It creates an unordered list of operations and mongodb executes this list in parallel, so it's fast and you don't have to take extra care of performance as mongo handles it.
For ordered insertions, you can use db.collection.initializeOrderedBulkOp().
e.g.
var newPost="New Post";
var bulk = db.followersCollection.initializeUnorderedBulkOp();
//get list of followers of user 1
var listOfFollowers = followersCollection.find({u:"1"});
for(var i=0;i<listOfFollowers.length;i++){
var followerId = listOfFollowers[i]._id;
//insert new post of user 1 to every follower feed
bulk.insert( {ownerId:followerId,authorId:"1",content:newPost});
}
bulk.execute();
If you are using Mongoose then checkout Mongoose docs for the same. In the above example, I have just trying to explain how you can do it using plain MongoDB.
Insert Many Read this document I think you will get the answer
Check this:
var newPost="New Post";
//Object Array
var collection = []
//get list of followers of user 1
var listOfFollowers = followersCollection.find({u:"1"});
for(var i=0;i<listOfFollowers.length;i++){
var followerId = listOfFollowers[i]._id;
collection.push({ownerId:followerId,authorId:"1",content:newPost})
}
feedsCollection.insert(collection); //Or use insertMany()
You can create your object array and insert it at once.
Check documentation :- https://docs.mongodb.com/manual/reference/method/db.collection.insert/#insert-multiple-documents
Even though this is a simple answer for your question, If the collection array has a large number of elements, there still might be performance issues. So the best way to handle this is using triggers. https://docs.mongodb.com/stitch/triggers/
I am trying to pull a URL for an image in storage that is currently logged in the firebase real time database.
This is for a game of snap - there will be two cards on the screen (left image and right image) and when the two matches the user will click snap.
All of my image urls are stored in the following way:
Each one has a unique child called "index" - I also have another tree that is just a running count of each image record. So currently I am running a function that checks the total of the current count, then performs a random function to generate a random number, then performs a database query on the images tree using orderByChild and an equalTo that contains the random index number.
If I log the datasnap of this I can see a full node for one record (So index, score, url, user and their values) however if I try to just pull the URL I get returned a value of Null. I can, rather annoyingly, return the term "URL" seemingly at my leisure but I can't get the underlying value. I've wondered if this is due to it being a string and not a numeric but I can't find anything to suggest that is a problem.
Please bare in mind I've only been learning Javascript for about a week at max, so if I'm making obvious rookie errors that's probably why!
Below is a code snippet to show you what I mean:
var indRef = firebase.database().ref('index')
var imgRef = firebase.database().ref('images')
var leftImg = document.getElementById('leftImg')
var rightImg = document.getElementById('rightImg')
document.addEventListener('DOMContentLoaded', function(){
indRef.once('value')
.then(function(snapShot){
var indMax = snapShot.val()
return indMax;
})
.then(function(indMax){
var leftInd = Math.floor(Math.random()* indMax + 1)
imgRef.orderByChild('index').equalTo(leftInd).once('value', function(imageSnap){
var image = imageSnap.child('url').val();
leftImg.src=image;
})
})
})
When you execute a query against the Firebase Database, there will potentially be multiple results. So the snapshot contains a list of those results. Even if there is only a single result, the snapshot will contain a list of one result.
Your code needs to cater for that list, by looping over Snapshot.forEach():
imgRef.orderByChild('index').equalTo(leftInd).once('value', function(imageSnap){
imageSnap.forEach(function(child) {
var image = child.child('url').val();
leftImg.src=image;
})
})
I feel like I'm going about this in all the wrong way. I'm trying to automate some of my workload here. I'm cleaning up spreadsheets with 4 columns (A-E), 2000+ rows. Column B contains website URLs, column D contains the URL's business name, generated from another source.
Sometimes the tool doesn't grab the name correctly or the name is missing, so it populates the missing entries in column D with "------" (6 hyphens). I've been trying to make a function that takes an input cell, checks if the contents of the cell are "------", and if it is the function changes the contents of the input cell to the contents of the cell two columns to the left (which is generally a website url). This is what I've come up with.
function replaceMissing(input) {
var sheet = SpreadsheetApp.getActiveSheet();
//sets active range to the input cell
var cell = sheet.getRange('"' + input + '"');
//gets cell to fill input cell
var urlCell = sheet.getRange(cell.getRow(), cell.getColumn() - 2);
//gets contents of input cell as String
var data = cell.getValue();
//gets contents of urlCell as String
var data2 = cell.getValue();
//checks if input cell should be replaced
if (data === "------") {
//set current cell's value to the value of the cell 2 columns to the left
cell.setValue(data2);
}
}
When I attempt to use my function in my sheet, the cell is returning the error
Error Range not found (line 4).
I'm assuming, based on similar questions people have asked, that this is how you use the A1 notation of the function with an argument. However, that doesn't seem to be the case, so I'm stuck. I also don't think my solution is very good period.
1) It's somewhat ambiguous in GAS documentation, but custom functions have quite a few limitations. They are better suited for scenarios where you need to perform a simple calculation and return a string or a number type value to the cell. While custom functions can call some GAS services, this practice is strongly discouraged by Google.
If you check the docs for the list of supported services, you'll notice that they support only some 'get' methods for Spreadsheet service, but not 'set' methods https://developers.google.com/apps-script/guides/sheets/functions
That means you can't call cell.setValue() in the context of a custom function. It makes sense if you think about it - your spreadsheet can contain 1000s of rows, each with its own custom function making multiple calls to the server. In JavaScript, every function call creates its own execution context, so things could get ugly very quickly.
2) For better performance, use batch operations and don't alternate between read / write actions. Instead, read all the data you need for processing into variables and leave the spreadsheet alone. After processing your data, perform a single write action to update values in the target range. There's no need to go cell by cell when you can get the entire range using GAS.
Google Apps Script - best practices
https://developers.google.com/apps-script/guides/support/best-practices
Below is a quick code example that runs onOpen and onEdit. If you need more flexibility in terms of when to run the script, look into dynamically-created triggers https://developers.google.com/apps-script/reference/script/script-app
Because your spreadsheets have lots of rows, you may hit the execution quota anyway - by using triggers you can work around the limitation.
Finally, if a cell containing '----' is a rare occurrence, it might be better to create another array variable with new values and row numbers to update than updating the entire range.
Personally, I think the single range update action would still be quicker, but you could try both approaches and see which one works best.
function onOpen(){
test();
}
function onEdit() {
test();
}
function test() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName('yourSheetName');
//range to replace values in
var range = sheet.getRange(2, 4, sheet.getLastRow() - 1, 1);
//range to get new values from
var lookupRange = range.offset(0, -2);
//2d array of values from the target range
var values = range.getValues();
//2d array of values from the source range
var lookupValues = lookupRange.getValues();
//looping through the values array and checking if array element meets our condition
for (var i=0; i < values.length; i++) {
values[i][0] = (values[i][0] == '------') ? lookupValues[i][0] : values[i][0];
}
// one method call to update the range
range.setValues(values);
}
I am new to node-mysql but I am having a speed issue that is worrysome because of how simple it is.
I am working with a fairly reasonable amount of data (2000 mysql entries) and entering it via connection.query() has become very slow when I use multiple queries. The code is this
var rows = xlsx.rows;
for (var i=0; i<rows.length; ++i) {
var escaped = '';
var values = [];
var row = rows[i];
escaped += 'INSERT INTO Main (name, value, enabled) VALUES ';
var name = row['name'];
var val = row['value'];
var enabled = rows['enabled'];
escaped += '(?,?,?);';
values.push(name);
values.push(val);
values.push(enabled);
connection.query(escaped, values);
}
It takes upwards of one minute to input all the rows. The same issue has arisen when I use a multiple statements inside one query. The only time I am able to enter all the rows quickly, and almost instantly, is if I use one string and one entry, a.k.a.
INSERT INTO Main (name, value, enabled) VALUES (?,?,?), (?,?,?), (?,?,?)...
Am I just using the queries in an inefficient manner? Or is there an actual issue with the speed of the queries here?
As mentioned in the comments, this is just a slow way of inserting mysql data, it is much easier to use
connection.query('INSERT INTO Table (col1, col2) VALUES ?', [insert], callback);
where insert is a variable containing the multiple entry values
I'm trying to test out Firebase to allow users to post comments using push. I want to display the data I retrieve with the following;
fbl.child('sell').limit(20).on("value", function(fbdata) {
// handle data display here
}
The problem is the data is returned in order of oldest to newest - I want it in reversed order. Can Firebase do this?
Since this answer was written, Firebase has added a feature that allows ordering by any child or by value. So there are now four ways to order data: by key, by value, by priority, or by the value of any named child. See this blog post that introduces the new ordering capabilities.
The basic approaches remain the same though:
1. Add a child property with the inverted timestamp and then order on that.
2. Read the children in ascending order and then invert them on the client.
Firebase supports retrieving child nodes of a collection in two ways:
by name
by priority
What you're getting now is by name, which happens to be chronological. That's no coincidence btw: when you push an item into a collection, the name is generated to ensure the children are ordered in this way. To quote the Firebase documentation for push:
The unique name generated by push() is prefixed with a client-generated timestamp so that the resulting list will be chronologically-sorted.
The Firebase guide on ordered data has this to say on the topic:
How Data is Ordered
By default, children at a Firebase node are sorted lexicographically by name. Using push() can generate child names that naturally sort chronologically, but many applications require their data to be sorted in other ways. Firebase lets developers specify the ordering of items in a list by specifying a custom priority for each item.
The simplest way to get the behavior you want is to also specify an always-decreasing priority when you add the item:
var ref = new Firebase('https://your.firebaseio.com/sell');
var item = ref.push();
item.setWithPriority(yourObject, 0 - Date.now());
Update
You'll also have to retrieve the children differently:
fbl.child('sell').startAt().limitToLast(20).on('child_added', function(fbdata) {
console.log(fbdata.exportVal());
})
In my test using on('child_added' ensures that the last few children added are returned in reverse chronological order. Using on('value' on the other hand, returns them in the order of their name.
Be sure to read the section "Reading ordered data", which explains the usage of the child_* events to retrieve (ordered) children.
A bin to demonstrate this: http://jsbin.com/nonawe/3/watch?js,console
Since firebase 2.0.x you can use limitLast() to achieve that:
fbl.child('sell').orderByValue().limitLast(20).on("value", function(fbdataSnapshot) {
// fbdataSnapshot is returned in the ascending order
// you will still need to order these 20 items in
// in a descending order
}
Here's a link to the announcement: More querying capabilities in Firebase
To augment Frank's answer, it's also possible to grab the most recent records--even if you haven't bothered to order them using priorities--by simply using endAt().limit(x) like this demo:
var fb = new Firebase(URL);
// listen for all changes and update
fb.endAt().limit(100).on('value', update);
// print the output of our array
function update(snap) {
var list = [];
snap.forEach(function(ss) {
var data = ss.val();
data['.priority'] = ss.getPriority();
data['.name'] = ss.name();
list.unshift(data);
});
// print/process the results...
}
Note that this is quite performant even up to perhaps a thousand records (assuming the payloads are small). For more robust usages, Frank's answer is authoritative and much more scalable.
This brute force can also be optimized to work with bigger data or more records by doing things like monitoring child_added/child_removed/child_moved events in lieu of value, and using a debounce to apply DOM updates in bulk instead of individually.
DOM updates, naturally, are a stinker regardless of the approach, once you get into the hundreds of elements, so the debounce approach (or a React.js solution, which is essentially an uber debounce) is a great tool to have.
There is really no way but seems we have the recyclerview we can have this
query=mCommentsReference.orderByChild("date_added");
query.keepSynced(true);
// Initialize Views
mRecyclerView = (RecyclerView) view.findViewById(R.id.recyclerView);
mManager = new LinearLayoutManager(getContext());
// mManager.setReverseLayout(false);
mManager.setReverseLayout(true);
mManager.setStackFromEnd(true);
mRecyclerView.setHasFixedSize(true);
mRecyclerView.setLayoutManager(mManager);
I have a date variable (long) and wanted to keep the newest items on top of the list. So what I did was:
Add a new long field 'dateInverse'
Add a new method called 'getDateInverse', which just returns: Long.MAX_VALUE - date;
Create my query with: .orderByChild("dateInverse")
Presto! :p
You are searching limitTolast(Int x) .This will give you the last "x" higher elements of your database (they are in ascending order) but they are the "x" higher elements
if you got in your database {10,300,150,240,2,24,220}
this method:
myFirebaseRef.orderByChild("highScore").limitToLast(4)
will retrive you : {150,220,240,300}
In Android there is a way to actually reverse the data in an Arraylist of objects through the Adapter. In my case I could not use the LayoutManager to reverse the results in descending order since I was using a horizontal Recyclerview to display the data. Setting the following parameters to the recyclerview messed up my UI experience:
llManager.setReverseLayout(true);
llManager.setStackFromEnd(true);
The only working way I found around this was through the BindViewHolder method of the RecyclerView adapter:
#Override
public void onBindViewHolder(final RecyclerView.ViewHolder holder, int position) {
final SuperPost superPost = superList.get(getItemCount() - position - 1);
}
Hope this answer will help all the devs out there who are struggling with this issue in Firebase.
Firebase: How to display a thread of items in reverse order with a limit for each request and an indicator for a "load more" button.
This will get the last 10 items of the list
FBRef.child("childName")
.limitToLast(loadMoreLimit) // loadMoreLimit = 10 for example
This will get the last 10 items. Grab the id of the last record in the list and save for the load more functionality. Next, convert the collection of objects into and an array and do a list.reverse().
LOAD MORE Functionality: The next call will do two things, it will get the next sequence of list items based on the reference id from the first request and give you an indicator if you need to display the "load more" button.
this.FBRef
.child("childName")
.endAt(null, lastThreadId) // Get this from the previous step
.limitToLast(loadMoreLimit+2)
You will need to strip the first and last item of this object collection. The first item is the reference to get this list. The last item is an indicator for the show more button.
I have a bunch of other logic that will keep everything clean. You will need to add this code only for the load more functionality.
list = snapObjectAsArray; // The list is an array from snapObject
lastItemId = key; // get the first key of the list
if (list.length < loadMoreLimit+1) {
lastItemId = false;
}
if (list.length > loadMoreLimit+1) {
list.pop();
}
if (list.length > loadMoreLimit) {
list.shift();
}
// Return the list.reverse() and lastItemId
// If lastItemId is an ID, it will be used for the next reference and a flag to show the "load more" button.
}
I'm using ReactFire for easy Firebase integration.
Basically, it helps me storing the datas into the component state, as an array. Then, all I have to use is the reverse() function (read more)
Here is how I achieve this :
import React, { Component, PropTypes } from 'react';
import ReactMixin from 'react-mixin';
import ReactFireMixin from 'reactfire';
import Firebase from '../../../utils/firebaseUtils'; // Firebase.initializeApp(config);
#ReactMixin.decorate(ReactFireMixin)
export default class Add extends Component {
constructor(args) {
super(args);
this.state = {
articles: []
};
}
componentWillMount() {
let ref = Firebase.database().ref('articles').orderByChild('insertDate').limitToLast(10);
this.bindAsArray(ref, 'articles'); // bind retrieved data to this.state.articles
}
render() {
return (
<div>
{
this.state.articles.reverse().map(function(article) {
return <div>{article.title}</div>
})
}
</div>
);
}
}
There is a better way. You should order by negative server timestamp. How to get negative server timestamp even offline? There is an hidden field which helps. Related snippet from documentation:
var offsetRef = new Firebase("https://<YOUR-FIREBASE-APP>.firebaseio.com/.info/serverTimeOffset");
offsetRef.on("value", function(snap) {
var offset = snap.val();
var estimatedServerTimeMs = new Date().getTime() + offset;
});
To add to Dave Vávra's answer, I use a negative timestamp as my sort_key like so
Setting
const timestamp = new Date().getTime();
const data = {
name: 'John Doe',
city: 'New York',
sort_key: timestamp * -1 // Gets the negative value of the timestamp
}
Getting
const ref = firebase.database().ref('business-images').child(id);
const query = ref.orderByChild('sort_key');
return $firebaseArray(query); // AngularFire function
This fetches all objects from newest to oldest. You can also $indexOn the sortKey to make it run even faster
I had this problem too, I found a very simple solution to this that doesn't involved manipulating the data in anyway. If you are rending the result to the DOM, in a list of some sort. You can use flexbox and setup a class to reverse the elements in their container.
.reverse {
display: flex;
flex-direction: column-reverse;
}
myarray.reverse(); or this.myitems = items.map(item => item).reverse();
I did this by prepend.
query.orderByChild('sell').limitToLast(4).on("value", function(snapshot){
snapshot.forEach(function (childSnapshot) {
// PREPEND
});
});
Someone has pointed out that there are 2 ways to do this:
Manipulate the data client-side
Make a query that will order the data
The easiest way that I have found to do this is to use option 1, but through a LinkedList. I just append each of the objects to the front of the stack. It is flexible enough to still allow the list to be used in a ListView or RecyclerView. This way even though they come in order oldest to newest, you can still view, or retrieve, newest to oldest.
You can add a column named orderColumn where you save time as
Long refrenceTime = "large future time";
Long currentTime = "currentTime";
Long order = refrenceTime - currentTime;
now save Long order in column named orderColumn and when you retrieve data
as orderBy(orderColumn) you will get what you need.
just use reverse() on the array , suppose if you are storing the values to an array items[] then do a this.items.reverse()
ref.subscribe(snapshots => {
this.loading.dismiss();
this.items = [];
snapshots.forEach(snapshot => {
this.items.push(snapshot);
});
**this.items.reverse();**
},
For me it was limitToLast that worked. I also found out that limitLast is NOT a function:)
const query = messagesRef.orderBy('createdAt', 'asc').limitToLast(25);
The above is what worked for me.
PRINT in reverse order
Let's think outside the box... If your information will be printed directly into user's screen (without any content that needs to be modified in a consecutive order, like a sum or something), simply print from bottom to top.
So, instead of inserting each new block of content to the end of the print space (A += B), add that block to the beginning (A = B+A).
If you'll include the elements as a consecutive ordered list, the DOM can put the numbers for you if you insert each element as a List Item (<li>) inside an Ordered Lists (<ol>).
This way you save space from your database, avoiding unnecesary reversed data.