We Keep Coding

Adding only Distinct numbers during map reduce

I have a csv as to which number called which number and the call details(duration and time etc.)
i want to have all the numbers a particular number called in an array.
that array should be an array of documents and so,in each document i can have all the call details also.
so finally i need documents with a "caller" number and a "called" array(that array is as defined above).
for this i had come up with a map reduce solution.(quite basic and intuitive).
but my problem is that i need only distinct numbers that a "caller" number has called.
my current mapreduce script repeats the dialled numbers.
how can i only consider unique numbers during the reduce phase?
my code looks like this:(i enter this in the mongo shell)
db.contacts.mapReduce(
function(){
numbers = [];
value={phone:this.<<called_number>>};
numbers.push(value);
emit(this.<<caller_number>>,{called:numbers});
},
function(key,values) {
result={called:[]};
values.forEach(function (v) {
var i,j;
for(i=0;i<v.called.length;i++) {
var flag=0;
for(j=0;j<result.called.length;j++) {
if(v.called[i].phone==result.called[j].phone){
flag=1;
}
}
if(flag==0) {
result.called.push(v.called[i])
}
}
});
return result;
},
{"query": {},"out":"new_collection"}
)
I understand that the map and reduce functions are java script functions.
so even the javascript coders can help me out here(to create the reduce function).

Try this.
db.contacts.mapReduce(function(){
emit(this.<<caller_number>>, {called:this.<<called_number>>, callDuration:this.<<callDuration>>,...});}
,function(key,values)
{
var map = {};
var called=values.filter(function removeDuplicated(it){
if (!map[it.called]){
map[it.called] = 1;
return true;
}
return false;
})
return {caller:key, called:called};},
{"query": {},"out":"new_collection"})

Meteor 1.0 - Mongo queries using variables as key, including $inc

I'm working with a large dataset that needs to be efficient with its Mongo queries. The application uses the Ford-Fulkerson algorithm to calculate recommendations and runs in polynomial time, so efficiency is extremely important. The syntax is ES6, but everything is basically the same.
This is an approximation of the data I'm working with. An array of items and one item being matched up against the other items:
let items = ["pen", "marker", "crayon", "pencil"];
let match = "sharpie";
Eventually, we will iterate over match and increase the weight of the pairing by 1. So, after going through the function, my ideal data looks like this:
{
sharpie: {
pen: 1,
marker: 1,
crayon: 1,
pencil: 1
}
}
To further elaborate, the value next to each key is the weight of that relationship, which is to say, the number of times those items have been paired together. What I would like to have happen is something like this:
// For each in the items array, check to see if the pairing already
// exists. If it does, increment. If it does not, create it.
_.each(items, function(item, i) {
Database.upsert({ match: { $exist: true }}, { match: { $inc: { item: 1 } } });
})
The problem, of course, is that Mongo does not allow bracket notation, nor does it allow for variable names as keys (match). The other problem, as I've learned, is that Mongo also has problems with deeply nested $inc operators ('The dollar ($) prefixed field \'$inc\' in \'3LhmpJMe9Es6r5HLs.$inc\' is not valid for storage.' }).
Is there anything I can do to make this in as few queries as possible? I'm open to suggestions.
EDIT
I attempted to create objects to pass into the Mongo query:
_.each(items, function(item, i) {
let selector = {};
selector[match] = {};
selector[match][item] = {};
let modifier = {};
modifier[match] = {};
modifier[match]["$inc"] = {};
modifier[match]["$inc"][item] = 1
Database.upsert(selector, modifier);
Unfortunately, it still doesn't work. The $inc breaks the query and it won't let me go more than 1 level deep to change anything.
Solution
This is the function I ended up implementing. It works like a charm! Thanks Matt.
_.each(items, function(item, i) {
let incMod = {$inc:{}};
let matchMod = {$inc:{}};
matchMod.$inc[match] = 1;
incMod.$inc[item] = 1;
Database.upsert({node: item}, matchMod);
Database.upsert({node: match}, incMod);
});

I think the trouble comes from your ER model. a sharpie isn't a standalone entity, a sharpie is an item. The relationship between 1 item and other items is such that 1 item has many items (1:M recursive) and each item-pairing has a weight.
Fully normalized, you'd have an items table & a weights table. The items table would have the items. The weights table would have something like item1, item2, weight (in doing so, you can have asymmetrical weighting, e.g. sharpie:pencil = 1, pencil:sharpie = .5, which is useful when calculating pushback in the FFA, but I don't think that applies in your case.
Great, now let's mongotize it.
When we say 1 item has many items, that "many" is probably not going to exceed a few thousand (think 16MB document cap). That means it's actually 1-to-few, which means we can nest the data, either using subdocs or fields.
So, let's check out that schema!
doc =
{
_id: "sharpie",
crayon: 1,
pencil: 1
}
What do we see? sharpie isn't a key, it's a value. This makes everything easy. We leave the items as fields. The reason we don't use an array of objects is because this is faster & cleaner (no need to iterate over the array to find the matching _id).
var match = "sharpie";
var items = ["pen", "marker", "crayon", "pencil"];
var incMod = {$inc:{}};
var matchMod = {$inc:{}};
matchMod.$inc[match] = 1;
for (var i = 0; i < items.length; i++) {
Collection.upsert({_id: items[i]}, matchMod);
incMod.$inc[items[i]] = 1;
}
Collection.upsert({_id: match}, incMod);
That's the easy part. The hard part is figuring out why you want to use an FFA for a suggestion engine :-P.

MongoDB: How to rename a field using regex

I have a field in my documents, that is named after its timestamp, like so:
{
_id: ObjectId("53f2b954b55e91756c81d3a5"),
domain: "example.com",
"2014-08-07 01:25:08": {
A: [
"123.123.123.123"
],
NS: [
"ns1.example.com.",
"ns2.example.com."
]
}
}
This is very impractical for queries, since every document has a different timestamp.
Therefore, I want to rename this field, for all documents, to a fixed name.
However, I need to be able to match the field names using regex, because they are all different.
I tried doing this, but this is an illegal query.
db['my_collection'].update({}, {$rename:{ /2014.*/ :"201408"}}, false, true);
Does someone have a solution for this problem?
SOLUTION BASED ON NEIL LUNN'S ANSWER:
conn = new Mongo();
db = conn.getDB("my_db");
var bulk = db['my_coll'].initializeOrderedBulkOp();
var counter = 0;
db['my_coll'].find().forEach(function(doc) {
for (var k in doc) {
if (k.match(/^2014.*/) ) {
print("replacing " + k)
var unset = {};
unset[k] = 1;
bulk.find({ "_id": doc._id }).updateOne({ "$unset": unset, "$set": { WK1: doc[k]} });
counter++;
}
}
if ( counter % 1000 == 0 ) {
bulk.execute();
bulk = db['my_coll'].initializeOrderedBulkOp();
}
});
if ( counter % 1000 != 0 )
bulk.execute();

This is not a mapReduce operation, not unless you want a new collection that consists only of the _id and value fields that are produced from mapReduce output, much like:
"_id": ObjectId("53f2b954b55e91756c81d3a5"),
"value": {
"domain": "example.com",
...
}
}
Which at best is a kind of "server side" reworking of your collection, but of course not in the structure you want.
While there are ways to execute all of the code in the server, please don't try to do so unless you are really in a spot. These ways generally don't play well with sharding anyway, which is usually where people "really are in a spot" for the sheer size of records.
When you want to change things and do it in bulk, you generally have to "loop" the collection results and process the updates while having access to the current document information. That is, in the case where your "update" is "based on" information already contained in fields or structure of the document.
There is therefore not "regex replace" operation available, and there certainly is not one for renaming a field. So let's loop with bulk operations for the "safest" form of doing this without running the code all on the server.
var bulk = db.collection.initializeOrderedBulkOp();
var counter = 0;
db.collection.find().forEach(function(doc) {
for ( var k in doc ) {
if ( doc[k].match(/^2014.*/) ) {
var update = {};
update["$unset"][k] = 1;
update["$set"][ k.replace(/(\d+)-(\d+)-(\d+).+/,"$1$2$3") ] = doc[k];
bulk.find({ "_id": doc._id }).updateOne(update);
counter++;
}
}
if ( counter % 1000 == 0 ) {
bulk.execute();
bulk = db.collection.initializeOrderedBulkOp();
}
});
if ( counter % 1000 != 0 )
bulk.execute();
So the main things there are the $unset operator to remove the existing field and the $set operator to create the new field in the document. You need the document content to examine and use both the "field name" and "value", so hence the looping as there is no other way.
If you don't have MongoDB 2.6 or greater on the server then the looping concept still remains without the immediate performance benefit. You can look into things like .eval() in order to process on the server, but as the documentation suggests, it really is not recommended. Use with caution if you must.

As you already recognized, value-keys are indeed very bad for the MongoDB query language. So bad that what you want to do doesn't work.
But you could do it with a MapReduce. The map and reduce functions wouldn't do anything, but the finalize function would do the conversion in Javascript.
Or you could write a little program in a programming language of your which reads all documents from the collection, makes the change, and writes them back using collection.save.

How do I access the nested keys in this nested javascript object?

I have the following javascript object:
result = {
"banking6dig":{
"GM-B-001":{
"releaseDate":"2/2/2012 14:44","noOfHex":"3","versInfo":"6 digit Banking"
},
"GM-B-002":{
"releaseDate":"1/2/2012 14:46","noOfHex":"3","versInfo":"6 digit Banking with changes"
}
},
"paynpark":[]
}
Explanation:
"banking6dig" and "paynpark" are applications
"banking6dig" has two subversions : "GM-B-001" and "GM-B-002"
"paynpark" has no subversions.
Additionally, each subversion has its own properties,viz, "releaseDate", "noOfHex", and "versInfo".
This object "result" is built after a php request, so it can have any number of apps and subversions; the format, however is ALWAYS the same.
I have tried this on jsFiddle: http://jsfiddle.net/2JLtZ/1/
How do I find out the number of "subversions" in each "app"? (I get some 40 "subversions"!)
How do I add an app to the object "result", e.g., "electricity" with its properties reset?
How do I add a subversion to "paynpark", e.g. "fixedRate"?
How do I modify a subversions properties, for example change "releaseDate" of "GM-B-001" to "3/12/2012 14:46"?

1.
var i=0;
for (var j in result['banking6dig']) i++;
alert('banking6dig has '+i+' subversions'):
2.
result.electricity={}
3.
paynpark.fixedRate={};
Thanks AlienWebguy for pointing this out.
4.
result['banking6dig']["GM-B-001"]["releaseDate"]="3/12/2012 14:46";

result = {"banking6dig":{"GM-B-001":{"releaseDate":"2/2/2012 14:44","noOfHex":"3","versInfo":"6 digit Banking"},"GM-B-002":{"releaseDate":"1/2/2012 14:46","noOfHex":"3","versInfo":"6 digit Banking with changes"}},"paynpark":[]};
var app = [],
svn = {};
for (var _app in result) {
if(result.hasOwnProperty(_app)){
app.push(_app);
svn[_app] = [];
for (var _svn in result[_app]){
if(result[_app].hasOwnProperty(_svn)){
svn[_app].push(_svn);
}
}
}
}
// How man apps?
alert(app.length);
// How many subversions?
alert(svn['banking6dig'].length);
// Add an app 'electricity'
result.electricity = {};
// Add subversion to paynpark
result.paynpark.fixedRate = {}
// Add electricity
result.electricity = {};
app.push('electricity');
// Modify subversion
result.banking6dig['GM-B-001'].releaseDate = '3/12/2012';
console.log(result);
Demo: http://jsfiddle.net/AlienWebguy/DpCTP/

What's the best way to query an array in javascript to get just the items from it I want?

I have an array like this (with just over 3000 objects instead of the 3 here):
items = [{name:'charlie', age:'16'}, {name:'ben', age:'18'}, {name:'steve', age:'18'}]
What's the best way to return an array with just the objects of people who are 18? So I want:
items = [{name:'ben', age:'18'}, {name:'steve', age:'18'}]
The best I can think of is this (using jQuery):
newArray = []
$.each(items, function(index, item) {
if(item.age=='18') {
newArray.push(item)
}
})
Considering that there's 3000 thousand objects, and also that I'll be doing that comparison up to fifty times in one go, that's a lot of looping. Is there a better way?

You can use pure javascript
var wanted = items.filter( function(item){return (item.age==18);} );
And if your browser does not support the 1.6 version of javascript you can find an implementation of the filter method at https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Array/filter
Update
Speedwise there is a huge varying (had an error in the test) difference from a normal loop (depending on browser).. Have a look at this little test i made at http://jsperf.com/array-filter-vs-loop/3

Get matched item and items using find() and filter() method
If you want first matched single item, use find() method which returns single object.
If you want all matched , use filter() method which returns array of objects.
let items = [{name:'charlie', age:'16'},
{name:'ben', age:'18'},
{name:'steve', age:'18'}]
let all = items.filter(item=> item.age==='18')
console.log(all);
let single = items.find(item=> item.age==='18')
console.log(single);

If you're going to do the search often it may be best to keep a version of your data in a form that is quick to access.
I've used underscore.js (http://documentcloud.github.com/underscore/) to make it easy for myself, but this code here will create an object that holds your data indexed by the age field.
You end up with something that looks like this:
{
"16": [
{
"name": "charlie",
"age": "16"
}
],
"18": [
{
"name": "ben",
"age": "18"
},
{
"name": "steve",
"age": "18"
}
]
}
The code:
var itemsByAge = _(items).reduce(function(memo, item) {
memo[item.age] = memo[item.age] || [];
memo[item.age].push(item);
return memo;
}, {});
alert(JSON.stringify(itemsByAge["18"]));

No matter which method you choose (items.filter or any "query language" for json), a for loop is inevitable.
If performance is a concern, I would recommend you to use pure javascript instead of libraries like jQuery which will add overheads to the whole processing as is evident here.
Thus, your code would look like:
var newArray = [];
for(var i=0;i<items.length;i++) {
var item = items[i];
if(item.age == '18') {
newArray.push(item);
}
});

making use of javascript magnificent function eval() which evaluates string as code at runtime, we can define a prototype method for Array type
Array.prototype.where = function (query) {
var newArray = [];
for(var i=0; i<this.length; i++) {
var item = this[i];
if(eval( "item" + query )) {
newArray.push(item);
}
}
return newArray;
};
and use it with any array, passing the query as string
var newArray= items.where('.age >= 18');

Use the filter method of the array, it calls the provided callbackfunction once for each element in an array.
array.filter(<callbackfucntion>[, <Object to use>])

once i had such problem and i solved it like this
1- create an array of array
2- each index create an Index record
e.g.
var pAry=[];
var cAry=[{name:'ben', age:'18'}, {name:'steve', age:'18'}]
pAry[17]=cAry;
This way when u require person with age 18, you will get on index 17.