We Keep Coding

In JS which one makes sense: searching for a value in object collection by foreach vs keeping multiple collections with different keys

I'm working on some kind of 1:1 chat system, the environment is Node.JS
For each country, there is a country room (lobby), for each socket client there is a js class/object is being created and each object is in a list with their unique user id.
This unique id is preserved even users logged in from different browser tabs etc..
Each object stored in collections like: "connections" (all of them), "operators"(only operators), "{countryISO}_clients" (users) and the reference key is their unique id.
In some circumstances, I need to access these connections by their socket ids.
At this point, I can think of 2 resolutions.
Using a for each loop to find the desired object
Creating another collection, this time instead of unique id use socket id (or something else.)
Which one makes sense? Because in JS since this collection will be a reference list instead of a copy, it feels like it makes sense (and beautiful looking) but I can't be sure. Which one is expensive in memory/performance terms?
I can't make thorough tests since I don't know how to create dummy (simultaneous) socket connections.
Expected connected socket client count: 300 - 1000 (depends on the time of the day)
e.g. user:
"qk32d2k":{
"uid":"qk32d2k",
"name":"Josh",
"socket":"{socket.io's socket reference}",
"role":"user",
"rooms":["room1"],
"socketids":["sid1"]
"country":"us",
...
info:() => { return gatherSomeData(); },
update:(obj) => { return updateSomeData(obj); },
send:(data)=>{ /*send data to this user*/ }
}
e.g. Countries collection:
{
us:{
"qk32d2k":{"object above."}
"l33t71":{"another user object."}
},
ca:{
"asd231":{"other user object."}
}
}

Pick Simple Design First that Optimizes for Most Common Access
There is no ideal answer here in the absolute. CPUs are wicked fast these days, so if I were you I'd start out with one simple mechanism of storing the sockets that you can access both ways you want, even if one way is kind of a brute force search. Pick the data structure that optimizes the access mechanism that you expect to be either most common or most sensitive to performance.
So, if you are going to be looking up by userID the most, then I'd probably store the sockets in a Map object with the userID as the key. That will give you fast, optimized access to get the socket for a given userID.
For finding a socket by some other property of the socket, you will just iterate the Map item by item until you find the desired match on some other socket property. I'd probably use a for/of loop because it's both fast and easy to bail out of the loop when you've found your match (something you can't do on a Map or Array object with .forEach()). You can obviously make yourself a little utility function or method that will do the brute force lookup for you and that will allow you to modify the implementation later without changing much calling code.
Measure and Add Further Optimization Later (if data shows you need to)
Then, once you get up to scale (or simulated scale in pre-production test), you take a look at the performance of your system. If you have loads of room to spare, you're done - no need to look further. If you have some operations that are slower than desired or higher than desired CPU usage, then you profile your system and find out where the time is going. It's most likely that your performance bottlenecks will be elsewhere in your system and you can then concentrate on those aspects of the system. If, in your profiling, you find that the linear lookup to find the desired socket is causing some of your slow-down, then you can make a second parallel lookup Map with the socketID as the key in order to optimize that type of lookup.
But, I would not recommend doing this until you've actually shown that it is an issue. Premature optimization before you have actual metrics that prove it's worth optimizing something just add complexity to a program without any proof that it is required or even anywhere close to a meaningful bottleneck in your system. Our intuition about what the bottlenecks are is often way, way off. For that reasons, I tend to pick an intelligent first design that is relatively simple to implement, maintain and use and then, only when we have real usage data by which we can measure actual performance metrics would I spend more time optimizing it or tweaking it or making it more complicated in order to make it faster.
Encapsulate the Implementation in Class
If you encapsulate all operations here in a class:
Adding a socket to the data structure.
Removing a socket from the data structure.
Looking up by userID
Looking up by socketID
Any other access to the data structure
Then, all calling code will access this data structure via the class and you can tweak the implementation some time in the future (to optimize based on data) without having to modify any of the calling code. This type of encapsulation can be very useful if you suspect future modifications or change of modifications to the way the data is stored or accessed.
If You're Still Worried, Design a Quick Bench Measurement
I created a quick snippet that tests how long a brute force lookup is in a 1000 element Map object (when you want to find it by something other than what the key is) and compared it to an indexed lookup.
On my computer, a brute force lookup (non-indexed lookup) takes about 0.002549 ms per lookup (that's an average time when doing 1,000,000 lookups. For comparison an indexed lookup on the same Map takes about 0.000017 ms. So you save about 0.002532 ms per lookup. So, this is fractions of a millisecond.
function addCommas(str) {
var parts = (str + "").split("."),
main = parts[0],
len = main.length,
output = "",
i = len - 1;
while(i >= 0) {
output = main.charAt(i) + output;
if ((len - i) % 3 === 0 && i > 0) {
output = "," + output;
}
--i;
}
// put decimal part back
if (parts.length > 1) {
output += "." + parts[1];
}
return output;
}
let m = new Map();
// populate the Map with objects that have a property that
// you have to do a brute force lookup on
function rand(min, max) {
return Math.floor((Math.random() * (max - min)) + min)
}
// keep all randoms here just so we can randomly get one
// to try to find (wouldn't normally do this)
// just for testing purposes
let allRandoms = [];
for (let i = 0; i < 1000; i++) {
let r = rand(1, 1000000);
m.set(i, {id: r});
allRandoms.push(r);
}
// create a set of test lookups
// we do this ahead of time so it's not part of the timed
// section so we're only timing the actual brute force lookup
let numRuns = 1000000;
let lookupTests = [];
for (let i = 0; i < numRuns; i++) {
lookupTests.push(allRandoms[rand(0, allRandoms.length)]);
}
let indexTests = [];
for (let i = 0; i < numRuns; i++) {
indexTests.push(rand(0, allRandoms.length));
}
// function to brute force search the map to find one of the random items
function findObj(targetVal) {
for (let [key, val] of m) {
if (val.id === targetVal) {
return val;
}
}
return null;
}
let startTime = Date.now();
for (let i = 0; i < lookupTests.length; i++) {
// get an id from the allRandoms to search for
let found = findObj(lookupTests[i]);
if (!found) {
console.log("!!didn't find brute force target")
}
}
let delta = Date.now() - startTime;
//console.log(`Total run time for ${addCommas(numRuns)} lookups: ${delta} ms`);
//console.log(`Avg run time per lookup: ${delta/numRuns} ms`);
// Now, see how fast the same number of indexed lookups are
let startTime2 = Date.now();
for (let i = 0; i < indexTests.length; i++) {
let found = m.get(indexTests[i]);
if (!found) {
console.log("!!didn't find indexed target")
}
}
let delta2 = Date.now() - startTime2;
//console.log(`Total run time for ${addCommas(numRuns)} lookups: ${delta2} ms`);
//console.log(`Avg run time per lookup: ${delta2/numRuns} ms`);
let results = `
Total run time for ${addCommas(numRuns)} brute force lookups: ${delta} ms<br>
Avg run time per brute force lookup: ${delta/numRuns} ms<br>
<hr>
Total run time for ${addCommas(numRuns)} indexed lookups: ${delta2} ms<br>
Avg run time per indexed lookup: ${delta2/numRuns} ms<br>
<hr>
Net savings of an indexed lookup is ${(delta - delta2)/numRuns} ms per lookup
`;
document.body.innerHTML = results;

How to implement a robust hash table like v8

Looking to learn how to implement a hash table in a decent way in JavaScript.
I would like for it to be able to:
Efficiently resolve collisions,
Be space efficient, and
Be unbounded in size (at least in principle, like v8 objects are, up to the size of the system memory).
From my research and help from SO, there are many ways to resolve collisions in hash tables. The way v8 does it is Quadratic probing:
hash-table.h
The wikipedia algorithm implementing quadratic probing in JavaScript looks something like this:
var i = 0
var SIZE = 10000
var key = getKey(arbitraryString)
var hash = key % SIZE
if (hashtable[hash]) {
while (i < SIZE) {
i++
hash = (key + i * i) % SIZE
if (!hashtable[hash]) break
if (i == SIZE) throw new Error('Hashtable full.')
}
hashtable[hash] = key
} else {
hashtable[hash] = key
}
The elements that are missing from the wikipedia entry are:
How to compute the hash getKey(arbitraryString). Hoping to learn how v8 does this (not necessarily an exact replica, just along the same lines). Not being proficient in C it looks like the key is an object, and the hash is a 32 bit integer. Not sure if the lookup-cache.h is important.
How to make it dynamic so the SIZE constraint can be removed.
Where to store the final hash, and how to compute it more than once.
V8 allows you to specify your own "Shape" object to use in the hash table:
// The hash table class is parameterized with a Shape.
// Shape must be a class with the following interface:
// class ExampleShape {
// public:
// // Tells whether key matches other.
// static bool IsMatch(Key key, Object* other);
// // Returns the hash value for key.
// static uint32_t Hash(Isolate* isolate, Key key);
// // Returns the hash value for object.
// static uint32_t HashForObject(Isolate* isolate, Object* object);
// // Convert key to an object.
// static inline Handle<Object> AsHandle(Isolate* isolate, Key key);
// // The prefix size indicates number of elements in the beginning
// // of the backing storage.
// static const int kPrefixSize = ..;
// // The Element size indicates number of elements per entry.
// static const int kEntrySize = ..;
// // Indicates whether IsMatch can deal with other being the_hole (a
// // deleted entry).
// static const bool kNeedsHoleCheck = ..;
// };
But not sure what the key is and how they convert that key to the hash so keys are evenly distributed and the hash function isn't just a hello-world example.
The question is, how to implement a quick hash table like V8 that can efficiently resolve collisions and is unbounded in size. It doesn't have to be exactly like V8 but have the features outlined above.
In terms of space efficiency, a naive approach would do var array = new Array(10000), which would eat up a bunch of memory until it was filled out. Not sure how v8 handles it, but if you do var x = {} a bunch of times, it doesn't allocate a bunch of memory for unused keys, somehow it is dynamic.
I'm stuck here essentially:
var m = require('node-murmurhash')
function HashTable() {
this.array = new Array(10000)
}
HashTable.prototype.key = function(value){
// not sure if the key is actually this, or
// the final result computed from the .set function,
// and if so, how to store that.
return m(value)
}
HashTable.prototype.set = function(value){
var key = this.key(value)
var array = this.array
// not sure how to get rid of this constraint.
var SIZE = 10000
var hash = key % SIZE
var i = 0
if (array[hash]) {
while (i < SIZE) {
i++
hash = (key + i * i) % SIZE
if (!array[hash]) break
if (i == SIZE) throw new Error('Hashtable full.')
}
array[hash] = value
} else {
array[hash] = value
}
}
HashTable.prototype.get = function(index){
return this.array[index]
}

This is a very broad question, and I'm not sure what exactly you want an answer to. ("How to implement ...?" sounds like you just want someone to do your work for you. Please be more specific.)
How to compute the hash
Any hash function will do. I've pointed out V8's implementation in the other question you've asked; but you really have a lot of freedom here.
Not sure if the lookup-cache.h is important.
Nope, it's unrelated.
How to make it dynamic so the SIZE constraint can be removed.
Store the table's current size as a variable, keep track of the number of elements in your hash table, and grow the table when the percentage of used slots exceeds a given threshold (you have a space-time tradeoff there: lower load factors like 50% give fewer collisions but use more memory, higher factors like 80% use less memory but hit more slow cases). I'd start with a capacity that's an estimate of "minimum number of entries you'll likely need", and grow in steps of 2x (e.g. 32 -> 64 -> 128 -> etc.).
Where to store the final hash,
That one's difficult: in JavaScript, you can't store additional properties on strings (or primitives in general). You could use a Map (or object) on the side, but if you're going to do that anyway, then you might as well use that as the hash table, and not bother implementing your own thing on top.
and how to compute it more than once.
That one's easy: invoke your hashing function again ;-)
I just want a function getUniqueString(string)
How about this:
var table = new Map();
var max = 0;
function getUniqueString(string) {
var unique = table.get(string);
if (unique === undefined) {
unique = (++max).toString();
table.set(string, unique);
}
return unique;
}
For nicer encapsulation, you could define an object that has table and max as properties.

Create a new collection from an existing one backbone.js

I have a view that should render models for the player who has the maximum points between all the teams. There are many ways to do this but here is the path I am leading down.
getMax : function(attribute) {
return this.collection.max(function (team) {
//return team.get('players').get(attribute);
var test = new PlayersCollection(team.get('players'));
console.log(test)
}, this);
},
This is in a marionette collectionView for teams (well composite, but it works like a collection). I understand why test returns the players for each team, but I can't think of a way to merge all the players into one collection then query who is the max points leader.
That said I may be able to avoid merging them in the first place if there is a way to determine who is the leader, but since the collection is nested I am a little stumped.
Since this.collection are the Teams, I thought something like this.collection.get('players').get('points') would allow me to get the max value of all the teams, but that didn't work.
Weird solution 1 I did a little hacking and came up with this. Alot of problems with this because Its stripped of backbone functionality meaning I cant return the model of the max player, only the points of that player, thats it.. still thinking (brain bleeding lol)
teams = App.data.teams
var points1 = teams.get('5368dcc1227a937829b2cb4a').players.pluck('points')
console.log(points1)
var points2 = teams.get('5368dcd9227a937829b2cb4c').players.pluck('points')
console.log(points2)
var test = points1.concat(points2);
console.log(test)
var maxi = _.max(test);
console.log(maxi)
Slightly better solution 2 merging the object arrays
teams = App.data.teams
var home = teams.get('5368dcc1227a937829b2cb4a').players.models;
var away = teams.get('5368dcd9227a937829b2cb4c').players.models;
all = home.concat(away);
console.log(all)

I think what you are looking for is something like this:
_.max(this.collection.get('players').pluck('points'));

Okay so I think I managed to create a somewhat elegant solution, playing in the console can really teach you a lot (highly recommended if you want to get better).
teams = App.data.teams
var home = teams.get('5368dcc1227a937829b2cb4a').players.models;
var away = teams.get('5368dcd9227a937829b2cb4c').players.models;
all = home.concat(away);
leaders = new PlayersCollection(all)
function mostPoints() {
return leaders.max(function(leader) {
return leader.get('points');
});
}
mostPoints();
Now the function will return the model of the player who has the most points out of everyone, pretty cool!

Breaking a cycle in FRP snake in Bacon.js

I've been following this snake example and decided to modify it to generate new apples only in empty (i.e. non-snake) cells. However, that's introduced a cyclic dependency between Observables, since generating new apples now depends not only on the last position but on the whole snake:
// stream of last `length` positions -- snake cells
var currentSnake = currentPosition.slidingWindowBy(length);
// stream of apple positions
var apples = appleStream(currentSnake);
// length of snake
var length = apples.scan(1, function(l) { return l + 1; });
Is there a nice way to resolve the cycle?
I can imagine how this would work in a messy state machine but not with clean FRP.
The closest I can think of is coalescing apples and length into one stream and making that stream generate its own "currentSnake" from currentPosition.
applesAndLength --> currentPosition
^ ^
| /
currentSnake
I haven't thought about the implementation much, though.

Once it has been constructed, Bacon can usually handle a cyclic dependency between Observables. It is constructing them that's a bit tricky.
In a language like Javascript, to create a structure with a cycle in it (i.e. a doubly-linked list), you need a mutable variable. For regular objects you use a regular variable or field to do that, e.g.
var tail = { prev: null, next: null };
var head = { prev: null, next: tail };
tail.prev = head; // mutating 'tail' here!
In Bacon, we operate on Observables instead of variables and objects, so we need some kind of a mutable observable to reach the same ends. Thankfully, Bacon.Bus is just the kind of observable we need:
var apples = new Bacon.Bus(); // plugged in later
var length = apples.scan(1, function(l) { return l + 1; });
var currentSnake = currentPosition.slidingWindowBy(length);
apples.plug(appleStream(currentSnake)); // mutating 'apples' here!
In my experience, it is preferrable to cut the cycles at EventStreams instead of Properties, because initial values tend to get lost otherwise; thus the reordering of apples and length.

JavaScript anti-flood spam protection?

I was wondering if it were possible to implement some kind of crude JavaScript anti-flood protection.
My code receives events from a server through AJAX, but sometimes these events can be quite frequent (they're not governed by me).
I have attempted to come up with a method of combating this, and I've written a small script: http://jsfiddle.net/Ry5k9/
var puts = {};
function receiverFunction(id, text) {
if ( !puts[id] ) {
puts = {};
puts[id] = {};
}
puts[id].start = puts[id].start || new Date();
var count = puts[id].count = puts[id].count + 1 || 0;
var time = (new Date() - puts[id].start) * 0.001;
$("text").set("text", (count / time.toFixed()).toString() + " lines/second");
doSomethingWithTextIfNotSpam(text);
}
};
which I think could prove effective against these kinds of attacks, but I'm wondering if it can be improved or perhaps rewritten?
So far, I think everything more than 3 or 2.5 lines per second seems like spam, but as time progresses forward (because start mark was set... well... at the start), an offender could simply idle for a while and then commence the flood, effectively never passing 1 line per minute.
Also, I would like to add that I use Mootools and Lo-Dash libraries (maybe they provide some interesting methods), but it would be preferable if this can be done using native JS.
Any insight is greatly appreciated!

If you are concerned about the frequency a particular javascript function fires, you could debounce the function.
In your example, I guess it would be something like:
onSuccess: function(){ _.debounce(someOtherFunction, timeOut)};
where timeout is the maximum frequency you want someOtherFunction to be called.

I know you asked about native JavaScript, but maybe take a look at RxJS.
RxJS or Reactive Extensions for JavaScript is a library for
transforming, composing, and querying streams of data. We mean all
kinds of data too, from simple arrays of values, to series of events
(unfortunate or otherwise), to complex flows of data.
There is an example on that page which uses the throttle method to "Ignores values from an observable sequence which are followed by another value before dueTime" (see source).
keyup = Rx.Observable.fromEvent(input, 'keyup').select(function(ev) {
return ev.target.value;
}).where(function(text) {
return text.length > 2;
}).throttle(500)
.distinctUntilChanged()
There might be a similar way to get your 2.5-3 per second and ignore the rest of the events until the next second.

I've spent many days pondering on effective measures to forbid message-flooding, until I came across the solution implemented somewhere else.
First, we need three things, penalty and score variables, and a point in time where last action occured:
var score = 0;
var penalty = 200; // Penalty can be fine-tuned.
var lastact = new Date();
Next, we decrease score by the distance between the previous message and current in time.
/* The smaller the distance, more time has to pass in order
* to negate the score penalty cause{d,s}.
*/
score -= (new Date() - lastact) * 0.05;
// Score shouldn't be less than zero.
score = (score < 0) ? 0 : score;
Then we add the message penalty and check if it crosses the threshold:
if ( (score += penalty) > 1000 ) {
// Do things.
}
Shouldn't forget to update last action afterwards:
lastact = new Date();