We Keep Coding

Optimizing hash table implementation to accommodate large amount of elements

Consider the following scenario:
One million clients visit a store and pay an amount of money using their credit card. The credit card codes are generated using a 16-digit number, and replacing 4 of its digits (randomly) with the characters 'A', 'B', 'C', 'D'. The 16-digit number is generated randomly once, and is used for every credit card, the only change between cards being the positions in the string of the aforementioned characters (that's ~40k possible distinct codes).
I have to organize the clients in a hash table, using a hash function of my choosing and also using open addressing (linear probing) to deal with the collisions. Once organized in the table, I have to find the client who
paid the most money during his purchases.
visited the store the most times.
My implementation of the hash table is as follows, and seems to be working correctly for the test of 1000 clients. However once I increase the number of clients to 10000 the page never finishes loading. This is a big issue since the total number of "shopping sessions" has to be one million, and I am not even getting close to that number.
class HashTable{
constructor(size){
this.size = size;
this.items = new Array(this.size);
this.collisions = 0;
}
put(k, v){
let hash = polynomial_evaluation(k);
//evaluating the index to the array
//using modulus a prime number (size of the array)
//This works well as long as the numbers are uniformly
//distributed and sparse.
let index = hash%this.size;
//if the array position is empty
//then fill it with the value v.
if(!this.items[index]){
this.items[index] = v;
}
//if not, search for the next available position
//and fill that with value v.
//if the card already is in the array,
//update the amount paid.
//also increment the collisions variable.
else{
this.collisions++;
let i=1, found = false;
//while the array at index is full
//check whether the card is the same,
//and if not then calculate the new index.
while(this.items[index]){
if(this.items[index] == v){
this.items[index].increaseAmount(v.getAmount());
found = true;
break;
}
index = (hash+i)%this.size;
i++;
}
if(!found){
this.items[index] = v;
}
found = false;
}
return index;
}
get(k){
let hash = polynomial_evaluation(k);
let index = hash%this.size, i=1;
while(this.items[index] != null){
if(this.items[index].getKey() == k){
return this.items[index];
}
else{
index = (hash+i)%this.size;
i++;
}
}
return null;
}
findBiggestSpender(){
let max = {getAmount: function () {
return 0;
}};
for(let item of this.items){
//checking whether the specific item is a client
//since many of the items will be null
if(item instanceof Client){
if(item.getAmount() > max.getAmount()){
max = item;
}
}
}
return max;
}
findMostFrequentBuyer(){
let max = {getTimes: function () {
return 0;
}};
for(let item of this.items){
//checking whether the specific item is a client
//since many of the items will be null
if(item instanceof Client){
if(item.getTimes() > max.getTimes()){
max = item;
}
}
}
return max;
}
}
To key I use to calculate the index to the array is a list of 4 integers ranging from 0 to 15, denoting the positions of 'A', 'B', 'C', 'D' in the string
Here's the hash function I am using:
function polynomial_evaluation(key, a=33){
//evaluates the expression:
// x1*a^(d-1) + x2*a^(d-2) + ... + xd
//for a given key in the form of a tuple (x1,x2,...,xd)
//and for a nonzero constant "a".
//for the evaluation of the expression horner's rule is used:
// x_d + a*(x_(d-1) + a(x_(d-2) + .... + a*(x_3 + a*(x_2 + a*x1))... ))
//defining a new key with the elements of the
//old times 2,3,4 or 5 depending on the position
//this helps for "spreading" the values of the keys
let nKey = [key[0]*2, key[1]*3, key[2]*4, key[3]*5];
let sum=0;
for(let i=0; i<nKey.length; i++){
sum*=a;
sum+=nKey[i];
}
return sum;
}
The values corresponding to the keys generated by the hash function are instances of a Client class which contains the fields amount (the amount of money paid), times (the times this particular client shopped), key (the array of 4 integers mentioned above), as well as getter functions for those fields. In addition there's a method that increases the amount when the same client appears more than once.
The size of the hash table is 87383 (a prime number) and the code in my main file looks like this:
//initializing the clients array
let clients = createClients(10000);
//creating a new hash table
let ht = new HashTable(N);
for(let client of clients){
ht.put(client.getKey(), client);
}
This keeps running until google chrome gives a "page not responding" error. Is there any way I can make this faster? Is my approach on the subject (perhaps even my choice of language) correct?
Thanks in advance.

The page is not responding since the main (UI) thread is locked. Use a WebWorker or ServiceWorker to handle the calculations, and post them as messages to the main thread.
Regarding optimizing your code, one thing I see is in findBiggestSpender. I'll break it down line-by-line.
let max = {getAmount: function () {
return 0;
}};
This is a waste. Just assign a local variable, no need to keep calling max.getAmount() in every iteration.
for(let item of this.items){
The fastest way to iterate a list in Javascript is with a cached length for loop: for (let item, len = this.items.length; i < len; i ++)
if(item instanceof Client){
This is slower than a hard null check, just use item !== null.

What is the time complexity of this for loop nested in a while loop?

I am trying to optimize a function. I believe this nested for loop is quadratic, but I'm not positive. I have recreated the function below
const bucket = [["e","f"],[],["j"],[],["p","q"]]
let totalLettersIWantBack = 4;
//I'm starting at the end of the bucket
function produceLetterArray(bucket, limit){
let result = [];
let countOfLettersAccumulated = 0;
let i = bucket.length - 1;
while(i > 0){
if(bucket[i].length > 0){
bucket[i].forEach( (letter) =>{
if(countOfLettersAccumulated === totalLettersIWantBack){
return;
}
result.push(letter);
countOfLettersAccumulated++;
})
}
i--;
}
return result;
}
console.log(produceLetterArray(bucket, totalLettersIWantBack));

Here is a trick for such questions. For the code whose complexity you want to analyze, just write the time that it would take to execute each statement in the worst case assuming no other statement exists. Note the comments begining with #operations worst case:
For the given code:
while(i > 0){ //#operations worst case: bucket.length
if(bucket[i].length > 0){ //#operations worst case:: 1
bucket[i].forEach( (letter) =>{ //#operations worst case: max(len(bucket[i])) for all i
if(countOfLettersAccumulated === totalLettersIWantBack){ //#operations worst case:1
return;
}
result.push(letter); //#operations worst case:1
countOfLettersAccumulated++; //#operations worst case:1
})
}
i--; ////#operations worst case:: 1
}
We can now multiply all the worst case times (since they all can be achieved in the worst case, you can always set totalLettersIWantBack = 10^9) to get the O complexity of the snippet:
Complexity = O(bucket.length * 1 * max(len(bucket[i])) * 1 * 1 * 1 * 1)
= O(bucket.length * max(len(bucket[i]))
If the length of each of the bucket[i] was a constant, K, then your complexity reduces to:
O(K * bucket.length ) = O(bucket.length)
Note that the complexity of the push operation may not remain constant as the number of elements grow (ultimately, the runtime will need to allocate space for the added elements, and all the existing elements may have to be moved).

Whether or not this is quadratic depends on what you consider N and how bucket is organized. If N is the total number of letters, then the runtime is bound by either the number of bins in your bucket, if that is larger than N, or it is bound by the number of letters in the bucket, if N is larger. In either case, the search time increases linearly with the larger bound, if one would dominate the other the time complexity is O(N). This is effectively a linear search with "turns" in it, scrunching a linear search and spacing it out does not change the time complexity. The existence of multiple loops in a piece of code does not alone make it non linear. Take the linear search example again. We search a list until we've found the largest element.
//12 elements
var array = [0,1,2,3,4,5,6,7,8,9,10,11];
var rows = 3;
var cols = 4;
var largest = -1;
for(var i = 0; i < rows; ++i){
for(var j = 0; j < cols; ++j){
var checked = array[(i * cols) + j];
if (checked > largest){
largest = checked;
}
}
}
console.log("found largest number (eleven): " + largest.toString());
Despite this using two loops instead of one, the runtime complexity is still O(N) where N is the number of elements in the input. Scrunching this down so each index is actually an array to multiple elements, or separating relevant elements by empty bins doesn't change the fact the runtime complexity is bound linearly.

This is technically linear with n being the number of elements total in your matrix. This is because the exit condition is the length of bucket and for each array in bucket you check if countOfLettersAccumulated is equal to totalLettersIWantBack. Continually looking at values.
It gets a lot more complicated if you are looking for an answer matching the dimensions of your matrix because it looks like the dimensions of bucket are not fixed.
You can turn this bit of code into constant by adding an additional check outside the bucket foreach which if countOfLettersAccumulated is equal to
totalLettersIWantBack then you do a break.

I like #axiom's explanation of complexity analyze.
Just would like to add possible optimized solution.
UPD .push (O(1)) is faster that .concat (O(n^2))
also here is test Array push vs. concat
const bucket = [["e","f"],[],["j", 'm', 'b'],[],["p","q"]]
let totalLettersIWantBack = 4;
//I'm starting at the end of the bucket
function produceLetterArray(bucket, limit){
let result = [];
for(let i = bucket.length-1; i > 0 && result.length < totalLettersIWantBack; i--){
//previous version
//result = result.concat(bucket[i].slice(0, totalLettersIWantBack-result.length));
//faster version of merging array
Array.prototype.push.apply(result, bucket[i].slice(0, totalLettersIWantBack-result.length));
}
return result;
}
console.log(produceLetterArray(bucket, totalLettersIWantBack));

Identifying edge cases of a one-dimensional array in Javascript

I'm creating a 2-dimensional heat map which has functionality when you click on any pixel. It grabs data associated with the index of every pixel (including adjacent pixels) and plots it. It currently looks like this:
The problem that I'm encountering is when I click on a left or right edge pixel, since it grabs data from adjacent pixels, it can retrieve data from the opposite side of the graph since it is all within a one-dimensional array. I am trying to create a conditional which checks if the clicked pixel is an edge case, and then configures the magnified graph accordingly to not show points from the other side of the main graph. This is the code I have so far:
// pushes all dataMagnified arrays left and right of i to magMainStore
var dataGrabber = function(indexGrabbed, arrayPushed) {
// iterates through all 5 pixels being selected
for (var b = -2; b <= 2; b++) {
var divValue = toString(i / cropLength + b);
// checks if selected index exists, and if it is not in the prior row, or if it is equal to zero
if (dataMagnified[indexGrabbed + b] != undefined && (& divValue.indexOf(".")!=-1)) {
dataMagnified[indexGrabbed + b].forEach(function(z) {
arrayPushed.push(z);
})
}
}
};
I am trying to get the same result as if I had a two dimensional array, and finding when the adjacent values within a single array is undefined. This is the line where I'm creating a conditional for that
if (dataMagnified[indexGrabbed + b] != undefined && (& divValue.indexOf(".")!=-1)) {
The second condition after the and is my attempts so far trying to figure this out. I'm unsure if I can even do this within a for loop that iterates 5 times or if I have to create multiple conditions for this. In addition, here's an image displaying what I'm trying to do:
Thank you!

Your approach looks overly complex and will perform rather slowly. For example, converting numbers to strings to be able to use .indexOf() to find a decimal point just for the sake of checking for integer numbers doesn't seem right.
A much simpler and more elegant solution might be the following function which will return the selection range bounded by the limits of the row:
function getBoundedSelection(indexGrabbed, selectionWidth) {
return dataMagnified.slice(
Math.max(Math.floor(indexGrabbed/cropLength) * cropLength, indexGrabbed - selectionWidth),
Math.min(rowStartIndex + cropLength, indexGrabbed + selectionWidth)
);
}
Here, to keep it as flexible as possible, selectionWidth determines the width of the selected range to either side of indexGrabbed. This would be 2 in your case.
As an explanation of what this does, I have broken it down:
function getBoundedSelection(indexGrabbed, selectionWidth) {
// Calculate the row indexGrabbed is on.
var row = Math.floor(indexGrabbed/cropLength);
// Determine the first index on that row.
var rowStartIndex = row * cropLength;
// Get the start index of the selection range or the start of the row,
// whatever is larger.
var selStartIndex = Math.max(rowStartIndex, indexGrabbed - selectionWidth);
// Determine the last index on that row
var rowEndIndex = rowStartIndex + cropLength;
// Get the end index of the selection range or the end of the row,
//whatever is smaller.
var selEndIndex = Math.min(rowEndIndex, indexGrabbed + selectionWidth);
// Return the slice bounded by the row's limits.
return dataMagnified.slice(selStartIndex, selEndIndex);
}

So I discovered that since the results of the clicked position would create a variable start and end position in the for loop, the only way to do this was as follows:
I started the same; all the code is nested in one function:
var dataGrabber = function(indexGrabbed, arrayPushed) {
I then create a second function that takes a start and end point as arguments, then passes them as the for loop starting point and ending condition:
var magnifyCondition = function (start, end) {
for (var b = start; b <= end; b++) {
if (dataMagnified[indexGrabbed + b] != undefined) {
dataMagnified[indexGrabbed + b].forEach(function (z) {
arrayPushed.push(z);
})
}
}
};
After that, I created 5 independent conditional statements since the start and end points can't be easily iterated through:
if (((indexGrabbed - 1) / cropLength).toString().indexOf(".") == -1) {
magnifyCondition(-1, 2);
}
else if ((indexGrabbed / cropLength).toString().indexOf(".") == -1) {
magnifyCondition(0, 2);
}
else if (((indexGrabbed + 1) / cropLength).toString().indexOf(".") == -1) {
magnifyCondition(-2, 0);
}
else if (((indexGrabbed + 2) / cropLength).toString().indexOf(".") == -1) {
magnifyCondition(-2, 1);
}
else {
magnifyCondition(-2, 2);
}
};
Lastly, I pass the index grabbed (i of the on clicked function) and an arbitrary array where the values get stored.
dataGrabber(i, magMainStore);
If there's a better way instead of the if statements, please let me know and I'd be happy to organize it better in the future!

Numeric sort of a stream that keeps alpha order when the number is the same

I tried a few of the regex sorts I found on SO, but I think they may not like the + symbol in the stream i'm needing to sort.
So I'm getting a data stream that looks like this (3 to 30 letters '+' 0 to 64000 number)
userString = "AAA+800|BBB+700|CCC+600|ZZZ+500|YYY+400|XXX+300|XXA+300|XXZ+300";
the output needs to be in the format:
array[0] = "XXA+300" // 300 being the lowest num and XXA being before XXX
array[...]
array[7] = "AAA+800"
I wish to order it from lowest num to highest num and reversed.
Here is my inefficient code. which loops 8x8 times. (my stream maybe 200 items long)
It works, but it looks messy. Can someone help me improve it so it uses less iterations?
var array = userString.split('|');
array.sort();
for(var i=0; i<len; i++) { // array2 contains just the numbers
bits = array[i].split('+');
array2[i] = bits[1];
}
array2.sort();
if(sort_order==2)
array2.reverse();
var c=0;
for(var a=0;a<len;a++) { // loop for creating array3 (the output)
for(var i=0; i<len ; i++) { // loop thru array to find matching score
bits = array[i].split('+');
if(bits[1] == array2[a]) { // found matching score
array3[c++] = bits[0]+'+'+bits[1]; // add to array3
array[i]='z+z'; // so cant rematch array position
}
}
}
array = array3;
Kind Regards

Please forgive the terse answer (and lack of testing), as I'm typing this on an iPhone.
var userArr = userString.split('|');
userArr.sort(function(a, b) {
var aArr = a.split('+'),
bArr = b.split('+'),
aLetters = aArr[0],
bLetters = bArr[0],
aNumbers = parseInt( aArr[1] ),
bNumbers = parseInt( bArr[1] );
if (aNumbers == bNumbers) {
return aLetters.localeCompare( bLetters );
}
return aNumbers - bNumbers;
/*
// Or, for reverse order:
return -(aNumbers - bNumbers);
// or if you prefer to expand your terms:
return -aNumbers + bNumbers;
*/
});
Basically we're splitting on | then doing a custom sort in which we split again on +. We convert the numbers to integers, then if they differ (e.g. 300 and 800) we compare them directly and return the result (because in that case the letters are moot). If they're the same, though (300 and 300) we compare the first parts (XXA and XXX) and return that result (assuming you want an ordinary alphabetical comparison). In this fashion the whole array is sorted.
I wasn't entirely sure what you meant by "and reversed" in your question, but hopefully this will get you started.
As you may've guessed this isn't totally optimal as we do split and parseInt on every element in every iteration, even if we already did in a previous iteration. This could be solved trivially by pre-processing the input, but with just 200 elements you probably won't see a huge performance hit.
Good luck!

alternatives for excessive for() looping in javascript

Situation
I'm currently writing a javascript widget that displays a random quote into a html element. the quotes are stored in a javascript array as well as how many times they've been displayed into the html element. A quote to be displayed cannot be the same quote as was previously displayed. Furthermore the chance for a quote to be selected is based on it's previous occurences in the html element. ( less occurrences should result in a higher chance compared to the other quotes to be selected for display.
Current solution
I've currently made it work ( with my severely lacking javascript knowledge ) by using a lot of looping through various arrays. while this currently works ( !! ) I find this solution rather expensive for what I want to achieve.
What I'm looking for
Alternative methods of removing an array element from an array, currently looping through the entire array to find the element I want removed and copy all other elements into a new array
Alternative method of calculating and selecting a element from an array based on it's occurence
Anything else you notice I should / could do different while still enforcing the stated business rules under Situation
The Code
var quoteElement = $("div#Quotes > q"),
quotes = [[" AAAAAAAAAAAA ", 1],
[" BBBBBBBBBBBB ", 1],
[" CCCCCCCCCCCC ", 1],
[" DDDDDDDDDDDD ", 1]],
fadeTimer = 600,
displayNewQuote = function () {
var currentQuote = quoteElement.text();
var eligibleQuotes = new Array();
var exclusionFound = false;
for (var i = 0; i < quotes.length; i++) {
var iteratedQuote = quotes[i];
if (exclusionFound === false) {
if (currentQuote == iteratedQuote[0].toString())
exclusionFound = true;
else
eligibleQuotes.push(iteratedQuote);
} else
eligibleQuotes.push(iteratedQuote);
}
eligibleQuotes.sort( function (current, next) {
return current[1] - next[1];
} );
var calculatePoint = eligibleQuotes[0][1];
var occurenceRelation = new Array();
var relationSum = 0;
for (var i = 0; i < eligibleQuotes.length; i++) {
if (i == 0)
occurenceRelation[i] = 1 / ((calculatePoint / calculatePoint) + (calculatePoint / eligibleQuotes[i+1][1]));
else
occurenceRelation[i] = occurenceRelation[0] * (calculatePoint / eligibleQuotes[i][1]);
relationSum = relationSum + (occurenceRelation[i] * 100);
}
var generatedNumber = Math.floor(relationSum * Math.random());
var newQuote;
for (var i = 0; i < occurenceRelation.length; i++) {
if (occurenceRelation[i] <= generatedNumber) {
newQuote = eligibleQuotes[i][0].toString();
i = occurenceRelation.length;
}
}
for (var i = 0; i < quotes.length; i++) {
var iteratedQuote = quotes[i][0].toString();
if (iteratedQuote == newQuote) {
quotes[i][1]++;
i = quotes.length;
}
}
quoteElement.stop(true, true)
.fadeOut(fadeTimer);
setTimeout( function () {
quoteElement.html(newQuote)
.fadeIn(fadeTimer);
}, fadeTimer);
}
if (quotes.length > 1)
setInterval(displayNewQuote, 10000);
Alternatives considered
Always chose the array element with the lowest occurence.
Decided against this as this would / could possibly reveal a too obvious pattern in the animation
combine several for loops to reduce the workload
Decided against this as this would make the code to esoteric, I'd probably wouldn't understand the code anymore next week
jsFiddle reference
http://jsfiddle.net/P5rk3/
Update
Rewrote my function with the techniques mentioned, while I fear that these techniques still loop through the entire array to find it's requirements, at least my code looks cleaner : )
References used after reading the answers here:
http://www.tutorialspoint.com/javascript/array_map.htm
http://www.tutorialspoint.com/javascript/array_filter.htm
http://api.jquery.com/jQuery.each/

I suggest array functions that are mostly supported (and easily added if not):
[].splice(index, howManyToDelete); // you can alternatively add extra parameters to slot into the place of deletion
[].indexOf(elementToSearchFor);
[].filter(function(){});
Other useful functions include forEach and map.
I agree that combining all the work into one giant loop is ugly (and not always possible), and you gain little by doing it, so readability is definitely the winner. Although you shouldn't need too many loops with these array functions.

The answer that you want:
Create an integer array that stores the number of uses of every quote. Also, a global variable Tot with the total number of quotes already used (i.e., the sum of that integer array). Find also Mean, as Tot / number of quotes.
Chose a random number between 0 and Tot - 1.
For each quote, add Mean * 2 - the number of uses(*1). When you get that that value has exceeded the random number generated, select that quote.
In case that quote is the one currently displayed, either select the next or the previous quote or just repeat the process.
The real answer:
Use a random quote, at the very maximum repeat if the quote is duplicated. The data usages are going to be lost when the user reloads/leaves the page. And, no matter how cleverly have you chosen them, most users do not care.
(*1) Check for limits, i.e. that the first or last quota will be eligible with this formula.

Alternative methods of removing an array element from an array
With ES5's Array.filter() method:
Array.prototype.without = function(v) {
return this.filter(function(x) {
return v !== x;
});
};
given an array a, a.without(v) will return a copy of a without the element v in it.
less occurrences should result in a higher chance compared to the other quotes to be selected for display
You shouldn't mess with chance - as my mathematician other-half says, "chance doesn't have a memory".
What you're suggesting is akin to the idea that numbers in the lottery that haven't come up yet must be "overdue" and therefore more likely to appear. It simply isn't true.

You can write functions that explicitly define what you're trying to do with the loop.
Your first loop is a filter.
Your second loop is a map + some side effect.
I don't know about the other loops, they're weird :P
A filter is something like:
function filter(array, condition) {
var i = 0, new_array = [];
for (; i < array.length; i += 1) {
if (condition(array[i], i)) {
new_array.push(array[i]);
}
}
return new_array;
}
var numbers = [1,2,3,4,5,6,7,8,9];
var even_numbers = filter(numbers, function (number, index) {
return number % 2 === 0;
});
alert(even_numbers); // [2,4,6,8]
You can't avoid the loop, but you can add more semantics to the code by making a function that explains what you're doing.

If, for some reason, you are not comfortable with splice or filter methods, there is a nice (outdated, but still working) method by John Resig: http://ejohn.org/blog/javascript-array-remove/