I have two intervals that need access to the same data.
So in one interval I want to push() an element to an array and then in the other interval I want to get the first element from the array and then remove it.
for example:
let array = [];
let count = 0;
setInterval( () => {
array.push(count);
count++;
}, 1000);
setInterval( () => {
let data = array[0];
array.shift();
console.log("received data: "+data);
}, 1000);
the output of this is:
received data: 0
received data: 1
received data: 2
received data: 3
....
Does this also work with more complex functions and bigger arrays?
Could this cause any weird behaviour? Maybe it could shift and push at the same time and mess up the array?
Is this a good way to do that? Are there better ways?
EDIT:
The reason i want to do this. Is because I want to download data from many different links. So inside my script i call a download(link) function, but this will result in the script trying to download a lot of links at the same time. So i want to create a buffer, so that the script only downloads from 100 links at the same time.
Inside the script i want to call download(link) wherever i want and then let an interval take care of downloading only 100 links at the same time. So it removes 100 links from a buffer and downloads them. While the script pushes new links to the same array.
My main concern is that while i am doing a shift() the array will reorganize itself somehow. Might js try to make a push() in between this reorganization phase? Or will js not do any array operations on the array until shift() is completed?
Your general idea of pushing links to an array asynchronously and then removing them from the array in a separate asynchronous task is fine, it'll work.
My main concern is that while i am doing a shift() the array will reorganize itself somehow. Might js try to make a push() in between this reorganization phase? Or will js not do any array operations on the array until shift() is completed?
Javascript is single-threaded, so this isn't something to worry about - if one interval triggers a function that does stuff, that function's synchronous actions (like manipulating an array) will run to the end before any other interval callbacks can run.
The issue of shared mutable state is a problem for many other languages, but not for Javascript, at least in most cases.
Related
I have certain requirements , I wanted to do the following in quickest way as possible.
I have 1000's of objects like below
{id:1,value:"value1"} . . {id:1000,value:"value1000"}
I want to access above objects by id
I want to clean the objects Lesser than certain id every few minutes (Because it generates 1000's of objects every second for my high frequency algorithm)
I can clean easily by using this.
myArray = myArray.filter(function( obj ) {
return obj.id > cleanSize;
});
I can find the object by id using
myArray.find(x => x.id === '45');
Problem is here , I feel that find is little slower when there is larger sets of data.So I created some objects of object like below
const id = 22;
myArray["x" + id] = {};
myArray["x" + id] = { id: id, value:"test" };
so I can access my item by id easily by myArray[x22]; , but problem is i am not able find the way to remove older items by id.
someone guide me better way to achieve the three points I mentioned above using arrays or objects.
The trouble with your question is, you're asking for a way to finish an algorithm that is supposed to solve a problem of yours, but I think there's something fundamentally wrong with the problem to begin with :)
If you store a sizeable amount of data records, each associated with an ID, and allow your code to access them freely, then you cannot have another part of your code dump some of them to the bin out of the blue (say, from within some timer callback) just because they are becoming "too old". You must be sure nobody is still working on them (and will ever need to) before deleting any of them.
If you don't explicitly synchronize the creation and deletion of your records, you might end up with a code that happens to work (because your objects happen to be processed quickly enough never to be deleted too early), but will be likely to break anytime (if your processing time increases and your data becomes "too old" before being fully processed).
This is especially true in the context of a browser. Your code is supposed to run on any computer connected to the Internet, which could have dozens of reasons to be running 10 or 100 times slower than the machine you test your code on. So making assumptions about the processing time of thousands of records is asking for serious trouble.
Without further specification, it seems to me answering your question would be like helping you finish a gun that would only allow you to shoot yourself in the foot :)
All this being said, any JavaScript object inherently does exactly what you ask for, provided you're okay with using strings for IDs, since an object property name can also be used as an index in an associative array.
var associative_array = {}
var bob = { id:1456, name:"Bob" }
var ted = { id:2375, name:"Ted" }
// store some data with arbitrary ids
associative_array[bob.id] = bob
associative_array[ted.id] = ted
console.log(JSON.stringify(associative_array)) // Bob and Ted
// access data by id
var some_guy = associative_array[2375] // index will be converted to string anyway
console.log(JSON.stringify(some_guy)) // Ted
var some_other_guy = associative_array["1456"]
console.log(JSON.stringify(some_other_guy)) // Bob
var some_AWOL_guy = associative_array[9999]
console.log(JSON.stringify(some_AWOL_guy)) // undefined
// delete data by id
delete associative_array[bob.id] // so long, Bob
console.log(JSON.stringify(associative_array)) // only Ted left
Though I doubt speed will really be an issue, this mechanism is about as fast as you will ever get JavaScript to run, since the underlying data structure is a hash table, theoretically O(1).
Anything involving array methods like find() or filter() will run in at least O(n).
Besides, each invocation of filter() would waste memory and CPU recreating the array to no avail.
I post data to backend, processing data takes some time and long polling is not a solution in my particular case, so I send request each 5 seconds with expand operator
this.apiService.postData(data).pipe(
expand((status) =>
status.comptete? this.apiService.askStatus(status.request_id).pipe(delay(5000)) : empty()
),
map((result) => {
// processing result here
})
);
The question is how can I make delay be dynamic (e.g. at first time I want to ask for status in 1 second, at second time in 2 seconds and so on)? And two more questions. Have I understood correctly that if I add take(N) operator that will limit askStatus calls to N? Have I understood correctly that I don't need to do any sort of unsubscription here?
expand() passes also index every time it calls the project function so you can calculate delay based on that:
expand((status, index) =>
status.comptete ? this.apiService.askStatus(...).pipe(delay(...)) : empty()
Using take(N) inside expand() won't help because expand() calls the project function on every emission from both source and inner Observables. But you can of course use take(N) after expand().
You don't have to unsubscribe from askStatus() manually if you handle unsubscription later where you also subscribe.
I'm making a function on a node.js server which reads a CSV file, I need to read all lines and execute several promised operations (MySQL queries) for each one (update or insert, then set an specified column which identifies this item as "modified in this execution") and once this finishes change another column on those not updated or inserted to identify this items as "deleted"
At first, the problem I had was that this CSV has millions of lines (literally) and a hundred of columns, so I run out of memory quite easily, and this number of lines can grow or decrease so I cannot know the amount of lines I will have to process every time I receive it.
I made a simple program that allows me to separate this CSV in some others with a readable amount of lines so my server can work with each one of them without dying, thus making an unknown amount of files each new file is processed, so now I have a different problem.
I want to read all of those CSVs, make those operations, and, once those operations are finished, execute the final one which will change those not updated/inserted. The only issue is that I need to read all of them and I cannot do this simultaneously, I have to make it sequentially, no matter how many they are (as said, after separating the main CSV, I may have 1 million lines divided into 3 files, or 2 millions into 6 files).
At first I though about using a forEach loop, but the problem is that, foreach doesn't respects the promisifying, so it will launch all of them, server will run out of memory when loading all those CSVs and then die. Honestly, using a while(boolean) on each iteration of the foreach to wait for the resolve of each promisified function seems pretty.... smelly for me, plus I feel like that solution will stop the server from working properly so I'm looking for a different solution.
Let me give you a quick explanation of what I want:
const arrayReader=(([arrayOfCSVs])=>{
initialFunction();
functions(arrayOfCSVs[0])
.then((result)=>{
functions(arrayOfCSVs[1])
.then((result2)=>{
functions(arrayOfCSVs[2])
(OnAndOnAndOnAndOn...)
.then((resultX)=>{
executeFinalFunction();
});
});
});
You can use Array.reduce to get the previous promise and queue new promise, without the need for waiting.
const arrayReader = ([arrayOfCSVs]) => {
initialFunction();
return arrayOfCSVs.reduce((prom, csv) => {
return prom.then(() => functions(csv));
}, Promise.resolve()).then(resultX => executeFinalFunction());
}
I am writing a scheduling program that returns JSON data about courses. I just started Node.js a week ago so I'm not sure if I'm thinking right.
I am trying to find a better way to write this code and avoid callback hell. I have already written the getJSON method.
/*getJSON(course-name, callback(JSONretrieved)): gets JSON info about name and
takes in a callback to manipulate retrieved JSON data*/
Now I want to get multiple course-name from a course array and check for time conflicts between them. I will then add all viable combinations to an answer array. My current idea is:
/*courseArray: array of classes to be compared
answers: array of all nonconflicting classes*/
var courseArray = ['MATH-123','CHEM-123']
var answers=[]
getJSON(courseArray[0],function(class1data){
getJSON(courseArray[1],function(class2data){
if(noConflict) answers.push( merge(class1data,class2data))
})
)
})
);
Finally, to access the answer array we wrap the entire code from above:
function getAnswers(cb){
/*courseArray: array of classes to be compared
answers: array of all nonconflicting classes*/
var courseArray = ['MATH-123','CHEM-123']
var answers=[]
getJSON(courseArray[0],function(class1data){
getJSON(courseArray[1],function(class2data){
/check for time conflicts between class1data and class2 data
if(noConflict(class1data,class2data)) answers.push( merge(class1data,class2data))
})
)
})
);
cb(answers)
}
and we call the function
getAnswers(function(ans){
//do processing of answers
console.log(ans)
})
My main question is if there is any way to make this code shorter, more readable, or less callback hecky.
You can use a promise library to make things easier for yourself. The way you're doing it can quickly get out of hand if the user selects more than a handful of courses to compare.
With something like async, you can make parallel calls to getJSON and your conflict code will run inside a single callback once all of the getJSON calls have returned. Your code will be much more readable and maintainable for large arrays of courses.
I am using async.js to work on an array that holds roughly 12'000 values. The problem I'm currently facing is that I am not 100% certain what I can take for granted from async.js. First let me show you an abstract of the iterator function im using (coffeescript):
baseFace = 0
dataBuffer = new DataBuffer() # my own class that wraps DataView on an Uint8Array
async.each(#blocks,
(block, callback) =>
numFaces = writeToBuffer(dataBuffer, block, baseFace)
baseFace += numFaces
callback(null)
,
() =>
console.log 'done'
)
So essentially the order in which the blocks are written to the buffers is unimportant, but there are two things I need to be assured of:
The entire writeToBuffer has to be done atomic, its not reentrant since it writes to the end of the given buffer and the block has to be written as one (multiple calls to DataView.setXXXX)
The baseFace variable too must be accessed atomically inside the function. it cant be that writeToBuffer is called for another element before the face count of the previous call to writeToBuffer has been added to baseFace
Basically you can say my iterator cannot be evaluated multiple times at the same time, not even with interleaving (like, writeToBuffer, writeToBuffer, baseFace += numFaces, baseFace += numFaces). Coming from C++/C# etc I always fear that something goes wrong when methods access data like in the above example.
Thank you for any insight or tips about that topic, Cromon