We Keep Coding

Unknown length array used to execute promised functions sequentially

I'm making a function on a node.js server which reads a CSV file, I need to read all lines and execute several promised operations (MySQL queries) for each one (update or insert, then set an specified column which identifies this item as "modified in this execution") and once this finishes change another column on those not updated or inserted to identify this items as "deleted"
At first, the problem I had was that this CSV has millions of lines (literally) and a hundred of columns, so I run out of memory quite easily, and this number of lines can grow or decrease so I cannot know the amount of lines I will have to process every time I receive it.
I made a simple program that allows me to separate this CSV in some others with a readable amount of lines so my server can work with each one of them without dying, thus making an unknown amount of files each new file is processed, so now I have a different problem.
I want to read all of those CSVs, make those operations, and, once those operations are finished, execute the final one which will change those not updated/inserted. The only issue is that I need to read all of them and I cannot do this simultaneously, I have to make it sequentially, no matter how many they are (as said, after separating the main CSV, I may have 1 million lines divided into 3 files, or 2 millions into 6 files).
At first I though about using a forEach loop, but the problem is that, foreach doesn't respects the promisifying, so it will launch all of them, server will run out of memory when loading all those CSVs and then die. Honestly, using a while(boolean) on each iteration of the foreach to wait for the resolve of each promisified function seems pretty.... smelly for me, plus I feel like that solution will stop the server from working properly so I'm looking for a different solution.
Let me give you a quick explanation of what I want:
const arrayReader=(([arrayOfCSVs])=>{
initialFunction();
functions(arrayOfCSVs[0])
.then((result)=>{
functions(arrayOfCSVs[1])
.then((result2)=>{
functions(arrayOfCSVs[2])
(OnAndOnAndOnAndOn...)
.then((resultX)=>{
executeFinalFunction();
});
});
});

You can use Array.reduce to get the previous promise and queue new promise, without the need for waiting.
const arrayReader = ([arrayOfCSVs]) => {
initialFunction();
return arrayOfCSVs.reduce((prom, csv) => {
return prom.then(() => functions(csv));
}, Promise.resolve()).then(resultX => executeFinalFunction());
}

js asynchronously using array push() shift()

I have two intervals that need access to the same data.
So in one interval I want to push() an element to an array and then in the other interval I want to get the first element from the array and then remove it.
for example:
let array = [];
let count = 0;
setInterval( () => {
array.push(count);
count++;
}, 1000);
setInterval( () => {
let data = array[0];
array.shift();
console.log("received data: "+data);
}, 1000);
the output of this is:
received data: 0
received data: 1
received data: 2
received data: 3
....
Does this also work with more complex functions and bigger arrays?
Could this cause any weird behaviour? Maybe it could shift and push at the same time and mess up the array?
Is this a good way to do that? Are there better ways?
EDIT:
The reason i want to do this. Is because I want to download data from many different links. So inside my script i call a download(link) function, but this will result in the script trying to download a lot of links at the same time. So i want to create a buffer, so that the script only downloads from 100 links at the same time.
Inside the script i want to call download(link) wherever i want and then let an interval take care of downloading only 100 links at the same time. So it removes 100 links from a buffer and downloads them. While the script pushes new links to the same array.
My main concern is that while i am doing a shift() the array will reorganize itself somehow. Might js try to make a push() in between this reorganization phase? Or will js not do any array operations on the array until shift() is completed?

Your general idea of pushing links to an array asynchronously and then removing them from the array in a separate asynchronous task is fine, it'll work.
My main concern is that while i am doing a shift() the array will reorganize itself somehow. Might js try to make a push() in between this reorganization phase? Or will js not do any array operations on the array until shift() is completed?
Javascript is single-threaded, so this isn't something to worry about - if one interval triggers a function that does stuff, that function's synchronous actions (like manipulating an array) will run to the end before any other interval callbacks can run.
The issue of shared mutable state is a problem for many other languages, but not for Javascript, at least in most cases.

Putting Firestore data into array with Angular

I have created a small application for keeping track of how much time I spend in different courses (as a teacher) using Angular 5 and putting my data in Firestore via AngularFire2. Most things have worked out nicely, but for the final part of the application I am having serious problems. I am quite a newbie at Angular/Firebase and most importantly (perhaps) with reactive programming.
The data that is stored in Firestore is quite simple:
Data stored in Firebase It consists of six fields for information about the course, when the course was held and most importantly what I did (called Element) and for how long (called Duration).
Now, what I would like to do is to make a report that summaries all the durations for each thing I did (Element). In essence, what I would like to do is the equivalent of a GroupBy in SQL. As I understand it Firebase does not have an operation for grouping.
My, so far failed approach to this, has been to try to create an array into which I put the data and make calculations on that. It is here that I fail.
My first attempt is based on creating an observer as most examples for retrieving data in tutorials use. WorkItem is a simple class that has the same content as in Firebase.
this.lectureDoc = this.afs.collection(`users/${this.afAuth.auth.currentUser.uid}/regTime`,
ref => ref.where('course', '==', this.selectedCourse.courseName).orderBy('element') );
this.lectureItems = this.lectureDoc.snapshotChanges().pipe(
map (courses => courses.map(a => {
const data = a.payload.doc.data() as WorkItem;
const id = a.payload.doc.id;
return { id, ...data };
})
));
this.lectureItems.subscribe(item => {
item.forEach(i => {
this.allElements.push(i);
});
});
console.error(this.allElements);
console.error(this.allElements[0]);
The two last lines show the problem, namely that the first (of the two last lines) will return the complete array while the second will return 'undefined'.
I understand that reactive programming will make asynchronous calls and that I therefore cannot know for sure when the data is filled. However, I do not understand why I can see the content of the array in the first of the two last lines, but not in the second.
My second attempt is based on getting the data from the documents of the collection itself and also, to prevent empty arrays, using .then() but again the two last lines show exactly the same output as previously.
this.lectureDoc.ref.get().then(item => {
item.forEach(i => {
this.lectures.push(i.data() as WorkItem);
});
});
console.error(this.lectures);
console.error(this.lectures[0]);
The output can be seen in this image: Output from the running program
So, to recap, what I would like to do is to collect all the data in an array of WorkItem and then calculate how much time has been spent on different tasks and then display it in list on the web page. I have not come to the display part and I suspect that I will have trouble again with the array not being populated when bound to the list. I have a hard time to understand reactive programming...
Any help would be greatly appreciated!
//Tobias

This is a pretty common behaviour of the console as far as I'm aware, at least in Chrome. When you log data that's async but haven't been populated yet, the console will still show the array once the async operation is completed.
But when you log an object in the array, at the time that you log it that object isn't there yet, therefore the log will show as undefined.
Simply do the logging inside of the forEach and you will see the data being pushed properly.
An another case, example there: Console.log behavior

Better design for data stored using HTML5 localStorage

I have a scenario on my web application and I would like suggestions on how I could better design it.
I have to steps on my application: Collection and Analysis.
When there is a collection happening, the user needs to keep informed that this collection is going on, and the same with the analysis. The system also shows the 10 last collection and analysis performed by the user.
When the user is interacting with the system, the collections and analysis in progress (and, therefore, the last collections/analysis) keep changing very frequently. So, after considering different ways of storing these informations in order to display them properly, as they are so dynamic, I chose to use HTML5's localStorage, and I am doing everything with JavaScript.
Here is how they are stored:
Collection in Progress: (set by a function called addItem that receives ITEMNAME)
Key: c_ITEMNAME_Storage
Value: c_ITEMNAME
Collection Finished or Error: (set by a function called editItem that also receives ITEMNAME and changes the value of the corresponding key)
Key: c_ITEMNAME_Storage
Value: c_Finished_ITEMNAME or c_Error_ITEMNAME
Collection in the 10 last Collections (set by a function called addItemLastCollections that receives ITEMNAME and prepares the key with the current date and time)
Key: ORDERNUMBER_c_ITEMNAME_DATE_TIME
Value: c_ITEMNAME
Note: The order number is from 0 to 9, and when each collection finishes, it receives the number 0. At the same time, the number 9 is deleted when the addItemLastCollections function is called.
For the analysis is pretty much the same, the only thing that changes is that the "c" becomes an "a".
Anyway, I guess you understood the idea, but if anything is unclear, let me know.
What I want is opinions and suggestions of other approaches, as I am considering this inefficient and impractical, even though it is working fine. I want something easily maintained. I think that sticking with localStorage is probably the best, but not this way. I am not very familiar with the use of Design Patterns in JavaScript, although I use some of them very frequently in Java. If anyone can give me a hand with that, it would be good.
EDIT:
It is a bit hard even for me to explain exactly why I feel it is inefficient. I guess the main reason is because for each case (Progress, Finished, Error, Last Collections) I have to call a method and modify the String (adding underline and more information), and for me to access any data (let's say, the name or the date) of each one of them I need to test to see which case is it and then keep using split( _ ). I know this is not very straightforward but I guess that this whole approach could be better designed. As I am working alone on this part of the software, I don't have anyone that I can discuss things with, so I thought here would be a good place to exchange ideas :)
Thanks in advance!

Not exactly sure what you are looking for. Generally I use localStorage just to store stringified versions of objects that fit my application. Rather than setting up all sorts of different keys for each variable within localStore, I just dump stringified versions of my object into one key in localStorage. That way the data is the same structure whether it comes from server as JSON or I pull it from local.
You can quickly save or retrieve deeply nested objects/arrays using JSON.stringify( object) and JSON.parse( 'string from store');
Example:
My App Object as sent from server as JSON( I realize this isn't proper quoted JSON)
var data={ foo: {bar:[1,2,3], baz:[4,5,6,7]},
foo2: {bar:[1,2,3], baz:[4,5,6,7]}
}
saveObjLocal( 'app_analysis', data);
function saveObjLocal( key, obj){
localStorage.set( key, JSON.stringify(obj)
}
function getlocalObj( key){
return JSON.parse( localStorage.get(key) );
}
var analysisObj= =getlocalObj('app_analysis');
alert( analysisObj.foo.bar[2])

Whats the best way to resolve name conflicts in javascript?

I recently wrote some javascript code that filled a drop down list based on some XML, pretty simple stuff. The problem was I had to write similar code to do almost the same thing on a different page.
Because the code was almost identical I named most of the functions the same, thinking that they would never be included in the same page. However, naming conflicts arose because both javascript files were eventually included in the same HTML page.
When I had to go back and change the names I simply added first_ or second_ to the method's names. This was a pain and it doesn't seem very elegant to me. I was wondering if there is a better way to resolve name conflicts in javascript?

Try the JavaScript module pattern (or namespaces) used in various libraries.
Try to be DRY (don't repeat yourself) so you can avoid name collisions. If the code is almost the same you better avoid code duplication by creating a function which can handle both cases. The function can take two parameters: which dropdown to populate and with what data. This helps maintainability as well.
update: I assume that you take the XML from an AJAX request. In this case you can create on-the-fly anonymous functions with the appropriate parameters for callback inside a loop.

I would look at how I could merge the two pieces of code (functions?) into a single function. If you need to populate a list box, then pass the list box id into the function, so you are not hard-coded to operate on one single control only...
I did this on my rocket business's web site where I sold rocket motors with different delay values, but in essence, they were the same product, just a different delay value.
Perhaps this might try and explain what I'm trying to say... I use this if an image file happens to be missing, it will display a "no image" image in place of the real image.
function goBlank(image)
{
if(image) {
var imgobj = document[image];
imgobj.src="/images/blank.png";
}
}
In this case, you call it with:
<img src="/images/aerotech.png" name="header" onError="goBlank('header');">
If you need more example with things like list boxes used, let me know. Perhaps even post some sample code of yours.

Another option (if possible) is to carefully tie the code to the element itself.
e.g.
<input type="text" name="foo" id="foo" value="World" onchange="this.stuff('Hello ' + this.value);"/>
<script>
document.getElementById('foo').stuff = function(msg){
//do whatever you want here...
alert('You passed me: ' + msg);
};
</script>