We Keep Coding

Axios GET in for loop causing "Sort exceeded memory limit" error

I'm trying to load entries from my mongoDB database one at a time. It works for about 400/1000 entries then breaks with the following error:
Executor error during find command :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.
My Axios get request looks like:
for (let i = 0; i < total; i++) {
await axios({
method: "GET",
url: "/api/entries/",
params: { from: index + i, _limit: 1 },
})
.then((res) => {
setPics((prev) => {
return [...new Set([...prev, ...res.data])];
});
setIndex((prev) => prev++)
setMoreEntries(res.data.length > 0)
})
}
and in my controller my GET function looks like:
const getEntries = asyncHandler(async (req, res) => {
const entries = await Entry.find().allowDiskUse(true).sort({ createdAt: 'desc' }).skip(req.query.from).limit(req.query._limit)
res.status(200).json(entries)
})
Everything works perfectly until about half the entries have loaded, then it breaks. I thought adding allowDiskUse(true) would fix it, but same result.
Edit: I should mention, if I take out .sort({ createdAt: 'desc' }) it works perfectly, but loads in the opposite order.

The syntax itself is fine, I am not quite sure why it's not working for you.
I have two theory's that could be possible:
You are hosting your db on Atlas and are using unqualified instances for this operations as specified in their docs:
Atlas M0 free clusters and M2/M5 shared clusters don't support the allowDiskUse
It seems you're using mongoose, maybe you are using an older version that has an issue with this cursor flag or has different syntax.
Regardless of the source of the issue, I recommend you solve this by either:
Create an index on createdAt, sorting when an index exists does not scan documents into memory, which will make both the query more efficient and solve this issue.
Just use the _id index and sort with { _id: -1}, from your comment about removing the sort operation and getting documents in reverse order, it seems that your createdAt corresponds document creation date ( makes sense ), the ObjectId _id is monotonically increasing which means you can just use that field to sort.

I ended up just doing a backwards for-loop, and using skip instead of sort, like
const getOne = asyncHandler(async (req, res) => {
const entry = await Entry.findOne({}, {}).allowDiskUse(true).skip(req.query.from)
res.status(200).json(entry)
})
Which worked perfectly for me, but Tom's answer is what I was looking for, thank you :)

Node/Apollo/Sequelize onboxiously slow (>7 seconds)

I'm no expert in these things (I'm used to Laravel), but running one query in Apollo Server is taking ~7.2 seconds, for maybe 300 items total.
The entire resolver is below - as you can see there's essentially no logic aside from running a query. It's just humongously slow.
getMenu: async (parent, {
slug
}, { models, me }) => {
const user = await models.User.findByPk(me.id)
const account = await models.Account.findByPk(user.accountId, {
include: [{
model: models.Venue,
as: 'venues',
include: getVenueIncludes(models)
}],
minifyAliases: true
})
return account.venues.find(venue => venue.slug === slug): null
},
I realise this is rather vague, but does anyone happen to know where I'd look to try and improve this? I understand they're different, but in a Laravel app I can load 10 times that amount (with more nesting) in under a second...

Aha!!
separate: true on your hasMany relationships. Good grief, cut request times from 7.2 seconds to 500ms.
Amazing.

Cannot Delete Fields with Period in Key Name

I've tried every solution in this other StackOverflow thread, and none of them have worked for me. At this point I'm absolutely stumped, and have no idea on what to try next.
The data that I'm trying to access is this:
The key I'm trying to use is a URL, which is inside the accounts map. This is the code I'm running to try and delete the key:
var userRef = db.collection('userAccounts').doc(userEmail)
let dynamicKey = `accounts.${accountURL}`
console.log(dynamicKey)
userRef.set({
[dynamicKey]: firebase.firestore.FieldValue.delete()
}, { merge: true})
.then((result) => {
console.log(result)
})
.catch((error) => {
console.log(error)
})
Looking in console, accounts.www.stackoverflow.com is printed:
So, it looks like the path should match. A note to make is that a URL with no periods works just fine, so it seems like the path IS correct, and that periods are in fact the issue.
The accepted answer by J Livengood simply doesn't work for keys with periods in the name:
[`hello.${world}`]: firebase.firestore.FieldValue.delete()
The code posted by Sam Stern simply doesn't run, and I get an error mentioning that update only takes one parameter. Contrary to the last poster (ishandutta2007), adding a 'new' before the FieldPath doesn't fix the error:
doc.update(
firebase.firestore.FieldPath("hello.world"),
firebase.firestore.FieldValue.delete());
This code (in the comments) posted by OP, Sandeep Dinesh, just doesn't work at all, even when trying to delete with a key with no period. My code is the following, and the returned Promise is undefined in the "then" portion of the code:
var userRef = db.collection('accounts').doc(userEmail)
let dynamicKey = `accounts.${accountURL}`
userRef.set({
[dynamicKey]: firebase.firestore.FieldValue.delete()
}, { merge: true})
.then((result) => {
console.log(result)
})
.catch((error) => {
console.log(error)
})

The problem you have here is a little bit different than you described. Specifically, you're trying to delete a nested map key with periods in it. It's important to realize that this nesting requires special treatment that you didn't see in other questions, which were dealing only with top-level fields.
You will need to use FieldPath here, and specify the path of the field as array elements in the constructor. From the linked API documentation (emphasis mine):
Creates a FieldPath from the provided field names. If more than one field name is provided, the path will point to a nested field in a document.
So, you should set or update with a field of new firebase.firestore.FieldPath(["accounts", "www.stackoverflow.com"]) and a value of firebase.firestore.FieldValue.delete().
documentReference.update(
new firebase.firestore.FieldPath(["accounts", "www.stackoverflow.com"]),
firebase.firestore.FieldValue.delete()
);

More efficient way to handle an extremely specific response from a Rest API in JavaScript

I'm working with a Dungeons & Dragons 5e API and want that an especific result be treated in a special way. The user can choose what to search from a range of options, and in only one of them would I need to take care of the answer in a different way. In this option, I get the answer in JSON that contains a 'name' field, which stores a String, but in this specific case this String comes with an acronym, and I would like to transform it into the full name.I'm afraid to just put am 'if' statement in the middle of the code and deal with the situation inefficiently, even more so that I did not find similar situations to have any reference.
This is part of the result of the API I want to handle in a special way:
{"count":6,
"results":[
{"name":"STR",
"url":"http://www.dnd5eapi.co/api/ability-score/1"},
{"name":"DEX",
"url":"http://www.dnd5eapi.co/api/ability-scores2"},
....
]
}
This is how I handle the answer:
fetch(fullAPIURL)
.then(result => result.json())
.then(data => {
let resultContainer = document.getElementById('resultContainer');
//Cleaning the result container from previous results
document.querySelectorAll('#resultContainer article').forEach(container =>
resultContainer.removeChild(container));
spanSearchResult.classList.remove('invisible', 'searchFail');
spanSearchResult.classList.add('searchSucess');
spanSearchResult.innerHTML = `Search returned ${data.count} results`;
for (element of data.results) {
let containerTitle = element.name != undefined ? element.name : element.class;
resultContainer.appendChild(createResultContainer(containerTitle));
}
})
.catch(err => {
spanSearchResult.classList.remove('invisible');
spanSearchResult.classList.add('searchFail');
spanSearchResult.innerHTML = 'Something went wrong! Details in the console';
console.log(err);
});
Is putting a condition in this snippet of code really the most efficient way to solve this situation?
Thanks in advance.

You could just make a lookup call, actually. In fact, that'd be preferable if you ever want to port your application to another language, for example.
Define the following:
var retrieve = (function() {
var items = {
"STR": "Strength",
"DEX": "Dexterity"
};
return function(item) {
return items[item] || item;
}
})();
console.log(retrieve("DEX"));
With this, you can simply call retrieve(element.name) to retrieve its "actual" name. You can add elements to the object to create new translations, and if you ever need to support multiple languages, you can even replace the function entirely.

Async/Await chaining

Guess it's my turn to create a user at stackoverflow :)
I've tried and solve this assignment for 2 days now, i can get it to show the first two informations, but i have no idea how i access the last two...
Plus i have a slight idea, that i am doing it completely wrong...
This is what i have to do:
Use information from this link to find the first movie in which Luke Skywalker appeared
Use information from this link to find the first species, which appeared in this movie
Use information from this link to find the planet (homeworld) for this species
Now, Implement a method getPlanetforFirstSpeciesInFirstMovieForPerson(id){} which for id = 1 (Luke Skywalker) should log this info:
Name: Luke Skywalker
First film: The Empire Strikes Back
First species: Yoda's species
Homeworld for Specie: unknown
Hints:
This requires you to make a number of REST-requests (using fetch), read a value from the request, and use this value to perform a new request.
The lists in the responses are not sorted. For this exercise it’s ok to just use the first URL in the list: Like films[0] will actually give you the second movie, see below:
Data:
"films": [
"https://swapi.co/api/films/2/",
"https://swapi.co/api/films/6/",
"https://swapi.co/api/films/3/",
"https://swapi.co/api/films/1/",
"https://swapi.co/api/films/7/"
],
This is what I have so far:
const fetch = require('node-fetch')
const URL = "https://swapi.co/api/people/";
async function getPlanetForFirstSpeciesInFirstMovieForPersonID(id) {
const result = await fetch(URL);
const data = await result.json().then((nameData) => nameData.results[id]);
const name = data.name
const film = await fetch(data.films[id])
const filmdata = await film.json().then((movietitle) => movietitle.title)
console.log(name)
console.log(filmdata)
}
getPlanetForFirstSpeciesInFirstMovieForPersonID(0).catch((e) => {
console.log('There was an error :', e)
});
But I guess there has to be a cleaner and smarter way to do this.
I can do it without problems using promises the old way, but I have to use async/await.