We Keep Coding

How do I search through raw data for matching data?

Right now, I coded a function to go like this
async function checkPlayerScam(ign) {
const UUID = await getUUID(ign);
if(MATCHING){
playerIsScammer = true
}
else {
playerIsScammer = false
}
}
The MATCHING is just a placeholder at the moment. I want to check their UUID, and make sure it isn't in this list: https://raw.githubusercontent.com/skyblockz/pricecheckbot/master/scammer.json
Any idea how? It needs to be relatively fast
EDIT: It'd also be cool if I could get the reason from the list, but that's not as necessary

https://lodash.com/docs/#find
Use lodash _.find to
const uuid = '000c97aaf948417a9a74d6858c01aaae'; // uuid you want to find
const scammer = _.find(scammersList, o => o.uuid === uuid);
if (scammer) { // if scammer found
console.log(scammer);
console.log(scammer.reason)
}

For anyone wondering, this is how I solved it:
async function checkPlayerScam(ign) {
const UUID = await getUUID(ign);
const response = await fetch(`https://raw.githubusercontent.com/skyblockz/pricecheckbot/master/scammer.json`);
const result = await responsejson();
if (result[UUID] = null) {
playerIsScammer == False
}
else{
playerIsScammer == True
}
}

This function will fetch the data, then check if the uuid 1d0c0ef4295047b39f0fa899c485bd00 exists. Assuming that you already fetched the data somewhere else and stored it, all you need to do is check if a given uuid exists by adding the following line where you please:
!!data[uuidToCheck]
uuidToCheck should be the uuid string that you are looking for.
This line will return true if the uuid exists and false otherwise.
In terms of the spacetime complexity, this function runs in constant time [O(1)] and O(N) space. This is the fastest time you can get it to run.
data[uuidToCheck].reason will return the reason.
async function playerIsScammer(uuidToCheck) {
uuidToCheck = '1d0c0ef4295047b39f0fa899c485bd00';
const response = await fetch('https://raw.githubusercontent.com/skyblockz/pricecheckbot/master/scammer.json');
if (response.ok){
let data = await response.json();
if(!!data[uuidToCheck])
return data[uuidToCheck].reason;
return false
}
}

async functions not executing in the correct order inside a map function

I have created an async function that will extra the data from the argument, create a Postgres query based on a data, then did some processing using the retrieved query data. Yet, when I call this function inside a map function, it seemed like it has looped through all the element to extra the data from the argument first before it proceed to the second and the third part, which lead to wrong computation on the second element and onwards(the first element is always correct). I am new to async function, can someone please take at the below code? Thanks!
async function testWeightedScore(test, examData) {
var grade = [];
const testID = examData[test.name];
console.log(testID);
var res = await DefaultPostgresPool().query(
//postgres query based on the score constant
);
var result = res.rows;
for (var i = 0; i < result.length; i++) {
const score = result[i].score;
var weightScore = score * 20;
//more computation
const mid = { "testID": testID, "score": weightScore, more values...};
grade.push(mid);
}
return grade;
}
(async () => {
const examSession = [{"name": "Sally"},{"name": "Bob"},{"name": "Steph"}]
const examData = {
"Sally": 384258,
"Bob": 718239,
"Steph": 349285,
};
var test = [];
examSession.map(async sesion => {
var result = await testWeightedScore(sesion,examData);
let counts = result.reduce((prev, curr) => {
let count = prev.get(curr.testID) || 0;
prev.set(curr.testID, curr.score + count);
return prev;
}, new Map());
let reducedObjArr = [...counts].map(([testID, score]) => {
return {testID, score}
})
console.info(reducedObjArr);
}
);
})();
// The console log printed out all the tokenID first(loop through all the element in examSession ), before it printed out reducedObjArr for each element

The async/await behaviour is that the code pause at await, and do something else (async) until the result of await is provided.
So your code will launch a testWeightedScore, leave at the postgresql query (second await) and in the meantime go to the other entries in your map, log the id, then leave again at the query level.
I didn't read your function in detail however so I am unsure if your function is properly isolated or the order and completion of each call is important.
If you want each test to be fully done one after the other and not in 'parallel', you should do a for loop instead of a map.

Push inside forEach with query not working properly

I'm working with mongodb stitch/realm and I'm trying to modify objects inside an array with a foreach and also pushing ids into a new array.
For each object that i'm modifying, I'm also doing a query first, after the document is found I start modifying the object and then pushing the id into another array so I can use both arrays later.
The code is something like this:
exports = function(orgLoc_id, data){
var HttpStatus = require('http-status-codes');
// Access DB
const db_name = context.values.get("database").name;
const db = context.services.get("mongodb-atlas").db(db_name);
const orgLocPickupPointCollection = db.collection("organizations.pickup_points");
const orgLocStreamsCollection = db.collection("organizations.streams");
const streamsCollection = db.collection("streams");
let stream_ids = [];
data.forEach(function(stream) {
return streamsCollection.findOne({_id: stream.stream_id}, {type: 1, sizes: 1}).then(res => { //if I comment this query it will push without any problem
if(res) {
let newId = new BSON.ObjectId();
stream._id = newId;
stream.location_id = orgLoc_id;
stream.stream_type = res.type;
stream.unit_price = res.sizes[0].unit_price_dropoff;
stream._created = new Date();
stream._modified = new Date();
stream._active = true;
stream_ids.push(newId);
}
})
})
console.log('stream ids: ' + stream_ids);
//TODO
};
But when I try to log 'stream_ids' it's empty and nothing is shown. Properties stream_type and unit_price are not assigned.
I've tried promises but I haven't had success

It's an asynchronous issue. You're populating the value of the array inside a callback. But because of the nature of the event loop, it's impossible that any of the callbacks will have been called by the time the console.log is executed.
You mentioned a solution involving promises, and that's probably the right tack. For example something like the following:
exports = function(orgLoc_id, data) {
// ...
let stream_ids = [];
const promises = data.map(function(stream) {
return streamsCollection.findOne({ _id: stream.stream_id }, { type: 1, sizes: 1 })
.then(res => { //if I comment this query it will push without any problem
if (res) {
let newId = new BSON.ObjectId();
// ...
stream_ids.push(newId);
}
})
})
Promise.all(promises).then(function() {
console.log('stream ids: ' + stream_ids);
//TODO
// any code that needs access to stream_ids should be in here...
});
};
Note the change of forEach to map...that way you're getting an array of all the Promises (I'm assuming your findOne is returning a promise because of the .then).
Then you use a Promise.all to wait for all the promises to resolve, and then you should have your array.
Side note: A more elegant solution would involve returning newId inside your .then. In that case Promise.all will actually resolve with an array of the results of all the promises, which would be the values of newId.

Javascript object retaining "old" properties, can't override?

I have the following code:
const readDataFromSql = () => {
// going to have to iterate through all known activities + load them here
let sql = "[...]"
return new Promise((resolve, reject) => {
executeSqlQuery(sql).then((dict) => {
let loadedData = [];
for (let key in dict) {
let newItemVal = new ItemVal("reading hw", 7121, progress.DONE);
loadedData.push(newItemVal);
}
resolve(loadedData);
});
});
}
ItemVal implementation:
class ItemVal {
constructor(name, time, type) {
this.name = name
this.time = time
this.type = type
}
}
Let's assume that newItemVal = "reading hwj", 5081, progress.PAUSED when readDataFromSql() first runs.
readDataFromSql() is then again called after some state changes -- where it repulls some information from a database and generates new values. What is perplexing, however, is that when it is called the second time, newItemVal still retains its old properties (attaching screenshot below).
Am I misusing the new keyword?

From what I can see in your example code, you are not mutating existing properties but creating a new object with the ItemVal constructor function and adding them to an array, that you then return as a resolved promise. Are you sure the examples you give a correct representation of what you are actually doing
Given that, I'm not sure what could be causing the issue you are having, but I would at least recommend a different structure for your code, using a simpler function for the itemVal.
Perhaps with this setup, you might get an error returned that might help you debug your issue.
const itemVal = (name, time, type) => ({ name, time, type })
const readDataFromSql = async () => {
try {
const sql = "[...]"
const dict = await executeSqlQuery(sql)
const loadedData = dict.map((key) =>
ItemVal("reading hw", 7121, progress.DONE)
)
return loadedData
} catch (error) {
return error
}
};
If the issue is not in the function, then I would assume that the way you handle the data, returned from the readDataFromSql function, is where the issue lies. You need to then share more details about your implementation.

const readDataFromSql = async () => {
let sql = "[...]"
------> await executeSqlQuery(sql).then((dict) => {
Use the await keyword instead of creating a new promise.

I did some modification and found that below code is working correctly, and updating the new values on each call.
const readDataFromSql = () => {
return new Promise((resolve, reject) => {
let loadedData = [];
let randomVal = Math.random();
let newItemVal = new ItemVal(randomVal*10, randomVal*100, randomVal*1000);
loadedData.push(newItemVal);
resolve(loadedData);
});
}
Could you recheck if you are using below line in the code, as it will instantiate object with same properties again and again.
let newItemVal = new ItemVal("reading hw", 7121, progress.DONE);

You can modify your code as below to simplify the problem.
const readDataFromSql = async () => {
// going to have to iterate through all known activities + load them here
let sql = "[...]" // define sql properly
let result = await executeSqlQuery(sql);
let loadedData = [];
for (let row in result) {
let newItemVal = new ItemVal(row.name, row.time, row.type);
loadedData.push(newItemVal);
}
return loadedData;
}
class ItemVal {
constructor(name, time, type) {
this.name = name
this.time = time
this.type = type
}
}
What you are talking about is an issue related to Object mutation in Redux, however, you didn't add any redux code. Anyway, you might be making some mistake while recreating(not mutating) the array.
General solution is the use spread operator as:
loadedData = [ ...loadedData.slice(0) , ...newloadedData]

In Dropdown.js line 188 instead of console.log-ing your variable write debugger;
This will function as a breakpoint. It will halt your code and you can inspect the value by hovering your mouse over the code BEFORE the newItemVal is changed again.
I can see in your screenshot that the newItemVal is modified again after you log it.

Continue on Null Value of Result (Nodejs, Puppeteer)

I'm just starting to play around with Puppeteer (Headless Chrome) and Nodejs. I'm scraping some test sites, and things work great when all the values are present, but if the value is missing I get an error like:
Cannot read property 'src' of null (so in the code below, the first two passes might have all values, but the third pass, there is no picture, so it just errors out).
Before I was using if(!picture) continue; but I think it's not working now because of the for loop.
Any help would be greatly appreciated, thanks!
for (let i = 1; i <= 3; i++) {
//...Getting to correct page and scraping it three times
const result = await page.evaluate(() => {
let title = document.querySelector('h1').innerText;
let article = document.querySelector('.c-entry-content').innerText;
let picture = document.querySelector('.c-picture img').src;
if (!document.querySelector('.c-picture img').src) {
let picture = 'No Link'; } //throws error
let source = "The Verge";
let categories = "Tech";
if (!picture)
continue; //throws error
return {
title,
article,
picture,
source,
categories
}
});
}

let picture = document.querySelector('.c-picture img').src;
if (!document.querySelector('.c-picture img').src) {
let picture = 'No Link'; } //throws error
If there is no picture, then document.querySelector() returns null, which does not have a src property. You need to check that your query found an element before trying to read the src property.
Moving the null-check to the top of the function has the added benefit of saving unnecessary calculations when you are just going to bail out anyway.
async function scrape3() {
// ...
for (let i = 1; i <= 3; i++) {
//...Getting to correct page and scraping it three times
const result = await page.evaluate(() => {
const pictureElement = document.querySelector('.c-picture img');
if (!pictureElement) return null;
const picture = pictureElement.src;
const title = document.querySelector('h1').innerText;
const article = document.querySelector('.c-entry-content').innerText;
const source = "The Verge";
const categories = "Tech";
return {
title,
article,
picture,
source,
categories
}
});
if (!result) continue;
// ... do stuff with result
}
Answering comment question: "Is there a way just to skip anything blank, and return the rest?"
Yes. You just need to check the existence of each element that could be missing before trying to read a property off of it. In this case we can omit the early return since you're always interested in all the results.
async function scrape3() {
// ...
for (let i = 1; i <= 3; i++) {
const result = await page.evaluate(() => {
const img = document.querySelector('.c-picture img');
const h1 = document.querySelector('h1');
const content = document.querySelector('.c-entry-content');
const picture = img ? img.src : '';
const title = h1 ? h1.innerText : '';
const article = content ? content.innerText : '';
const source = "The Verge";
const categories = "Tech";
return {
title,
article,
picture,
source,
categories
}
});
// ...
}
}
Further thoughts
Since I'm still on this question, let me take this one step further, and refactor it a bit with some higher level techniques you might be interested in. Not sure if this is exactly what you are after, but it should give you some ideas about writing more maintainable code.
// Generic reusable helper to return an object property
// if object exists and has property, else a default value
//
// This is a curried function accepting one argument at a
// time and capturing each parameter in a closure.
//
const maybeGetProp = default => key => object =>
(object && object.hasOwnProperty(key)) ? object.key : default
// Pass in empty string as the default value
//
const getPropOrEmptyString = maybeGetProp('')
// Apply the second parameter, the property name, making 2
// slightly different functions which have a default value
// and a property name pre-loaded. Both functions only need
// an object passed in to return either the property if it
// exists or an empty string.
//
const maybeText = getPropOrEmptyString('innerText')
const maybeSrc = getPropOrEmptyString('src')
async function scrape3() {
// ...
// The _ parameter name is acknowledging that we expect a
// an argument passed in but saying we plan to ignore it.
//
const evaluate = _ => page.evaluate(() => {
// Attempt to retrieve the desired elements
//
const img = document.querySelector('.c-picture img');
const h1 = document.querySelector('h1')
const content = document.querySelector('.c-entry-content')
// Return the results, with empty string in
// place of any missing properties.
//
return {
title: maybeText(h1),
article: maybeText(article),
picture: maybeSrc(img),
source: 'The Verge',
categories: 'Tech'
}
}))
// Start with an empty array of length 3
//
const evaluations = Array(3).fill()
// Then map over that array ignoring the undefined
// input and return a promise for a page evaluation
//
.map(evaluate)
// All 3 scrapes are occuring concurrently. We'll
// wait for all of them to finish.
//
const results = await Promise.all(evaluations)
// Now we have an array of results, so we can
// continue using array methods to iterate over them
// or otherwise manipulate or transform them
//
results
.filter(result => result.title && result.picture)
.forEach(result => {
//
// Do something with each result
//
})
}

Try-catch worked for me:
try {
if (await page.$eval('element')!==null) {
const name = await page.$eval('element')
}
}catch(error){
name = ''
}