We Keep Coding

Run code after executing promise in Javascript

I am trying to save to json the values returned from indeed api. I use indeed-scraper code from github https://github.com/rynobax/indeed-scraper
My code:
... required files ...
const parsedResults = []
indeed.query(queryOptions).then(response => {
response.forEach((res,i) => {
setTimeout(function(){
let url = res.url
let resultCount = 0
console.log(`\n Scraping of ${url} initiated...\n`)
const getWebsiteContent = async (url) => {
try {
const response = await axios.get(url)
const $ = cheerio.load(response.data)
...get scraped data...
parsedResults.push(metadata)
} catch (error) {
exportResults(parsedResults)
console.error(error)
}
}
getWebsiteContent(url)
}
, i*3000);
});
});
const outputFile = 'data.json'
const fs = require('fs');
const exportResults = (parsedResults) => {
fs.writeFile(outputFile, JSON.stringify(parsedResults, null, 4), (err) => {
if (err) {
console.log(err)
}
console.log(`\n ${parsedResults.length} Results exported successfully to ${outputFile}\n`)
})
}
parsedResults is not accessible in last portion of script, so to save as json file.
Any help appreciated!

Firebase React native Expo file upload and download issue resolving promise

I'm having a problem where the Array is not filled out, I think its something to do with the promoses resolving.
const UploadFile = async ({
imageName = `${Date.now()}`,
imageUris,
imageFolder = '',
metadata,
}: IFile) => {
if (imageUris) {
const promises: any[] = [];
const imageURLs: string[] = [];
imageUris.forEach(async (uri) => {
const randomNumber = Randomize('0', 10);
const finalImageName = `${Slugify(imageName)}`.toLowerCase();
const imagePath = `${imageFolder}/${finalImageName}-${randomNumber}`;
const imageRef = storageRef.child(imagePath);
const blob = (await fetch(uri)).blob();
const uploadTask = imageRef.put(await blob, metadata);
uploadTask.on(
firebase.storage.TaskEvent.STATE_CHANGED,
(snapshot) => {
const progress =
(snapshot.bytesTransferred / snapshot.totalBytes) * 100;
console.log('Upload is ' + progress + '% done');
},
(error) => console.log('Error:', error),
() => {
uploadTask.snapshot.ref.getDownloadURL().then((downloadURL) => {
console.log('File available at', downloadURL);
imageURLs.push(downloadURL);
});
},
);
promises.push(uploadTask);
});
// Not sure promise is resolving
Promise.all(promises).then((i) => {
console.log('All files uploaded', i);
});
Promise.all(imageURLs).then((i) => {
console.log('All imageURLs', i);
});
}
}
Output:
Retrieved listings
All files uploaded Array []
All imageURLs Array []
imageURLs Contents undefined
Upload is 0% done
Upload is 0% done
Upload is 100% done
File available at https://firebasestorage.googleapis.com/v0/b/wrecknet-ab69d.appspot.com/o/listings%2Fcar-5701393331?alt=media&token=ccfda911-36fb-4305-b6d7-0ee06fc824e1
Listing was successfully created
Upload is 100% done
File available at https://firebasestorage.googleapis.com/v0/b/wrecknet-ab69d.appspot.com/o/listings%2Fcar-4491812919?alt=media&token=03f72706-4201-4652-9172-8bcefaeb3e1f
As you can see the "All files uploaded Array []" and "All imageURLs Array []" arrays are empty, I suspect the Promise is not resolving.

As far as I know you can either listen to the on() of the UploadTask or to its then(), but not to both. Luckily you don't do anything meaningful in the on handling, so the entire code can be simplified down to:
const UploadFile = async ({
imageName = `${Date.now()}`,
imageUris,
imageFolder = '',
metadata,
}: IFile) => {
if (imageUris) {
const promises: any[] = [];
imageUris.forEach(async (uri) => {
const randomNumber = Randomize('0', 10);
const finalImageName = `${Slugify(imageName)}`.toLowerCase();
const imagePath = `${imageFolder}/${finalImageName}-${randomNumber}`;
const imageRef = storageRef.child(imagePath);
const blob = (await fetch(uri)).blob();
promises.push(imageRef.put(await blob, metadata));
});
Promise.all(promises).then((imageURLs) => {
console.log('All imageURLs', imageURLs);
});
}
}

Retrieve data from url with axios (scraping)

I try to retrieve data (scraping) from a url with Get method (axios) but it doesn't retrieve the data what i looking for (piano) and no error is show. I use json to retrieve the data. Any idea what i'm doing wrong .dddddddddddddddddddddddddddd
const axios = require('axios');
const cheerio = require ('cheerio');
const fs = require('fs');
const baseUrl = '**********/';
const axiosCookieJarSupport = require('axios-cookiejar-support').default;
const tough = require('tough-cookie');
axiosCookieJarSupport(axios);
const cookieJar = new tough.CookieJar();
function scrapeUrl(url, items=[]) {
const params = {
keywords: "piano",
latitude:40.489353,
longitude:-3.6827461,
};
return axios
.get(baseUrl + "search", {
params,
jar: cookieJar,
withCredentials: true,
})
.then(response => {
const $ = cheerio.load(response.data);
const pageItems = $('.container-wall .card-product-content').toArray()
tracked').toArray()
.map(item => {
const $item = $(item);
// console.log('todo', $item);
return {
// id: $item.attr('data-adid'),
title: $item.find('.card-product-image').attr('alt'),
link: baseUrl + $item.parent('a').attr('href'),
image: $item.find('.card-product-image').attr('src'),
price: $item.find('.product-info-price').text(),
};
});
const allItems = items.concat(pageItems);
console.log(pageItems.length,'items retrieved', allItems.length,
'acumulated');
const nextUrl = $('.pagination .next a').attr('href');
return nextUrl ? scrapeUrl(baseUrl + nextUrl, allItems) : allItems;
})
.catch(error => {
console.log('error', error);
return items;
});
}
scrapeUrl(baseUrl + initialUrl)
.then(items => {
process.stdout.write(JSON.stringify(items));
fs.writeFile('./items.json', JSON.stringify(items), 'utf8', function(error) {
if (error) return console.log('error', error);
console.log(items.length, 'items saved');
});
});

i switch to puppeteer and headless Chrome as Chris sayed it's better than axios/cheerio becouse nowadays when many of the websites are built as a single page application and gets rendered dynamically on the client it might not be possible to get the content.

Rename photo taken Expo

I created an application and in this app I take a photo with an expo-camera and save it with MediaLibrary, but I can't rename the photo. How can I rename the photo? I tried to use String.slice and change the name at the end of the uri, but it doesn't work because to use String.slice uri has to be created before the photo is taken.
How i take the photo:
takePicture = async () => {
if (this.camera) {
let photo = await this.camera.takePictureAsync();
this.setState({
image: photo.uri,
modalOpen: true
});
}
console.log(this.state.image)
}
How i save:
saveImage = async (image, imageName) => {
const asset = await MediaLibrary.createAssetAsync(image);
MediaLibrary.createAlbumAsync('Geocad', asset)
.then(() => {
console.log('Album created!');
})
.catch(error => {
console.log('err', error);
});
this.setState({
modalOpen: false
});
}
What i try:
var str = image;
var indices = [];
for(var i=0; i<str.length;i++) {
if (str[i] === "/") indices.push(i);
}
let indexNameSlice = indices.slice(-1,)[0]
let oldImageName = image.slice(0,indexNameSlice)
let newImageName = oldImageName+'/'+imageName+'.jpg'
console.log('NEWIMAGENAME', newImageName+'/'+imageName+'.jpg')
const asset = await MediaLibrary.createAssetAsync(newImageName);

Imported async data from another module in Node does not resolve, how can I fix this?

This is the module that collections and exports async data: scraper.js
const express = require('express')
const cheerio = require('cheerio')
const request = require("tinyreq")
const fs = require('fs')
const _ = require('lodash')
const uuid = require('uuid/v4')
const async = require('async')
const mental_models = {
url: 'https://www.farnamstreetblog.com/mental-models/',
data: {}
}
const decision_making = {
url: 'https://www.farnamstreetblog.com/smart-decisions/',
data: {}
}
const cognitive_bias = {
url: 'https://betterhumans.coach.me/cognitive-bias-cheat-sheet-55a472476b18',
data: {}
}
const DATA_URLS = [mental_models, decision_making, cognitive_bias]
const filterScrape = async (source, params) => {
let filtered_data = {
topics: [],
content: [],
additional_content: []
}
let response = await scrape(source)
try {
let $ = cheerio.load(response)
params.forEach((elem) => {
let headers = ['h1', 'h2', 'h3']
if ($(elem) && headers.includes(elem)) {
let topic = {}
let content = {}
let id = uuid()
topic.id = id
topic.text = $(elem).text()
if ($(elem).closest('p')) {
content.text = $(elem).closest('p').text()
content.id = id
}
filtered_data.topics.push(topic)
filtered_data.content.push(content)
} else if ($(elem) && !headers.includes(elem)) {
let content = {}
let id = uuid()
content.text = $(elem).text()
content.id = id
filtered_data.additional_content.push(content)
} else {
}
})
}
catch (err) {
console.log(err)
}
return filtered_data
}
const scrape = (source) => {
return new Promise((resolve, reject) => {
request(source.url, function (err, body) {
if (err) {
reject(err)
return
}
resolve(body)
})
})
}
const DATA = _.map(DATA_URLS, async (source) => {
let params = ['h1', 'h2', 'h3', 'p']
let new_data = await filterScrape(source, params)
try {
source.data = new_data
}
catch (err) {
console.log(err)
}
})
module.exports = DATA
This is the module that imports the data: neural.js
const brain = require('brain')
const neural_net = new brain.NeuralNetwork()
const DATA = require('./scraper')
console.log(DATA)
Obviously not much going on, I've removed the code since the variable doesn't resolve. When logged it logs a promise but the promise does not resolve. However in the imported module, the promise is logged and then resolves. What gives? Should I import a function that resolves the data?

Of course it would be best to import that function, however it won't change the issue in your code which is here:
const DATA = _.map(DATA_URLS, async (source) => {
Lodash doesn't support async iteration - so you need to have some other method, one would be to use the newest nodejs version (10.x) and make use of async iteration - but that won't use the full power of asynchronous code.
You can also use scramjet - a framework my company is supporting. The code above would take the following form:
const {DataStream} = require("scramjet");
const DATA_URLS = [mental_models, decision_making, cognitive_bias];
module.exports = async () => DataStream.fromArray(DATA_URLS)
.setOptions({maxParallel: 2}) // if you need to limit that at all.
.map(async ({url}) => {
let params = ['h1', 'h2', 'h3', 'p']
let data = await filterScrape(source, params);
return { url, data };
})
.toArray();
The other file would take the following form:
const brain = require('brain')
const neural_net = new brain.NeuralNetwork()
const scraper = require('./scraper')
(async (){
const DATA = await scraper();
console.log(DATA); // or do whatever else you were expecting...
})();