We Keep Coding

VueJS get request every 20 minutes

I fetch data from a get request to display coin name information. I want this to update every 20 minutes.
For test-drive, I keep it 500 milliseconds. It always and always fetch data and add current list, so I am getting Duplicate keys detected: 'BUSDRON'. This may cause an update error this error and also my app is freezing.
How can I getting data from above link request every 20 minute, without dublicating values?
Also it
My fetch codes is here
methods: {
async fetchApi() {
const response = await fetch(
'https://api2.binance.com/api/v3/ticker/24hr'
);
const data = await response.json();
await data.forEach(element => {
this.chartData.symbols = [...this.chartData.symbols, element.symbol];
this.chartData.price = [...this.chartData.price, +element.lastPrice];
});
},
}
data: () => ({
timer: '',
)}
async created() {
this.loaded = false;
try {
this.timer = setInterval(this.fetchApi, 500);
this.loaded = true;
} catch (e) {
console.error(e);
}
},

Use uniqBy from lodash to remove duplicates.
Not sure the uniqueness would be based on the latest data. For safety, we reverse the array. After filtering the unique date, we reverse the array again in the order it should be.
methods: {
async fetchApi() {
const response = await fetch(
'https://api2.binance.com/api/v3/ticker/24hr'
);
const data = await response.json();
this.chartData = _.uniqBy([...this.data, data].reverse(), 'symbols').reverse()
}
}

Before your forEach loop, reset the arrays for symbols and price to an empty array, then push the new values to them inside of the forEach loop.
const response = await fetch(
'https://api2.binance.com/api/v3/ticker/24hr'
);
const data = await response.json();
const symbols = [];
const price = [];
data.forEach(element => {
symbols.push(element.symbol);
price.push(element.lastPrice);
});
this.chartData.symbols = symbols;
this.chartData.price = price;

Merging various backend requests in the express res.send()

I'm trying to make several asynchronous backend calls to generate a JSON response in my express API. Because of the nature of the API, I have 3 requests that are being made that are dependent on each other in some way.
Request 1: Returns an Array of values that are used to make request 2. Each value will be used as a mapping for the remaining requests. That is to say, it will be a unique identifier used to map the response from the requests in Request 3.
Request 2 (Parallel Batch): A request is made using each value from the Array returned in request 1. Each of these returns a value to be used in each of the Request 3s. That is to say, it's a 1-to-1
Request 3 (Parallel Batch): This request takes the response from Request 2, and makes a 1-to-1 follow up request to get more data on that specific mapping (the id from request 1)
I would like the final data I send to the consumer to look like this:
{
id1: details1,
id2: details2,
id3: details3,
...
}
Here is the code I have so far...
app.get("/artists/:artist/albums", (req, res) => {
console.log("#############")
const artistName = req.params.artist
let response = {};
let s3Promise = s3.listAlbums(artistName)
let albumDetailsPromises = []
s3Promise
.then((data) => {
data.map((album) => {
// Each album name here will actually be used as the unique identifier for
// the final response
// Build an Array of promises that will first fetch the albumId, then use
// that album id to fetch the details on the album
albumDetailsPromises.push(
discogs.getAlbumId(artistName, album).then( // Returns a promise
({ data }) => {
let masterId = data.results[0].id
let recordName = data.results[0].title
// Storing the album name to carry as a unique id alongside the promise
return [album, discogs.getAlbumDetails(masterId) // Returns a promise ]
}
)
)
})
})
.then(() => {
// When all the albumIds have been fetched, there will still exist a promise in the
// second index of each element in the albumDetailsPromises array
Promise.all(albumDetailsPromises)
.then((namedPromises) => {
namedPromises.map(
(album) => {
let albumName = album[0] // Unique Id
let albumDetailPromise = album[1]
// Resolving the albumDetailsPromise here, and storing the value on
// a response object that we intend to send as the express response
albumDetailPromise
.then(
({ data }) => {
response[albumName] = data
})
.catch(err => response[albumName] = err)
})
})
})
.catch((err) => console.log(err))
})
As of now, everything seems to be working as expected, I just can't seem to figure out how to "await" the response object being updated at the end of all these Promises. I've omitted res.send(response) from this example because it's not working, but that's of course my desired outcome.
Any advice is appreciated! New to javascript...

I would recommend rewriting this using async/await as it helps to reduce nesting. You can also extract the logic the get the album-details into a separate function, as this also increases the readability of the code. Something like this (this still needs error-handling, but it should give you a start):
app.get("/artists/:artist/albums", async (req, res) => {
const artistName = req.params.artist;
const albumNames = await s3.listAlbums(artistName);
const result = {};
const albumDetailPromises = albumNames.map(albumName => requestAlbumDetails(discogs, artistName, albumName));
const resolvedAlbumDetails = await Promise.all(albumDetailPromises);
// map desired response structure
for(const albumDetail of resolvedAlbumDetails) {
result[albumDetail.albumName] = albumDetail.albumDetails;
}
res.json(result);
});
async function requestAlbumDetails(service, artistName, albumName) {
const albumInfo = await service.getAlbumId(artistName, albumName);
const masterId = albumInfo.results[0].id;
const albumDetails = await service.getAlbumDetails(masterId);
return { albumName, albumDetails };
}
To answer your question how you could do it with your code:
You'd need to wait for all details to be fulfilled using another Promise.all call and then just send the response in the then-handler:
Promise.all(albumDetailsPromises)
.then((namedPromises) => {
const detailsPromises = namedPromises.map(
(album) => {
let albumName = album[0];
let albumDetailPromise = album[1];
return albumDetailPromise
.then(({ data }) => {
response[albumName] = data;
})
.catch(err => response[albumName] = err);
});
return Promise.all(detailsPromises)
.then(() => res.json(response));
})

Refactored using async/await...
app.get("/artists/:artist/albums", async (req, res) => {
const artistName = req.params.artist
let response = {};
let albums = await s3.listAlbums(artistName)
const promises = albums.map(async (album) => {
let result = await discogs.getAlbumId(artistName, album)
try {
let masterId = result.data.results[0].id
let tempRes = await discogs.getAlbumDetails(masterId)
return [album, tempRes.data]
} catch (error) {
return [album, { "msg": error.message }]
}
})
responses = await Promise.all(promises)
responses.map(data => { response[data[0]] = data[1] })
res.send(response)
})

React & Firebase - making multiple Firebase calls and waiting for all promises to be complete

I would like to change the following code as it reaches Firebases 10 item in an array limit. I want to loop through all teamIds and make a Firebase query for each individual teamId. The issue is, I'm not sure how to do this in a way that it waits until all promises are complete before continuing.
This is the current code;
const unsubscribe = Firebase.firestore().collection('invites').where('teamId', 'in', teamIds).onSnapshot(snapshot => {
const invites = [];
snapshot.docs.forEach(doc => {
const data = doc.data();
if (!invites[data.teamId]) {
invites[data.teamId] = [];
}
invites[data.teamId].push(Object.assign({}, { id: doc.id }, doc.data()));
});
setTeamInvites(invites);
setLoading(false);
setError(false);
});
I've like to change it to something like this;
teamIds.forEach(teamId => {
Firebase.firestore().collection('invites').where('teamId', '==', teamId).onSnapshot(snapshot => {
// Map the results to an array that will be stored in the pageState when all promises are complete
});
});
How can I do this?

I figured out I can do this with Promise.all, this is what I ended up with.
const promises = [];
teamIds.forEach(teamId => {
promises.push(new Promise((resolve, reject) => {
Firebase.firestore().collection('invites').where('teamId', '==', teamId).onSnapshot(snapshot => {
let invites = [];
invites[teamId] = [];
snapshot.docs.forEach(doc => {
const data = doc.data();
invites[data.teamId].push(Object.assign({}, { id: doc.id }, data));
resolve(invites);
});
});
}));
});
Promise.all(promises).then(allInvites => {
// Do what I needed to do with all of the invites here
});

Chaining mapped axios/API calls

For the API I'm using, I have to bounce off 3 separate endpoints to get the data I need. I'm stuck on the very last endpoint and it's driving me crazy. Here's the gist of what I'm currently doing (or attempting to do).
Make direct call to ENDPOINT 1. Process data through map (to return data I need specifically) then push to ARRAY 1.
Once ARRAY 1 is done processing, I map ARRAY 1's data to make an API call for each of it's IDs to ENDPOINT 2, then push this to ARRAY 2.
Once ARRAY 2 is done processing, I map ARRAY 2's data to make an API call for each of it's IDs to ENDPOINT 3, then push this to ARRAY 3.
All of these steps are wrapped in a promise that resolves with the 3 completed arrays.
Steps 1 and 2 get done just fine, but I have tried a million different things for step 3 and it keeps returning . What would be the best way for this to be handled? Any help would be MUCH appreciated!
router.get("/", (req, res) => {
let artists = [];
let albums = [];
let tracks = [];
const options = {
headers: {
'Authorization': `Bearer ${token}`,
},
};
function getArtists(url) {
return new Promise(resolve => {
axios.get(url, options).then(response => {
artists.push(...response.data.artists.items.map(artist => ({
url: artist.external_urls.spotify,
name: artist.name,
images: artist.images,
id: artist.id,
genres: artist.genres,
})));
let next = response.data.artists.next;
if (next !== null) {
getArtists(next);
} else {
resolve(getAlbums().then(() => getTracks().then(() => res.send({artists, albums, tracks}))));
};
});
});
};
let getAlbums = () => {
return new Promise(resolve => {
const requests = artists.map(item => {
return axios.get(`https://api.spotify.com/v1/artists/${item.id}/albums?market=us&include_groups=single,appears_on`, options).then(response => {
albums.push(...response.data.items);
});
});
Promise.all(requests).then(() => {
const filtered = albums.filter((curr, index, self) => self.findIndex(t => t.id === curr.id) === index);
const sorted = filtered.sort((a, b) => (b.release_date > a.release_date) ? 1 : -1); // change data to filtered to filter duplicates
const sliced = sorted.slice(0, 50);
albums = sliced;
// res.send({artists, albums});
resolve();
});
});
};
let getTracks = () => {
albums.map(item => {
return axios.get(`https://api.spotify.com/v1/albums/${item.id}/tracks`, options).then(response => {
tracks.push(...response.data.items);
});
});
};
if (token) {
const url = 'https://api.spotify.com/v1/me/following?type=artist&limit=50';
getArtists(url);
} else {
res.send({
message: 'Please login to retrieve data',
});
};
});

I think you might be better to follow a simpler approach, using async/await. This allows us to structure the code in an easier to follow way. If we have loads of .then() and new Promises etc, it gets very confusing, very quickly!
I've restructured like so, I think this is easier to follow and hopefully to debug!
We can loop over each item in the getXXX functions, or we can use Promise.all, either approach works, though the latter may be more performant.
I've used this in getTracks()
router.get("/", async (req, res) => {
let options = {
headers: {
'Authorization': `Bearer ${token}`,
}
}
if (!token) {
res.send({
message: 'Please login to retrieve data',
});
return;
}
const url = 'https://api.spotify.com/v1/me/following?type=artist&limit=50';
let artists = await getArtists(url, options);
console.log ("Artists (length):", artists.length );
let albums = await getAlbums(artists, options);
console.log ("Albums (length):", albums.length );
let tracks = await getTracks(albums, options);
console.log("Tracks (length):", tracks.length);
res.send( { albums, artists, tracks } );
}
async function getArtists(url, options, maxLoopCount = 100) {
let artists = [];
let count = 0;
do {
console.log(`getArtists: Page #${++count}...`);
let artistResp = await getArtistsPage(url, options);
artists.push(...artistResp.artistList);
url = artistResp.next;
} while (url && count < maxLoopCount) ;
return artists;
}
async function getArtistsPage(url, options) {
let response = await axios.get(url, options);
let artistList = response.data.artists.items.map(artist => ({
url: artist.external_urls.spotify,
name: artist.name,
images: artist.images,
id: artist.id,
genres: artist.genres,
}));
let next = response.data.artists.next;
return { artistList, next}
};
async function getAlbums(artists, options, sliceCount = 50) {
let albums = [];
for(let artist of artists) {
let response = await axios.get(`https://api.spotify.com/v1/artists/${artist.id}/albums?market=us&include_groups=single,appears_on`, options);
albums.push(...response.data.items);
}
const filtered = albums.filter((curr, index, self) => self.findIndex(t => t.id === curr.id) === index);
const sorted = filtered.sort((a, b) => (b.release_date > a.release_date) ? 1 : -1); // change data to filtered to filter duplicates
const sliced = sorted.slice(0, sliceCount);
return sliced;
}
async function getTracks(albums, options) {
let promises = albums.map(album => axios.get(`https://api.spotify.com/v1/albums/${album.id}/tracks`, options));
let responseList = await Promise.all(promises);
return responseList.map(response => response.data.items).flat();
}

RXJS Stream - How to wait for my STREAM to end and return DATA and then do my OPERATIONS

I have a Websocket Endpoint I am subscribing. I want to get that Data, and then operate on them.
CODE:
// Simple HTTP POST Request. Works Perfectly. I am Logged In to the API
const authenticationRequest = () => axios.post(authenticationUrl, {
user: username, password
})
.then((response) => response.data)
.catch((error) => console.error(console.error('Error Response', error)));
// WS Request. I need to wait for this to return my Data and then operate on them
const wsRequest = async () => {
// Getting the Auth Token. Working Perfectly.
const reqToken = await authenticationRequest();
// Hitting the ws Endplint. Working Perfectly.
const webSocketRequest = new WebSocket(topicDataUrl);
// Setting the Data for the First Message. Works Perfectly.
const firstMessage = {
token: reqToken,
stats: 2,
sql: "SELECT * FROM cc_payments LIMIT 100",
live: false
};
// Initialising an Empty Array. Works.
let websocketData = [];
// Opening the Endpoint
webSocketRequest.onopen = () => {
// Sending the first Message
webSocketRequest.send(JSON.stringify(firstMessage));
// On Each Message
webSocketRequest.onmessage = (streamEvent) => {
of(streamEvent).pipe(
map(event => JSON.parse(event.data)), // Parse the Data
filter(message => message.type === 'RECORD') // Filter the Data
).subscribe(
message => websocketData.push(message.data.value)// Adding each Value from each message to the Array.
);
};
};
console.log(JSON.stringify(websocketData), 'Websocket DATA'); // Empty Array
return websocketData;
};
Here I am calling it a few lines down, but still with no results. I get an empty Array.
(async function () {
const data = await wsRequest();
console.log(JSON.stringify(data), 'Data'); // Still Empty
}());
So, what am I doing wrong? Can someone, explain to me the problem? I mean I get the asynchronisity of things, but I am awaiting. I even tried setting a timeout but didn't work.
Is my stream correct? Maybe there is a problem there??
So, the RXJS Actions are asynchronous. So, I would need 2 Things.
- Close the Stream when Operationg Completed. (Tried takeUntil, takeWhile, but obviously was doing something wrong)
- Wait in order to return the Actual Data(WebsocketData).
UPDATE:
async function authenticationRequest() {
const AuthenticateWith = await axios.post(authenticationUrl, {
user: username,
password
})
.then(response => response.data)
.catch((error) => console.error('Error:', error));
return AuthenticateWith;
}
const webSocketRequest = new WebSocket(topicDataUrl);
const websocketData = new Array;
const subject = new Subject();
async function requestToWSEndpoint() {
const reqToken = await authenticationRequest();
const firstMessage = {
token: reqToken,
stats: 2,
sql: "SELECT * FROM cc_payments LIMIT 100",
live: false
};
webSocketRequest.onopen = () => {
webSocketRequest.send(JSON.stringify(firstMessage));
webSocketRequest.onmessage = (streamEvent) => {
JSON.parse(streamEvent.data).type === 'RECORD' && websocketData.push(JSON.parse(streamEvent.data).data.value);
subject.next(websocketData);
JSON.parse(streamEvent.data).type === 'END' && subject.complete();
};
};
};
(async function () {
requestToWSEndpoint();
const chartData = subject.subscribe((event) => console.log(event, 'Event')); // Event Adds to the Array and at the End I have all my Items(Filtered). It printed 100 Times.
console.log('ARRAY', chartData); // This returns [Subscriber {closed: false, _parentOrParents: null, _subscriptions: Array(1), syncErrorValue: null, syncErrorThrown: false, …}]. This is what I want. The Array.
}());

My suggestion as outlined in my comment:
const subject = new Subject();
...
}
(async function () {
wsRequest();
subject.pipe(finalize(()=> {console.log('ARRAY', websocketData);})).subscribe();
}());
Actually you dont even need a function for what you do in wsRequest, except for const reqToken = await authenticationRequest();. The rest can easily be globally scoped.
You should take a look at the documentation for rxjs operators.