Trouble Fetching JSON files on server into Javascript Array - javascript

I'm quite new to using Javascript and particularly JSON, and I've been struggling to do this:
There is a JSON file on my web server which I am trying to access and parse into a JavaScript object. What I am trying to do is parse that JSON into an array and then further manipulate that array based on other user variables.
This is a sample of what the JSON looks like:
{"log":
[{
"name":"Al",
"entries":[8,12,16,19]},
{"name":"Steve",
"entries":[11,17,22]}]}
What I need to do is retrieve the array for one of the entries and store it in an array as a JavaScript object. What I have tried to do is this:
var entriesLogged;
fetch ('url to the json file').then(function(response){
return response.json();
}).then(function(data){
entriesLogged = data.log[0].entries;
});
However, I can't seem to get this to work and to assign the value to the variable in a way that persists outside of this scope. I have been able to output the value of the array using console.log, but I have not been able to actually work with and manipulate that data like an object. I'd ideally like to parse the JSON file from the server onto a global array.
Most of the tutorials I've come across so far have used the JSON file to output console logs or change the contents of html elements, however I need to retrieve the values from the JSON into a global array first.
Am I missing something here? Does anyone have advice on how to do this?
Best wishes,
Dom

Are you trying to manipulate the entriesLogged variable after the fetch promise chain? The fetch is asynchronous so this means that any code after the fetch chain will run before the fetch chain finishes.
var entriesLogged;
fetch('url to the json file').then(function(response){
return response.json();
}).then(function(data){
entriesLogged = data.log[0].entries;
});
// Code down here will run before the fetch chain finishes
// So if you're using entriesLogged down here, it will be null
console.log(entriesLogged); // entriesLogged is null
You might want to do something like:
.then(function(data){
entriesLogged = data.log[0].entries;
myFunction(entriesLogged);
});
function myFunction(myData) {
console.log(myData); // can access entriesLogged here
}

What is happening is that the call is async, it means that when you make that call a new thread is created and javascript automatically passes to the next line.
var entriesLogged;
fecth(something).then(function(data){entriesLogged = data[0]}); // this goes to another thread an takes e.g. 5 miliseconds but javascript does not wait for this to complete, it goes to the next line
console.log(entriesLogged); // this is executed before the call is finished

There are TONS of "answers" for variations of this question that don't actually solve the problem, they just describe the issue. (tl;dr solution is last block of code)
The issue is that .fetch('URL.json') and .json() are "asynchronous" calls. So the script will start working on that call, then continue on to the rest of the commands in your script while the .fetch() and .json() calls are still being worked on. If you then hit a command BEFORE one of these asynchronous calls is done, the rest of your script will have no data to work with.
There are tons of places to look for fully understanding the ins and outs of how async calls work in JS, for example here:
How do I return the response from an asynchronous call?
If you're like me and you just want the code to function synchronously, the quick and dirty solution is to create an async main loop and then to make sure you use the await keyword when calling asynchronous functions that you need to WAIT for the data to be populated on.
Example main loop:
(async () => {
// put main loop code here
})();
You would then put your code inside that main loop BUT you have to make sure that you're adding the await keyword in front of every asynchronous call you use. in the original example provided, you'll have to declare your .then() functions that need to use await as async as well. This is confusing if you don't fully understand the asynchronous nature of javascript and I'll simplify it later, but I wanted to provide a working version of the original code provided so you can understand what changed:
(async () => {
var entriesLogged;
await fetch('url_to_the_file.json')
.then( async function(response){
return await response.json();
})
.then( function(data){
entriesLogged = data;
});
})();
The .then() call is typically used for post-processing so you can do work inside of the asynchronous call... but since in this example we're deliberately making it synchronous, we can avoid all of that and clean up the code by doing individual simple commands:
// Start main loop
(async () => {
let response = await fetch('url_to_the_file.json');
let entriesLogged = await response.json();
})();
That should get you what you're looking for and hopefully save anyone like us the heartache of spending hours of time trying track down something that there is a simple solution for.
Also want to give a huge call out to the place I got the solution for this:
https://www.sitepoint.com/delay-sleep-pause-wait/

Related

How to control C# Task (async/await in same way as javascript Promise)?

I'm js developer and jumped to C#. I'm learning async/await/Task in C# and can't understand how to control C# Task.
In javascript I can save Promise resolve handler to call it later (and resolve promise), for example, to organize communication between some "badly connected" app parts (request doesn't returns result, but result is sent separately and fires some "replyEvent"), and do something like this:
// javascript
const handlers = {}
// event firing on reply
const replyEventListener = (requestId, result) => {
if (handlers[requestId]) {
handlers[requestId](result)
delete handlers[requestId]
}
}
const requestFunction = (id, params) => new Promise(resolve => {
// handler is saved to use later from "outside"
handlers[id] = resolve;
// do required request
makeRequest(params)
})
// .. somewhere in code request becomes simple awaitable
const resultOfRequest = await requestFunction(someUniqueId)
How can I do same with C# Task?
I think the philosophy behind Promise (in JS) and Task (in C#) is different.
We don't implement them exactly the same way.
Read the documentation on the MSDN :
Task Class - Remarks
Task<TResult> Class
You must empty your cup to be able to fill it again.
A good article in MSDN is the following which shows examples of how Tasks can be used like Promises in the sense that you can start them and then await them later.
Asynchronous programming with async and await
If you want to await a Task in C# like you would a Promise in JS, you would do the following:
Create a method or function that returns a Task. Usually this will be a Task of some class object. In the article above, it references creating breakfast as an analogy, so you can define FryEggsAsync as a method that takes in an int number of how many eggs to fry and returns a Task<Egg>. The C# convention is to end any function or method name with Async to indicate that a Task is being returned.
Create your Task, which is similar to a Promise in JS, so you could do var eggsTask = FryEggsAsync(2), which would store a Task<Egg> in eggsTask, that you can then await or pass to other functions as needed.
To await your Task to resolve it and get the result, you would simply do await eggsTask.
You can also use Task.WaitAll to await multiple Tasks at once, similar to Promise.all in JS, by passing in an array of Tasks as an argument to that method call. See the Task.WaitAll documentation for some examples of this. The difference here though is that the results are stored separately in each task passed to Task.WaitAll, instead of aggregated together into a results array like in JS with Promise.all, but if you need to await multiple Tasks then this will be an easier way to do so.

Firestore not writing all documents

I'm using forEach to write over 300 documents with data from an object literal.
It works 80% of the time -- all documents get written, the other times it only writes half or so, before the response gets sent and the function ends. Is there a way to make it pause and always work correctly?
Object.entries(qtable).forEach(([key, value]) => {
db.collection("qtable").doc(key).set({
s: value.s,
a: value.a
}).then(function(docRef) {
console.log("Document written with ID: ", docRef.id);
res.status(200).send(qtable);
return null;
})
Would it be bad pratice to just put a 2 second delay?
You are sending the response inside your loop, before the loop is complete. If you are using Cloud Functions (you didn't say), sending the response will terminate the function an clean up any extra work that hasn't completed.
You will need to make sure that you only send the response after all the async work is complete. This means you will have to pay attention to the promises returned by set() and use them to determine when to finally send the response. Leaning how promises work in JavaScript is crucial to writing functions that work properly.
You need to wait for the set() calls to conclude. They return a promise that you should deal with.
For instance, you can do this by pushing the return of set() to a promise array and awaiting for them outside the loop (with Promise.all()).
Or you can await each individual call, but in this case you need to change the forEach() to a normal loop, otherwise the await will not work inside a forEach() arrow function.
Also, you should probably set the response status just once, and outside the loop.

How to handle async code within a loop in JavaScript?

I am reaching out to a SongKick's REST API in my ReactJS Application, using Axios.
I do not know how many requests I am going to make, because in each request there is a limit of data that I can get, so in each request I need to ask for a different page.
In order to do so, I am using a for loop, like this:
while (toContinue)
{
axios.get(baseUrl.replace('{page}',page))
.then(result => {
...
if(result.data.resultsPage.results.artist === undefined)
toContinue = false;
});
page++;
}
Now, the problem that I'm facing is that I need to know when all my requests are finished. I know about Promise.All, but in order to use it, I need to know the exact URLs I'm going to get the data from, and in my case I do not know it, until I get empty results from the request, meaning that the last request is the last one.
Of course that axios.get call is asynchronous, so I am aware that my code is not optimal and should not be written like this, but I searched for this kind of problem and could not find a proper solution.
Is there any way to know when all the async code inside a function is finished?
Is there another way to get all the pages from a REST API without looping, because using my code, it will cause extra requests (because it is async) being called although I have reached empty results?
Actually your code will crash the engine as the while loop will run forever as it never stops, the asynchronous requests are started but they will never finish as the thread is blocked by the loop.
To resolve that use async / await to wait for the request before continuing the loop, so actually the loop is not infinite:
async function getResults() {
while(true) {
const result = await axios.get(baseUrl.replace('{page}', page));
// ...
if(!result.data.resultsPage.results.artist)
break;
}
}

Get asynchronous response of http post request into variable with Node.js

I'm pretty new to Javascript and am trying to wrap my head around the Javascript's asynchronicity concept.
my goal is,
responseStr = httpPost(parameters)
// then store responseStr in DB
However, the Javascript I can do is,
var callback = function(response) {
var str = '';
//another chunk of data has been recieved, so append it to `str`
response.on('data', function(chunk) {
str += chunk;
});
//the whole response has been recieved, so we just print it out here
response.on('end', function() {
console.log(str);
});
};
http.request(options, callback).end();
So I get the http post request's response into the local variable str, from above code, but I still cannot imagine how to pass it back to my httpPost caller into responseStr
Basically, the flow is,
Within my httpPost function, I sent the http request via http.request,
and it returns immediately,
somehow sometime later, the callback function populated the str variable bits by bits
but how to pass it back to my httpPost caller into responseStr when all done so that I can store it away? Thx.
Summary:
Thank you all who answered. I enjoy all your sense of humor greatly, upvoting you all.
PS. sharing a piece of comment I read the other day, about the Javascript's asynchronicity,
Today, we will examine four different methods of performing the same task asynchronously, in node.js...:
In parallel, using callback functions
Sequentially, using callback functions
In parallel, using promises
Sequentially, using promises
This will help you decide which to use for your particular situation. It is simply a matter of taste. If you want to know which is the best method -- the absolute best way to go -- is probably to switch to Golang.
:-)
There are a couple of problems with what you are trying to do. The first problem being that it is impossible to return a value from a non blocking function that is only available in the future. Don't fret though, once time travel is possible we won't have to worry about this issue ever again :) But until then, where do we go from here? There are a couple of things we can do to get the value.
The first (preferred) solution is to modify your program by simply moving all functionality that uses the value in the place where the value becomes available e.g. the callback. For example, suppose the only functionality that uses value is the function call do_stuff_with(value)
// original program
var value = get_value('param', callback);
do_stuff_with(value);
// changes to
get_value('param', function (value) {
do_stuff_with(value);
});
Usually the reponse I get from new programmers is "Oh no! That means I need to change my entire program! Is there anything else we can do. I don't want to change the code. It is perfect and is now a part of my family. You monster!" I must admit, this is exactly how I felt after being proposed this same solution. There is another solution to this problem you might want to use right away, but it will drastically affect the performance of your program.
The second (terrible, naive, you'll be sorry, don't do it) solution that successfully returns the value from the function call requires you to use a method that supports synchronous functionality. I only see synchronous code in all languages other than javascript, because javascript is single threaded. "Blasphemy, What the hell does that mean?" That means the entire javascript program needs to pause the entire process in order to wait for an http request to complete. It is not possible to open an http request without opening a new thread. Is that bad? Hell yea it is. That means the entire user interface will freeze during an http request. That means other tasks scheduled tasks will not be called on time. You know what it's like to play a video game that freezes all the time. It's terrible.
"But I have no imagination and I really want to write a bad program." - Fine, just modify your program to use a method that implements synchronous functionality instead of an asynchronous method that takes a callback. I have no idea where to get this, because I've never needed it.
// replace this method
var value = get_str_async_method('param', callback);
console.log(value) // undefined
// use one that is synchronous
// a callback is not needed for this method
var value = get_str_sync_method('param');
console.log(value); // reponse
It's pretty straightforward: you can't do that. 😀
This is one of those problems that you have to cope with in JavaScript. Your callbacks get called after code that got pushed into the event loop was executed, so the surrounding code in your program has already executed synchronously and is long gone.
You should dig into the event loop to get a better feel for this.
As things are arranged currently in your code, nested code is the most straightforward way to handle this. Whatever depends on str gets moved to or called from within the callback where str gets it's value.
Another way to handle this is to find a package that supports a promise based interface. This allows your asynchronous code to transform from the nested structure asynchronous callbacks force you into and flattens the code. It can even allow you to do something like what you're trying to do here.
const context = {};
doAsync()
.then(function(result) {
// Do something with result
context.str = result;
return doSomethingElseAsync();
})
.then(function(result) {
// No more nesting, more readable, easier to reason about
context.str2 = result;
return doAsync3(context.str);
});
if you want to store all data after getting it you can add it to the nested function as down below
var callback = function(response) {
var str = '';
//another chunk of data has been recieved, so append it to `str`
response.on('data', function(chunk) {
str += chunk;
});
//the whole response has been recieved, so we just print it out here
response.on('end', function() {
console.log(str);
responseStr = httpPost(parameters); //this line should be added here
});
};
http.request(options, callback).end();

Get content from .json and write to variable [duplicate]

This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 7 years ago.
What is the best way to just grab the content of a .json and store it to a var ?
I've seen some threads but i just can't manage on my code...
function retrieve_json() {
var content;
$.when(
$.getJSON("http://localhost/browsergame/data/test.json", function(data) {
content = data;
})
).then(function () {
console.log(content);
return content;
});
}
var json_content = retrieve_json();
console.log (json_content);
the console.log within the function displays the correct content of the json, the console.log at the very end of my code shows 'undefined'. I guess the async is the main issue here. I have tried doing it with $.ajax but couldn't manage either.
So, what is the best way to just grab the content of a .json file and store it to a var ?
The problem is that retrieve_json wont stop and wait until the json arrives from the server. The execution continues with whatever is after the statement starting with your $.when. In this case it's the end of the function. Since there is no return statement here, the function returns with the value 'undefined', what you store in json_content.
If you want to stop execution until the answer returns you have to wrap the next steps into a function and for example call it in the 'then' method of your example or just pass it to the getJSON where you store it in the content variable now.
And make sure you read up on how promises work.
Once code is async, it can't be made synchronous. You can only reliably use the content of your JSON in functions that you pass to promises or as callbacks, otherwise you have no guarantee that your JSON has been fetched when your code gets executed.
If you find even the promises syntax horrible you might be interested in async/await, a proposed syntax for ES7 that can be transpiled to ES5 right now with Babel.
Here's a good article which demonstrates the different approaches to asynchronous programming in JavaScript: The long road to Async/Await in JavaScript.

Categories