I am creating a small application to in order to test a larger application I am building. This is reading from a CSV file and attempting to POST that data to my API endpoint (this is being done to send a large dataset over time).
Everything works perfectly well with Postman, but I am having trouble getting node to play nice. A sample json object I am posting looks like this
{
meta:
[ 'array','of','headers'],
data: [
[1,2,3],
[4,5,6],
[7,8,9]]
}
When I receive the data in express, it comes through as meta=[array, of, headers], data=[0=1, 1=2, 2=3, 0=4, 1=5, 2=6, 0=7, 1=8, 2=9]
I assume this is something to do with the way I am sending my JSON (as I said, this renders fine from Postman).
I am stringifying the data before it's sent, setting headers to
"Content-type":"application/json" and setting the "content-length".
After vanilla http request I tried using request-json, but I am getting the same result.
Any help is greatly appreciated!
It turns out that the data was in fact being transferred properly, it was the fact that I was sending strings that led me to believe that there were errors here.
Related
Context
A shiny application I work on sends processed data through:
session$sendCustomMessage("handler", data)
The Javascript on the front end catches this message and does some visualizations:
Shiny.addCustomMessageHandler('handler', (data) => visualizer(data))
When the fetch data button is clicked this data is sent.
The shiny server is currently hosted on an Amazon EC2 R6a Large instance.
Issue
Depending on the parameters the user selects, the data can be small or huge. When the parameters are such that the data is a huge JSON object, there is an error given and the server disconnects with the following messages:
I have tested the same scenarios but without sending the data through session$sendCustomMessage("handler", data) and am not able to reproduce the error. It seems like sending a huge amount of data through the network causes the shiny server to disconnect.
What could be the fix for this?
It seems like increasing the sockjs_heartbeat_delay and sockjs_disconnect_delay helped with this. After increasing this, the error seemed to be gone.
The connection simply times out when the huge data is loading and the comment above was correct about it.
https://docs.posit.co/shiny-server/
Simply add these directives in the shiny-server.conf with a reasonable value.
I have a more generic question about receiving a stream with JavaScript
Given the generic use case: I want to get data from backend (csv file, ...) to frontend.
Given the following restrictions:
Backend just accepts 'POST' (non negotiable)
Frontend uses 'Axios' (could be adjusted)
How would I do that?
All the examples in the network work with stream to get data like csv etc., I'm convinced that that's the way to go but couldn't explain it.
My situation
So far I tried a lot to make it work, I did actually but I don't think it was really "Streaming".
Backend: I pushed my stream into the OutputStream of the response and returned the stream. That means the data is visible in the devTools which shoudn't be. Without that, just returning the stream and not writing it to the OutputStream, I always got an "Network error", the backend returned successfully (I guess) but still I got the error.
Frontend: After some time I assume I realized why I got probably got the "Network error": Axios does not support stream
Furthermore I use a fileDownloader that simulates a download click to download the response (POST fetch with Axios) which I formatted into a Blob.
What is necessary to make streaming work? Is there an agenda?
I don't want to fabricate stupid stuff and go somewhere I shouldn't go. That's why I would like to get some information from people who have already some experience with that.
Do I need 'GET' to make streaming work? Or does it work with 'POST' too?
In the backend returning the stream without writing it to the output of the response body is enough?
Is Streaming the right way?
What do I use? Frontend: React, TypeScript, Axios | Backend: C#
I can't for the life of me figure this out, it seems like it should be straight forward but it's just not clicking.
I have an ES6 app that I created using create-react-app. I've got all the templates and layouts set up for the project and came to trying to pull in data from an API that I want to sit inside the app - like a botched MVC where React handles the views and I run the models and controllers in PHP.
So I have a function in one of my components that I want to fetch some data. I use the fetch() function (I know this isn't yet compatible with a number of browsers but that's a problem for another day) to fetch a relative path from the component to the model I want to load, however the fetch function treats my path as a call to the base URL followed by the request. So with the site running on localhost:3000, I run the following code in my getData() function...
let test = fetch('../models/overall-stats.php').then(function(response) {
console.log(response);
return response;
});
...the URL that fetch hits is then http://localhost:3000/models/overall-stats.php which simply resolves back to the index.html file and loads the app, rather than the PHP file I'm requesting.
If I need to hit that PHP file to get my data, am I wrong in using fetch? Or am I just using it incorrectly? If I shouldn't be using fetch what's a better approach to this problem I'm having?
When I run this on an apache server (after building and deploying) I can get the fetches to work fine (apache recognizes the structure of the URL and hits it as I am expecting) and I hit the file no issues, but I need to be able to work in a local development environment and have the same functionality. The app will end up being deployed live on an apache server.
Any help would be greatly appreciated.
I knew after sleeping on this it would be very straight-forward... I simply had to move my models and controllers into the public directory for them to be accessible. I'll be putting in authentication to the models so that they can't be hit directly, but only through GET requests.
Why don't you just use something like ${baseUrl}/models/... ?
Also for solving browsers problem with fetch you can import the Polyfill or simply use axios (my choice)!
Maybe you can try to use ajax to get or post the data from server, just like this:
$.ajax({
url: '../models/overall-stats.php',
data: {
},
type: 'GET',
dataType : 'json',
success : function(res){
let obj = parseJSON(res)
}
})
or add this on top in your php file because the CORS :
header('Access-Control-Allow-Origin: *');
I'm relatively new to full-stack development, and currently trying to figure out an effective way to send and fetch large data between my front-end (React) and back-end (Express) while minimizing memory usage. Specifically, I'm building a mapping app which requires me to play around with large JSON files (10-100mb).
My current setup works for smaller JSON files:
Backend:
const data = require('../data/data.json');
router.get('/', function(req, res, next) {
res.json(data);
});
Frontend:
componentDidMount() {
fetch('/')
.then(res => res.json())
.then(data => this.setState({data: data}));
}
However, if data is bigger than ~40mb, the backend would crash if I test on local due to running out of memory. Also, holding onto the data with require() takes quite a bit of memory as well.
I've done some research and have a general understanding of JSON parsing, stringifying, streaming, and I think the answer lies somewhere with using chunked json stream to send the data bit by bit, but am pretty much at a loss on its implementation, especially using a single fetch() to do so (is this even possible?).
Definitely appreciate any suggestions on how to approach this.
First off, 40mb is huge and can be inconsiderate to your users especially if there' s a high probability of mobile use.
If possible, it would be best to collect this data on the backend, probably put it onto disk, and then provide only the necessary data to the frontend as it's needed. As the map needs more data, you would make further calls to the backend.
If this isn't possible, you could load this data with the client-side bundle. If the data doesn't update too frequently, you can even cache it on the frontend. This would at least prevent the user from needing to fetch it repeatedly.
Alternatively, you can read the JSON via a stream on the server and stream the data to the client and use something like JSONStream to parse the data on the client.
Here's an example of how to stream JSON from your server via sockets: how to stream JSON from your server via sockets
Hi guys - Super Quick TL;DR version:
My Circle CI endpoint to return artifacts in a JSON contains a first entry, That entry has a URL to a code-coverage JSON file.
That is the data that i need returned via my axios, ajax or fetch API.
I've tried all three.
On Postman, via webserver or in the browser console... they all come back with "Not Logged In" sort of errors.
So this is an issue with tokens, access etc I guess?
[ Another person experiencing the same issue - https://discuss.circleci.com/t/circle-token-param-ignored-when-using-api-url-to-fetch-latest-artifact/3197 ]
Has anyone managed to resolve this yet?
Can anyone advise how I can get that data (not just the artifacts) returned if Circle CI's endpoint only works in the browser? And if the URL for that one-level-deeper sort of information doesn't accept tokens?
Ummmm. Help?
Inside The React Component:
componentDidMount() {
axios.get("https://circleci.com/api/v1.1/project/github/ORG/REPO/Latest/artifacts?circle-token=<My TOKEN>")
.then(function(result){
console.log('API result===>', result);
})
}
What kind of artifact URLs are you using? You should be using the *.circle-artifacts.com/* type URLs.
The endpoint you provided in your question should work fine as long as you lowercase Lower. It should look like this:
componentDidMount() {
axios.get("https://circleci.com/api/v1.1/project/github/<org>/<repo>/latest/artifacts?circle-token=<my-token>")
.then(function(result){
console.log('API result===>', result);
})
}