My application needs to read in a large dataset and pipe it to the client to manipulate with D3.js. The problem is, on large datasets, the reading/loading of the file contents could take a while. I want to solve this using streams. However, I'm unsure of how to do so in the context of the Sails framework.
What I want to do is read the contents of the file and pipe it to a rendered page. However, I can't figure out how to pipe it through if I use something like res.view('somePage', { data: thePipedData });.
I currently have something like this:
var datastream = fs.createReadStream(path.resolve(DATASET_EXTRACT_PATH, datatype, dataset, dataset + '.csv'));
datastream.pipe(res);
...
return res.view('analytics', { title: 'Analytics', data: ??? });
What's the best way to approach this?
Based on your example it seems like the best course of action would be to set up a separate endpoint to serve just the data, and include it on the client via a regular <script> tag.
MyDataController.js
getData: function(req, res) {
/* Some code here to determine datatype and dataset based on params */
// Wrap the data in a Javascript string
res.write("var theData = '");
// Open a read stream for the file
var datastream = fs.createReadStream(
path.resolve(DATASET_EXTRACT_PATH, datatype, dataset, dataset + '.csv')
);
// Pipe the file to the response. Set {end: false} so that res isn't closed
// when the file stream ends, allowing us to continue writing to it.
datastream.pipe(res, {end: false});
// When the file is done streaming, finish the Javascript string
datastream.on('end', function() {
res.end("';");
});
}
MyView.ejs
<script language="javascript" src="/mydata/getdata?datatype=<%=datatype%>&etc.."></script>
MyViewController.js
res.view('analytics', {datatype: 'someDataType', etc...});
A slight variation on this strategy would be to use a JSONP-style approach; rather than wrapping the data in a variable in the data controller action, you would wrap it in a function. You could then call the endpoint via AJAX to get the data. Either way you'd have the benefit of a quick page load since the large data set is loaded separately, but with the JSONP variation you'd also be able to easily show a loading indicator while waiting for the data.
Related
I use an html table where it's content can be changed with mouse drag and drop implemented. Technically, you can move the data from any table cell to another. The table size 50 row * 10 column with each cell given a unique identifier. I want to export it to .xlsx format with C# EPPlus library, and give back the exported file to client.
So I need the pass the whole table data upon a button press and post it to either a web api or an mvc controller, create an excel file (like the original html table data) and send it back to download with browser.
So the idea is to create an array which contains each of table cell's value ( of course there should be empty cells in that array), and post that array to controller.
The problem with that approach lies in the download, if I call the api or mvc controller with regular jquery's ajax.post it did not recognize the response as a file.
C# code after ajax post:
[HttpPost]
public IHttpActionResult PostSavedReportExcel([FromBody]List<SavedReports> savedReports, [FromUri] string dateid)
{
//some excel creation code
HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StreamContent(new MemoryStream(package.GetAsByteArray()))
};
response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
response.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment")
{
FileName = dateid + "_report.xlsx"
};
ResponseMessageResult responseMessageResult = ResponseMessage(response);
return responseMessageResult;
}
Usually, for this kind of result I could use window.location = myurltocontroller to download properly , but that is only for GET requests, POST anything is not possible.
I found some answers which could help me in this topic:
JavaScript post request like a form submit
This points out I should go with creating a form, which passes the values, but I do not know how to do so in case of arrays (the table consists 50*10 = 500 values which I have to pass in the form)
I tried some only frontend solutions to the html-excel export problem, which of course does not require to build files on api side, but free jquery add-ins are deprecated, not customizeable, handle only .xls formats, etc.
I found EPPlus nuget package a highly customizeable tool, that is why I want to try this is at first place.
So the question is: how can I post an array of 500 elements, that the controller will recognize, generate the file, and make it automatically download from browser?
If you can provide some code that would be fantastic, but giving me the right direction is also helpful.
Thank you.
You can use fetch() (docs) to send the request from the JS frontend. When the browser (JS) has received the response, it can then offer its binary content as a download. Something like this:
fetch("http://your-api/convert-to-excel", // Send the POST request to the Backend
{
method:"POST",
body: JSON.stringify(
[[1,2],[3,4]] // Here you can put your matrix
)
})
.then(response => response.blob())
.then(blob => {
// Put the response BLOB into a virtual download from JS
if (navigator.appVersion.toString().indexOf('.NET') > 0) {
window.navigator.msSaveBlob(blob, "my-excel-export.xlsx");
} else {
var a = window.document.createElement('a');
a.href = URL.createObjectURL(blob);
a.download = "my-excel-export.xlsx";
a.click();
}});
So the JS part of the browser actually first downloads the file behind the scenes, and only when it's done, it's triggering the "download" from the browsers memory into a file on the HD.
This is a quite common scenario with REST APIs that require bearer token authentication.
Is there any way to redirect to an HTML file from a Node.JS application with something like: res.sendFile of express and pass a JSON data along to the html file?
I know this is late but I wanted to offer a solution which no one else has provided. This solution allows a file to be streamed to the response while still allowing you to modify the contents without needing a templating engine or buffering the entire file into memory.
Skip to the bottom if you don't care about "why"
Let me first describe why res.sendFile is so desirable for those who don't know. Since Node is single threaded, it works by performing lots and lots of very small tasks in succession - this includes reading from the file system and replying to an http request. At no point in time does Node just stop what it's doing and read an entire from the file system. It will read a little, do something else, read a little more, do something else. The same goes for replying to an http request and most other operations in Node (unless you explicitly use the sync version of an operation - such as readFileSync - don't do that if you can help it, seriously, don't - it's selfish).
Consider a scenario where 10 users make a request for for the same file. The inefficient thing to do would be to load the entire file into memory and then send the file using res.send(). Even though it's the same file, the file would be loaded into memory 10 separate times before being sent to the browser. The garbage collector would then need to clean up this mess after each request. The code would be innocently written like this:
app.use('/index.html', (req, res) => {
fs.readFile('../public/index.html', (err, data) => {
res.send(data.toString());
});
});
That seems right, and it works, but it's terribly inefficient. Since we know that Node does things in small chunks, the best thing to do would be to send the small chunks of data to the browser as they are being read from the file system. The chunks are never stored in memory and your server can now handle orders of magnitude more traffic. This concept is called streaming, and it's what res.sendFile does - it streams the file directly to the user from the file system and keeps the memory free for more important things. Here's how it looks if you were to do it manually:
app.use('/index.html', (req, res) => {
fs.createReadStream('../public/index.html')
.pipe(res);
});
Solution
If you would like to continue streaming a file to the user while making slight modifications to it, then this solution is for you. Please note, this is not a replacement for a templating engine but should rather be used to make small changes to a file as it is being streamed. The code below will append a small script tag with data to the body of an HTML page. It also shows how to prepend or append content to an http response stream:
NOTE: as mentioned in the comments, the original solution could have an edge case where this would fail. For fix this, I have added the new-line package to ensure data chunks are emitted at new lines.
const Transform = require('stream').Transform;
const parser = new Transform();
const newLineStream = require('new-line');
parser._transform = function(data, encoding, done) {
let str = data.toString();
str = str.replace('<html>', '<!-- Begin stream -->\n<html>');
str = str.replace('</body>', '<script>var data = {"foo": "bar"};</script>\n</body>\n<!-- End stream -->');
this.push(str);
done();
};
// app creation code removed for brevity
app.use('/index.html', (req, res) => {
fs
.createReadStream('../public/index.html')
.pipe(newLineStream())
.pipe(parser)
.pipe(res);
});
You get one response from a given request. You can either combine multiple things into one response or require the client to make separate requests to get separate things.
If what you're trying to do is to take an HTML file and modify it by inserting some JSON into it, then you can't use just res.sendFile() because that just reads a file from disk or cache and directly streams it as the response, offering no opportunity to modify it.
The more common way of doing this is to use a template system that lets you insert things into an HTML file (usually replacing special tags with your own data). There are literally hundreds of template systems and many that support node.js. Common choices for node.js are Jade (Pug), Handlebars, Ember, Dust, EJS, Mustache.
Or, if you really wanted to do so, you could read the HTML file into memory, use some sort of .replace() operation on it to insert your own data and then res.send() the resulting changed file.
Well, it's kinda old, but I didn't see any sufficient answer, except for "why not". You DO have way to pass parameters IN static file. And that's quite easy. Consider following code on your origin (using express):
let data = fs.readFileSync('yourPage.html', 'utf8');
if(data)
res.send(data.replace('param1Place','uniqueData'));
//else - 404
Now for example, just set a cookie, in yourPage.html, something like:
<script>
var date = new Date();
document.cookie = "yourCookieName='param1Place';" +
date.setTime(date.getTime() + 3600) + ";path=/";
</script>
And you can plainly pull content of uniqueData from yourCookieName wherever you want in your js
I think the answer posted by Ryan Wheale is the best solution if you actually want to modify something within an HTML file. You could also use cheerio for working with complex logic.
But in regards to this particular question where we just want to pass some data to the client from the server, there's actually no need to read index.html into memory at all.
You can simply add the following script tag somewhere at the top of your HTML file:
<script src="data.js"></script>
And then let Express serve that file with whatever data needed:
app.get("/data.js", function (req, res) {
res.send('window.SERVER_DATA={"some":"thing"}');
});
This data can then easily be referenced anywhere in your client application using the window object as: window.SERVER_DATA.some
Additional context for a React frontend:
This approach is especially useful during development if your client and server are running on different ports such as in the case of create-react-app because the proxied server can always respond to the request for data.js but when you're inserting something into index.html using Express then you always need to have your production build of index.html ready before inserting any data into it.
Why not just read the file, apply transformations and then set up the route in the callback?
fs.readFile(appPath, (err, html) => {
let htmlPlusData = html.toString().replace("DATA", JSON.stringify(data));
app.get('/', (req, res) => {
res.send(htmlPlusData);
});
});
Note that you can't dynamically change data, you'd have to restart the node instance.
You only have one response you can return from the server. The most common thing to do would be to template your file on the server with nunjucks or jade. Another choice is to render the file on the client and then to use javascript to make an ajax call to the server to get additional data. I suppose you could also set some data in a cookie and then read that on the client side via javascript as well.
(Unless you want to template the html file to insert the json data into a script tag). You'll need to expose an api endpoint in express the send along the data to the page, and have a function on the page to access it. for example,
// send the html
app.get('/', (req, res) => res.sendFile('index'));
// send json data
app.get('/data', (req, res) => res.json(data));
Now on the client side you can create a request to access this endpoint
function get() {
return new Promise((resolve, reject) => {
var req = new XMLHttpRequest();
req.open('GET', '/data');
req.onload = () => resolve(req.response);
});
}
// then to get the data, call the function
get().then((data) => {
var parsed = JSON.parse(data);
// do something with the data
});
EDIT:
So arrow functions probably don't work client side yet. make sure to replace them with function(){} in your real code
This is pretty easy to do using cookies. Simply do this:
On the server side -
response.append('Set-Cookie', 'LandingPage=' + landingPageCode);
response.sendFile(__dirname + '/mobileapps.html');
On client side -
<!DOCTYPE html>
<html>
<body onload="showDeferredLandingPageCode()">
<h2>Universal Link Mobile Apps Page</h2>
<p>This html page is used to demostrate deferred deeplinking with iOS</p>
</body>
<script language="javascript">
function showDeferredLandingPageCode() {
alert(document.cookie);
}
</script>
</html>
I have to load some data stored in tsv file to create a bar chart with d3js.
I use this code to read the file:
d3.tsv("data.tsv", function(error, data) {
The data inside the file change at every click of a button that invokes a servlet function to update these data.
The problem is that I can't get the updating data, so I am stuck on the fist data stored in the file.
I avoid this problem creating n-files and reading these different files.
But I want to use the same file.
So here you go,
Onclick get the new file
Once you are sure you have the updatedFile read it using d3.tsv (you can use callback here) and inside that give a call to the function which does drawing
This is because your browser is caching the file and not actually reloading it. You can avoid this by adding a changing query parameter to the call, which doesn't do anything but prevents the browser from caching:
var counter = 0;
// ...
d3.tsv("data.tsv?foo=" + i++, function(error, data) {
// ...
});
I'm trying to create a note taking web app that will simply store notes client side using HTML5 local storage. I think JSON is the way to do it but unsure how to go about it.
I have a simple form set up with a Title and textarea. Is there a way I can submit the form and store the details entered with several "notes" then list them back?
I'm new to Javascript and JSON so any help would be appreciated.
there are many ways to use json.
1> u can create a funciton on HTML page and call ajax & post data.
here you have to use $("#txtboxid").val(). get value and post it.
2> use knock out js to bind two way.and call ajax.
here is simple code to call web app. using ajax call.
var params = { "clientID": $("#txtboxid") };
$.ajax({
type: "POST",
url: "http:localhost/Services/LogisticsAppSuite.svc/Json/GetAllLevelSubClients",
contentType: 'application/json',
data: JSON.stringify(params),
dataType: 'json',
async: false,
cache: false,
success: function (response) {
},
error: function (ErrorResponse) {
}
I have written a lib that works just like entity framework. I WILL put it here later, you can follow me there or contact me to get the source code now. Then you can write js code like:
var DemoDbContext = function(){ // define your db
nova.data.DbContext.call(this);
this.notes=new nova.data.Repository(...); // define your table
}
//todo: make DemoDbContext implement nova.data.DbContext
var Notes = function(){
this.id=0; this.name="";
}
//todo: make Note implement nova.data.Entity
How to query data?
var notes = new DemoDbContext().notes.toArray(function(data){});
How to add a note to db?
var db = new DemoDbContext();
db.notes.add(new Note(...));
db.saveChanges(callback);
Depending on the complexity of the information you want to store you may not need JSON.
You can use the setItem() method of localStorage in HTML5 to save a key/value pair on the client-side. You can only store string values with this method but if your notes don't have too complicated a structure, this would probably be the easiest way. Assuming this was some HTML you were using:
<input type="text" id="title"></input>
<textarea id="notes"></textarea>
You could use this simple Javascript code to store the information:
// on trigger (e.g. clicking a save button, or pressing a key)
localStorage.setItem('title', document.getElementById('title').value);
localStorage.setItem('textarea', document.getElementById('notes').value);
You would use localStorage.getItem() to retrieve the values.
Here is a simple JSFiddle I created to show you how the methods work (though not using the exact same code as above; this one relies on a keyup event).
The only reason you might want to use JSON, that I can see, is if you needed a structure with depth to your notes. For example you might want to attach notes with information like the date they were written and put them in a structure like this:
{
'title': {
'text':
'date':
}
'notes': {
'text':
'date':
}
}
That would be JSON. But bear in mind that the localStorage.setItem() method only accepts string values, you would need to turn the object into a string to do that and then convert it back when retrieving it with localStorage.getItem(). The methods JSON.stringify will do the object-to-string transformation and JSON.parse will do the reverse. But as I say this conversion means extra code and is only really worth it if your notes need to be that complicated.
I have an ExtJS based application. When editing an object, an ExtJS window appears with a number of tabs. Three of these tabs have Ext GridPanels, each showing a different type of data. Currently each GridPanel has it's own JsonStore, meaning four total AJAX requests to the server -- one for the javascript to create the window, and one for each of the JsonStores. Is there any way all three JsonStores could read from one AJAX call? I can easily combine all the JSON data, each one currently has a different root property.
Edit: This is Ext 2.2, not Ext 3.
The javascript object created from the JSON response is available in yourStore.reader.jsonData when the store's load event is fired. For example:
yourStore.on('load', function(firstStore) {
var data = firstStore.reader.jsonData;
otherStore.loadData(data);
thirdStore.loadData(data);
}
EDIT:
To clarify, each store would need a separate root property (which you are already doing) so they'd each get the data intended.
{
"firstRoot": [...],
"secondRoot": [...],
"thirdRoot": [...]
}
You could get the JSON directly with an AjaxRequest, and then pass it to the loadData() method of each JSONStore.
You may be able to do this using Ext.Direct, where you can make multiple requests during a single connection.
Maybe HTTP caching can help you out. Combine your json data, make sure your JsonStores are using GET, and watch Firebug to be sure the 2nd and 3rd requests are not going to the server. You may need to set a far-future expires header in that json response, which may be no good if you expect that data to change often.
Another fantastic way is to use Ext.Data.Connection() as shown below :
var conn = new Ext.data.Connection();
conn.request({
url: '/myserver/allInOneAjaxCall',
method: 'POST',
params: {
// if you wish too
},
success: function(responseObj) {
var json = Ext.decode(responseObj.responseText);
yourStore1.loadData(json.dataForStore1);
yourStore2.loadData(json.dataForStore2);
},
failure: function(responseObj) {
var message = Ext.decode(responseObj.responseText).message;
alert(message);
}
});
It worked for me.