We Keep Coding

Send some data to server without REST in JS

As far as i understand all what REST do is standartize a data sended to server by adding some headers. For example REST request can generate a line of bytes like so: POST /qwe HTTP/1.1 Host: 127.0.0.1 Connection: keep-alive and finish it with some user input.
Now im just playing with writing my own JS server and here is my question: is there a way in JS to send some data(bytes) without this REST addings like headers/method and will it work for browsers and HTTP protocol itself?
For example instead of sending POST /qwe HTTP/1.1 Host: 127.0.0.1 Connection: keep-alive MY DATA OVER THERE!!! just send MY DATA OVER THERE!!! so my server can read only user data without everything else.
Iv tried to google and end up that XMLhttpRequest and fetch both require some CRUD method to be specified and adding some headers in request anyway.

HTTP requests:
Need to specify the method
Need to specify the Host as a header (in HTTP 1.1. and newer)
Will include some other request headers automatically when make using JS from a browser
This has nothing to do with REST. It's just how HTTP works.
A non-HTTP protocol could avoid having that. JavaScript in a browser has no mechanisms that allow making non HTTP requests.
You might want to research WebSocket which allows two way communication over a single connection … but that is a bootstrapped by HTTP so doesn't really fulfil your requirement.

For example instead of sending POST /qwe HTTP/1.1 Host: 127.0.0.1 Connection: keep-alive MY DATA OVER THERE!!! just send MY DATA OVER THERE!!! so my server can read only user data without everything else.
I suspect you're misunderstanding what a request is, on a fundamental level. Without POST (the method), /qwe (the path), HTTP/1.1 (the protocol) and 127.0.0.1 (the address) there is no way for your computer to know where and how to send the data. These are necessary if you want to communicate with a server, and removing them will mean your code no longer works.
You're working with very low-level data here, which is probably not what you actually want to be doing. There are some packages which will let you ignore the how and what of the request, and focus on just the data inside it. Express might be a good place to start. You can set up a simple express server to handle requests on specific paths, and reply with data that your frontend can then use.
A REST API is a high-level concept and largely unrelated to what you're asking about.

Why are my server sent events arriving as a batch?

I have a Java 8 / Spring4-based web application that is reporting the progress of a long-running process using Server Sent Events (SSEs) to a browser-based client running some Javascript and updating a progress bar. In my development environment and on our development server, the SSEs arrive in near-real-time at the client. I can see them arriving (along with their timestamps) using Chrome dev tools and the progress bar updates smoothly.
However, when I deploy to our production environment, I observe different behaviour. The events do not arrive at the browser until the long-running process completes. Then they all arrive in a burst (the events all have the timestamps within a few hundred milliseconds of each other according to dev tools). The progress bar is stuck at 0% for the duration and then skips to 100% really quickly. Meanwhile, my server logs tell me the events were generated and sent at regular intervals.
Here's the relevant server side code:
public class LongRunningProcess extends Thread {
private SseEmitter emitter;
public LongRunningProcess(SseEmitter emitter) {
this.emitter = emitter;
}
public void run() {
...
// Sample event, representing 10% progress
SseEventBuilder event = SseEmitter.event();
event.name("progress");
event.data("{ \"progress\": 10 }"); // Hand-coded JSON
emitter.send(event);
...
}
}
#RestController
public class UploadController {
#GetMapping("/start")
public SseEmitter start() {
SseEmitter emitter = new SseEmitter();
LongRunningProcess process = new LongRunningProcess(emitter);
process.start();
return emitter;
}
}
Here's the relevant client-side Javascript:
EventSource src = new EventSource("https://www.example.com/app/start");
src.addEventListener('progress', function(event) {
// Process event.data and update progress bar accordingly
});
I believe my code is fairly typical and it works just fine in DEV. However if anyone can see an issue let me know.
The issue could be related to the configuration of our production servers. DEV and PROD are all running the same version of Tomcat. However, some of them are accessed via a load balancer (F5 in out case). Almost all of them are behind a CDN (Akamai in our case). Could there be some part of this setup that causes the SSEs to be buffered (or queued or cached) that might produce what I'm seeing?
Following up on the infrastructure configuration idea, I've observed the following in the response headers. In the development environment, my browser receives:
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Connection: Keep-Alive
Content-Type: text/event-stream;charset=UTF-8
Keep-Alive: timeout=15, max=99
Pragma: no-cache
Server: Apache
Transfer-Encoding: chunked
Via: 1.1 example.com
This is what I'd expect for an event stream. A chunked response of an unknown content length. In the production environment, my browser receives something different:
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Connection: keep-alive
Content-Type: text/event-stream;charset=UTF-8
Content-Encoding: gzip
Content-Length: 318
Pragma: no-cache
Vary: Accept-Encoding
Here the returned content has a known length and is compressed. I don't think this should happen for an event stream. It would appear that something is converting my event stream into single file. Any thoughts on how I can figure out what's doing this?

It took a significant amount of investigation to determine that the cause of the issue was the elements in our network path. So the code above is correct and safe to use. If you find SSE buffering you will most likely want to check the configuration of key networking elements.
In my case, it was Akamai as our CDN and the use of an F5 device as a load balancer. Indeed it was the fact that both can introduce buffering that made it quite difficult to diagnose the issue.
Akamai Edge servers buffer event streams by default. This can be disabled through the use of Akamai's advanced metadata and controlled via custom behaviours. At this time, this cannot be controlled directly through Amakai's portal, so you will need to get their engineers to do some of the work for you.
F5 devices appear to default to buffering response data as well. Fortunately, this is quite simple to change and can be done yourself via the device's configuration portal. For the virtual device in question, go to Profile : Services : HTTP and change the configuration of Response Chunking to Preserve (in our case it had defaulted to Selective).
Once I made these changes, I began to receive SSEs in near real-time from our PROD servers (and not just our DEV servers).

Have you tried alternative browsers? I'm trying to debug a similar problem in which SSE works on an iPhone client but not on MacOS/Safari or Firefox.
There may be a work-around for your issue - if the server sends "Connection: close" instead of keep-alive, or even closes the connection itself, the client should re-connect in a few seconds and the server will send the current progress bar event.
I'm guessing that closing the connection will flush whatever buffer is causing the problem.

This is not a solution to this question exactly, but related to SSE, Spring and use of compression.
In my case I had ziplet CompressionFilter configured in my Spring application and it was closing the Http Response and causing SSE to fail. This seems to be related to an open issue in the ziplet project. I disabled the filter and enabled Tomcat compression in application.properties (server.compression.enabled=true) and it solved the SSE issue.
Note that I did not change the default compressionMinSize setting, which may have something to do with SSE traffic not getting compressed and passing through.

The webpack dev server also buffers server sent events when using the proxy setting.

Unable to download file from client side using Node JS and Express JS

I am new to nodejs/expressjs, and I am building a web app and require file download functionality
I have a simple code to allow the client side to download files from the server. The GET call is as follows:
app.get('/FileDownload', function(req, res){
var sourcePath = /mydrive/pic.png;
res.download(sourcePath, 'pic.png');
});
Upon clicking download from the frontend, a status of 200 was returned, and using the developer tools I was able to see that the response returned the picture in binary format. However, no files were downloaded to my client.
Network Headers:
Request URL: HOST:PORT/FileDownload
Request Method: GET
Status Code: 200 OK
Response:
�PNG
IHDR�I�9usRGB���gAMA���a pHYs���o�d�IDATx^��y�$�u�Y EHKW��յ���$M��#R�(i)�J�̖Z���8=
However, when typing in the get request into the browser directly, the file was able to download.
Any advice on how I can make the download prompt appear?

What do I put in my HTML to ensure users get latest version of my page, not old version?

I have a mostly static HTML website served from CDN (plus a bit of AJAX to the server), and do want user's browsers to cache everything, until I update any files and then I want the user's browsers to get the new version.
How do I do achieve this please, for all types of static files on my site (HTML, JS, CSS, images etc.)? (settings in HTML or elsewhere). Obviously I can tell the CDN to expire it's cache, so it's the client side I'm thinking of.
Thanks.

One way to achieve this is to make use of the HTTP Last-Modified or ETag headers. In the HTTP headers of the served file, the server will send either the date when the page was last modified (in the Last-Modified header), or a random ID representing the current state of the page (ETag), or both:
HTTP/1.1 200 OK
Content-Type: text/html
Last-Modified: Fri, 18 Dec 2015 08:24:52 GMT
ETag: "208f11-52727df9c7751"
Cache-Control: must-revalidate
If the header Cache-Control is set to must-revalidate, it causes the browser to cache the page along with the Last-Modified and ETag headers it received with it. On the next request, it will send them as If-Modified-Since and If-None-Match:
GET / HTTP/1.1
Host: example.com
If-None-Match: "208f11-52727df9c7751"
If-Modified-Since: Fri, 18 Dec 2015 08:24:52 GMT
If the current ETag of the page matches the one that comes from the browser, or if the page hasn’t been modified since the date that was sent by the browser, instead of sending the page, the server will send a Not Modified header with an empty body:
HTTP/1.1 304 Not Modified
Note that only one of the two mechanisms (ETag or Last-Modified) is required, they both work on their own.
The disadvantage of this is that a request has to be sent anyways, so the performance benefit will mostly be for pages that contain a lot of data, but particularly on internet connections with high latency, the page will still take a long time to load. (It will for sure reduce your traffic though.)
Apache automatically generates an ETag (using the file’s inode number, modification time, and size) and a Last-Modified header (based on the modification time of the file) for static files. I don’t know about other web-servers, but I assume it will be similar. For dynamic pages, you can set the headers yourself (for example by sending the MD5 sum of the content as ETag).
By default, Apache doesn’t send a Cache-Control header (and the default is Cache-Control: private). This example .htaccess file makes Apache send the header for all .html files:
<FilesMatch "\.html$">
Header set Cache-Control "must-revalidate"
</FilesMatch>
The other mechanism is to make the browser cache the page by sending Cache-Control: public, but to dynamically vary the URL, for example by appending the modification time of the file as a query string (?12345). This is only really possible if your page/file is only linked from within your web application, in which case you can generate the links to it dynamically. For example, in PHP you could do something like this:
<script src="script.js?<?php echo filemtime("script.js"); ?>"></script>

To achieve what you want on the client side, you have to change the url of your static files when you load them in HTML, i.e. change the file name, add a random query string like unicorn.css?p=1234, etc. An easy way to automate this is to use a task runner such as Gulp and have a look at this package gulp-rev.
In short, if you integrate gulp-rev in your Gulp task, it will automatically append a content hash to all the static files piped into the task stream and generate a JSON manifest file which maps the old files to newly renamed files. So a file like unicorn.css will become unicorn-d41d8cd98f.css. You can then write another Gulp task to crawl through your HTML/JS/CSS files and replace all the urls or use this package gulp-rev-replace.
There should be plenty of online tutorial that shows you how to accomplish this. If you use Yeoman, you can check out this static webapp generator I wrote here which contains a Gulp routine for this.

This is what the HTML5 Application Cache does for you. Put all of your static content into the Cache Manifest and it will be cached in the browser until the manifest file is changed. As an added bonus, the static content will be available even if the browser is offline.
The only change to your HTML is in the <head> tag:
<!DOCTYPE HTML>
<html manifest="cache.appcache">
...
</html>

Backbone in sails.js

I am creating a site using sails and passport for authentication purposes. I've got problems when it come to the use of Jquery and backbone in my code though. It seems that both are down when i tried to use them with sails. What I am trying to do, after user authentication I route the user to home page where all the scripts exists. I put all .js files in the layout.ejs, for exapmle:
<link rel="stylesheet" href="styles/bootstrap-theme.min.css">
<script type="text/javascript" src="js/jquery.js"></script>
Most of css and js files works, but I run into problems with backbone and jquery (and jquerymobile). I am using jQuery 1.10.2 and backbone 1.1.0. Any idea what might be wrong?
In backbone code, I a trying to make Ajax requests via a php file.
var ProfileList = Backbone.Collection.extend({
model: ProfileModel,
url: 'data.php'
});
What exactly i ve got to do? To add url in routes? Or where should I place data.php file?
EDIT:I ve changed gruntfile.js putting jquery at the top and works fine. Now the problem remains the backbone. I am guessing that my troubles arises since I request access to different domains using passport and backbone. When I am requesting data with data.php I am calling the following jquery code:
xhr.send( ( s.hasContent && s.data ) || null ): jquery.js (line 8706)
This is an Ajax XMLHttpRequest send request http://www.w3schools.com/ajax/ajax_xmlhttprequest_send.asp. I generally try to request data from two domains localhost and localhost:1337, so a cross domain issue arises.
I ve found that cors(cross origin recource sharing) can handle this issue. Any idea how to allow cors in sails.js??
EDIT: In the route.js file I turn in homepage cors to true:
'/secondscreen': {
view: 'secondsocialscreen/index',
cors: true
}
I still receiving the same error. Also i changed the variable AllRoutes to true in config/cors.js file. My backbone file works(checked using console), however I cant fetch data. Config/cors.js file is the following:
module.exports.cors = {
// Allow CORS on all routes by default? If not, you must enable CORS on a
// per-route basis by either adding a "cors" configuration object
// to the route config, or setting "cors:true" in the route config to
// use the default settings below.
allRoutes: true,
// Which domains which are allowed CORS access?
// This can be a comma-delimited list of hosts (beginning with http:// or https://)
// or "*" to allow all domains CORS access.
origin: '*',
// Allow cookies to be shared for CORS requests?
credentials: true,
// Which methods should be allowed for CORS requests? This is only used
// in response to preflight requests (see article linked above for more info)
methods: 'GET, POST, PUT, DELETE, OPTIONS, HEAD',
// Which headers should be allowed for CORS requests? This is only used
// in response to preflight requests.
headers: 'content-type'
};
Am I missing something?? Ok I think I ve found something. Actually I am trying to direct to a home.ejs whrere I define a fetching.js (fetching data from data.php and definition of model-controller-views) where backbone code exists. Is it possible this file to work as it is, or I ve to define my MVC from sails?? Also I found that I am trying to get data from localhost:1337 (sails server). However I want to fetch data from apache localhost. How is it possible to request data to apache server while running sails?? In firebug I received the above:
GET http://localhost/sitec/fetchdata.php?widget=highlights 200 OK 47ms jquery.js (line 8706)
GET http://localhost/sitec/fetchdata.php?widget=sentiment 200 OK 47ms jquery.js (line 8706)
GET http://localhost/sitec/fetchdata.php?widget=tagsCloud 200 OK 47ms jquery.js (line 8706)
In response instead of having the object of json data (data.php returns json file), actually I can see the code of data.php file. Weird

We use CORS in sails by modifying the config/routes.js
module.exports.routes = {
'/*': {
cors: true
}
}
Documented here: http://sailsjs.org/#!documentation/config.routes

Sails.js by default use grunt to add CSS and JS files. Adding something by hand to layout isn't a good idea. In file Gruntfile.js that is in main root you will see two important arrays - cssFilesToInject and jsFilesToInject. To load files in proper order, just add them here in order that you need.
var jsFilesToInject = [
// Below, as a demonstration, you'll see the built-in dependencies
// linked in the proper order order
// Bring in the socket.io client
'linker/js/socket.io.js',
// then beef it up with some convenience logic for talking to Sails.js
'linker/js/sails.io.js',
// A simpler boilerplate library for getting you up and running w/ an
// automatic listener for incoming messages from Socket.io.
'linker/js/app.js',
// *-> put other dependencies here <-*
'path_to_jquery',
'path_to_underscore',
'path_to_backbone',
// All of the rest of your app scripts imported here
'linker/**/*.js'
];