First of all - I am aware of this answer to a kind of similar problem.
Problem
I have a third party protocol, that uses TCP\IP. This protocol defines that the server replies to every message received. On the client side (which I try to implement) I have to wait for the answer from the server.
The problem occurs, when I try to send messages. I need to wait for the answer from the first message before I send the second one (like ping-pong).
I tried to do multiple writes on my NodeJS tcp-client like this, which understandably fails due to async:
client.connect(connectOptions, function () {
client.write(message1);
client.write(message2);
});
Like I said before, I have a third party component, which responses to both messages with a numeric value. So when
client.on('data',function (data) {});
fires an event, I can't distinguish which message, was responsible for the answer. Unlike the linked answer I don't have the ability, to tag the answer on the server side.
I am new to node.js, so I try to figure out the best way to solve this kind of problem, as it´s of the nature: do synchronous things in the async environment.
One way would be to use a common list of handlers to keep track of requests and responses:
var handlers = [];
client.connect(connectOptions, function () {
client.write(message1);
handlers.push(function msg1_handler(data) {});
client.writewrite(message2);
handlers.push(function msg2_handler(data) {});
});
client.on('data',function(data) {
var handler = handlers.shift();
handler(data);
});
All of this should obviously be wrapped in a separate class containing both handlers an client objects. It's just an example of how to do it. The drawback is that if the server fails to respond to some request then you have a complete mess, hard to make it right.
Another idea is to buffer requests:
function BufferedClient(cli) {
this.cli = cli;
this.buffer = [];
this.waiting_for_response = false;
var that = this;
cli.on('data', function(data) {
that.waiting_for_response = false;
var pair = that.buffer.shift();
var handler = pair[0];
process.nextTick(function() {
// we use .nextTick to avoid potential
// exception in handler which would break
// BufferedClient
handler(data);
});
that.flush();
});
};
BufferedClient.prototype = {
request: function(msg, handler) {
this.buffer.push([handler, msg]);
this.flush();
},
flush: function() {
var pair = this.buffer[0];
if (pair && !this.waiting_for_response) {
this.cli.write(pair[1]);
this.waiting_for_response = true;
}
}
};
This time you send requests sequentially (so like synchronous) due to how .request() and .on('data') handler work together with .flush() function. Usage:
client.connect(connectOptions, function () {
var buff_cli = new BufferedClient(client);
buff_cli.request(message1, function(data) { });
buff_cli.request(message2, function(data) { });
});
Now even if the server fails to respond you don't have a mess. However if you issue buff_cli.request parallely and one of them fails then you will have a memory leak (since this.buffer is getting bigger while nothing is draining it because the BufferedClient is waiting for a response). This can be fixed by adding some timeouts on the socket.
Note that both solutions assume that the server never pushes anything to the client without a request.
If I were you I would go with second solution. Note that I haven't tested the code so it might be buggy but the general idea should be ok.
Side note: When you implement a server (and I know that you don't in this case) you should always have a protocol that matches each request with a response in a unique way. One way would be to send a unique ID with each request so that the server would be respond with the same ID. In such scenario matching request with response is very easy and you avoid all that mess.
Related
How can I get the full URL of the page which is being serviced by a service worker's 'fetch' event?
The "self.location" property seems to only refer to the root URL of the site. For example, if page https://example.com/folder/pagename.html is performing a fetch which the service worker is intercepting, the service worker's 'self.location' property returns "https://example.com".
event.currentTarget.location and event.explicitOriginalTarget.location, event.originalTarget, and event.target all return the URL of the service worker .js file.
How can I get the full URL of the page that triggered the fetch event?
You've got two general approaches, depending on how involved you want to get:
Use the 'Referer' header info
If the request is for a subresource and includes a Referer header, then there's a decent chance that the value of that header is the URL of the page that made the request. (There are some caveats; read this background info to delve into that.)
From within a fetch handler, you can read the value of that header with the following:
self.addEventListener('fetch', event => {
const clientUrl = event.request.referrer;
if (clientUrl) {
// Do something...
}
});
Use the clientId value
Another approach is to use the clientId value that (might) be exposed on the FetchEvent, and then use clients.get(id) or loop through the output of clients.matchAll() to find the matching WindowClient. You could then read the url property of that WindowClient.
One caveat with this approach is that the methods which look up the WindowClient are all asynchronous, and return promises, so if you're somehow using the URL of the client window to determine whether or not you want to call event.respondWith(), you're out of luck (that decision needs to be made synchronously, when the FetchEvent handler is first invoked).
There's a combination of different things that need to be supported in order for this approach to work, and I'm not sure which browsers currently support everything I mentioned. I know Chrome 67 does, for instance (because I just tested it there), but you should check in other browsers if this functionality is important to you.
self.addEventListener('fetch', async event => {
const clientId = event.clientId;
if (clientId) {
if ('get' in clients) {
const client = await clients.get(clientId);
const clientUrl = client.url;
// Do something...
} else {
const allClients = await clients.matchAll({type: 'window'});
const filtered = allClients.filter(client => client.id === clientId);
if (filtered.length > 0) {
const clientUrl = filtered[0].url;
// Do something...
}
}
}
});
I have a node application handling some ZeroMQ events coming from another application utilizing the Node-ZMQ bindings found here: https://github.com/JustinTulloss/zeromq.node
The issue I am running into is one of the operations from an event takes a long time to process and this appears to be blocking any other event from being processed during this time. Although the application is not currently clustered, doing so would only afford a few more threads and doesn't really solve the issue. I am wondering if there is a way of allowing for these async calls to not block other incoming requests while they process, and how I might go about implementing them.
Here is a highly condensed/contrived code example of what I am doing currently:
var zmq = require('zmq');
var zmqResponder = zmq.socket('rep');
var Client = require('node-rest-client').Client;
var client = new Client();
zmqResponder.on('message', function (msg, data) {
var parsed = JSON.parse(msg);
logging.info('ZMQ Request received: ' + parsed.event);
switch (parsed.event) {
case 'create':
//Typically short running process, not an issue
case 'update':
//Long running process this is the issue
serverRequest().then(function(response){
zmqResponder.send(JSON.stringify(response));
});
}
});
function serverRequest(){
var deferred = Q.defer();
client.get(function (data, response) {
if (response.statusCode !== 200) {
deferred.reject(data.data);
} else {
deferred.resolve(data.data);
}
});
return deferred.promise;
}
EDIT** Here's a gist of the code: https://gist.github.com/battlecow/cd0c2233e9f197ec0049
I think, through the comment thread, I've identified your issue. REQ/REP has a strict synchronous message order guarantee... You must receive-send-receive-send-etc. REQ must start with send and REP must start with receive. So, you're only processing one message at a time because the socket types you've chosen enforce that.
If you were using a different, non-event-driven language, you'd likely get an error telling you what you'd done wrong when you tried to send or receive twice in a row, but node lets you do it and just queues the subsequent messages until it's their turn in the message order.
You want to change REQ/REP to DEALER/ROUTER and it'll work the way you expect. You'll have to change your logic slightly for the ROUTER socket to get it to send appropriately, but everything else should work the same.
Rough example code, using the relevant portions of the posted gist:
var zmqResponder = zmq.socket('router');
zmqResponder.on('message', function (msg, data) {
var peer_id = msg[0];
var parsed = JSON.parse(msg[1]);
switch (parsed.event) {
case 'create':
// build parsedResponse, then...
zmqResponder.send([peer_id, JSON.stringify(parsedResponse)]);
break;
}
});
zmqResponder.bind('tcp://*:5668', function (err) {
if (err) {
logging.error(err);
} else {
logging.info("ZMQ awaiting orders on port 5668");
}
});
... you need to grab the peer_id (or whatever you want to call it, in ZMQ nomenclature it's the socket ID of the socket you're sending from, think of it as an "address" of sorts) from the first frame of the message you receive, and then use send it as the first frame of the message you send back.
By the way, I just noticed in your gist you are both connect()-ing and bind()-ing on the same socket (zmq.js lines 52 & 143, respectively). Don't do that. Inferring from other clues, you just want to bind() on this side of the process.
There are many SO questions how to get http headers with javascript, but for some reason they don't show up HTTP_CF_IPCOUNTRY header.
If I try to do with php echo $_SERVER["HTTP_CF_IPCOUNTRY"];, it works, so CF is working just fine.
Is it possible to get this header with javascript?
#Quentin's answer stands correct and holds true for any javascript client trying to access server header's.
However, since this question is specific to Cloudlfare and specific to getting the 2 letter country ISO normally in the HTTP_CF_IPCOUNTRY header, I believe I have a work-around that best befits the question asked.
Below is a code excerpt that I use on my frontend Ember App, sitting behind Cloudflare... and varnish... and fastboot...
function parseTrace(url){
let trace = [];
$.ajax(url,
{
success: function(response){
let lines = response.split('\n');
let keyValue;
lines.forEach(function(line){
keyValue = line.split('=');
trace[keyValue[0]] = decodeURIComponent(keyValue[1] || '');
if(keyValue[0] === 'loc' && trace['loc'] !== 'XX'){
alert(trace['loc']);
}
if(keyValue[0] === 'ip'){
alert(trace['ip']);
}
});
return trace;
},
error: function(){
return trace;
}
}
);
};
let cfTrace = parseTrace('/cdn-cgi/trace');
The performance is really really great, don't be afraid to call this function even before you call other APIs or functions. I have found it to be as quick or sometimes even quicker than retrieving static resources from Cloudflare's cache. You can run a profile on Pingdom to confirm this.
Assuming you are talking about client side JavaScript: no, it isn't possible.
The browser makes an HTTP request to the server.
The server notices what IP address the request came from
The server looks up that IP address in a database and finds the matching country
The server passes that country to PHP
The data never even goes near the browser.
For JavaScript to access it, you would need to read it with server side code and then put it in a response back to the browser.
fetch('https://cloudflare-quic.com/b/headers').then(res=>res.json()).then(data=>{console.log(data.headers['Cf-Ipcountry'])})
Reference:
https://cloudflare-quic.com/b
https://cloudflare-quic.com/b/headers
Useful Links:
https://www.cloudflare.com/cdn-cgi/trace
https://github.com/fawazahmed0/cloudflare-trace-api
Yes you have to hit the server - but it doesn't have to be YOUR server.
I have a shopping cart where pretty much everything is cached by Cloudflare - so I felt it would be stupid to go to MY server to get just the countrycode.
Instead I am using a webworker on Cloudflare (additional charges):
addEventListener('fetch', event => {
event.respondWith(handleRequest(event.request))
})
async function handleRequest(request) {
var countryCode = request.headers.get('CF-IPCountry');
return new Response(
JSON.stringify({ countryCode }),
{ headers: {
"Content-Type": "application/json"
}});
}
You can map this script to a route such as /api/countrycode and then when your client makes an HTTP request it will return essentially instantly (for me it's about 10ms).
/api/countrycode
{
"countryCode": "US"
}
Couple additional things:
You can't use webworkers on all service levels
It would be best to deploy an actual webservice on the same URL as a backup (if webworkers aren't enabled or supported or for during development)
There are charges but they should be neglibible
It seems like there's a new feature where you can map a single path to a single script. That's what I am doing here. I think this used to be an enterprise only feature but it's now available to me so that's great.
Don't forget that it may be T1 for TOR network
Since I wrote this they've exposed more properties on Request.cf - even on lower priced plans:
https://developers.cloudflare.com/workers/runtime-apis/request#incomingrequestcfproperties
You can now get city, region and even longitude and latitude, without having to use a geo lookup database.
I've taken Don Omondi's answer, and converted it to a promise function for ease of use.
function get_country_code() {
return new Promise((resolve, reject) => {
var trace = [];
jQuery.ajax('/cdn-cgi/trace', {
success: function(response) {
var lines = response.split('\n');
var keyValue;
for (var index = 0; index < lines.length; index++) {
const line = lines[index];
keyValue = line.split('=');
trace[keyValue[0]] = decodeURIComponent(keyValue[1] || '');
if (keyValue[0] === 'loc' && trace['loc'] !== 'XX') {
return resolve(trace['loc']);
}
}
},
error: function() {
return reject(trace);
}
});
});
}
usage example
get_country_code().then((country_code) => {
// do something with the variable country_code
}).catch((err) => {
// caught the error, now do something with it
});
I'm testing a web page where the user can send a message to another via a textinput. A POST request is then send on the server and the message is dumped on the disk in the var/mail/new folder.
After automatising the sending of the message in the page with Protractor I'm calling browser.waitForAngular() and browser.driver.sleep(4000) to leave time for the backend to write the mail on the disk.
After these calls the check of the email's presence fails. When looking in the Unix shell, I can confirm that the email was sent and also the next test marked with in Jasmine with it confirms the presence of the email.
Why is browser.driver.sleep(4000) not effective to wait for the backend to proceed? How can I correct the following code?
it("is possible to send a message", function() {
shared.loginContributor();
var mailsBeforeMessaging =
fs.readdirSync(browser.params.mail.queue_path + "/new");
console.log('mailsBeforeMessaging');
console.log(mailsBeforeMessaging.length);
console.log(fs.lstatSync(browser.params.mail.queue_path + "/new"));
var usersListing = new UserPages.UsersListing().get();
var annotatorPage = usersListing.getUserPage("annotator");
annotatorPage.sendMessage("title5", "content64");
exec("/tmp/check.sh");
// we expect the message widget to disappear
var button = element(by.css(".user-profile-info-button"));
console.log('waiting');
browser.wait(EC.elementToBeClickable(button), 5000);
console.log('waiting is finished');
expect(EC.elementToBeClickable(button)).toBeTruthy();
// wait for mail to be dumped on the disk?
browser.waitForAngular();
browser.driver.sleep(4000);
exec("/tmp/check.sh");
var mailsAfterMessaging =
fs.readdirSync(browser.params.mail.queue_path + "/new");
console.log('mailsAfterMessaging');
// ERROR: here the number of emails is NOT incremented
console.log(mailsAfterMessaging.length);
console.log(fs.lstatSync(browser.params.mail.queue_path + "/new"));
});
it("xyz", function() {
console.log(fs.lstatSync(browser.params.mail.queue_path + "/new"));
// here the number of emails is incremented
var mailsAfterMessaging =
fs.readdirSync(browser.params.mail.queue_path + "/new");
console.log('mailsAfterMessaging');
console.log(mailsAfterMessaging.length);
});
Most of the Protractor functions do not do anything. They queue something up to be done later, and return promise to do it. After an it block schedules a bunch of things to do, they actually start happening (via the promises they registered in the ControlFlow).
Your checks, however, are all executing immediately. So, they are happening before any of the protractor calls accomplish anything.
Use then to make the waiting and dependencies explicit in your test. Like this:
annotatorPage.sendMessage("title5", "content64").then(function() {
exec("/tmp/check.sh");
});
or:
browser.wait(EC.elementToBeClickable(button), 5000).then(function() {
console.log('wait-for-clickable has completed'); // B
});
console.log('wait-for-clickable has been scheduled'); // A
See the Protractor Control Flow documentation and the Webdriver JS API doc.
Its not you. This is a crazy API to learn because it does not act at all like anyone familiar with normal synchronous programming would expect.
I have a strange issue with a basic pubsub application with node and zmq:
a client is publishing strings to a broker, the problem is that the broker only receives the first line. At network level I've noticed that only the first message is sent then the next calls to .send() function have no effect (no packets are sent) so I suppose the problem is in the client/publisher.
I used the example code provided in the official guide which works perfectly, the only difference in my code is that I use prototype to have a reusable structure.
(I didn't paste subscriber's code because is not relevant and took some other not relevant stuff out)
relevant part of the client/publisher:
Publisher = function(zmq, pport) {
this.logread = spawn('tail', ['-n0', '-f', '/var/log/auth.log']);
this.publisher = zmq.socket('req');
this.pport = pport;
};
Publisher.prototype.start = function() {
var self = this;
this.publisher.connect('tcp://127.0.0.1:' + this.pport);
this.logread.stdout.on('data', function(data){
self.publisher.send(data.toString());
console.log(data.toString());
});
};
relevant part of the broker:
Broker = function(zmq, bpport, bsport) {
this.server = zmq.socket('rep');
this.bpport = bpport;
this.bsport = bsport;
};
Broker.prototype.start = function() {
this.server.on('message', function(request) {
console.log(request.toString());
});
this.server.bind('tcp://127.0.0.1:' + this.bsport, function(err) {
if (err)
console.log(err);
});
};
You are talking about publish subscribe pattern, but in your code, you create a req socket, and in the broker a rep socket, which is for the request-reply pattern. The request-reply pattern is strictly need to send first, than receive, see the api docs docs, or read more from the guide
I suppose you should use pub socket on the client side, and a sub socket on the other side, but don't know what do you want to achieve, maybe a different pattern would fit your needs better.
so I'll answer my question:
the server must send a reply to the client, until then the client will not send more messages
server.send('OK');
I also suppose there is a different way to achieve this