We Keep Coding

Axios GET request returning ECONNRESET for some URLs

I am working on an application that uses an express server to reach out to an API to fetch data. In our organisation outbound traffic requires a proxy which I have supplier to axios like below (not the real one):
let response = await axios.get(endpointUrl, {
proxy: {
host: "123.45.678.90",
port: 0000,
},
})
Passing various URLs into the axios get function returns varied results, with the following URLs returning a result:
https://www.boredapi.com/api/activity
https://api.ipify.org?format=json
https://jsonplaceholder.typicode.com/todos/1
Whereas the following URLs are returning an ECONNRESET error almost instantly:
https://api.publicapis.org/entries
https://randomuser.me/api/
https://reqres.in/api/users
I can't see any pattern between the URLs that are/are not working so wondered if a fresh set of eyes could spot the trait in them? It's important to note that all these URLs return successfully in the browser, just through this axios call being the problem.
To add to the mystery, the URLs that do work work on my machine, do work on a machine outside our organisation - so potentially a clue there?
Any help/guidance of course would be appreciated, thank you.

This error simply means that the other party closed the connection in a way that was probably not normal (or perhaps in a hurry).
For example, a socket connection may be closed by the other party abruptly for various reasons or you may have lost your wifi signal while running your application. You will then see this error/exception on your end.
What could also be the case: at random times, the other side is overloaded and simply kills the connection as a result. If that's the case, depends on what you're connecting to exactly…
Solution - This is happening because you are not listening to/handling the 'error' event. To fix this, you need to implement a listener that can handle such errors.

If the URL that work on your machine also work outside your organization and the other don't, it is most likely a problem with your proxy.
Some proxies might have configurations that makes them remove headers or change the request in a way that the target does not receive it as intended.
I also encountered a problem with axios and proxies once. I had to switch libs to make it work. To be sure, I would recommand using a lib like "request" (deprecated) juste to make sure it is not a problem with axios. There are multiple open issues on the axios repository for proxy issues.

ECONNRESET is likely occurring either because the proxy runs into some sort of error and drops the connection or the target host finds something wrong with the incoming connection and decides to immediately drop it.
That target host may either be finding a problem because of the proxy or it may be expecting something in the request that it finds is missing.
Since you have evidence that all the requests work fine when running from a different location (not through your proxy) and I can confirm that your code works fine from my location (also not running through your proxy), it definitely seems like the evidence points at your proxy as causing some problem in some requests.
One way to debug proxy issues like this is to run a request through the proxy that ends up going to some server you can debug on and see exactly what the proxy has done to the incoming request, compared to a request to that same host that doesn't go through the proxy. That will hopefully highlight some difference that you can then test to see if that's causing the problem and then eventually work on the configuration of the proxy to correct.

Minimal example of "websocket wrapping" for web server using websockify

some webdev basic questions for a noob.
background: I have a javascript client that uses the websocket protocol, and a good old java server, which I'm perfectly able to interact with via telnet. I want them to be able to communicate to each other locally (passing strings would suffice).
For that, I definitely don't want to make deep changes in the client. I tried to adapt the server to websocket following this guide before that, and even passed the handshake but it got pretty messy.
So at this point I believe that the best way to deal with this would be to serve through a TCP->WebSocket proxy like websockify. If I got the explanation right, it would "wrap" the server's process, opening the "upgraded" connection in a new port. And the websocket client should be able to directly speak to the server through that port. But I've been trying to implement this with different approaches without success, therefore my questions:
Is there a better way to deal with this problem? (that would explain google's sparseness, I can imagine this happens very often!)
If "wrapping" is the best way and I got it right, how to implement it?
Implementation - edited: For the sake of testing, I've tried to build a simple echo server:
I'm running the very simple command line echo server suggested in this post: ncat -l 2000 -k -c xargs -n1 echo. Talking to it via telnet localhost 2000 returns the desired echo. So far so good.
I installed websockify using apt-get,and run it as follows: websockify 2023: 2000, this should open the same server via a websocket connection on port 2023.
At this point I'm still able to comunicate via telnet localhost 2000, but I'm not having that much luck with port 2023. This is what I tried so far:
Downloaded the telsocket binary, and called ./telsocket -url ws://127.0.0.1:2023. Response: errrr dial tcp 127.0.0.1:2023: connection refused
Cloned wssh (a command line shell for websocket), which seems very promising. I was able to install it, but running wssh -l localhost:2023 returns NameError: name 'PROTOCOL_SSLv3' is not defined. Apparently some problems with gevent (didn't look much further into that).
Tested the several examples in websockify's repo, like wstelnet, or the ones in tests. All of them got me a code 400, message Invalid protocol selected.
Follow-up - edited: Diving deeper into the websock.js and wstelnet.js files, I was able to get more specific results by issuing the following into the JS console:
ws = new Websock()
ws.open("ws://127.0.0.1:2023", "binary") // connects, proxy says: connecting to: :2000, Plain non-SSL (ws://) WebSocket connection, Version hybi-13, base64: 'False'
ws.send_string("hello server") // send and wait for echo
ws.get_rQ() // empty??
So, as you can see, I'm able to establish the connection, but still don't get an echo. The mistake might be as well in the server's side since every tool I tried is failing. Help! D:
Remarks: Since it is intended to work locally, I wouldn't care having either ws or wss. I also don't have a preferred way to do this as long as it is viable and works. In case it is relevant, I'm on Ubuntu15.10 + Firefox47.0

finally, I got the TCP echo server communicating with websockify's websocket telnet emulator through websockify's proxy. This is how:
run the echo server ncat -l 2000 -k -c 'xargs -n1 echo echoServer received'
in a separate process, run the proxy: websockify 2023 :2000
clone the repo: git clone https://github.com/novnc/websockify.git, and make the following changes to wstelnet.js:
3.1 changing the line ws.open(uri) to ws.open(uri, "binary") allowed to overcome the code 400 problem.
3.2 in the definition of do_recv (this is a permalink), add the following line after the initial var statement: arr = Array.from(arr); this is a conversion to Array since Uint8Array didn't apparently support the shift method.
open wstelnet.html with firefox, select Host: localhost, Port: 2023, no encryption, press Connect and type into the black field.
The screen should reply with echoServer received: <YOUR_MESSAGE>. hurray!

Server stops responding to requests after too many in a session?

I have web app that uses frequent $.ajax() calls to transmit data to and from the server. This runs locally between a virtual machine host and client.
The problem I'm having is that it seems to cut out after making certain number of consecutive calls in a session (no actual number has been determined). This is can be seconds or minutes.
I tried assigning my $.ajax() calls to objects so they could be deleted, eg.:
myApp.ajaxRegistry.myAjax = $.ajax({
url: '/path/to/server',
error: function() {
delete myApp.ajaxRegistry.myAjax;
}
success: function() {
delete myApp.ajaxRegistry.myAjax;
}
});
I thought that may have improved it, but it could just be coincidence. It still fails frequently.
I've monitored the server access log when these failures occur, I can see that it's not even making the request. There are no Javascript errors in the browser console.
EDIT
The browser's network logger indicates that it is making the request, but server is not responding (according to apache's access log). After a few minutes, it starts responding again, so I'm thinking there is configuration on the server.
It might also be worth noting that the virtual machine server frequently loses time (some sort of annoying VirtualBox "feature"), so I wonder if that might be related.
UPDATE
I think my hunch about the server time may have been right. I finally managed to get ntp to work properly on the VM and I haven't encountered this problem for a few weeks now.

Just to have the answer in a separate post: the server time needs to be accurate (at least in this context) or the AJAX requests get confused.

Setting timeout for vertx-eventbus.js

I'm currently working on a Vertx based application that is making use of vertx-eventbus.js as described here: http://vertx.io/docs/vertx-web/java/#_sockjs_event_bus_bridge
So the client side is using vertx-eventbus.js to send a message to server side which has SockJSHandler doing the 'bridge' to receive the message. The whole mechanism has been working fine for few months. Only very recently we now potentially need to have higher timeout limit for certain messages. If run right now, those messages always cause the following warning on the server side (and also caused the eventbus to attempt to return too early on client side with "undefined"):
Message reply handler timed out as no reply was received - it will be removed
My question is then how do you increase the timeout for the message sending in this scenario? It seems to be using the default 30 seconds timeout and I have looked everywhere (online searches Google, here at Stackoverflow, etc) with no luck. I know there is a way to set timeout when you are sending using Vertx on the backend but not when sending from vertx-eventbus.js.
Any input is very much appreciated. Let me know if there is anything else you need to better understand what I'm trying to achieve.
Thank you,
Tom

Websocket: Browser dosn't seem to receive data from python server

I'm working on creating a websocket server via python (I'm kinda new to python) and I've made a significant progress, but I am unable to send data to the web browser. I can establish a connection and receive data from the browser, but I cannot send back data. The browser just ignores it. I would assume that if the browser received a package that didn't follow the specifications, it would terminate the connection, but the connection stays active.
Here is the method I am using to encode the data into the frame:
def encode_message(data):
frame = "\x81"
size = len(data)
if size * 8 <= 125:
frame += chr(size)
else:
raise Exception("Uh, oh. Strings larger than 125 bits are not supported")
return frame + data
I am sending the data using sock.sendall(framed_data). What could be the problem? The data for a message like "yo" ends up being 10000001 00000010 01111001 01101111 (spaces added for improved readability). Why doesn't the browser accept a message like this? Doesn't it follow the guidelines outlined in the specification? I am trying to support the most recent websocket version which I believe to be version 13. I am using python version 2.7.3.
I have tried to look at python websocket libraries' source code, but all of them seem to implement a deprecated version of the websocket protocol that has been shown to have vulnerabilities.
Here is the code that calls the function above:
def send(data):
frame = encode_message(data)
print "Sending all..."
sock.sendall(frame) #Socket that handles all communications with client
print "Frame sent :)"
return
I also downloaded wireshark to sniff the packages sent between the server and the socket. The packages sent by my server are identical to those sent from a server that is accepted by the browser. I couldn't see any difference at all. (I looked directly at the hex source)

The second byte of your transmitted message (and the length check in your code) looks wrong. The length of a message is in bytes, not bits.
From RFC6455 §5.2 (my emphasis)
Payload length: 7 bits, 7+16 bits, or 7+64 bits
The length of the "Payload data", in bytes: if 0-125, that is the
payload length.
The reason that nothing is received in the browser is that your message claims to have a 16 byte body. The browser will read the 2 additional bytes you send then block waiting for another 14 bytes that it expects but you don't send.
If you change the second byte to the number of bytes in the message - 0x2 or 00000010 binary - then things should work.

I finally figured out the problem! It took hours of unfun debugging and messing with my code. After closely examining the packages sent back and forth between the server and client I finally realized that there was a problem with my server's connection upgrade response. Whenever it computed a hash, it also added a \n to the end of it. That resulted in a \n\r\n at the end of one of the lines. The client interpreted that as the end of that transmission and everything that followed was parsed using WebSocket protocol. I had another line after that in the header, so it totally messed up my communications with the client. I could still read from the client, but if I tried to write to the client, the data would get messed up.