I have this code to pull down a news feed from a third party website using an API. Its setup to run every 5 seconds, and pull any news transactions that may occur. The problem seems to be when there is no new transactions that occur.
By adding the process.on('uncaught exception', function(error){ console.log("hmph") }) the cron job is able to continue 5 seconds later, so I was tempted to leave it as is; however, I added console.log("hmph") to the and now I'm confused.
The first time, the console will write hmph. 5 seconds later it will write
hmph
hmph
and so on. I know I must be missing something, but I'm not quite sure what it is. I've tried in the else statement to do request.end() but the error still fires.
Without the process.on('uncaught...') the error thrown is:
events.js:71
throw arguments[1]; // Unhandled 'error' event
^
Error: Parse Error
at Socket.socketOnData (http.js:1367:20)
at TCP.onread (net.js:403:27)
with the proccess.on('uncaught...') the console.log(error) is:
{ [Error: Parse Error] bytesParsed: 161, code: 'HPE_INVALID_CONSTANT' }
How do I properly handle this error?
Abbreviated code:
var job = new cronJob('*/5 * * * * *', function(){
var request = http.get({
host: 'www.example.com',
path: '/news_feed?apicode=myapicode',
port: 80,
headers: { 'accept-encoding': 'gzip' }
})
request.on('response', function(response){
if (response.statusCode == 200){
// gunzip response, save response to mongodb
}
else
{
// here is where the error is occuring
process.on('uncaughtException',function(error){
console.log(error);
console.log("hmph");
}
});
}, null, true);
Every time you make a request, you are binding a new uncaughtException handler, so when the first request is sent, you bind the first one, and when it fails it prints an error, and then at the next request, you add another handler, and when that fails both the first and second handlers will run.
Examining the error, and a conversation about this type of error here: https://github.com/joyent/node/issues/3354 It seems like the server you are connecting to is probably doing something weird. The easiest solution for you probably is to use the uncaughtException handler for now. That said, it is less than ideal and you should not do it as a general solution to future problems like this.
var job = new cronJob('*/5 * * * * *', function(){
var request = http.get({
host: 'www.example.com',
path: '/news_feed?apicode=myapicode',
port: 80,
headers: { 'accept-encoding': 'gzip' }
});
request.on('response', function(response){
if (response.statusCode == 200){
// gunzip response, save response to mongodb
}
});
}, null, true);
process.on('uncaughtException',function(error){
console.log(error);
console.log("hmph");
});
Related
I’m using the Got library for HTTP requests within my application. I’m making HTTP requests to an API where I can expect an HTTP code 404 under certain conditions. I’d like to use Got's internal retry functionality for rerunning the request until the 404 error is gone (which will happen; I just don't know if it takes 1 minute or 30 minutes).
From the documentation I know that HTTP code 404 is not a supported statusCode for the built-in retry functionality and therefore I cannot perform any action within the beforeRetry hook of Got; see here.
I’m extending a Got instance to allow some presets for the API I’m calling. For now I have not found a clean way to extend the existing retry statusCodes with 404.
const gitlabOnPrem = got.extend({
prefixUrl: ".." + "..",
mutableDefaults: true,
responseType: 'json',
//retry: { statusCode: got.defaults.options.retry.statusCode.push(404) }, //does not work | not clean
https: { rejectUnauthorized: false },
headers: {
'PRIVATE-TOKEN': "..",
Accept: 'application/json',
},
hooks: {
beforeError: [
(error) => {
const { response } = error;
console.log(response);
/*
Another idea: if I cannot extend retry statusCodes then I´d like to somehow force a retry from here
if (response.statusCode === 404 && response.path.endsWith('/export/download')) {
console.log('FORCE A RETRY AS THE DOWNLOAD MIGHT NOT BE READY YET');
}
*/
if (response && response.body) {
error.name = 'GitLabOnPremError';
error.message = `${response.body.message !== undefined
? response.body.message
: response.body.error
} (${response.statusCode})`;
}
return error;
},
],
},
});
How can I extend the HTTP statusCodes that allow running a retry?
If this is not possible, see my comment in the code. Is it somehow possible to force a retry manually by just using Got?
Sometimes it's best to just write it yourself. Trying to get a library to work in a way that it isn't totally made to do can be more pain than it's worth. It's also usually very brittle in the long run.
Why not just wrap it yourself? Something like this:
async function MyGotFn() {
let retries = 100;
let statusCode = 404;
let response;
while (statusCode === 404 && --retries > 0) {
response = await got('...');
statusCode = response.statusCode;
}
if (response.statusCode === 404) throw new Error('max retries reached');
return response;
}
I am trying to increase the timeout time of my React app. I am using axios, so initially I tried:
axios.post('/gene_info', postData, {timeout: timeoutVal});
It did not work, and there is the respective thread that deals with it:
https://github.com/axios/axios/issues/647
So, I tried the following code:
let CancelToken = axios.CancelToken;
const source = CancelToken.source();
try {
let response = null;
setTimeout(() => {
if (response === null) {
source.cancel();
}
}, 60 * 1500 * 1000);
response = await axios.post('/gene_info', postData, {cancelToken: source.token});
console.log(response);
} catch (error) {
console.log(error);
}
And it is not working either. The request times out and I see the empty response error, even though on the Node.js backend I see that the result is returned correctly. On the backend I am making a very long running request to Neo4j database. I got a suspicion that maybe it timeouts, so I added to neo4j.config file the following lines:
unsupported.dbms.executiontime_limit.enabled=true
unsupported.dbms.executiontime_limit.time=99999999999999s
That I found here:
How to configure a query timeout in Neo4j 3.0.1
and restarted neo4j but it did not help either. Here is what I see in the terminal:
I am not sure what this POST /gene_info - - ms - - means, whether the problem is still on the front end, or the back end, but I have a suspicion that neo4j now times out, but it is still calculating the result which I see using console.log() statements. Any suggestions would be greatly appreciated.
Update
I tried using Reacts fetch, but still not working. Here is the code:
fetchWithTimeout = (url, postData, timeout) => {
let didTimeOut = false;
new Promise(function(resolve, reject) {
const timeout = setTimeout(function() {
didTimeOut = true;
reject(new Error('Request timed out'));
}, timeout);
fetch(url, {
method: 'POST',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json',
},
timeout: timeout,
body: JSON.stringify(postData)
})
.then(function(response) {
// Clear the timeout as cleanup
clearTimeout(timeout);
if(!didTimeOut) {
console.log('fetch good! ', response);
resolve(response);
}
})
.catch(function(err) {
console.log('fetch failed! ', err);
// Rejection already happened with setTimeout
if(didTimeOut) return;
// Reject with error
reject(err);
});
})
.then(function() {
// Request success and no timeout
console.log('good promise, no timeout! ');
})
.catch(function(err) {
// Error: response error, request timeout or runtime error
console.log('promise error! ', err);
});
}
Then I am calling this function like that:
let postData = {"jsonData": geneNameArr,
"datasetName": this.props.datasetName};
this.fetchWithTimeout('/gene_info', postData, timeout).then((response) => {
console.log("fetchWithTimeout is done!");
console.log(response);
});
Update
I tried using axios.create() function with no success:
const axiosInstance = axios.create({
baseURL: '/gene_info',
timeout: timeout
});
axiosInstance.post('', postData).then((response) => {
console.log("axios request is done with create() method");
console.log(response);
});
If nothing seems to work on the front end, I would think it is the timeout that comes from the neo4j driver, even though somehow the results are returned. Here is the code I am using for the driver:
router.post('/gene_info', function(req, res) {
...
...
var driver = dbUtils.driver;
const session = driver.session();
session.run(
full_query,
{}
).then(result => {
const exprData = chartService.prepareGeneInfoData(result, '');
res.json({
exprData
});
session.close();
});
})
Or maybe it can also be express.Router(); that I am using for treating get and post requests on the backend with Node.js
If you want to configure your timeout in axios, you can use,
const axiosInstance = axios.create({
baseURL: "http://example.com/api/",
timeout: 5000
});
Replace 5000 with your timeout value needed.
Ultimately I found the solution that worked here:
Node Express specific timeout value per route
And I used the setConnectionTimeout() function in the following way:
router.post('/gene_info', setConnectionTimeout('12h'), function(req, res) {
...
})
Situation
From my Meteor.js website I'm calling my own REST service. Here's a code sample from my server side
function (question) {
var r = Async.runSync(function (done) {
HTTP.get(URL, {
params: {q: question}, headers: {
"Accept": "application/json",
}
}, function (err, result) {
done(err, result);
});
});
if (r.err) {
console.log("Failed to smartSearch ... ", r.err);
return null;
} else if (r.result.content) {
console.log("Success ... ");
return JSON.parse(r.result.content);
}
}
This works great but there is also some crucial information in the response headers which I'm unable to find.
What I've tried so far
I viewed everything within r.result.content, but this only contains my request headers.
I've installed https://atmospherejs.com/gadicohen/headers and tried everything the site said.
But still not seeing my response headers.
Additional Information
I'm fairly new to Meteor.js so I don't really have an idea what I might be doing wrong but getting response headers doesn't see like a strange thing to me.
There is no need to wrap the request as an async call, as it already is.
You can use a try..catch block to handle both successful and failed requests.
try {
var result = HTTP.get(...);
var responseHeaders = result.headers;
} catch (e) {
// handle error
}
If the response headers indicate JSON response, it will be parsed and available as result.data. The response will be available as a string in result.content.
More details are available in the HTTP package API docs.
i'm having a specific problem with the error "Can't set headers after they are sent".
The code is this one:
create: (request, response, next) ->
socket = #app.socket
#
# This method will be used to call the right method inside
# the emails service.
data = JSON.stringify(
Object.assign request.body.data, method: 'server'
)
socket.send data
socket.on 'message', (result) =>
result = JSON.parse(
result.toString()
)
if result.code is 'success'
#model.insertAsync request.body
.then (result) ->
response.json data: result
return
.catch next
return
return
I'm using two servers and socket connection to communication between them. When i want create a email, i send a message for this other server and wait for the result, if the result is "success", i send the data back (i'm using Ember, so i need send the data after saving it). Now the problem: when i send the email for the first time everything works normal, when i try for the second time, a error message shows in my terminal:
Error: Can't set headers after they are sent.
According to express, the error is in this line:
response.json data: result
or in JS:
response.json({data: result});
The code in pure JS:
create: function(request, response, next) {
var data, socket;
socket = this.app.socket;
data = JSON.stringify(Object.assign(request.body.data, {
method: 'server'
}));
socket.send(data);
socket.on('message', (function(_this) {
return function(result) {
result = JSON.parse(result.toString());
if (result.code === 'success') {
_this.model.insertAsync(request.body).then(function(result) {
response.json({
data: result
});
})["catch"](next);
}
};
})(this));
}
Thanks in advance, guys!
Most likely what is happening is you are receiving more than one successful "message" and so it's calling response.json() multiple times.
You could fix this by changing on('message') to once('message') so that the event handler only executes once. However, if that one message is not successful, a response won't be sent. So you may need to either add an else to your if (result.code === 'success') or leave on('message') and introduce some sort of guard variable so that the code inside the if is only executed once.
I am using the got library to make some http requests. A typical got request looks like this:
got.post(url, options, function(err, body, res) {
...
}
The callback has errors as well as the body content and the full response. In some cases when connecting to certain 3rd party APIs, if I get an error response, I find both the err and body both contain important error information I would want to pass back up the chain of callbacks:
API.prototype.makeRequest = function(callback) {
got.post(url, {}, function(err, body, res) {
if (err) {
callback(err);
return false;
}
// Otherwise do stuff
});
}
// app.js
var myAPI = new API();
myAPI.makeRequest(function(err, data){
if (err) {
console.log(err);
return false;
}
// Do stuff...
});
So in this example, I'm obviously only returning the error for the response. An example of this error:
{ [HTTPError: Response code 400 (Bad Request)]
message: 'Response code 400 (Bad Request)',
code: undefined,
host: 'api.someservice.com',
hostname: 'api.someservice.com',
method: 'POST',
path: '/oauth2/token_access',
statusCode: 400,
statusMessage: 'Bad Request' }
But if I examine the body of the error request, I see even more important and relevant error information:
{ error_description: 'Grant type is not supported: some_type_of_grant', error: 'unsupported_grant_type' }
So I have useful information in both the err and body variables that I want to pass back up the callback chain since both are useful.
What is the best approach to this? Typical Node.js approach seems to be having a single err value be the first value in the callback. Should I combine them into a single object or combine them and formulate my own error? Any other good approaches?