So I want to fetch a large amount of data on Parse, a good solution I found is to make a recursive function: when the data are successfully found, launch another request. The way I'm doing it is pretty simple:
var containedLimit = 1000, // Also tried with 500, 300, and 100
parseUsersWaiting = {
// A lot of Users
},
parseUsers = {}, // Recipt for Users
getPlayers = function (containedIn) {
var count = 0;
if (containedIn == undefined) {
containedIn = [];
for (var i in parseUsersWaiting) {
count++;
if (count > containedLimit) {
break;
}
containedIn.push(parseUsersWaiting[i].id);
delete parseUsersWaiting[i];
}
}
var UserObject = Parse.Object.extend('User'),
UserQuery = new Parse.Query(UserObject);
UserQuery.limit(containedLimit);
UserQuery.containedIn('objectId', containedIn);
UserQuery.find({
success: function (results) {
if (results) {
for (var i in results) {
if (parseUsers[results[i].id] == undefined) {
parseUsers[results[i].id] = results[i];
}
// Other stuff
if (results.length < containedLimit) {
// End of process
} else {
getPlayers();
}
}
} else {
// Looks like an end of process too
}
}, error: function (error) {
console.log(error.message);
getPlayers(containedIn);
}
});
};
Now, here is what I would like to call the "issue": it happen, very frequently, that the "error" callback is launched with this:
Received an error with invalid JSON from Parse: Error: getaddrinfo ENOTFOUND api.parse.com
at errnoException (dns.js:44:10)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:94:26)
(With code 107, of course) So I searched on the Parse Documentation, and it says the exact same thing: "Received an error with invalid JSON". Yeah.
I'm using the Parse SDK provided by Parse.com (npm install parse)
I also tried to modify a bit the Parse code with replacing the host key by 'hostname' on line 361 of the file parse/node_modules/xmlhttprequest/lib/XMLHttpRequest.js (Parse package version: 1.5.0), and it didn't worked, so I removed my changes.
(By the way, I found a solution talking about using ulimit to change memory usage limit that could solve the problem, but actually, I haven't the necessary rights to execute this command)
This error occurs when it can not connect to the API (technically when it cannot lookup the IP address of the API server). Perhaps your internet connection was lost for a brief moment or their SDK server were unavailable (or maybe the server is denying your request due to rate limits).
Either way it is good to code some resilience into your application. I see your on error function retries the API call, but perhaps it would be worth adding a timeout before you do that to give the problem a chance to recover?
error: function (error) {
console.log(error.message);
setTimeout(getPlayers, 10000, containedIn);
}
If the sever is rate limiting your requests, this will also help.
Related
I'm creating a REST API to store setting from a specific camera sending data through TCP.
Camera communicate through TCP with a request/response pattern.
e.g : You can send "Get rate\n" to the camera and it responds "100 fps", in order to get the current framerate of the camera.
How to get data and store it in API?
var command = "rate"; //Array with setting I want to store in my API
function getDataFromCamera(command) { // This function will be iterate in another function
async.series([
console.log('Test1'); // console log to test output
function(callback) {
CAMERA_TCP_SOCKET.send("get "+command+"\n");
callback(null,command);
},
function(callback) {
console.log('Test2'); // console log to test output
CAMERA_TCP_SOCKET.onmessage = function(event) {
callback(null, event.data); // THIS line is the problem. I can't retrieve event.data because i don't know how to callback this variable
}
}],
function(err, results) {
if (err) {
//Handle the error in some way. Here we simply throw it
throw err;
}
if (results) {
console.log(results); // expected output : // ['rate','100']
}
});
}
At the moment, I get this error:
Test1
Test2
/Users/maximecongi/Desktop/Hologram/node_modules/async/dist/async.js:966
if (fn === null) throw new Error("Callback was already called.");
^
Error: Callback was already called.
at /Users/maximecongi/Desktop/Hologram/node_modules/async/dist/async.js:966:32
at /Users/maximecongi/Desktop/Hologram/node_modules/async/dist/async.js:3885:13
at W3CWebSocket.CAMERA_TCP_SOCKET.onmessage (/Users/maximecongi/Desktop/Hologram/core/api.js:91:13)
at W3CWebSocket._dispatchEvent [as dispatchEvent] (/Users/maximecongi/Desktop/Hologram/node_modules/yaeti/lib/EventTarget.js:107:17)
at W3CWebSocket.onMessage (/Users/maximecongi/Desktop/Hologram/node_modules/websocket/lib/W3CWebSocket.js:234:14)
at WebSocketConnection.<anonymous> (/Users/maximecongi/Desktop/Hologram/node_modules/websocket/lib/W3CWebSocket.js:205:19)
at WebSocketConnection.emit (events.js:188:13)
at WebSocketConnection.processFrame (/Users/maximecongi/Desktop/Hologram/node_modules/websocket/lib/WebSocketConnection.js:554:26)
at /Users/maximecongi/Desktop/Hologram/node_modules/websocket/lib/WebSocketConnection.js:323:40
at process.internalTickCallback (internal/process/next_tick.js:70:11)
[nodemon] app crashed - waiting for file changes before starting...
How to solve my problem?
It's because you are listening on every message, and looks like you're sending 3 commands, and I guess, every command get a response. This callback after first serie is not destroyed, its alive. For me, you should check if the response is for this specific command, then execute the problematic callback.
We've got a Node.js script that is run once a minute to check the status of our apps. Usually, it works just fine. If the service is up, it exits with 0. If it's down, it exits with 1. All is well.
But every once in a while, it just kinda stops. The console reports "Calling status API..." and stops there indefinitely. It doesn't even timeout at Node's built-in two-minute timeout. No errors, nothing. It just sits there, waiting, forever. This is a problem, because it blocks following status check jobs from running.
At this point, my whole team has looked at it and none of us can figure out what circumstance could make it hang. We've built in a start-to-finish timeout, so that we can move on to the next job, but that essentially skips a status check and creates blind spots. So, I open the question to you fine folks.
Here's the script (with names/urls removed):
#!/usr/bin/env node
// SETTINGS: -------------------------------------------------------------------------------------------------
/** URL to contact for status information. */
const STATUS_API = process.env.STATUS_API;
/** Number of attempts to make before reporting as a failure. */
const ATTEMPT_LIMIT = 3;
/** Amount of time to wait before starting another attempt, in milliseconds. */
const ATTEMPT_DELAY = 5000;
// RUNTIME: --------------------------------------------------------------------------------------------------
const URL = require('url');
const https = require('https');
// Make the first attempt.
make_attempt(1, STATUS_API);
// FUNCTIONS: ------------------------------------------------------------------------------------------------
function make_attempt(attempt_number, url) {
console.log('\n\nCONNECTION ATTEMPT:', attempt_number);
check_status(url, function (success) {
console.log('\nAttempt', success ? 'PASSED' : 'FAILED');
// If this attempt succeeded, report success.
if (success) {
console.log('\nSTATUS CHECK PASSED after', attempt_number, 'attempt(s).');
process.exit(0);
}
// Otherwise, if we have additional attempts, try again.
else if (attempt_number < ATTEMPT_LIMIT) {
setTimeout(make_attempt.bind(null, attempt_number + 1, url), ATTEMPT_DELAY);
}
// Otherwise, we're out of attempts. Report failure.
else {
console.log("\nSTATUS CHECK FAILED");
process.exit(1);
}
})
}
function check_status(url, callback) {
var handle_error = function (error) {
console.log("\tFailed.\n");
console.log('\t' + error.toString().replace(/\n\r?/g, '\n\t'));
callback(false);
};
console.log("\tCalling status API...");
try {
var options = URL.parse(url);
options.timeout = 20000;
https.get(options, function (response) {
var body = '';
response.setEncoding('utf8');
response.on('data', function (data) {body += data;});
response.on('end', function () {
console.log("\tConnected.\n");
try {
var parsed = JSON.parse(body);
if ((!parsed.started || !parsed.uptime)) {
console.log('\tReceived unexpected JSON response:');
console.log('\t\t' + JSON.stringify(parsed, null, 1).replace(/\n\r?/g, '\n\t\t'));
callback(false);
}
else {
console.log('\tReceived status details from API:');
console.log('\t\tServer started:', parsed.started);
console.log('\t\tServer uptime:', parsed.uptime);
callback(true);
}
}
catch (error) {
console.log('\tReceived unexpected non-JSON response:');
console.log('\t\t' + body.trim().replace(/\n\r?/g, '\n\t\t'));
callback(false);
}
});
}).on('error', handle_error);
}
catch (error) {
handle_error(error);
}
}
If any of you can see any places where this could possibly hang without output or timeout, that'd be very helpful!
Thank you,
James Tanner
EDIT: p.s. We use https directly, instead of request so that we don't need to do any installation when the script runs. This is because the script can run on any build machine assigned to Jenkins without a custom installation.
Aren't you missing the .end()?
http.request(options, callback).end()
Something like explained here.
Inside your response callback your not checking the status..
The .on('error', handle_error); is for errors that occur connecting to the server, status code errors are those that the server responds with after a successful connection.
Normally a 200 status response is what you would expect from a successful request..
So a small mod to your http.get to handle this should do..
eg.
https.get(options, function (response) {
if (response.statusCode != 200) {
console.log('\tHTTP statusCode not 200:');
callback(false);
return; //no point going any further
}
....
Is it possible to perform a lookup of a Bluetooth device given its address in a Nodejs script?
There are a few packages out there, the main one being Noble. However, they all focus around scanning, and not looking up a known address (as far as i can tell anyway!).
What i want to achieve, is to look up a known address, to see if the device can be found.
Much like PyBluez does for Python:
bluetooth.lookup_name('CC:20:E8:8F:3A:1D', timeout=5)
In Python, this can find the device even if it is undiscoverable, unlike a typical inquiry scan would.
I had this same problem and just found the btwatch lib, but it isn't working for me on the latest raspbian. But the source is just calling l2ping and looking for a string that I'm guessing no longer prints on success, so the modified code below works instead, similar to the lookup_name method, once you have l2ping installed (I think npm bluetooth or pybluez has it)
var Spawn = require('child_process').spawn;
function detectMacAddress(macAddress, callback)
{
//var macAddress = '72:44:56:05:79:A0';
var ls = Spawn('l2ping', ['-c', '1', '-t', '5', macAddress]);
ls.stdout.on('data', function (data) {
console.log("Found device in Range! " + macAddress);
callback(true);
});
ls.on('close', function () {
console.log("Could not find: " + macAddress);
callback(false);
});
}
Or, a synchronous way,
var execSync = require('child_process').execSync;
function detectMacAddressSync(macAddress)
{
var cmd = 'l2ping -c 1 -t 5 ' + macAddress;
try
{
var output = execSync(cmd );
console.log("output : "+ output );
return true;
}
catch(e)
{
console.log("caught: " + e);
return false;
}
}
As far as I have understood the problem you want to connect to the device using address. Then, I would suggest using node-bluetooth-serial-port.
var btSerial = new (require('bluetooth-serialport')).BluetoothSerialPort();
btSerial.on('found', function(address, name) {
btSerial.findSerialPortChannel(address, function(channel) {
btSerial.connect(address, channel, function() {
console.log('connected');
btSerial.write(new Buffer('my data', 'utf-8'), function(err, bytesWritten) {
if (err) console.log(err);
});
btSerial.on('data', function(buffer) {
console.log(buffer.toString('utf-8'));
});
}, function () {
console.log('cannot connect');
});
// close the connection when you're ready
btSerial.close();
}, function() {
console.log('found nothing');
});
});
BluetoothSerialPort.findSerialPortChannel(address, callback[, errorCallback])
Checks if a device has a serial port service running and if it is found it passes the channel id to use for the RFCOMM connection.
callback(channel) - called when finished looking for a serial port on the device.
errorCallback - called the search finished but no serial port channel was found on the device. Connects to a remote bluetooth device.
bluetoothAddress - the address of the remote Bluetooth device.
channel - the channel to connect to.
[successCallback] - called when a connection has been established.
[errorCallback(err)] - called when the connection attempt results in an error. The parameter is an Error object.
I have used pouchDB in one application and now I want to introduce couchDB to sync the document to remote server. Hence i followed this link http://pouchdb.com/getting-started.html i used the below code to replicate the data to couchDB
var db2 = new PouchDB('todos');
var remoteCouch = 'http://localhost:5984/_utils/database.html?couchdb_sample';
db2.changes({
since: 'now',
live: true
}).on('change', showTodos);
sync();
function sync() {
//alert("sync");
//syncDom.setAttribute('data-sync-state', 'syncing');
//var opts = {live: true};
db2.replicate.to(remoteCouch).on('complete', function () {
console.log("done");
}).on('error', function (err) {
console.log(err);
});
function addTodo(text) {
var todo = {
_id: $("#eid").val()+$("#version").val(),
title: text,
name: $("#nameid").val(),
version: $("#version").val(),
completed: false
};
db2.put(todo, function callback(err, result) {
if (!err) {
console.log('Successfully posted a todo!');
}
else{
console.log(err);
}
});}
here the title has an xml string as value. But i am facing below error
SyntaxError: Unexpected token <
at Object.parse (native)
for this line db2.replicate.to(remoteCouch). I manually created a new document in couchDb database and entered the same data it gave no error but when i try replicating it shows syntax error. Can anyone please hint me where I have gone wrong
http://localhost:5984/_utils/database.html?couchdb_sample
Points to a HTML site (copied over from the browsers address bar, right?). Remove the middle part:
http://localhost:5984/couchdb_sample
It look like you have not defined the remote database in the way PouchDb is expecting. You should use the "new PouchDb" call. The second line of your code is:
var remoteCouch = 'http://localhost:5984/_utils/database.html?couchdb_sample';
but I think it should be like this:
var remoteCouch = new PouchDB('http://localhost:5984/couchdb_sample');
I am not clear from your code what the name of the remote database is, but it would not normally end in ".html" as Ingo Radatz pointed out, so I have assumed it is couchdb_sample above. There is more information about replication on the PouchDb site.
I have the following code in server/statusboard.js;
var require = __meteor_bootstrap__.require,
request = require("request")
function getServices(services) {
services = [];
request('http://some-server/vshell/index.php?type=services&mode=json', function (error, response, body) {
var resJSON = JSON.parse(body);
_.each(resJSON, function(data) {
var host = data["host_name"];
var service = data["service_description"];
var hardState = data["last_hard_state"];
var currState = data["current_state"];
services+={host: host, service: service, hardState: hardState, currState: currState};
Services.insert({host: host, service: service, hardState: hardState, currState: currState});
});
});
}
Meteor.startup(function () {
var services = [];
getServices(services);
console.log(services);
});
Basically, it's pulling some data from a JSON feed and trying to push it into a collection.
When I start up Meteor I get the following exception;
app/packages/livedata/livedata_server.js:781
throw exception;
^
Error: Meteor code must always run within a Fiber
at [object Object].withValue (app/packages/meteor/dynamics_nodejs.js:22:15)
at [object Object].apply (app/packages/livedata/livedata_server.js:767:45)
at [object Object].insert (app/packages/mongo-livedata/collection.js:199:21)
at app/server/statusboard.js:15:16
at Array.forEach (native)
at Function.<anonymous> (app/packages/underscore/underscore.js:76:11)
at Request._callback (app/server/statusboard.js:9:7)
at Request.callback (/usr/local/meteor/lib/node_modules/request/main.js:108:22)
at Request.<anonymous> (/usr/local/meteor/lib/node_modules/request/main.js:468:18)
at Request.emit (events.js:67:17)
Exited with code: 1
I'm not too sure what that error means. Does anyone have any ideas, or can suggest a different approach?
Just wrapping your function in a Fiber might not be enough and can lead to unexpected behavior.
The reason is, along with Fiber, Meteor requires a set of variables attached to a fiber. Meteor uses data attached to a fiber as a dynamic scope and the easiest way to use it with 3rd party api is to use Meteor.bindEnvironment.
T.post('someurl', Meteor.bindEnvironment(function (err, res) {
// do stuff
// can access Meteor.userId
// still have MongoDB write fence
}, function () { console.log('Failed to bind environment'); }));
Watch these videos on evented mind if you want to know more:
https://www.eventedmind.com/posts/meteor-dynamic-scoping-with-environment-variables
https://www.eventedmind.com/posts/meteor-what-is-meteor-bindenvironment
As mentioned above it is because your executing code within a callback.
Any code you're running on the server-side needs to be contained within a Fiber.
Try changing your getServices function to look like this:
function getServices(services) {
Fiber(function() {
services = [];
request('http://some-server/vshell/index.php?type=services&mode=json', function (error, response, body) {
var resJSON = JSON.parse(body);
_.each(resJSON, function(data) {
var host = data["host_name"];
var service = data["service_description"];
var hardState = data["last_hard_state"];
var currState = data["current_state"];
services+={host: host, service: service, hardState: hardState, currState: currState};
Services.insert({host: host, service: service, hardState: hardState, currState: currState});
});
});
}).run();
}
I just ran into a similar problem and this worked for me. What I have to say though is that I am very new to this and I do not know if this is how this should be done.
You probably could get away with only wrapping your insert statement in the Fiber, but I am not positive.
Based on my tests you have to wrap the insert in code I tested that is similar to the above example.
For example, I did this and it still failed with Fibers error.
function insertPost(args) {
if(args) {
Fiber(function() {
post_text = args.text.slice(0,140);
T.post('statuses/update', { status: post_text },
function(err, reply) {
if(reply){
// TODO remove console output
console.log('reply: ' + JSON.stringify(reply,0,4));
console.log('incoming twitter string: ' + reply.id_str);
// TODO insert record
var ts = Date.now();
id = Posts.insert({
post: post_text,
twitter_id_str: reply.id_str,
created: ts
});
}else {
console.log('error: ' + JSON.stringify(err,0,4));
// TODO maybe store locally even though it failed on twitter
// and run service in background to push them later?
}
}
);
}).run();
}
}
I did this and it ran fine with no errors.
function insertPost(args) {
if(args) {
post_text = args.text.slice(0,140);
T.post('statuses/update', { status: post_text },
function(err, reply) {
if(reply){
// TODO remove console output
console.log('reply: ' + JSON.stringify(reply,0,4));
console.log('incoming twitter string: ' + reply.id_str);
// TODO insert record
var ts = Date.now();
Fiber(function() {
id = Posts.insert({
post: post_text,
twitter_id_str: reply.id_str,
created: ts
});
}).run();
}else {
console.log('error: ' + JSON.stringify(err,0,4));
// TODO maybe store locally even though it failed on twitter
// and run service in background to push them later?
}
}
);
}
}
I thought this might help others encountering this issue. I have not yet tested calling the asynchy type of external service after internal code and wrapping that in a Fiber. That might be worth testing as well. In my case I needed to know the remote action happened before I do my local action.
Hope this contributes to this question thread.