I am using web3 version 1.0.0-beta.27 where all accesses to the blockchain will be asynchronous, clearly this opens up the possibility of race conditions, ie:
var Web3 = require("web3");
// connect to etherum blockchain
var ether_port = 'http://localhost:8545'
var web3 = new Web3(new Web3.providers.HttpProvider(ether_port));
// this is how we set the value, note there is the possiblity of race condidtions here
var accounts = []
web3.eth.getAccounts().then(function(accts){
console.log("printing account: ", accts)
accounts = accts
})
// observe race condition
console.log("assert race condition: ", accounts[0])
The last line above is contrived, it is there to demonstrate that I would like to use accounts after it has been evaluated. Ie, eventually I would like modify/read the blockchain from a front end express.js web app or even a mobile app, so in the interest of being rigorous, what are the common tools in node.js to ensure race conditions never occur? Do these tools exist? If not what are some common practices. I am new to node.js as well.
One idea is to not attempt to directly store the data because code trying to access the data has no idea when it's valid due to the uncertain nature of asynchronous results. So, instead you store the promise and any code that wants access to the data, just uses .then()/.catch() on the promise. This will always work, regardless of the async timing. If the data is already there, the .then() handler will be called quickly. If the data is not yet there, then the caller will be in line to be notified when the data arrives.
let accountDataPromise = web3.eth.getAccounts().then(function(accts){
console.log("printing account: ", accts)
return accts;
});
// then, elsewhere in the code
accountDataPromise.then(accts => {
// use accts here
}).catch(err => {
// error getting accts data
});
FYI, assigning data from a .then() handler to a higher scoped variable that you want to generally use in other code outside the promise chain is nearly always a sign of troublesome code - don't do it. This is because other code outside the promise chain has no idea when that data will or will not be valid.
Related
I have a simple Web app running with NodeJS and Express. It has a route where an outside 3rd party can POST us a XML document, which we then convert to JSON then save to our MongoDB database. A few things can go wrong:
The XML can be malformed
The request might be empty
The outside 3rd party might send us duplicate documents
Rather than having an endless series of then() blocks, going deeper and deeper, indented further and further, I wanted to throw an exception for each possible error, and then catch those errors at the top level and process them there.
So we find a unique id and then check to see if this unique id is already in MongoDB:
// will throw an error if there is a duplicate
document_is_redundant(AMS_945, unique_id);
The function looks like this:
function document_is_redundant(this_model, this_unique_id) {
return this_model.findOne({ unique_id : this_unique_id })
.exec()
.then((found_document) => {
// 2021-11-28 -- if we find a duplicate, we throw an error and handle it at the end
// But remember, we want to return a HTTP status code 200 to AMS, so they will stop
// re-sending this XML document.
if (found_document != 'null') {
throw new DocumentIsRedundantException(this_unique_id);
}
});
// no catch() block because we want the exception to go to the top level
}
This gives me: UnhandledPromiseRejectionWarning
Maybe I'm thinking too much like Java instead of Javascript, but I was assuming that if I didn't catch() the exception in that function, it would bubble up to the top level, which is where I want to deal with it. Also assumed it would interrupt the flow of the code at the line where I call the function.
Sadly, the uncaught exception does not interrupt the main thread of execution, so the document is saved, even when it is a duplicate.
So I'm left thinking the only way I can make this work is to return the Promise from the function and then have a then() block after the call to the document_is_duplicate function?
I dislike having to nest then() blocks inside of then() blocks, several levels deep. This seems like bad code. Is there another way?
Not sure why you want to throw an error if your document exists. Look for it, Mongoose will return a document if it exists, or null if it does not. Then simply await the result. Mongoose methods can be awaited, and if you add .exec() they even return a true Promise, which makes your life even easier :
const document_is_redundant = (this_model, unique_id) => this_model.findOne({ unique_id }).lean().exec();
// Now you use it this way
if( !(await document_is_redundant(AMS_945, unique_id))){ // If the returned value is not null
console.log("Document is redundant! Aborting")
return;
}
// Returned value was null
console.log("The document doesn't exist yet!")
I am having a problem where I am making a bulk insert of multiple elements into a table, then I immediatly get the last X elements from that table that were recently inserted but when I do that it seems that the elements have not yet been inserted fully even thought I am using async await to wait for the async operations.
I am making a bulk insert like
const createElements = elementsArray => {
return knex
.insert(elementsArray)
.into('elements');
};
Then I have a method to immediately access those X elements that were inserted:
const getLastXInsertedElements = (userId, length, columns=['*']) => {
return knex.select(...columns)
.from('elements').where('userId', userId)
.orderBy('createdAt', 'desc')
.limit(length);
}
And finally after getting those elements I get their ids and save them into another table that makes use of element_id of those recently added elements.
so I have something like:
// A simple helper function that handles promises easily
const handleResponse = (promise, message) => {
return promise
.then(data => ([data, undefined]))
.catch(error => {
if (message) {
throw new Error(`${message}: ${error}`);
} else {
return Promise.resolve([undefined, `${message}: ${error}`])
}
}
);
};
async function service() {
await handleResponse(createElements(list), 'error text'); // insert x elements from the list
const [elements] = await handleResponse(getLastXInsertedElements(userId, list.length), 'error text') // get last x elements that were recently added
await handleResponse(useElementsIdAsForeignKey(listMakingUseOfElementsIds), 'error text'); // Here we use the ids of the elements we got from the last query, but we are not getting them properly for some reason
}
So the problem:
Some times when I execute getLastXInsertedElements it seems that the elements are not yet finished inserting, even thought I am waiting with async/await for it, any ideas why this is? maybe something related to bulk inserts that I don't know of? an important note, all the elements always properly inserted into the table at some point, it just seems like this point is not respected by the promise (async operation that returns success for the knex.insert).
Update 1:
I have tried putting the select after the insert inside a setTimeout of 5 seconds for testing purposes, but the problem seems to persist, that is really weird, seems one would think 5 seconds is enough between the insert and the select to get all the data.
I would like to have all X elements that were just inserted accessible in the select query from getLastXInsertedElements consistently.
Which DB are you using, how big list of data are you inserting? You could also test if you are inserting and getLastXInsertedElements in a transaction if that hides your problem.
Doing those operations in transaction also forces knex to use the same connection for both queries so it might lead to a tracks where is this coming from.
Another trick to force queries to use the same connection would be to set pool's min and max configuration to be 1 (just for testing is parallelism is indeed the problem here).
Also since you have not provided complete reproduction code for this, I'm suspecting there is something else here in the mix which causes this odd behavior. Usually (but not always) this kind of weird cases that shouldn't happen are caused by user error in elsewhere using the library.
I'll update the answer if there is more information provided. Complete reproduction code would be the most important piece of information.
I am not 100% sure but I guess the knex functions do not return promise by default (but a builder object for the query). That builder has a function called then that transforms the builder into a promise. So you may try to add a call to that:
...
limit(length)
.then(x => x); // required to transform to promise
Maybe try debugging the actual type of the returned value. It might happen that this still is not a promise. In this case you may not use async await but need to use the then Syntax because it might not be real js promises but their own implementation.
Also see this issue about standard js promise in knex https://github.com/knex/knex/issues/1588
In theory, it should work.
You say "it seems"... a more clear problem explanation could be helpful.
I can argue the problem is that you have elements.length = list.length - n where n > 0; in your code there are no details about userId property in your list; a possible source of the problem could be that some elements in your list has a no properly set userId property.
I'm running into problems with the validated method package in my app tests. I'm calling my methods through the _execute function in order to be able to pass a userId to simulate a logged-in user while testing. The problem is that my asserts right underneath that _execute are called before the method has a chance of completing. I know my test works though because it only happens sometimes, mostly because mongo isn't always returning results quite as fast.
I looked around and found a todos app that uses the _execute function in its tests. I can't get those tests to fail no matter how many times I rerun them, though.
This is an example of my test code.
describe('clients.add', function() {
it('should add an empty (draft) client', function() {
const res = clients_add._execute({ userId: 'CURRENTUSERID' }, { company_id: c1._id });
assert.instanceOf(res, Mongo.ObjectID, 'method returns the newly created clients ID');
const db_client = Clients.findOne(res);
assert.isTrue(db_client.draft, 'client is drafted');
assert.isDefined(db_client.created, 'there\'s a created date');
});
});
clients_add does quite a few permission checks and can therefor take a little while before completing. Rerunning this test 20 times will fail about 5 times and pass the other 15.
Shouldn't the _execute function be synchronous? How do I make it? What am I missing?
In server code, if you provide a callback to database modification functions like insert, it returns the created ID instantaneously, and runs the callback only once the database has acknowledged the write. If you don't provide a callback, the insert call is synchronous and throws an error if the operation fails. See more about this in Meteor docs.
It seems that you have provided an error-handling callback to the insert-function in your method code. This causes the inconsistent behavior, since the database might not actually have had time to do the write before you call findOne in your test. Also, this is redundant since if an error occurs in the insert, the method has already returned and the error is never shown to the user. It's better to simply omit the error-handling callback altogether:
return Clients.insert(new_client);
I'm having this "design" problem that's driving me crazy.
My goal is having a decoupled RabbitMQ client. It has to be able to init it's connection and "return" a created channel so my publishing module can use it.
Code looks like this (i know that is not the better code but i expect it serves for this explanation).
var createConnection = (function() {
var channel;
var connect = function(){
// amqp connect
// error handling
createChannel();
});
}
var createChannel = function(){
//amqpConn.createConfirmChannel...
}
//pseudo
return{
getChannel : function(){
if(!initiated)
connect();
return channel;
}
}
})();
module.exports = createConnection;
Now, important things:
1- I know this ain't gonna work and i know why, its a simplification.
2- I'm aware that i can accomplish my goals by using async or promises.
3- Maybe has no sense decoupling a rabbit client but is for understanding purposes
That said, my questions:
1- Is there any way i can accomplish this without using other modules?
2- if so, can be accomplished in a fancy and stylish way?
3- Is there any fancy solution that allows 3rd party code executing a simple "publish(exchange, channel, msg)" been sure that connection has been established?
I feel able to work with JS but sometimes you just need do things one way only to know that you can but this is giving me some headache.
Truly thanks and i hope the question was understood :)
One way I've found to handle this is to wrap your asynchronous object in an object that is aware of the asynchronous state of your object and presents the same API regardless of whether or not the asynchronous object has finished initializing.
For example, you could wrap your channel object in another object that presents the same methods but internally checks if the actual channel object is initialized. If it is, use it as normal. If it isn't, wait for it to be initialized and use it as normal. The user of the wrapper object wouldn't need to know if the channel is actually initialized. The main drawback of this is every wrapper method that needs to access channel must be asynchronous even if the method it's accessing on channel is synchronous.
Example:
function initializeChannel() {
return new Promise((resolve, reject) => {
// create, initialize, and resolve channel
});
}
module.exports = { // wrapper
channelPromise: initializeChannel(),
foo(a) {
return this.channelPromise.then((channel) => channel.foo(a));
}
};
I recently did a lot of coding in AngularJS. After some time it started to feel comfortable with it and also got really productive. But unfortunately there is this one thing I don't understand:
Within my project I need to get data through $http.get and a RESTful API server. This is where I started to stumble first. After implementing promise ($q.defer etc and .then) at functions which are processing data that's necessary to continue, I thought I conquered the problem.
But in this code:
$scope.getObservationsByLocations = function() {
var promise = $q.defer();
var locationCount = 0;
angular.forEach($scope.analysisData, function(loc) { // for each location
$http.get($scope.api + 'Device?_format=json', { // get all devices
params: {
location: loc.location.id
}
}).then(function (resultDevices) {
var data = angular.fromJson(resultDevices);
promise.resolve(data);
// for each device in this location
angular.forEach(angular.fromJson(resultDevices).data.entry.map(function (dev) {
http.get($scope.api + 'Observation?_format=json', { // get all observations
params: {
device: dev.resource.id
}
}).then(function (resultObservations) {
var observations = angular.fromJson(resultObservations);
// for each obervation of that device in this location
angular.forEach(observations.data.entry.map(function(obs) {
$scope.analysisData[locationCount].observations.push({observation: obs.resource});
}));
})
}))
});
locationCount++
});
return promise.promise
};
I can't understand in which order the commands are executed. Since I use the Webstorm IDE and it's debugging feature, it would be more accurate to say I don't know why the commands are executed in an order I don't understand.
Thinking simple, everything included in the forEach have to be executed before the return is reached, because $http.get's are connected through .then's. But following the debugging information, the function iterates over locationCount++ and even returns the promise before it goes deeper (meaning after the first .then() ).
What's that all about? Did I misunderstood this part of the AngularJS concept?
Or is this just really bad practice and I should reach out for a different solution?
If the context is important/interesting: Objects are based on i.e. https://www.hl7.org/fhir/2015May/location.html#5.15.3
With JavaScript you can create only single thread applications, although e.g. here they say it is not guaranteed to be like that.
But we are talking about the real world and real browsers, so your code sample is running as a single thread (by the way the same thread is also used for rendering your CSS and HTML, at least in Firefox).
When it comes to an asynchronous call
$http.get($scope.api + 'Device?_format=json', {
it says "hey, I can do that later". And it waits with that because it must go on with the current thread.
Then, once the current task is done with return it finally can start getting the remote data.
Proof? Check this fiddle:
console.log(1);
for (var i=0;i<1000000;i++) setTimeout(function(){
console.log(2);
},0);
console.log(3);
You see the spike with the for loop? This is the moment when it registers the setTimeout asynchronous calls. Still 3 is printed before 2 because the task is not done until the 3 is printed.
The $http.get is asynchronous, so depending on (among other things) how large the fetched data is, the time it takes to 'complete' the get is variable. Hence why there is no saying in what order they will be completed