This is a fairly weird thing, and it's hard to reproduce. Not the best state of a bug report, I apologize.
I'm using .transaction() to write a value to a location in Firebase. Here's some pseudo-code:
var ref = firebase.child('/path/to/location');
var storeSafely = function(val) {
ref.transaction(
function updateFunc(currentData) {
console.log('Attempting update: ' + JSON.stringify(val));
if (currentData) return;
return val;
},
function onTransactionCompleteFunc(err, isCommitted, snap) {
if (err) {
console.log('Error in onTransactionCompleteFunc: ' + JSON.stringify(err));
return;
}
if (! isCommitted) {
console.log('Not committed');
return;
}
ref.onDisconnect().remove();
doSomeStuff();
});
};
var doSomeStuff = function() {
// Things get done, time passes.
console.log('Cleaning up');
ref.onDisconnect().cancel();
ref.set(
null,
function onSetCompleteFunc(err) {
if (err) {
console.log('Error in onSetCompleteFunc: ' + JSON.stringify(err));
}
});
};
storeSafely(1);
// later...
storeSafely(2);
// even later...
storeSafely(3);
I'm effectively using Firebase transactions as a sort of mutex lock:
Store a value at a location via transaction.
Set the onDisconnect for the location to remove the value in case my app dies while working.
Do some stuff.
Remove the onDisconnect for the location, because I'm done with the stuff.
Remove the value at the location.
I do this every few minutes, and it all works great. Things get written and removed perfectly, and the logs show me creating the lock, doing stuff, and then releasing the lock.
The weird part is what happens hours later. Occasionally Firebase has maintenance, and my app gets a bunch of permission denied errors. At the same time this happens, I suddenly start getting a bunch of this output in the logs:
Attempting update 1
Attempting update 2
Attempting update 3
...in other words, it looks like the transactions never fully completed, and they're trying to retry now that the location can't be read any more. It's almost like there's a closure in the transaction() code that never completed, and it's getting re-executed now for some reason.
Am I missing something really important here about how to end a transaction?
(Note: I originally posted this to the Firebase Google Group, but was eventually reminded that code questions are supposed to go to Stack Overflow. I apologize for the cross-posting.)
Just a guess, but I wonder if your updateFunc() function is being called with null when your app gets the permission-denied errors from Firebase. (If so, I could believe that's part of their "Offline Writes" support.)
In any case, you should handle null as a possible state. Saving Transactional Data says:
transaction() will be called multiple times and must be able to handle
null data. Even if there is existing data in your database it may not
be locally cached when the transaction function is run.
I don't know the intricacies of Firebase's transaction mechansim, but I would try changing your .set(null) to set the value to 0 instead, change your .remove() to also set the value with .set(0), and change your line in updateFunc() to:
if (currentData === null || currentData) return;
Unfortunately, that assumes that '/path/to/location' is initially set to 0 at some point. If that's a problem, maybe you can muck around with null versus undefined. For example, it would be nice if Firebase used one of those for non-existent data and another when it's offline.
Related
I have a Vue 2.0 app in which I use this line in order to call this.refreshState() every min.
this.scheduler = setInterval(() => this.refreshState(), 60 * 1000)
Later in the code I need to make sure that the execution loop is stopped and also that if there's an instance of this.refreshState() currently running (from the setInterval scheduler) it's stopped as well (even if it's in the middle of doing stuff).
So far I'm using :
clearInterval(this.scheduler)
as per (https://developer.mozilla.org/en-US/docs/Web/API/clearInterval)
The question I'm having is does clearInterval blocks the current execution if any? I can't find the answer in the doc unfortunately.
FYI the code of refreshState:
refreshState: function () {
// API call to backend
axios.get("/api/refreshState")
.then(response => {
this.states = response.data.states
})
.catch((err) => console.log(err)
}
Here's my use case :
alterState: function (incremental_state) {
clearInterval(this.scheduler) // ???
axios.post("/api/alterState", incremental_state)
.then(() => {
this.refreshState()
this.scheduler = setInterval(() => this.refreshState(), 60 * 1000)
})
.catch((err) => { console.log(error) })
}
I want to make sure that when i exit alterState , the variable this.states takes into account the addition of incremental state.
From...
I want to make sure that when i exit alterState, the variable this.states takes into account the addition of incremental state.
...I understand you're performing a change on backend and you want it reflected on frontend. And that currently that doesn't happen, although you're calling this.refreshState() right after getting a successful response from /api/alterState. 1
To achieve this functionality, it's not enough to call this.refreshState(), because your browser, by default, caches the result (it remembers the recent calls and their results, so it serves the previous result from cache, instead of calling the server again), unless the endpoint is specifically configured to disable caching.
To disable caching for a particular endpoint, you could either
configure the endpoint (server side) to tell browsers: "Hey, my stuff is time sensitive, don't cache it!" (won't go into how, as I have no idea what technology you're using on backend and it varies). Roughly it means setting appropriate response headers.
or call the endpoint with a unique param, each time. This makes the endpoint "change" from browser's POV, so it's always going to request from server:
axios
.get(`/api/refreshState?v=${Date.now()}`)
.then...
I recommend the second option, it's reliable, predictable and does not depend on server configuration.
And, unless something else, other than the current app instance (some other user, or other server scripts, etc...) make changes to the data, you don't actually need a setInterval. I suggest removing it.
But If you do have other sources changing server-side data, (and you do want to refresh it regardless of user interactions with the app), what you have works perfectly fine, there's no need to even cancel the existing interval when you make a change + refreshState()). 2
1 - if I misunderstood your question and that is not your problem, please clarify your question, right now it's a bit unclear
2 - as side-note and personal preference, I suggest renaming refreshState() to getState()
I am facing this issue for the past 1 week and I am just confused about this.
Keeping it short and simple to explain the problem.
We have an in memory Model which stores values like budget etc.Now when a call is made to the API it has a spent associated with it.
We then check the in memory model and add the spent to the existing spend and then check to the budget and if it exceeds we donot accept any more clicks of that model. for each call we also udpate the db but that is a async operation.
A short example
api.get('/clk/:spent/:id', function(req, res) {
checkbudget(spent, id);
}
checkbudget(spent, id){
var obj = in memory model[id]
obj.spent+= spent;
obj.spent > obj.budjet // if greater.
obj.status = 11 // 11 is the stopped status
update db and rebuild model.
}
This used to work fine but now with concurrent requests we are getting false spends out spends increase more than budget and it stops after some time. We simulated the call with j meter and found this.
As far as we could find node is async so by the time the status is updated to 11 many threads have already updated the spent for the campaign.
How to have a semaphore kind of logic for Node.js so that the variable budget is in sync with the model
update
db.addSpend(campaignId, spent, function(err, data) {
campaign.spent += spent;
var totalSpent = (+camp.spent) + (+camp.cpb);
if (totalSpent > camp.budget) {
logger.info('Stopping it..');
camp.status = 11; // in-memory stop
var History = [];
History.push(some data);
db.stopCamp(campId, function(err, data) {
if (err) {
logger.error('Error while stopping );
}
model.campMAP = buildCatMap(model);
model.campKeyMap = buildKeyMap(model);
db.campEventHistory(cpcHistory, false, function(err) {
if (err) {
logger.error(Error);
}
})
});
}
});
GIST of the code can anyone help now please
Q: Is there semaphore or equivalent in NodeJs?
A: No.
Q: Then how do NodeJs users deal with race condition?
A: In theory you shouldn't have to as there is no thread in javascript.
Before going deeper into my proposed solution I think it is important for you to know how NodeJs works.
For NodeJs it is driven by an event based architecture. This means that in the Node process there is an event queue that contains all the "to-do" events.
When an event gets pop from the queue, node will execute all of the required code until it is finished. Any async calls that were made during the run were spawned as other events and they are queued up in the event queue until a response is heard back and it is time to run them again.
Q: So what can I do to ensure that only 1 request can perform updates to the database at a time?
A: I believe there are many ways you can achieve this but one of the easier way out is to use the set_timeout API.
Example:
api.get('/clk/:spent/:id', function(req, res) {
var data = {
id: id
spending: spent
}
canProceed(data, /*functions to exec after canProceed=*/ checkbudget);
}
var canProceed = function(data, next) {
var model = in memory model[id];
if (model.is_updating) {
set_timeout(isUpdating(data, next), /*try again in=*/1000/*milliseconds*/);
}
else {
// lock is released. Proceed.
next(data.spending, data.id)
}
}
checkbudget(spent, id){
var obj = in memory model[id]
obj.is_updating = true; // Lock this model
obj.spent+= spent;
obj.spent > obj.budjet // if greater.
obj.status = 11 // 11 is the stopped status
update db and rebuild model.
obj.is_updating = false; // Unlock the model
}
Note: What I got here is pseudo code as well so you'll may have to tweak it a bit.
The idea here is to have a flag in your model to indicate whether a HTTP request can proceed to do the critical code path. In this case your checkbudget function and beyond.
When a request comes in it checks the is_updating flag to see if it can proceed. If it is true then it schedules an event, to be fired in a second later, this "setTimeout" basically becomes an event and gets placed into node's event queue for later processing
When this event gets fired later, the checks again. This occurs until the is_update flag becomes false then the request goes on to do its stuff and is_update is set to false again when all the critical code is done.
Not the most efficient way but it gets the job done, you can always revisit the solution when performance becomes a problem.
See jsbin.com/ceyiqi/edit?html,console,output for a verifiable example.
I have a reference listening to a database point
jobs/<key>/list
where <key> is the teams unique number
Under this entry point is a list of jobs
I have a listener on this point with
this.jobsRef.orderByChild('archived')
.equalTo(false)
.on('child_added', function(data) {
I also have a method that does the following transaction:
ref.transaction(function(post) {
// correct the counter
if(post)
{
// console.log(post);
if(active)
{
// if toggeling on
}
else
{
// if toggeling off
}
}
return post;
})
When invoking the transaction the child_added is also invoked again, giving me duplicate jobs.
Is this expected behavior?
Should I simply check to see if the item has been got before and add to the array accordingly?
Or am I doing something wrong?
Thanks in advance for your time
You're hitting an interesting edge case in how the Firebase client handles transactions. Your transaction function is running twice, first on null and then on the actual data. It's apparent to see if you listen for value events on /jobs, since then you'll see:
initial value
null (when transaction starts and runs on null)
initial value again (when transaction runs again on real data)
The null value in step 2 is the client's initial guess for the current value. Since the client doesn't have cached data for /jobs (it ignores the cached data for /jobs/list), it guesses null. It is clearly wrong. But unfortunately has been like this for a long time, so it's unlikely to change in the current major version of the SDK.
And because of the null in step 2, you'll get child_removed events (which you're not handling right now) and then in step 3 you'll get child_added events to re-add them.
If you handled the child_removed events, you're items wouldn't end up duplicated, but they would still disappear / reappear, which probably isn't desirable. Another workaround in the current setup is to explicitly tell the transaction to not run with the local estimate, which you can do by passing in false as the third parameter:
function transact() {
var path = 'jobs/';
var ref = firebase.database().ref(path);
ref.transaction(function(post) {
return post;
}, function(error, committed, snapshot) {
if (error) {
console.log('Transaction failed abnormally!', error);
} else if (!committed) {
console.log('Transaction aborted.');
} else {
console.log('Transaction completed.');
}
}, false);
}
I'm sorry I don't have a better solution. But as I said: it's very unlikely that we'll be able to change this behavior in the current generation of SDKs.
I am trying to implement a trigger on an Azure DocumentDb collection, which is supposed to auto-increment a version of a document, which is being inserted. The trigger is created as a pre-trigger.
The challenge I am facing is that collection class doesn't seem to provide a synchronous API for querying data. My plan for the trigger was to query existing documents, get the top version, increment, and assign the +1 value to the document, which is being inserted into the collection. But since the result of the query is only available asynchronously, by that time my trigger is completed and the document is inserted unmodified.
How can I await the query result?
Here is how my current trigger looks like:
// TRIGGER Auto increment version
function autoIncrementVersion() {
var collection = getContext().getCollection();
var request = getContext().getRequest();
var docToCreate = request.getBody();
// Reject documents that do not have a name property by throwing an exception.
if (!docToCreate.Version) {
throw new Error('Document must include a "Version" property.');
}
var lastVersion;
var filter = "SELECT TOP 1 d.Version FROM CovenantsDocuments d ORDER BY d.Version DESC";
var result = collection.queryDocuments(collection.getSelfLink(), filter, {},
function (err, documents, responseOptions) {
if (err) throw new Error("Error: " + err.message);
if (documents.length != 1 || !documents[0]) {
lastVersion = 0;
} else {
lastVersion = documents[0];
}
//By the time we reach this line, our trigger has already completed?
docToCreate.Version = lastVersion + 1;
});
if (!result) throw "Unable to read last version of the document";
}
UPDATE: The issue was with the way I was submitting request. Looks like triggers are not fired by default, their names need to be explicitly provided as an argument to the request.
In my case the trigger wasn't firing until I changed the client code to this:
RequestOptions options = new RequestOptions
{
PreTriggerInclude = new[] { "autoIncrementVersion"}
};
client.CreateDocumentAsync(url, document, options);
It will automatically wait until all pending async operations either complete, fail, or time out before returning. What you have is close. The only thing that I can see is missing is that you never call request.setBody(docToCreate) after you alter docToCreate.
That said, I'm not 100% certain that this approach is safe. All operations inside of a trigger, sproc, or UDF are atomic, but I'm not sure that the combination of a pre-trigger plus a write operation is atomic. The risk is that two simultaneous writes will both run and complete the trigger part which would give them a same .Version. You would probably have to ask the DocumentDB Product Managers to confirm this. They hang out here so they may respond here.
If you find that it's not atomic, then you can move everything (read to find latest version and write) into a stored procedure (sproc).
You might also consider creating a single document whose id you hard code to something like 'LAST_VERSION' to hold the last used version. That means that every write will result in a read + two writes (one for the document and one to update this document), but it may be more efficient than your query + one write approach. You could do all of this in one sproc or you could use a pre-trigger (to fetch the 'LAST_VERSION' + write operation + post-trigger (to update the 'LAST_VERSION' document) depending upon what the Product Managers say about atomicity.
One more caution about your current approach... Make sure the precision of the index on the Version field is set to -1 (Maximum precision).
I'm using Google App Engine with Java and Google Cloud Endpoints. In my JavaScript front end, I'm using this code to handle initialization, as recommended:
var apisToLoad = 2;
var url = '//' + $window.location.host + '/_ah/api';
gapi.client.load('sd', 'v1', handleLoad, url);
gapi.client.load('oauth2', 'v2', handleLoad);
function handleLoad() {
// this only executes once,
if (--apisToLoad === 0) {
// so this is not executed
}
}
How can I detect and handle when gapi.client.load fails? Currently I am getting an error printed to the JavaScript console that says: Could not fetch URL: https://webapis-discovery.appspot.com/_ah/api/static/proxy.html). Maybe that's my fault, or maybe it's a temporary problem on Google's end - right now that is not my concern. I'm trying to take advantage of this opportunity to handle such errors well on the client side.
So - how can I handle it? handleLoad is not executed for the call that errs, gapi.client.load does not seem to have a separate error callback (see the documentation), it does not actually throw the error (only prints it to the console), and it does not return anything. What am I missing? My only idea so far is to set a timeout and assume there was an error if initialization doesn't complete after X seconds, but that is obviously less than ideal.
Edit:
This problem came up again, this time with the message ERR_CONNECTION_TIMED_OUT when trying to load the oauth stuff (which is definitely out of my control). Again, I am not trying to fix the error, it just confirms that it is worth detecting and handling gracefully.
I know this is old but I came across this randomly. You can easily test for a fail (at least now).
Here is the code:
gapi.client.init({}).then(() => {
gapi.client.load('some-api', "v1", (err) => { callback(err) }, "https://someapi.appspot.com/_ah/api");
}, err, err);
function callback(loadErr) {
if (loadErr) { err(loadErr); return; }
// success code here
}
function err(err){
console.log('Error: ', err);
// fail code here
}
Example
Unfortunately, the documentation is pretty useless here and it's not exactly easy to debug the code in question. What gapi.client.load() apparently does is inserting an <iframe> element for each API. That frame then provides the necessary functionality and allows accessing it via postMessage(). From the look of it, the API doesn't attach a load event listener to that frame and rather relies on the frame itself to indicate that it is ready (this will result in the callback being triggered). So the missing error callback is an inherent issue - the API cannot see a failure because no frame will be there to signal it.
From what I can tell, the best thing you can do is attaching your own load event listener to the document (the event will bubble up from the frames) and checking yourself when they load. Warning: While this might work with the current version of the API, it is not guaranteed to continue working in future as the implementation of that API changes. Currently something like this should work:
var framesToLoad = apisToLoad;
document.addEventListener("load", function(event)
{
if (event.target.localName == "iframe")
{
framesToLoad--;
if (framesToLoad == 0)
{
// Allow any outstanding synchronous actions to execute, just in case
window.setTimeout(function()
{
if (apisToLoad > 0)
alert("All frames are done but not all APIs loaded - error?");
}, 0);
}
}
}, true);
Just to repeat the warning from above: this code makes lots of assumptions. While these assumptions might stay true for a while with this API, it might also be that Google will change something and this code will stop working. It might even be that Google uses a different approach depending on the browser, I only tested in Firefox.
This is an extremely hacky way of doing it, but you could intercept all console messages, check what is being logged, and if it is the error message you care about it, call another function.
function interceptConsole(){
var errorMessage = 'Could not fetch URL: https://webapis-discovery.appspot.com/_ah/api/static/proxy.html';
var console = window.console
if (!console) return;
function intercept(method){
var original = console[method];
console[method] = function() {
if (arguments[0] == errorMessage) {
alert("Error Occured");
}
if (original.apply){
original.apply(console, arguments)
}
else {
//IE
var message = Array.prototype.slice.apply(arguments).join(' ');
original(message)
}
}
}
var methods = ['log', 'warn', 'error']
for (var i = 0; i < methods.length; i++)
intercept(methods[i])
}
interceptConsole();
console.log('Could not fetch URL: https://webapis-discovery.appspot.com/_ah/api/static/proxy.html');
//alerts "Error Occured", then logs the message
console.log('Found it');
//just logs "Found It"
An example is here - I log two things, one is the error message, the other is something else. You'll see the first one cause an alert, the second one does not.
http://jsfiddle.net/keG7X/
You probably would have to run the interceptConsole function before including the gapi script as it may make it's own copy of console.
Edit - I use a version of this code myself, but just remembered it's from here, so giving credit where it's due.
I use a setTimeout to manually trigger error if the api hasn't loaded yet:
console.log(TAG + 'api loading...');
let timer = setTimeout(() => {
// Handle error
reject('timeout');
console.error(TAG + 'api loading error: timeout');
}, 1000); // time till timeout
let callback = () => {
clearTimeout(timer);
// api has loaded, continue your work
console.log(TAG + 'api loaded');
resolve(gapi.client.apiName);
};
gapi.client.load('apiName', 'v1', callback, apiRootUrl);