We Keep Coding

ClearInterval js stop current execution

I have a Vue 2.0 app in which I use this line in order to call this.refreshState() every min.
this.scheduler = setInterval(() => this.refreshState(), 60 * 1000)
Later in the code I need to make sure that the execution loop is stopped and also that if there's an instance of this.refreshState() currently running (from the setInterval scheduler) it's stopped as well (even if it's in the middle of doing stuff).
So far I'm using :
clearInterval(this.scheduler)
as per (https://developer.mozilla.org/en-US/docs/Web/API/clearInterval)
The question I'm having is does clearInterval blocks the current execution if any? I can't find the answer in the doc unfortunately.
FYI the code of refreshState:
refreshState: function () {
// API call to backend
axios.get("/api/refreshState")
.then(response => {
this.states = response.data.states
})
.catch((err) => console.log(err)
}
Here's my use case :
alterState: function (incremental_state) {
clearInterval(this.scheduler) // ???
axios.post("/api/alterState", incremental_state)
.then(() => {
this.refreshState()
this.scheduler = setInterval(() => this.refreshState(), 60 * 1000)
})
.catch((err) => { console.log(error) })
}
I want to make sure that when i exit alterState , the variable this.states takes into account the addition of incremental state.

From...
I want to make sure that when i exit alterState, the variable this.states takes into account the addition of incremental state.
...I understand you're performing a change on backend and you want it reflected on frontend. And that currently that doesn't happen, although you're calling this.refreshState() right after getting a successful response from /api/alterState. 1
To achieve this functionality, it's not enough to call this.refreshState(), because your browser, by default, caches the result (it remembers the recent calls and their results, so it serves the previous result from cache, instead of calling the server again), unless the endpoint is specifically configured to disable caching.
To disable caching for a particular endpoint, you could either
configure the endpoint (server side) to tell browsers: "Hey, my stuff is time sensitive, don't cache it!" (won't go into how, as I have no idea what technology you're using on backend and it varies). Roughly it means setting appropriate response headers.
or call the endpoint with a unique param, each time. This makes the endpoint "change" from browser's POV, so it's always going to request from server:
axios
.get(`/api/refreshState?v=${Date.now()}`)
.then...
I recommend the second option, it's reliable, predictable and does not depend on server configuration.
And, unless something else, other than the current app instance (some other user, or other server scripts, etc...) make changes to the data, you don't actually need a setInterval. I suggest removing it.
But If you do have other sources changing server-side data, (and you do want to refresh it regardless of user interactions with the app), what you have works perfectly fine, there's no need to even cancel the existing interval when you make a change + refreshState()). 2
1 - if I misunderstood your question and that is not your problem, please clarify your question, right now it's a bit unclear
2 - as side-note and personal preference, I suggest renaming refreshState() to getState()

How to cancel a wasm process from within a webworker

I have a wasm process (compiled from c++) that processes data inside a web application. Let's say the necessary code looks like this:
std::vector<JSONObject> data
for (size_t i = 0; i < data.size(); i++)
{
process_data(data[i]);
if (i % 1000 == 0) {
bool is_cancelled = check_if_cancelled();
if (is_cancelled) {
break;
}
}
}
This code basically "runs/processes a query" similar to a SQL query interface:
However, queries may take several minutes to run/process and at any given time the user may cancel their query. The cancellation process would occur in the normal javascript/web application, outside of the service Worker running the wasm.
My question then is what would be an example of how we could know that the user has clicked the 'cancel' button and communicate it to the wasm process so that knows the process has been cancelled so it can exit? Using the worker.terminate() is not an option, as we need to keep all the loaded data for that worker and cannot just kill that worker (it needs to stay alive with its stored data, so another query can be run...).
What would be an example way to communicate here between the javascript and worker/wasm/c++ application so that we can know when to exit, and how to do it properly?
Additionally, let us suppose a typical query takes 60s to run and processes 500MB of data in-browser using cpp/wasm.
Update: I think there are the following possible solutions here based on some research (and the initial answers/comments below) with some feedback on them:
Use two workers, with one worker storing the data and another worker processing the data. In this way the processing-worker can be terminated, and the data will always remain. Feasible? Not really, as it would take way too much time to copy over ~ 500MB of data to the webworker whenever it starts. This could have been done (previously) using SharedArrayBuffer, but its support is now quite limited/nonexistent due to some security concerns. Too bad, as this seems like by far the best solution if it were supported...
Use a single worker using Emterpreter and using emscripten_sleep_with_yield. Feasible? No, destroys performance when using Emterpreter (mentioned in the docs above), and slows down all queries by about 4-6x.
Always run a second worker and in the UI just display the most recent. Feasible? No, would probably run into quite a few OOM errors if it's not a shared data structure and the data size is 500MB x 2 = 1GB (500MB seems to be a large though acceptable size when running in a modern desktop browser/computer).
Use an API call to a server to store the status and check whether the query is cancelled or not. Feasible? Yes, though it seems quite heavy-handed to long-poll with network requests every second from every running query.
Use an incremental-parsing approach where only a row at a time is parsed. Feasible? Yes, but also would require a tremendous amount of re-writing the parsing functions so that every function supports this (the actual data parsing is handled in several functions -- filter, search, calculate, group by, sort, etc. etc.
Use IndexedDB and store the state in javascript. Allocate a chunk of memory in WASM, then return its pointer to JavaScript. Then read database there and fill the pointer. Then process your data in C++. Feasible? Not sure, though this seems like the best solution if it can be implemented.
[Anything else?]
In the bounty then I was wondering three things:
If the above six analyses seem generally valid?
Are there other (perhaps better) approaches I'm missing?
Would anyone be able to show a very basic example of doing #6 -- seems like that would be the best solution if it's possible and works cross-browser.

For Chrome (only) you may use shared memory (shared buffer as memory). And raise a flag in memory when you want to halt. Not a big fan of this solution (is complex and is supported only in chrome). It also depends on how your query works, and if there are places where the lengthy query can check the flag.
Instead you should probably call the c++ function multiple times (e.g. for each query) and check if you should halt after each call (just send a message to the worker to halt).
What I mean by multiple time is make the query in stages (multiple function cals for a single query). It may not be applicable in your case.
Regardless, AFAIK there is no way to send a signal to a Webassembly execution (e.g. Linux kill). Therefore, you'll have to wait for the operation to finish in order to complete the cancellation.
I'm attaching a code snippet that may explain this idea.
worker.js:
... init webassembly
onmessage = function(q) {
// query received from main thread.
const result = ... call webassembly(q);
postMessage(result);
}
main.js:
const worker = new Worker("worker.js");
const cancel = false;
const processing = false;
worker.onmessage(function(r) {
// when worker has finished processing the query.
// r is the results of the processing.
processing = false;
if (cancel === true) {
// processing is done, but result is not required.
// instead of showing the results, update that the query was canceled.
cancel = false;
... update UI "cancled".
return;
}
... update UI "results r".
}
function onCancel() {
// Occurs when user clicks on the cancel button.
if (cancel) {
// sanity test - prevent this in UI.
throw "already cancelling";
}
cancel = true;
... update UI "canceling".
}
function onQuery(q) {
if (processing === true) {
// sanity test - prevent this in UI.
throw "already processing";
}
processing = true;
// Send the query to the worker.
// When the worker receives the message it will process the query via webassembly.
worker.postMessage(q);
}
An idea from user experience perspective:
You may create ~two workers. This will take twice the memory, but will allow you to "cancel" "immediately" once. (it will just mean that in the backend the 2nd worker will run the next query, and when the 1st finishes the cancellation, cancellation will again become immediate).

Shared Thread
Since the worker and the C++ function that it called share the same thread, the worker will also be blocked until the C++ loop is finished, and won't be able to handle any incoming messages. I think the a solid option would minimize the amount of time that the thread is blocked by instead initializing one iteration at a time from the main application.
It would look something like this.
main.js -> worker.js -> C++ function -> worker.js -> main.js
Breaking up the Loop
Below, C++ has a variable initialized at 0, which will be incremented at each loop iteration and stored in memory.
C++ function then performs one iteration of the loop, increments the variable to keep track of loop position, and immediately breaks.
int x;
x = 0; // initialized counter at 0
std::vector<JSONObject> data
for (size_t i = x; i < data.size(); i++)
{
process_data(data[i]);
x++ // increment counter
break; // stop function until told to iterate again starting at x
}
Then you should be able to post a message to the web worker, which then sends a message to main.js that the thread is no longer blocked.
Canceling the Operation
From this point, main.js knows that the web worker thread is no longer blocked, and can decide whether or not to tell the web worker to execute the C++ function again (with the C++ variable keeping track of the loop increment in memory.)
let continueOperation = true
// here you can set to false at any time since the thread is not blocked here
worker.expensiveThreadBlockingFunction()
// results in one iteration of the loop being iterated until message is received below
worker.onmessage = function(e) {
if (continueOperation) {
worker.expensiveThreadBlockingFunction()
// execute worker function again, ultimately continuing the increment in C++
} {
return false
// or send message to worker to reset C++ counter to prepare for next execution
}
}
Continuing the Operation
Assuming all is well, and the user has not cancelled the operation, the loop should continue until finished. Keep in mind you should also send a distinct message for whether the loop has completed, or needs to continue, so you don't keep blocking the worker thread.

Error: 10 ABORTED: Too much contention on these documents. Please try again

What does this error mean?
Especially, what do they mean by : Please try again
Does it mean that the transaction failed I have to re-run the transaction manually?
From what I understood from the documentation,
The transaction read a document that was modified outside of the
transaction. In this case, the transaction automatically runs again.
The transaction is retried a finite number of times.
If so, on which documents?
The error do not indicate which document it is talking about. I just get this stack:
{ Error: 10 ABORTED: Too much contention on these documents. Please
try again.
at Object.exports.createStatusErrornode_modules\grpc\src\common.js:87:15)
at ClientReadableStream._emitStatusIfDone \node_modules\grpc\src\client.js:235:26)
at ClientReadableStream._receiveStatus \node_modules\grpc\src\client.js:213:8)
at Object.onReceiveStatus \node_modules\grpc\src\client_interceptors.js:1256:15)
at InterceptingListener._callNext node_modules\grpc\src\client_interceptors.js:564:42)
at InterceptingListener.onReceiveStatus\node_modules\grpc\src\client_interceptors.js:614:8)
at C:\Users\Tolotra Samuel\PhpstormProjects\CryptOcean\node_modules\grpc\src\client_interceptors.js:1019:24
code: 10, metadata: Metadata { _internal_repr: {} }, details: 'Too
much contention on these documents. Please try again.' }
To recreate this error, just run a for loop on the db.runTransaction method as indicated on the documentation

We run into the same problem with the Firebase Firestore database. Even small counters with less then 30 items to cound where running into this issue.
Our solution was not to distribute the counter but to increase the number of tries for the transaction and to add a deffer time for those retries.
The first step was to save the transaction action as const witch could be passed to another function.
const taskCountTransaction = async transaction => {
const taskDoc = await transaction.get(taskRef)
if (taskDoc.exists) {
let increment = 0
if (change.after.exists && !change.before.exists) {
increment = 1
} else if (!change.after.exists && change.before.exists) {
increment = -1
}
let newCount = (taskDoc.data()['itemsCount'] || 0) + increment
return await transaction.update(taskRef, { itemsCount: newCount > 0 ? newCount : 0 })
}
return null
}
The second step was to create two helper functions. One for waiting a specifix amount of time and the other one to run the transaction and catch errors. If the abort error with the code 10 occurs we just run the transaction again for a specific amount of retries.
const wait = ms => { return new Promise(resolve => setTimeout(resolve, ms))}
const runTransaction = async (taskCountTransaction, retry = 0) => {
try {
await fs.runTransaction(taskCountTransaction)
return null
} catch (e) {
console.warn(e)
if (e.code === 10) {
console.log(`Transaction abort error! Runing it again after ${retry} retries.`)
if (retry < 4) {
await wait(1000)
return runTransaction(taskCountTransaction, ++retry)
}
}
}
}
Now that we have all we need we can just call our helper function with await and our transaction call will run longer then a default one and it will deffer in time.
await runTransaction(taskCountTransaction)
What I like about this solution is that it doesn't mean more or complicated code and that most of the already written code can stay as it is. It also uses more time and resources only if the counter gets to the point that it has to count more items. Othervise the time and resources are the same as if you would have the default transactions.
For scaling up for large amounts of items we can increase eather the amount of retries or the waiting time. Both are also affecting the costs for Firebase. For the waiting part we also need to increase the timeout for our function.
DISCLAIMER: I have not stress tested this code with thousands or more of items. In our specific case the problems started with 20+ items and we need up to 50 items for a task. I tested it with 200 items and the problem did not apear again.

The transaction does run several times if needed, but if the values read continue to be updated before the write or writes can occur it will eventually fail, thus the documentation noting the transaction is retried a finite number of times. If you have a value that is updating frequently like a counter, consider other solutions like distributed counters. If you'd like more specific suggestions, I recommend you include the code of your transaction in your question and some information about what you're trying to achieve.

Firestore re-runs the transaction only a finite number of times. As of writing, this number is hard-coded as 5, and cannot be changed. To avoid congestion/contention when many users are using the same document, normally we use the exponential back-off algorithm (but this will result in transactions taking longer to complete, which may be acceptable in some use cases).
However, as of writing, this has not been implemented in the Firebase SDK yet — transactions are retried right away. Fortunately, we can implement our own exponential back-off algorithm in a transaction:
const createTransactionCollisionAvoider = () => {
let attempts = 0
return {
async avoidCollision() {
attempts++
await require('delay')(Math.pow(2, attempts) * 1000 * Math.random())
}
}
}
…which can be used like this:
// Each time we run a transaction, create a collision avoider.
const collisionAvoider = createTransactionCollisionAvoider()
db.runTransaction(async transaction => {
// At the very beginning of the transaction run,
// introduce a random delay. The delay increases each time
// the transaction has to be re-run.
await collisionAvoider.avoidCollision()
// The rest goes as normal.
const doc = await transaction.get(...)
// ...
transaction.set(...)
})
Note: The above example may cause your transaction to take up to 1.5 minutes to complete. This is fine for my use case. You might have to adjust the backoff algorithm for your use case.

I have implemented a simple back-off solution to share : maintain a global variable that assigns a different "retry slot" to each failed connection. For example if 5 connections came at the same time and 4 of them got a contention error, each would get a delay of 500ms, 1000ms, 1500ms, 2000ms until trying again, for example. So it could potentially all resolved at the same time without any more contention.
My transaction is a response of calling Firebase Functions. Each Functions computer instance could have a global variable nextRetrySlot that is preserved until it is shut down. So if error.code === 10 is caught for contention issue, the delay time can be (nextRetrySlot + 1) * 500 then you could for example nextRetrySlot = (nextRetrySlot + 1) % 10 so next connections get a different time round-robin in 500ms ~ 5000ms range.
Below are some benchmarks :
My situation is that I would like each new Firebase Auth registration to get a much shorter ID derived from unique Firebase UID, thus it has risk of collision.
My solution is simply to check all registered short ID and if the query returns something, just generate an another one until it is not. Then we register this new short ID to the database. So the algorithm cannot rely on only Firebase UID, but it is able to "move to the next one" in a deterministic way. (not just random again).
This is my transaction, it first read a database of all used short ID then write a new one atomically, to prevent an extremely unlikely event that 2 new registers came at the same time, with a different Firebase UID that derived into the same short ID, and both see that the short ID is vacant at the same time.
I run a test that intentionally register 20 different Firebase UIDs which all derived into the same short ID. (extremely unlikely situation) All that runs in burst at the same time. First I tried using the same delay on next retry, so I expect it to clash with each other again and again while slowly resolving some connections.
Same 500ms delay on retry : 45000ms ~ 60000ms
Same 1000ms delay on retry : 30000ms ~ 49000ms
Same 1500ms delay on retry : 43000ms ~ 49000ms
Then with distributed delay time in slots :
500ms * 5 slots on retry : 20000ms ~ 31000ms
500ms * 10 slots on retry : 22000ms ~ 23000ms
500ms * 20 slots on retry : 19000ms ~ 20000ms
1000ms * 5 slots on retry : ~29000ms
1000ms * 10 slots on retry : ~25000ms
1000ms * 20 slots on retry : ~26000ms
Confirming that different delay time definitely helps.

Found maxAttempts in the runTransaction code which should modify the 5 default attempts (but didn't tested yet).
Anyway, I think that random wait (plus eventually the queue) are still the better option.

Firestore now supports server-side increment() and decrement() atomic operations.
You can increment or decrement by any amount. See their blog post for full details. In many cases, this will remove the need for a client side transaction.
Example:
document("fitness_teams/Team_1").
updateData(["step_counter" : FieldValue.increment(500)])
This is still limited to a sustained write limit of 1 QPS per document so if you need higher throughput, consider using distributed counters. This will increase your read cost (as you'll need to read all the shard documents and compute a total) but allow you to scale your throughput by increasing the number of shards. Now, if you do need to increment a counter as part of a transaction, it's much less likely to fail due to update contention.

How to avoid multiple node processes doing repetitive things?

I have a module in Node.js which repeatedly pick a document from MongoDB and process it. One document should be processed only once. I also want to use multiple processes concept. I want to run the same module(process) on different processors, which run independently.
The problem is, there might be a scenario where the same document picked and processed by two different workers. How multiple processes can know that, a particular document is processed by some other worker so I should not touch it. And there is no way that my independent processes can communicate. I cannot use a parent which forks multiple processes and acts as a bridge between them. How to avoid this kind of problems in Node.js?

One way to do it is to assign an unique numeric ID to each of your MongoDB documents, and to assign an unique numeric identifier to each of your node.js workers.
For example, have an env var called NUM_WORKERS, and then in your node.js module:
var NumWorkers = process.env.NUM_WORKERS || 1;
You then need to assign an unique, contiguous instance number id (in the range 0 to NumWorkers-1) to each of your workers (e.g. via a command line parameter read by your node.js process when it initializes). You can store that in a variable called MyWorkerInstanceNum.
When you pick a document from MongoDB, call the following function (passing the document's unique documentId as a parameter):
function isMine(documentId){
//
// Example: documentId=10
// NumWorkers= 4
// (10 % 4) = 2
// If MyWorkerInstanceNum is 2, return true, else return false.
return ((documentId % NumWorkers) === MyWorkerInstanceNum);
}
Only continue to actually process the document if isMine() returns true.
So, multiple workers may "pick" a document, but only one worker will actually process it.

Simply keep a transaction log of the document being processed by its unique ID. In the transaction log table for the processed documents, write the status as one of the following (for example):
requested
initiated
processed
failed
You may also want a column in that table for stderr/stdout in case you want to know why something failed or succeeded, and timestamps - that sort of thing.
When you initialize the processing of the document in your Node app, look up the document by ID and check its status. If it doesn't exist, then you're free to process it.
Pseudo-code (sorry, I'm not a Mongo guy!):
db.collection.list('collectionName', function(err, doc) {
db.collection.find(doc.id, 'transactions', function(err, trx) {
if (trx === undefined || trx.status === 'failed') {
DocProcessor.child.process(doc)
} else {
// don't need to process it, it's already been done
}
})
})
You'll also want to enable concurrency locking on the transactions log collection so that you ensure a row (and subsequent job) can't be duplicated. If this becomes a challenge to ensure docs are being queued properly, consider adding in an AMQP service to handle queuing of the docs. Set up a handler to manage distribution of the child processes and transaction logging. Flow would be something like:
MQ ⇢ Log ⇢ Handler ⇢ Doc processor children

How to synchronously read data from Azure DocumentDb collection inside a server-side trigger?

I am trying to implement a trigger on an Azure DocumentDb collection, which is supposed to auto-increment a version of a document, which is being inserted. The trigger is created as a pre-trigger.
The challenge I am facing is that collection class doesn't seem to provide a synchronous API for querying data. My plan for the trigger was to query existing documents, get the top version, increment, and assign the +1 value to the document, which is being inserted into the collection. But since the result of the query is only available asynchronously, by that time my trigger is completed and the document is inserted unmodified.
How can I await the query result?
Here is how my current trigger looks like:
// TRIGGER Auto increment version
function autoIncrementVersion() {
var collection = getContext().getCollection();
var request = getContext().getRequest();
var docToCreate = request.getBody();
// Reject documents that do not have a name property by throwing an exception.
if (!docToCreate.Version) {
throw new Error('Document must include a "Version" property.');
}
var lastVersion;
var filter = "SELECT TOP 1 d.Version FROM CovenantsDocuments d ORDER BY d.Version DESC";
var result = collection.queryDocuments(collection.getSelfLink(), filter, {},
function (err, documents, responseOptions) {
if (err) throw new Error("Error: " + err.message);
if (documents.length != 1 || !documents[0]) {
lastVersion = 0;
} else {
lastVersion = documents[0];
}
//By the time we reach this line, our trigger has already completed?
docToCreate.Version = lastVersion + 1;
});
if (!result) throw "Unable to read last version of the document";
}
UPDATE: The issue was with the way I was submitting request. Looks like triggers are not fired by default, their names need to be explicitly provided as an argument to the request.
In my case the trigger wasn't firing until I changed the client code to this:
RequestOptions options = new RequestOptions
{
PreTriggerInclude = new[] { "autoIncrementVersion"}
};
client.CreateDocumentAsync(url, document, options);

It will automatically wait until all pending async operations either complete, fail, or time out before returning. What you have is close. The only thing that I can see is missing is that you never call request.setBody(docToCreate) after you alter docToCreate.
That said, I'm not 100% certain that this approach is safe. All operations inside of a trigger, sproc, or UDF are atomic, but I'm not sure that the combination of a pre-trigger plus a write operation is atomic. The risk is that two simultaneous writes will both run and complete the trigger part which would give them a same .Version. You would probably have to ask the DocumentDB Product Managers to confirm this. They hang out here so they may respond here.
If you find that it's not atomic, then you can move everything (read to find latest version and write) into a stored procedure (sproc).
You might also consider creating a single document whose id you hard code to something like 'LAST_VERSION' to hold the last used version. That means that every write will result in a read + two writes (one for the document and one to update this document), but it may be more efficient than your query + one write approach. You could do all of this in one sproc or you could use a pre-trigger (to fetch the 'LAST_VERSION' + write operation + post-trigger (to update the 'LAST_VERSION' document) depending upon what the Product Managers say about atomicity.
One more caution about your current approach... Make sure the precision of the index on the Version field is set to -1 (Maximum precision).