I create a simple queue-job system with using BullQueue in TypeScript and NestJS like below:
async addToQueue(): Promise<void> {
try {
await this.reportQueue.add('generate_report', {
//some data
})
this.logger.log(`Added to queue.`)
} catch (error) {
this.logger.error(`Not added to Queue.`);
}
}
#Process('generate_report')
async generateReport(job: Job<{options: ReportFilterDto, context: CustomContext, reportType: Type, worktime?: WorktimeDto}>): Promise<any> {
try {
//some working code
} catch (error) {
//
}
}
#OnQueueActive()
async onActive(job: Job): Promise<void> {
//
}
#OnQueueCompleted()
async onCompleted(job: Job): Promise<void> {
//
}
#OnQueueWaiting()
async onWaiting(job: Job): Promise<void> {
//
}
#OnQueueFailed()
onError(job: Job<any>, error: BadRequestException): void {
//
}
i run my function addToQueue() from my controller where i provide a parameters, buuuuuut
it is possible to return any response to the client from the queue about sucessfull or failed job?
If it is possible can someone show how to do it?
thanks for any help
.////////////////////////.
Immediately, there isn't really any way. A queue is there so you can run asynchronous tasks without blocking your main thread or holding up your server in any way. If you were to want to return the result, depending on the job the queue is running, you could be waiting for a few seconds to a few days (a bit of an exaggeration, but it gets the point across). What you could do instead is one of two things (at least that come immediately to mind)
return an Id that you create that relates back to this job. When the job finishes, save the result with that Id to the database. Then, the user can request the status at any time.
Take note of the user who sent in the request, and send an email to that user once the job completes. This will require more integrations, but can get more accurate results over "Ping every so often and we'll tell you if it's done".
Both approaches are valid, and you could even make a combination of the two (probably the best of both worlds so you can save the results and alert the user when it's done).
Related
in my Javascript application the user can press a button, in order to continuously fetch data from a server.
Pressing the button triggers a function, which performs a HTTP request to the server. After the request is completed successfully, this function calls itself recursively again and again. The function is escaped if the user presses the button again or a specific limit is reached.
Now my problem is, that by each recursion the delay between each request gets bigger. This can be visualized in the screenshot I took from the developer tools of my browser:
The blue lines are my sets of requests in each function. The white empty slots between the sets of requests get longer.
This is how my code is structured:
async function startContinousFetch() {
addContinousData(true);
}
async function addContinousData() {
addNewData().then((resp) => {
if (LOGSLIMIT != null && Logs.length > LOGSLIMIT) {
stopContinousFetch();
return;
}
//Unexpected error, promise didn't resolve with true;
if (resp == false) {
//Error handling
}
if (continousFetch) {
addContinousData();
}
})
}
function addNewData() {
return new Promise(async resolve => {
try {
//Various http requests are here
await fetchData();
//Add, organize and manipulate data in the global "Logs" array
handleLogs();
//Resolve promise
resolve(true)
} catch (e) {
resolve(false, e)
}
})
}
Why do the delays get longer? What is my mistake here?
It is clear to me that my function waits till the response, so there is a delay. But why does this delay get bigger every time.
In the dev tools, the request time does not get any longer, just the time in between.
Thanks a lot.
I have the following setup:
async MyFunction(param) {
//... Do some computation
await WriteToDB()
}
io.on('connection', (socket) => {
socket.on('AnEvent', (param) => MyFunction(param))
})
When an event comes in, it calls an asynchronous function which does some computation and in the end write the result to a database with another asynchronous call.
If MyFunction doesn't have an asynchronous call to write the database in the end, for example
MyFunction(param) {
//... Do some computation
}
then it is obviously that all events will be processed in their incoming order. The processing of the next event will only start after the processing of the previous one finishes. However, because of the asynchronous call to the database, I don't know if those incoming events will still be fully processed in order. I am afraid that the processing of the next event starts before the previous await WriteToDB() finishes. How do I change the code to fully processing them in order?
You are correct that there's no guarantee that incoming events will be processed in the order.
To achieve what you are asking, you would need a "Message Queue" that will periodically check for new messages and process them one by one.
const messageQueue = [];
// SocketIO adding Message to MessageQueue
const eventHandler = (message) => {
messageQueue.push(message);
}
const messageHandler = () => {
if (messageQueue.length === 0) {
return;
}
const message = messageQueue.shift();
// Handle Message
// If successful, ask for next message
return messageHandler();
}
Of course, my example is pretty naive, but I hope it will give you a general idea of how what you are asking is accomplished.
If you are finding yourself needing a more robust message queue, Look into RabbitMQ, BullMQ, Kafka
I am using node module pg in my application and I want to make sure it can properly handle connection and query errors.
The first problem I have is I want to make sure it can properly recover when postgres is unavailable.
I found there is an error event so I can detect if there is a connection error.
import pg from 'pg'
let pgClient = null
async function postgresConnect() {
pgClient = new pg.Client(process.env.CONNECTION_STRING)
pgClient.connect()
pgClient.on('error', async (e) => {
console.log('Reconnecting')
await sleep(5000)
await postgresConnect()
})
}
I don't like using a global here, and I want to set the sleep delay to do an small exponential backoff. I noticed "Reconnecting" fires twice immediately, then waits five seconds and I am not sure why it fired the first time without any waiting.
I also have to make sure the queries execute. I have something like this I was trying out.
async function getTimestamp() {
try {
const res = await pgClient.query(
'select current_timestamp from current_timestamp;'
)
return res.rows[0].current_timestamp
} catch (error) {
console.log('Retrying Query')
await sleep(1000)
return getTimestamp()
}
}
This seems to work, but I haven't tested it enough to make sure it will guarantee the query is executed or keep trying. I should look for specific errors and only loop forever on certain errors and fail on others. I need to do more research to find what errors are thrown. I also need to do a backoff on the delay here too.
It all "seems" to work, I don't want to fail victim to the Dunning-Kruger effect. I need to ensure this process can handle all sorts of situations and recover.
Nothing happens to my firestore when I call the function below. I also can't see "inside helloWorld" on my GCP logs.
exports.helloWorld = functions.https.onCall((data, context) => {
console.log("inside helloWorld);
const users = admin.firestore().collection('users');
users.doc(data.userId).get().then( // --------------------Line A
(snapshot) => {
if (snapshot.exists) {
console.log("snapshot exists");
return null;
} else {
console.log("inside else");
users.doc(data.userId).set({
name: data.name
});
return null;
}
}
).catch(() => 'obligatory catch');
return; //-----------------------------------------------Line B
});
However, when I place the return on Line A, the function works as expected and a document is created in my firestore. "inside helloWorld" is shown on my GCP logs.
Why is that so?
I really appreciate any levels of clarification.
According to the documentation for callable functions (particularly the part about sending back a result):
To return data after an asynchronous operation, return a promise. The
data returned by the promise is sent back to the client. For example,
you could return sanitized text that the callable function wrote to
the Realtime Database.
Even if you don't want to send any content in the response, Callable functions still need to respond to the HTTP request that originated them, as all HTTP transactions do.
In either case, you still need to make use of the promises in all your async calls so that Cloud Functions knows when to respond to the client, after all the work is complete. Placing the return statement on "line A" is effectively returning the promise from the async work started by get(), fulfilling this requirement. Without the return statement, Cloud Functions is terminated your function because it thinks there is no more work to complete in order to send the final response.
If you're not familiar about how promises work in JavaScript, watch my video tutorials here: https://firebase.google.com/docs/functions/video-series/
I've tried to read up on async methods and am now trying to create my own async method. The method is a webservice call that returns a list of error logs. I'm not sure that I've understood correctly so I thought I'd share my code to see if I should do anything different.
All I want the code to do is return a list of errorlogs by calling a method GetAllErrorLogs(), that is a synchronized method. Since it can take a second to fetch all the error logs I want to have the opportunity to do other stuff once I called the GetAllErrorLogs() method. Here is the code.
[WebMethod]
public async Task<List<ErrorLog>> GetAllErrorLogs()
{
List<ErrorLog> errorLogs = new List<ErrorLog>();
await System.Threading.Tasks.Task.Run(() => {
errorLogs = ErrorLogRepository.GetAllErrorLogs();
});
if (errorLogs == null)
return new List<ErrorLog>();
return errorLogs;
}
Thanks!
I recently gave a talk at ThatConference on async on the server side, and I address this issue in the slides.
On the server side, you want to avoid the use of Task.Run and other constructs that queue work to the thread pool. As much as possible, keep thread pool threads available for handling requests.
So, ideally your repository would have an asynchronous method GetAllErrorLogsAsync, which would itself be asynchronous. If GetAllErrorLogs cannot be asynchronous, then you may as well just call it directly (removing the await Task.Run).
Since it can take a second to fetch all the error logs I want to have the opportunity to do other stuff once I called the GetAllErrorLogs() method.
If you have a GetAllErrorLogsAsync available, then this can easily be done using Task.WhenAll. However, if GetAllErrorLogs is synchronous, then you can only do this by doing parallel work in your request (e.g., multiple calls to Task.Run followed by Task.WhenAll).
Parallel code on the server must be approached with great trepidation. It is only acceptable in a very limited set of scenarios. The entire point of async on the server side is to use fewer threads per request, and when you start parallelizing, you're doing the opposite: multiple threads per request. This is only appropriate if you know your user base is very small; otherwise, you'll kill your server scalability.
I found this great codeproject detailed article about how to achieve that
http://www.codeproject.com/Articles/600926/Asynchronous-web-services-call-in-ASP-NET
**This is potentially wrong, read comments or spinoff question at HttpContext.Current after an await
If ErrorLogRepository.GetAllErrorLogs() is not thread-safe, it will cause weird bugs and potentially exception out. Make sure your code is ready for multi-threaded operation before switching to async methods, this is obviously very trivial advice but often overlooked. For example, if you reference HttpContext.Current in your methods, your code will die in the async method, and sometimes even AFTER the await. The reason is that the code within the async block will potentially be run on a separate thread, which will not have access to the same HttpContext.Current thread-static property, and await gets compiled into two methods. All code before an await gets run on one thread, and then calls the code after an await keyword as a continuation, but potentially on yet another thread. So sometimes your code will even work in an async block, only to choke unexpectedly after it gets "out" of the async back to what you think is a synchronous part of your code (but in reality everything after an await keyword is already not guaranteed to be the original thread).
Here is some production code...
using System.Web.Http;
using AysncTask = System.Threading.Tasks.Task;
public class myController : ApiControllerBase
{
[HttpPut]
[Route("api/cleardata/{id}/{requestId}/")]
public async AysncTask ClearData(Guid id, Guid requestId)
{
try
{
await AysncTask.Run(() => DoClearData(id, requestId));
}
catch (Exception ex)
{
throw new Exception("Exception in myController.ClearData", ex);
}
}
}
Handling Async exceptions is also VERY VERY important.. although this is for a windows console app, the same principles should apply.
source: https://blogs.msdn.microsoft.com/ptorr/2014/12/10/async-exceptions-in-c/
using System;
using System.Runtime.CompilerServices;
using System.Threading;
using System.Threading.Tasks;
namespace AsyncAndExceptions
{
class Program
{
static void Main(string[] args)
{
AppDomain.CurrentDomain.UnhandledException += (s, e) => Log("*** Crash! ***", "UnhandledException");
TaskScheduler.UnobservedTaskException += (s, e) => Log("*** Crash! ***", "UnobservedTaskException");
RunTests();
// Let async tasks complete...
Thread.Sleep(500);
GC.Collect(3, GCCollectionMode.Forced, true);
}
private static async Task RunTests()
{
try
{
// crash
// _1_VoidNoWait();
// crash
// _2_AsyncVoidAwait();
// OK
// _3_AsyncVoidAwaitWithTry();
// crash - no await
// _4_TaskNoWait();
// crash - no await
// _5_TaskAwait();
// OK
// await _4_TaskNoWait();
// OK
// await _5_TaskAwait();
}
catch (Exception ex) { Log("Exception handled OK"); }
// crash - no try
// await _4_TaskNoWait();
// crash - no try
// await _5_TaskAwait();
}
// Unsafe
static void _1_VoidNoWait()
{
ThrowAsync();
}
// Unsafe
static async void _2_AsyncVoidAwait()
{
await ThrowAsync();
}
// Safe
static async void _3_AsyncVoidAwaitWithTry()
{
try { await ThrowAsync(); }
catch (Exception ex) { Log("Exception handled OK"); }
}
// Safe only if caller uses await (or Result) inside a try
static Task _4_TaskNoWait()
{
return ThrowAsync();
}
// Safe only if caller uses await (or Result) inside a try
static async Task _5_TaskAwait()
{
await ThrowAsync();
}
// Helper that sets an exception asnychronously
static Task ThrowAsync()
{
TaskCompletionSource tcs = new TaskCompletionSource();
ThreadPool.QueueUserWorkItem(_ => tcs.SetException(new Exception("ThrowAsync")));
return tcs.Task;
}
internal static void Log(string message, [CallerMemberName] string caller = "")
{
Console.WriteLine("{0}: {1}", caller, message);
}
}
}