I have a google cloud function that sends notifications to a firebase topic.
The function was working fine till suddenly, it start to send more than one notification 2 or 3 at the same time. After contacting the Firebase support team, they told may I should make the function Idempotent, but I don't know how, since it's a callable function.
for more details, this is a reference question containing more detail about the case.
below is the function's code.
UPDATE 2
it was a bug in the admin sdk and they resolved it in the last release.
UPDATE
the function is already idempotent because it is an event driven function
the link above contains the functions log as prof it runs only once.
after 2 month on go and back it appears the problem with firebase admin sdk
the function code getMessaging().sendToTopic() has retry 4 times and the origin request so its 5 times by default before throwing error and terminate the function. So the reason of duplicate notification is that the admin sdk from time to time cant reach the FCM server for some reason.it try to send notification to all subs but in half way or before send all notification it get error so it retry again from the beginning so some users receives one notification and some get 2, 3,4.
And Now the question is how to prevent these default retries or how to make the retry continue from where it get the error. probably Ill ask a separated question.
For now I did a naive solution by prevent the duplicate notification from the receiver( mobile client). if it get more than one notification has same content within a minute show only one.
const functions = require("firebase-functions");
// The Firebase Admin SDK to access Firestore.
const admin = require("firebase-admin");
const {getMessaging} = require("firebase-admin/messaging");
const serviceAccount = require("./serviceAccountKey.json");
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: "https://mylinktodatabase.firebaseio.com",
});
exports.callNotification = functions.https.onCall( (data) => {
// Grab the text parameter.
const indicator = data.indicator;
const mTitle = data.title;
const mBody = data.body;
// topic to send to
const topic = "mytopic";
const options = {
"priority": "high",
"timeToLive": 3600,
};
let message;
if (indicator != null ) {
message = {
data: {
ind: indicator,
},
};
} else {
message = {
data: {
title: mTitle,
body: mBody,
},
};
}
// Send a message to devices subscribed to the provided topic.
return getMessaging().sendToTopic(topic, message, options)
.then(() => {
if (indicator != null ) {
console.log("Successfully sent message");
return {
result: "Successfully sent message", status: 200};
} else {
console.log("Successfully sent custom");
return {
result: "Successfully sent custom", status: 200};
}
})
.catch((error) => {
if (indicator != null ) {
console.log("Error sending message:", error);
return {result: `Error sending message: ${error}`, status: 500};
} else {
console.log("Error sending custom:", error);
return {result: `Error sending custom: ${error}`, status: 500};
}
});
});
In this blog Cloud Functions pro tips: Building idempotent functions, shows how to do a function idempotent using two approaches:
Use your event IDs
One way to fix this is to use the event ID, a number that uniquely identifies an event that triggers a background function, and— this is important—remains unchanged across function retries for the same event.
To use an event ID to solve the duplicates problem, the first thing is to extract it from the event context that is accessed through function parameters. Then, we utilize the event ID as a document ID and write the document contents to Cloud Firestore. This way, a retried function execution doesn’t create a new document, just overrides the existing one with the same content. Similarly, some external APIs (e.g., Stripe) accept an idempotency key to prevent data or work duplication. If you depend on such an API, simply provide the event ID as your idempotency key.
A new lease on retries
While this approach eliminates the vast majority of duplicated calls on function retries, there’s a small chance that two retried executions running in parallel could execute the critical section more than once. To all but eliminate this problem, you can use a lease mechanism, which lets you exclusively execute the non-idempotent section of the function for a specific amount of time. In this example, the first execution attempt gets the lease, but the second attempt is rejected because the lease is still held by the first attempt. Finally, a third attempt after the first one fails re-takes the lease and successfully processes the event.
To apply this approach to your code, simply run a Cloud Firestore transaction before you send your email, checking to see if the event has been handled, but also storing the time until which the current execution attempt has exclusive rights to sending the email. Other concurrent execution attempts will be rejected until the lease expires, eliminating all duplicates for all intents and purposes.
Also, as stated in this other question:
Q: Is there a need to make these onCall functions idempotent or will they never perform retries?
A: Calls to onCall functions are not automatically retried. It's up to your application's client-side and server-side code, to agree on a retry strategy.
See also:
Retrying Event-Driven Functions - Best practices
Related
I found some old answers dealing with PHP and this code example, but I am not sure whether this is outdated now since the repo is archived and I know that generally PayPal moved to an approach that just uses the REST API.
I would love if somebody could give an update here on whats the latest recommendation is and whether the code here from 2015 is outdated now.
/* Copyright 2015-2016 PayPal, Inc. */
"use strict";
var paypal = require('../../../');
require('../../configure');
// Sends the webhook event data to PayPal to verify the webhook event signature is correct and
// the event data came from PayPal.
// Note this sample is only for illustrative purposes. You must have a valid webhook configured with your
// client ID and secret. This sample may not work due to other tests deleting and creating webhooks.
// Normally, you would pass all the HTTP request headers sent in the Webhook Event, but creating a
// JSON object here for the sample.
var certURL = "https://api.sandbox.paypal.com/v1/notifications/certs/CERT-360caa42-fca2a594-a5cafa77";
var transmissionId = "103e3700-8b0c-11e6-8695-6b62a8a99ac4";
var transmissionSignature = "t8hlRk64rpEImZMKqgtp5dlWaT1W8ed/mf8Msos341QInVn3BMQubjAhM/cKiSJtW07VwJvSX7X4+YUmHBrm5BQ+CEkClke4Yf4ouhCK6GWsfs0J8cKkmjI0XxfJpPLgjROEWY3MXorwCtbvrEo5vrRI2+TyLkquBKAlM95LbNWG43lxMu0LHzsSRUBDdt5IP1b2CKqbcEJKGrC78iw+fJEQGagkJAiv3Qvpw8F/8q7FCQAZ3c81mzTvP4ZH3Xk2/nNznEA7eMi3u1EjSpTmLfAb423ytX37Ts0QpmPNgxJe8wnMB/+fvt4xjYH6KNe+bIcYU30hUIe9O8c9UFwKuQ==";
var transmissionTimestamp = "2016-10-05T14:57:40Z";
var headers = {
'paypal-auth-algo': 'SHA256withRSA',
'paypal-cert-url': certURL,
'paypal-transmission-id': transmissionId,
'paypal-transmission-sig': transmissionSignature,
'paypal-transmission-time': transmissionTimestamp
};
// The eventBody parameter is the entire webhook event body.
var eventBody = '{"id":"WH-82L71649W50323023-5WC64761VS637831A","event_version":"1.0","create_time":"2016-10-05T14:57:40Z","resource_type":"sale","event_type":"PAYMENT.SALE.COMPLETED","summary":"Payment completed for $ 6.01 USD","resource":{"id":"8RS6210148826604N","state":"completed","amount":{"total":"6.01","currency":"USD","details":{"subtotal":"3.00","tax":"0.01","shipping":"1.00","handling_fee":"2.00","shipping_discount":"3.00"}},"payment_mode":"INSTANT_TRANSFER","protection_eligibility":"ELIGIBLE","protection_eligibility_type":"ITEM_NOT_RECEIVED_ELIGIBLE,UNAUTHORIZED_PAYMENT_ELIGIBLE","transaction_fee":{"value":"0.47","currency":"USD"},"invoice_number":"","custom":"Hello World!","parent_payment":"PAY-11X29866PC6848407K72RIQA","create_time":"2016-10-05T14:57:18Z","update_time":"2016-10-05T14:57:26Z","links":[{"href":"https://api.sandbox.paypal.com/v1/payments/sale/8RS6210148826604N","rel":"self","method":"GET"},{"href":"https://api.sandbox.paypal.com/v1/payments/sale/8RS6210148826604N/refund","rel":"refund","method":"POST"},{"href":"https://api.sandbox.paypal.com/v1/payments/payment/PAY-11X29866PC6848407K72RIQA","rel":"parent_payment","method":"GET"}]},"links":[{"href":"https://api.sandbox.paypal.com/v1/notifications/webhooks-events/WH-82L71649W50323023-5WC64761VS637831A","rel":"self","method":"GET"},{"href":"https://api.sandbox.paypal.com/v1/notifications/webhooks-events/WH-82L71649W50323023-5WC64761VS637831A/resend","rel":"resend","method":"POST"}]}';
// The webhookId is the ID of the configured webhook (can find this in the PayPal Developer Dashboard or
// by doing a paypal.webhook.list()
var webhookId = "3TR748995U920805P";
paypal.notification.webhookEvent.verify(headers, eventBody, webhookId, function (error, response) {
if (error) {
console.log(error);
throw error;
} else {
console.log(response);
// Verification status must be SUCCESS
if (response.verification_status === "SUCCESS") {
console.log("It was a success.");
} else {
console.log("It was a failed verification");
}
}
});
Those SDKs are abstractions for the REST API but are no longer being maintained, so it is best not to use them.
There are two possible ways to verify Webhooks
Posting the message back to PayPal with the verify webhook sygnature REST API call. You'll need to use a client_id and secret get an access token first, same as all other REST API calls.
Verifying the cryptographic signature yourself (Java pseudocode here).
For either method, the "webhookId" -- as opposed to each webhook event id -- is 17 alphadigits and for security (anti-spoof) reasons not part of the Webhook message itself (you get it when registering for webhooks or reviewing existing subscribed hooks in the REST app config)
As it can sometimes be a point of confusion, it's worth mentioning that verifying webhooks is for your own information -- to confirm the message did in fact originate from PayPal, and not some other (malicious) actor.
But for PayPal itself to consider the webhook message successfully delivered (and not keep retrying), all that needs to happen is for the listener URL it's posted to to respond with an HTTP 200 OK status. That concludes the webhook message delivery.
I'm working on a react-native app with spotify integration. I've set up the oAuth flow w/ auth code grant where I can get the authorization code. I've then set up cloud function on firebase to proxy the actual token exchange (I don't want to reveal my secret to the client!). I've added logs and can see that the function is correctly completing the exchange with the spotify token endpoint, and receiving a refresh and access token.
const tokenRequeset = functions.https.onCall(async (data, context) => {
// spotify network request, error handling, etc here ....
// I want to emphasize that this network request completes
// properly - my log statement below verifies in server logs
// that I'm getting the expected value.
const resp = await axios.post(
"https://accounts.spotify.com/api/token",
QueryString.stringify({
grant_type: "authorization_code",
code: code,
redirect_uri: redirectURI,
}),
{
headers: {
"Authorization": `Basic ${BEARER_TOKEN}`,
"Content-Type": "application/x-www-form-urlencoded",
},
}
);
console.log(resp.data.access_token);
return { status: "success", token: resp.data.access_token };
});
export default tokenRequest
resp.data.access_token is the JWT access token used to hit the spotify API - it's a string value according to the API. (I'd provide an example one, but it is an auth token)
However, when I try to use the firebase/functions package to call my function from my app, I will sometimes get a 'FirebaseError: Response is not valid JSON object.'
What makes this extra fun is that it's inconsistent - yesterday I had the issue, and then it went away (without changing my code!). I was able to hit both the local emulator function and then the deployed function no problem, but today the 'FirebaseError: Response is not valid JSON object.' error is back.
I have checked the logs for the failed invocations both locally and on the deployed function, and in both cases the spotify API call is working - I'm getting all the expected behavior right up until the return (which isn't working for some reason).
On the client side, I'm configuring firebase like so:
const firebaseConfig = {
// Shhhhhh
};
// Initialize Firebase
const app = initializeApp(firebaseConfig);
const functions = getFunctions(app);
// Uncomment to run locally
connectFunctionsEmulator(functions, "localhost", 5001);
export { app, functions };
And then exposing and calling it like so:
const SpotifyAuth = httpsCallable(functions, "spotify-auth");
const resp = await SpotifyAuth(code, redirectURI)
(I know this isn't full code - I grabbed the relevant portions. Happy to provide more if needed).
I tried looking up this error, and I found results from ~2018/2020 with the old style of firebase/functions, but they seem to be related to region and I'm deployed in the default us-central1 - according to the SDK documentation that means I shouldn't touch it.
The existing solutions to the problem also seem to be based on the old style of function calls, rather than the more recent httpsCallable() and getFunctions(app).
I'm going insane trying to figure out why sometimes I'm getting this error
EDIT:
More information on the error - I ran my code again today and didn't see the error locally, but I DID see it when I hit the deployed function.
Again, I want to emphasize that I think the error is in the firebase network response - if you look at the network request I receive a 200 but the response is empty.
Did an additional full mockup of a function to see what would happen:
const test = functions.https.onCall((data, context) => {
console.log("function call");
return { status: "success", token: "asdfasdfasdfasdfasfs" };
});
export default test;
I'm getting the same error.
UPDATE:
I've given up on using the sdk and onCall method for firebase cloud functions - all of my testing thus far indicates that this is a bug or error on the google cloud function side, and there's nothing I can do from my side.
The good news is the onRequest approach seems to not have this issue - it's behaving properly and reliably.
I really hope that I've messed up along the way and there's a solution I've missed - the SDK seems fantastic and I like the integration it (is supposed to) offer, but as far as I'm aware right now unless there's a bug fix (or update to the documentation if I'm doing something wrong) it seems like it simply won't work.
I'm still planning on using firebase, but from my experience thus far I'd advise anyone early in their server work to consider using another offering (at least if you need to use the functions - I was able to get storage working).
We are trying to setup error handling on a SignalR Core invoke method from a JavaScript client that needs to identify the error's type and act accordingly (e.g. if it is an authorization error, the user should be prompted to log in, etc.).
We've determined that the error returned from the hub contains a message and a stack property and have build the following that sets an error code based on the text contained in the message property:
Will the error text always be returned in English (and can thus be used to identify the error)? Or is there a better way to go about achieving this?
We are using .Net Core 3.1 and #microsoft/signalr 3.1.10.
According to aspnetcore/SignalR server-side code on GitHub, it seems that the invocation errors are indeed passed by string value.
The method responsible for send back error are defined as follow:
private async Task SendInvocationError(string invocationId, HubConnectionContext connection, string errorMessage)
{
if (string.IsNullOrEmpty(invocationId))
{
return;
}
await connection.WriteAsync(CompletionMessage.WithError(invocationId, errorMessage));
}
And a few examples on how it is being called:
if (!await IsHubMethodAuthorized(scope.ServiceProvider, connection, descriptor, hubMethodInvocationMessage.Arguments, hub))
{
Log.HubMethodNotAuthorized(_logger, hubMethodInvocationMessage.Target);
await SendInvocationError(hubMethodInvocationMessage.InvocationId, connection,
$"Failed to invoke '{hubMethodInvocationMessage.Target}' because user is unauthorized");
return;
}
var errorMessage = ErrorMessageHelper.BuildErrorMessage($"Failed to invoke '{bindingFailureMessage.Target}' due to an error on the server.",
bindingFailureMessage.BindingFailure.SourceException, _enableDetailedErrors);
return SendInvocationError(bindingFailureMessage.InvocationId, connection, errorMessage);
The only information about error is the string parameter of errorMessage.
On the other hand, the client-side javascript library source code:
HubConnection.prototype.connectionClosed = function (error) {
this.logger.log(_ILogger__WEBPACK_IMPORTED_MODULE_2__["LogLevel"].Debug, "HubConnection.connectionClosed(" + error + ") called while in state " + this.connectionState + ".");
// Triggering this.handshakeRejecter is insufficient because it could already be resolved without the continuation having run yet.
this.stopDuringStartError = this.stopDuringStartError || error || new Error("The underlying connection was closed before the hub handshake could complete.");
// If the handshake is in progress, start will be waiting for the handshake promise, so we complete it.
// If it has already completed, this should just noop.
if (this.handshakeResolver) {
this.handshakeResolver();
}
this.cancelCallbacksWithError(error || new Error("Invocation canceled due to the underlying connection being closed."));
...
};
This indicates that "Invocation canceled due to the underlying connection being closed." is the default error message when none was provided by server.
Therefore, I believe if the SignalR team didn't change the error message sending mechanism, your string.includes approach is reasonable.
I have been using pubnub for realtime communication between mobile devices.
The scenario I am creating is as follows:
The sender mobile device will publish a message to pubnub.
The message will call a On Before Function PubNub Function where original
message is sent to laravel endpoint where it is persisted and the
database record id is added to the message and published to the subscriber.
(Laravel endpoint is called using xhr module from PN Function)
The issue I am facing is that my laravel endpoint is called approximately 7 to 12 times for each publish the message.
Below is my onBefore PubNub function.
export default (request) => {
console.log('intial message: ',request.message);
const kvstore = require('kvstore');
const xhr = require('xhr');
const http_options = {
"timeout": 5000, // 5 second timeout.
"method": "POST",
"body": "foo=bar&baz=faz"
};
const url = "redacted_backend_url";
return xhr.fetch(url,http_options).then((x) => {
console.log('Messages after calling ajax: ',request.message.content);
// here i am changing the message
request.message.content = 'hello world';
return request.ok();
}).catch(err=>console.log(err)) ;
}
Please identify what exactly wrong.
PubNub Function Wildcard Channel Binding
Your Function is bound to channel '*' which of course means capture all publishes to all channels. What you don't know is that console.log in a Function publishes the message to a channel that looks like this: blocks-output-kpElEbxa9VOgYMJQ.77042688368579w9
And the Functions output window is subscribed to that channel to display your console.log's. So when the Function is invoked, it publishes a message to the console.logs channel which is captured by your Function, which calls console.log and eventually, a configured recursion limit is hit to protect you from getting into an infinite loop.
So if you were to change your channel binding to something like foo.* and publish to a channel like foo.bar, this undesired recursion would be avoided. In production, the console.logs should be removed, too, and it would not cause this to happen.
Additionally, you could implement some channel filter condition at the top of the your Function to prevent it from further execution your Function code:
if (channel.startsWith("blocks-output"))
return request.ok()
}
When awaiting on a promise as a result of adminuserglobalsignout the promise seems to return but the data contains nothing.
The next call after signout is to authenticate the user. correct accessToken is returned but it's already revoked which makes me think the promise is not awaiting correctly and the new credentials are getting signed out by the previous call which is still running.
We are using globalsignout to prevent users from having multiple sessions so the workflow is along lines of
authenticate -> success -> signout (to kill any other sessions) -> authenticate -> success -> return token
I have updated my lambda package to include the latest SDK version 2.469.0 and no improvement.
Sometimes the timing must be OK as the returned credentials are still valid and the token can be used.
In BOTH cases there appears to be zero data returned from the AWS call
section of lambda code that calls the signout method in the User library
try {
signOutResult = await User.globalSignOut(userId, process.env.COGNITO_POOL);
} catch (err) {
log.error("AWS Global Signout Error: " + JSON.stringify(err));
responseBody = Helper.buildCORSResponse(502, JSON.stringify({ message: err }));
return callback(null, responseBody);
}
globalsignout code in User library:
return new Promise((resolve, reject) => {
log.info(`globalSignOut: Signing ${Username} out from all devices in pool ${UserPoolId}`);
const signOutRequest = new AWS.CognitoIdentityServiceProvider({ apiVersion: "2016-04-18" }).adminUserGlobalSignOut({ Username, UserPoolId });
const signOutPromise = signOutRequest.promise();
signOutPromise.
then((data) => {
log.debug("globalSignOut: Cognito SignOut Success: " + JSON.stringify(data));
resolve(data);
}).catch((err) => {
log.error("globalSignOut: Cognito SignOut Error: " + err);
reject(err);
});
});
}
In every call, we reach the resolve with no issue and then we carry on to authenticate the user again.
log.debug("globalSignOut: Cognito SignOut Success: " + JSON.stringify(data));
resolve(data);
Does anyone see any issues that could be causing this? I've tried a few ways to specify the promise and using the same format that works fine for other services and waits for the promise of the result before code execution continues.
All advice greatly appreciated
Update from AWS Support on this behavior in case anyone else finds this issue. I can confirm that adding a small delay before re-authenticating the user after global signout works fine.
Thank you for getting back to us.
In order to troubleshoot this issue, I tried to replicate it on my end by testing the below mentioned flow (as provided by you in the ) :
Authenticate user —> Global Sign Out —> Authenticate again —-> Check the validity of the new token
I wrote a python code to implement the above flow. In the flow, after calling the globalSignOut method, I authenticated the user again and checked the validity of the token by making getUser API call. But, the getUser API call returned the following response : “An error occurred (NotAuthorizedException) when calling the GetUser operation: Access Token has been revoked”
Now, I added sleep function after the GlobalSignOut for 1 second and the flow worked correctly. I did a few tests with the sleep time and noticed that if we add a sleep period of 0.6 seconds or greater, the API works correctly. So, it seems that the GlobalSignOut API call returns the response immediately but, the global logging out process (revoking of tokens) still runs in the backend for approximately 0.6 seconds.
For this, I reached out to the Cognito development team to confirm this behavior of GlobalSignOut API call. The team has confirmed that this is an expected behavior of GlobalSignOut API call. When GlobalSignOut is called all the tokens that were issued before that time is considered invalid. If the gap between signout and authentication is very small ( from my tests, this is approximately 0.6 seconds ), the token issue after authentication can be treated to be issued before signout call and, for better security, is considered invalid.
I hope that the above information helps. If there is anything else I can do to help, please let me know. I will be more than happy to assist you.
Have a great day ahead.
Best regards,
Amazon Web Services