We Keep Coding

Why does my Firestore listener with offline persistence always read from the server?

I am using Firebase JavaScript Modular Web Version 9 SDK with my Vue 3 / TypeScript app.
My understanding is that when using Firestore real-time listeners with offline persistence it should work like this:
When the listener is started the callback fires with data read from the local cache, and then immediately after it also tries to read from the server to make sure the local cache has up to date values. If the server data matches the local cache the callback listener should only fire once with data read from the local cache.
When data changes, the callback listener fires with data read from the server. It uses that data to update the local cache.
When data doesn't change, all subsequent calls to the listener trigger a callback with data read from the local cache.
But I have setup offline persistence, created a listener for my Firestore data, and monitored where the reads were coming from...
And in my app I see an initial read from the local cache (expected), and then a second immediate read from the server (unexpected). And after that all subsequent reads are coming from the server (also unexpected).
During this testing none of my data has changed. So I expected all reads from the callback listener to be coming from the local cache, not the server.
And actually the only time I see a read from the local cache is when the listener is first started, but this was to be expected.
What could be the problem?
P.S. To make those "subsequent calls" I am navigating to a different page of my SPA and then coming back to the page where my component lives to trigger it again.
src/composables/database.ts
export const useLoadWebsite = () => {
const q = query(
collection(db, 'websites'),
where('userId', '==', 'NoLTI3rDlrZtzWCbsZpPVtPgzOE3')
);
const firestoreWebsite = ref<DocumentData>();
onSnapshot(q, { includeMetadataChanges: true }, (querySnapshot) => {
const source = querySnapshot.metadata.fromCache ? 'local cache' : 'server';
console.log('Data came from ' + source);
const colArray: DocumentData[] = [];
querySnapshot.docs.forEach((doc) => {
colArray.push({ ...doc.data(), id: doc.id });
});
firestoreWebsite.value = colArray[0];
});
return firestoreWebsite;
};
src/components/websiteUrl.vue
<template>
<div v-if="website?.url">{{ website.url }}</div>
</template>
<script setup lang="ts">
import { useLoadWebsite } from '../composables/database';
const website = useLoadWebsite();
</script>

Nothing is wrong. What you're describing is working exactly the way I would expect.
Firestore local persistence is not meant to be a full replacement for the backend. By default, It's meant to be a temporary data source in the case that the backend is not available. If the backend is available, then the SDK will prefer to ensure that the client app is fully synchronized with it, and serve all updates from that backend as long as it's available.
If you want to force a query to use only the cache and not the backend, you can programmatically specify the cache as the source for that query.
If you don't want any updates at all from the server for whatever reason, then you can disable network access entirely.
See also:
Firestore clients: To cache, or not to cache? (or both?)

I figured out why I was getting a result different than expected.
The culprit was { includeMetadataChanges: true }.
As explained here in the docs, that option will trigger a listener event for metadata changes.
So the listener callback was also triggering on each metadata change, instead of just data reads and writes, causing me to see strange results.
After removing that it started to work as expected, and I verified it by checking it against the Usage graphs in Firebase console which show the number of reads and snapshot listeners.
Here is the full code with that option removed:
export const useLoadWebsite = () => {
const q = query(
collection(db, 'websites'),
where('userId', '==', 'NoLTI3rDlrZtzWCbsZpPVtPgzOE3')
);
const firestoreWebsite = ref<DocumentData>();
onSnapshot(q, (querySnapshot) => {
const source = querySnapshot.metadata.fromCache ? 'local cache' : 'server';
console.log('Data came from ' + source);
const colArray: DocumentData[] = [];
querySnapshot.docs.forEach((doc) => {
colArray.push({ ...doc.data(), id: doc.id });
});
firestoreWebsite.value = colArray[0];
});
return firestoreWebsite;
};

Manage a long-running operation node.js

I am creating a telegram bot, which allows you to get some information about the destiny 2 game world, using the Bungie API. The bot is based on the Bot Framework and uses Telegram as a channel (as a language I am using JavaScript).
now I find myself in the situation where when I send a request to the bot it sends uses series of HTTP calls to the EndPoints of the API to collect information, format it and resubmit it via Adaptive cards, this process however in many cases takes more than 15 seconds showing in chat the message "POST to DestinyVendorBot timed out after 15s" (even if this message is shown the bot works perfectly).
Searching online I noticed that there doesn't seem to be a way to hide this message or increase the time before it shows up. So the only thing left for me to do is to make sure it doesn't show up. To do this I tried to refer to this documentation article. But the code shown is in C #, could someone give me an idea on how to solve this problem of mine or maybe some sample code?
I leave here an example of a call that takes too long and generates the message:
//Mostra l'invetraio dell'armaiolo
if (LuisRecognizer.topIntent(luisResult) === 'GetGunsmith') {
//Take more 15 seconds
const mod = await this.br.getGunsmith(accessdata, process.env.MemberShipType, process.env.Character);
if (mod.error == 0) {
var card = {
}
await step.context.sendActivity({
text: 'Ecco le mod vendute oggi da Banshee-44:',
attachments: [CardFactory.adaptiveCard(card)]
});
} else {
await step.context.sendActivity("Codice di accesso scaduto.");
await this.loginStep(step);
}
}

I have done something similar where you call another function and send the message once the function is complete via proactive message. In my case, I set up the function directly inside the bot instead of as a separate Azure Function. First, you need to save the conversation reference somewhere. I store this in conversation state, and resave it every turn (you could probably do this in onMembersAdded but I chose onMessage when I did it so it resaves the conversation reference every turn). You'll need to import const { TurnContext } = require('botbuilder') for this.
// In your onMessage handler
const conversationData = await this.dialogState.get(context, {});
conversationData.conversationReference = TurnContext.getConversationReference(context.activity);
await this.conversationState.saveChanges(context);
You'll need this for the proactive message. When it's time to send the API, you'll need to send a message (well technically that's optional but recommended), get the conversation data if you haven't gotten it already, and call the API function without awaiting it. If your API is always coming back around 15 seconds, you may just want a standard message (e.g. "One moment while I look that up for you"), but if it's going to be longer I would recommend setting the expectation with the user (e.g. "I will look that up for you. It may take up to a minute to get an answer. In the meantime you can continue to ask me questions."). You should be saving user/conversation state further down in your turn handler. Since you are not awaiting the call, the turn will end and the bot will not hang up or send the timeout message. Here is what I did with a simulation I created.
await dc.context.sendActivity(`OK, I'll simulate a long-running API call and send a proactive message when it's done.`);
const conversationData = await this.dialogState.get(context, {});
apiSimulation.longRunningRequest(conversationData.conversationReference);
// That is in a switch statement. At the end of my turn handler I save state
await this.conversationState.saveChanges(context);
await this.userState.saveChanges(context);
And then the function that I called. As this was just a simulation, I have just awaited a promise, but obviously you would call and await your API(s). Once that comes back you will create a new BotFrameworkAdapter to send the proactive message back to the user.
const request = require('request-promise-native');
const { BotFrameworkAdapter } = require('botbuilder');
class apiSimulation {
static async longRunningRequest(conversationReference) {
console.log('Starting simulated API');
await new Promise(resolve => setTimeout(resolve, 30000));
console.log('Simulated API complete');
// Set up the adapter and send the message
try {
const adapter = new BotFrameworkAdapter({
appId: process.env.microsoftAppID,
appPassword: process.env.microsoftAppPassword,
channelService: process.env.ChannelService,
openIdMetadata: process.env.BotOpenIdMetadata
});
await adapter.continueConversation(conversationReference, async turnContext => {
await turnContext.sendActivity('This message was sent after a simulated long-running API');
});
} catch (error) {
//console.log('Bad Request. Please ensure your message contains the conversation reference and message text.');
console.log(error);
}
}
}
module.exports.apiSimulation = apiSimulation;

Socket emitting event multiple times

I am working on socket for chatting. Here is my socket code in nodejs file which is working well.
The outer socket io.emit working good and emits the message to all the users which are connected to that conversationId.
But the socket.broadcast.emit(when user uses the app) which I am using to notify user, emits(socket.broadcast.emit) events multiple times. Why this is happening? Am I completely missing the socket approach.
socket.on('sendMessage', async(action2) => {
try {
action2.author = socket.decoded.id
action2.readBy = [socket.decoded.id]
action2.deliveredTo = [socket.decoded.id]
const createMessage = await Message.create(action2)
const sender = await User.findOne({ _id: socket.decoded.id }, { firstName: 1 })
const memebers = //some api call to get members
const promises = members.map(async(member) => {
// socket for message
const socketNotification = {
// my object
}
console.log(socketNotification, 'socketNotifcication')
socket.broadcast.emit(`messageSocket${member.memberId}`, socketNotification)
})
await Promise.all(promises)
io.emit(action2.conversationId, messages) // "newMessage"
} catch (err) {
throw err
}
})

From the Broadcast docs:
Sets a modifier for a subsequent event emission that the event data
will only be broadcast to every sockets but the sender.
https://socket.io/docs/server-api/#Flag-%E2%80%98broadcast%E2%80%99
So in your loop you are saying send this everyone but the original socket, and you call that multiple times. What you want to use it it.to
io.to(membersSocketID).emit('eventName', socketNotification)
It's unclear from your example if the messageSocket${member.memberId} is supposed to be the event name of if that is your specified socket.id.
This is a great cheatsheet for this https://socket.io/docs/emit-cheatsheet/
Side note, if your api call to get the member id's is significant you might be better off using rooms/namespaces and doing that query on connect to determine rooms.

use io.to(messageSocket${member.memberId}).emit() instead of socket.broadcast.emit()

Can't get custom push notification event working in PWA (Firebase)

I've been searching for a few hours on how to get my custom push notification working. Here is how I've set up my project: no front-end framework, a Node.js/Express.js back-end, Firebase Cloud Messaging (FCM) as push manager and a custom service worker. I am currently hosting the app on localhost and I have HTTPS set up and a manifest.json that contains the minimum amount of fields to get started. The manifest.json contains a start_url field that points to /index.html, the landing page for the app. The app is bundled with webpack v. 4.
Back-end
On the back-end, I have set up the Firebase Admin SDK in a specific router and I send a custom notification object a bit like the following to the FCM server:
let fcMessage = {
data : {
title : 'Foo',
tag : 'url to view that opens bar.html'
}
};
When an interesting event occurs on the back-end, it retrieves a list of users that contains the FCM registration tokens and sends the notification to the FCM servers. This part works great.
Front-end
I have two service workers on the front-end. Inside my front-end index.js, I register a custom service worker named sw.js in the domain root and tell firebase to use that service worker like so:
if ('serviceWorker' in navigator) {
navigator.serviceWorker.register('./sw.js')
.then(registration => {
messaging.useServiceWorker(registration);
})
.catch(err => console.error(`OOps! ${err}`));
}
FCM and its credentials are set up and the user can subscribe to push notifications. I won't show that code here since it works and I don't believe it is the issue.
Now on to the service workers themselves. I have a firebase-messaging-sw.js file at the root of my domain. It contains the following code:
importScripts('https://www.gstatic.com/firebasejs/4.8.1/firebase-app.js');
importScripts('https://www.gstatic.com/firebasejs/4.8.1/firebase-messaging.js');
firebase.initializeApp(configuration);
const messaging = firebase.messaging();
Configuration is just a placeholder for all of the creds. Again that stuff works.
What I want to do is to NOT use the FCM push notification and instead create my own push notification that contains a url to a view that the user can click on and go to that view. The following code almost works and is a hack I found on another site:
class CustomPushEvent extends Event {
constructor(data) {
super('push');
Object.assign(this, data);
this.custom = true;
}
}
self.addEventListener('push', (e) => {
console.log('[Service Worker] heard a push ', e);
// Skip if event is our own custom event
if (e.custom) return;
// Keep old event data to override
let oldData = e.data;
// Create a new event to dispatch
let newEvent = new CustomPushEvent({
data: {
json() {
let newData = oldData.json();
newData._notification = newData.notification;
delete newData.notification;
return newData;
},
},
waitUntil: e.waitUntil.bind(e),
})
// Stop event propagation
e.stopImmediatePropagation();
// Dispatch the new wrapped event
dispatchEvent(newEvent);
});
messaging.setBackgroundMessageHandler(function(payload) {
if (payload.hasOwnProperty('_notification')) {
return self.registration.showNotification(payload.data.title,
{
body : payload.data.text,
actions : [
{
action : `${payload.data.tag}`,
title : 'Go to link'
}
]
});
} else {
return;
}
});
self.addEventListener('notificationclick', function(e) {
console.log('CLICK');
e.notification.close();
e.waitUntil(clients.matchAll({ type : 'window' })
.then(function(clientList) {
console.log('client List ', clientList);
const cLng = clientList.length;
if (clientList.length > 0) {
for (let i = 0; i < cLng; i++) {
const client = clientList[i];
if (client.url === '/' && 'focus' in client) {
return client.focus();
}
}
} else {
console.log('no clients ', e.action);
clients.openWindow(e.action)
.then(client => {
console.log('client ', client);
return client.navigate(e.action);
})
.catch(err => console.error(`[Service Worker] cannot open client : ${err} `));
}
}))
});
The hack is meant to capture a push event and the FCM default notification payload and instead serve that payload through a custom one made via the Notification API.
The code above works great but ONLY if I put it in the firebase-messaging-sw.js file. That's not what I really want to do: I want to put it in the sw.js file instead but when I do, the sw.js cannot hear any push events and instead I get the default FCM push notification. I've also tried importing the entire firebase-messaging-sw scripts into the custom service worker and it still won't hear the message events.
Why do I want to use it in my service worker instead of the Firebase one? It's to be able to open the app on the view passed into the 'tag' field on the notification's body. If I use the Firebase service worker, it tells me that it's not the active registered service worker and though the app does open in a new window, it only opens on /index.html.
Some minor observations I've made: the clients array is always empty when the last bit of code is added to the firebase-messaging-sw.js file. The custom service worker is installed properly because it handles the app shell cache and listens to all of the other events normally. The firebase-messaging-sw service worker is also installed properly.

After much pain and aggravation, I figured out what the problem was. It was a combination of the architecture of the app (that is, a traditional Multi-Page App) and a badly-formed url in the custom service worker, sw.js as so:
sw.js
self.addEventListener('fetch', e => {
// in this app, if a fetch event url calls the back-end, it contains api and we
// treat it differently from the app shell
if (!e.request.url.includes('api')) {
switch(e.request.url) {
case `${endpoint}/bar`: // <-- this is the problem right here
matcher(e, '/views/bar.html');
break;
case `${endpoint}/bar.js`:
matcher(e, '/scripts/bar.js');
break;
case `${endpoint}/index.js`:
matcher(e, '/index.js');
break;
case `${endpoint}/manifest.json`:
matcher(e, '/manifest.json');
break;
case `${endpoint}/baz/`:
matcher(e, '/views/bar.html');
break;
case `${endpoint}/baz.js`:
matcher(e, '/scripts/bar.js');
break;
default:
console.log('default');
matcher(e, '/index.html');
}
}
});
Matcher is the function that matches the request url with the file on the server. If the file already exists in the cache, it returns what is in the cache but if it doesn't exist in the cache, it fetches the file from the server.
Every time the user clicks on the notification, it's supposed to take him/her to the 'bar' html view. In the switch it must be:
case `${endpoint}/bar/`:
and not
case `${endpoint}/bar`:
Even though the message-related code is still in the firebase-messaging-sw.js file, what happens is it creates a new WindowClient when the browser is in the background. That WindowClient is under the influence of sw.js, not firebase-messaging-sw.js. As a result, when the window is opened, sw.js intercepts the call and takes over from firebase-messaging-sw.js.

URL of page currently being served by Service Worker 'fetch' event

How can I get the full URL of the page which is being serviced by a service worker's 'fetch' event?
The "self.location" property seems to only refer to the root URL of the site. For example, if page https://example.com/folder/pagename.html is performing a fetch which the service worker is intercepting, the service worker's 'self.location' property returns "https://example.com".
event.currentTarget.location and event.explicitOriginalTarget.location, event.originalTarget, and event.target all return the URL of the service worker .js file.
How can I get the full URL of the page that triggered the fetch event?

You've got two general approaches, depending on how involved you want to get:
Use the 'Referer' header info
If the request is for a subresource and includes a Referer header, then there's a decent chance that the value of that header is the URL of the page that made the request. (There are some caveats; read this background info to delve into that.)
From within a fetch handler, you can read the value of that header with the following:
self.addEventListener('fetch', event => {
const clientUrl = event.request.referrer;
if (clientUrl) {
// Do something...
}
});
Use the clientId value
Another approach is to use the clientId value that (might) be exposed on the FetchEvent, and then use clients.get(id) or loop through the output of clients.matchAll() to find the matching WindowClient. You could then read the url property of that WindowClient.
One caveat with this approach is that the methods which look up the WindowClient are all asynchronous, and return promises, so if you're somehow using the URL of the client window to determine whether or not you want to call event.respondWith(), you're out of luck (that decision needs to be made synchronously, when the FetchEvent handler is first invoked).
There's a combination of different things that need to be supported in order for this approach to work, and I'm not sure which browsers currently support everything I mentioned. I know Chrome 67 does, for instance (because I just tested it there), but you should check in other browsers if this functionality is important to you.
self.addEventListener('fetch', async event => {
const clientId = event.clientId;
if (clientId) {
if ('get' in clients) {
const client = await clients.get(clientId);
const clientUrl = client.url;
// Do something...
} else {
const allClients = await clients.matchAll({type: 'window'});
const filtered = allClients.filter(client => client.id === clientId);
if (filtered.length > 0) {
const clientUrl = filtered[0].url;
// Do something...
}
}
}
});