I have a bunch of data that I want to send to a server through http. However in the server side I need to process the data in the same order as they were sent(e.g. if the order of sending is elem1, elem2 and elem3, I would like to process elem1 first, then elem2 and then elem3). Since in http, there is no grantee that the order will be maintained I need some way to maintain the order.
Currently I am keeping the data in a queue and I send one element and await for the response. Once the response reaches me I send the next element.
while (!queue.isEmpty()) {
let data = queue.dequeue();
await sendDataToServer(data);
}
I am not very sure if this will actually work in a production environment and what will be the impact on the performance.
Any sort of help is much appreciated. Thank you
Sorry, I don't have enough reputation to comment, thus I am posting this as an answer.
Firstly, your code will work as intended.
However, since the server has to receive them in order, the performance won't be good. If you can change the server, I suggest you implement it like this:
Add an ID to each data item.
Send all the data items, no need to ensure order.
Create a buffer on the server, the buffer will be able to contain all the data items.
The server receives the items and puts them into the buffer in the right position.
Example code:
Client (see Promise.all)
let i = 0;
let promises = [];
await sendDataLengthToServer(queue.length());
while (!queue.isEmpty()) {
let data = queue.dequeue();
data.id = i;
// no need to wait for a request to finish
promises.push(sendDataToServer(data));
}
await Promise.all(promises);
Server (pseudo-code)
length = receiveDataLengthFromClient()
buffer = new Array(length)
int received = 0
onDataReceivedFromClient(data, {
received = received + 1
buffer[data.id] = data
if (received == length) {
// the buffer contains the data in the right order
}
})
Related
Note:
The asynchronous nature of firebase has been discussed thousands of times here, but my low reputation number does not allow for a comment on an existing question. That's why I have asked this question.
I am a noob, so please help me understand the implementation in an easy to understand manner.
Steps to implement:
User enters a value in the HTML input box
Search the input value in the firebase db (showMessage() gets called)
Display an appropriate result based on the search result in step 2
Problem faced:
The message displayed in step 3 takes almost an average of 1.75 seconds to display. This experience is not user-friendly. I want to display the message as soon as possible i.e. want to reduce the fetch time.
Probable root causes:
Either my way of fetching the data from firebase dB is incorrect (I still don't understand how to keep a promise :()
Or The mechanism of search and display is not right
var full_name;
function showMessage(){
extractData();
}
function extractData(){
test(function(returnValue) {
custom_message = searchMessage(returnValue);
var container = document.querySelector('#placeholder');
var para = document.createElement('p');
var custom_message = "Happy happy, buds!";
para.innerHTML = custom_message;
para.className = "message";
container.appendChild(para);
});
}
function test(callback) {
var ref = firebase.database().ref();
ref.on('value', function(snapshot) {
var data = snapshot.val();
callback(data);
}, function (error) {
console.log("Error: " + error.code);
});
}
function searchMessage(data){
for(var i = 0; i < data.length; i++)
{
name_f_data = data[i].firstName.concat(" ", data[i].lastName);
if(full_name.toLowerCase() == name_f_data.toLowerCase())
{
console.log(name_f_data.toLowerCase());
console.log(full_name.toLowerCase());
return data[i].message;
}
}
}
The time a read operation takes depends on:
The latency of your connection to Firebase's servers
The amount of data you are reading
The bandwidth of your connection
The time it takes Firebase to process the request
In most cases, the time Firebase takes is only a very small portion of the total time, and most of your time actually goes to the data transfer, which depends purely on the bandwidth and amount of data. If this is the first time you're reading data from Firebase in the page, the latency also matters more, as Firebase has to establish a connection, which takes a few roundtrips.
Your current code is downloading all data from the database, and then searching in the JavaScript code for a child node that matches a certain value. The best way to reduce the time that takes (apart from upgrading to a fast connection) is to transfer less data, which you can do by using Firebase's query mechanism to do the filtering on the server.
You can get pretty close with:
var ref = firebase.database().ref();
var query = ref.orderByChild().startAt(firstName).endAt(firstName+"~");
query.once('value', function(snapshot) {
var data = snapshot.val();
callback(data);
This will significantly reduce the amount of data transferred. A few notes though:
The query returns just the people that have the first name you're looking for. It does not yet filter on the last name, so you'll still need to filter that in the client-side code.
To further optimize this, store the full name (which you now compose in the client-side code) in the database so that you can query on that and reduce data transfer even more.
Firebase queries are case sensitive, so the query only returns data where the case matches exactly. If you want to query case-indifferent, consider storing a toLowerCase() value in the database.
Be sure to define an index on firstName, as otherwise the Firebase database will still send all data to the client, and the SDK will perform the filtering client-side.
I'm trying to get users from database and then loop throug each user and get their images. However when I try to assign images array to user.images property nothing happens. I still get only users with empty images array.
Currently my code is the following:
user.getAll().then(result =>{
const userCollection = result[0];
for(let i=0; i < userCollection.length; i++){
userCollection[i].images = [{}];
image.getImagesByUserId(userCollection[i].Id).then(res =>{
userCollection[i].images = res[0];
}).catch(err =>{
console.log(err);
res.status(400).json(err);
})
}
res.json(userCollection);
})
.catch(err =>{
console.log(err);
res.status(400).json(err);
});
Why I can't assign images array to it's property?
It looks to me like you need to run your res.json(userCollection) from inside the callback.
This will of course mean you need to rename your res variable in the callback
This is assuming you are somehow using res.json(userCollection) to export the information. Which I am inferring based on the fact I don’t see res defined anywhere
May be a silly question, but are you assigning it? I only see
userCollection[i].images = [{}];
above and never see:
userCollection[i].images = res;
assignment once you grab the images.
The problem is that getImagesByUserId() is asynchronous so by the time you get to the assignment, you've already sent the response.
The order of events you have currently is:
Assign an empty image list to all the users
Queue up a load of database requests
Send the response (res.json())
Reassign the images for all the users once the database results come back.
The easy fix is to look at Promise.all() and aggregate the results when they all come back.
However, this isn't the most efficient way to deal with SQL databases, I would suggest you restructure your query so that you get all the images for all the users in 1 trip to the database and then format the response based on those results.
Say you have an image table and a user table, something like:
SELECT image_name, user_id
FROM image
then replace your loop in your js with:
getImagesForUsers().then(result => {
//create your response from the rows
res.json(response);
});
I'm sending data to the Firestore at 33Hz. This means that over time, a lot of data is stored.
I need to be able to download this data. For that, I have made a Http Function in firebase, that receives the user uid, device's serial number, start date/time and end date/time. The function then tests if the user exists and also if he does have that serial number. Then it queries firestore and append data to a JSON object, that eventually is send as a response.
But, depending on how long the query has been made, the function will timeout. It does work for short periods.
How should I do to make the function faster? Am I using the wrong tool?
[...]
const nD1 = db.collection('grind-bit').doc(req.query.serial).collection('history').where('date', '>=', startdate).where('date', '<=', enddate).get().then(snapshot => {
snapshot.forEach(doc => {
elem = {};
elem.date = doc.data().date;
elem.rms0 = doc.data().rms0;
elem.rms1 = doc.data().rms1;
elem.rms2 = doc.data().rms2;
data[key].push(elem);
});
if(data[key].length) {
let csv = json2csv(data[key]);
csv = JSON.stringify(csv);
res.set("Content-Type", "text/csv");
return promises.push(res.status(200).send(csv));
} else {
return promises.push(res.status(401).send('No data has been found in this interval.'));
}
[...]
If the timeout is caused by the fact that you're trying to read/return too much data in one go, you might want to consider limiting how much data you return. By adding a limit() clause to your query, you can limit how much data it returns at most.
const nD1 = db.collection('grind-bit').doc(req.query.serial).collection('history')
.where('date', '>=', startdate).where('date', '<=', enddate)
.limit(100)
.get().then(snapshot => {
snapshot.forEach(doc => {
...
This of course means that you may have to call the HTTP function multiple times to ensure you process all data. The simplest approach for that is: if you get 100 results (or whatever limit your specify), try another time after processing them.
Also see:
The Firebase documentation on limiting query results
I'm building an app which i need to stream data to client, my data is simply an array of objects .
this is the for loop which makes the array
for(let i =0;i<files.length;i++){
try {
let file = files[i]
var musicPath = `${baseDir}/${file}`
let meta = await getMusicMeta(musicPath)
musics.push(meta)
}
right now I wait for the loop to finish it's works then I send the whole musics array to client, I want to use stream to send musics array one by one to client instead of waiting for the loop to finish
Use scramjet and send the stream straight to the response:
const {DataStream} = require("scramjet");
// ...
response.writeHead(200);
DataStream.fromArray(files)
// all the magic happens below - flow control
.map(file => getMusicMeta(`${baseDir}/${file}`))
.toJSONArray()
.pipe(response);
Scramjet will make use of your flow control and most importantly - it'll get the result out faster than any other streaming framework.
Edit: I wrote a couple lines of code to make this use case easier in scramjet. :)
Is there a way to store all emits of socket.io in a global array so that I can loop through the emits when a new user joins the website and can pickup where the 'canvas drawing' currently is. I want the new user to see what work has already been done and then collaborate on it.
Any other ways I could approach towards this?
If you just want the emits stored for the duration of the server running, you can simply declare a module level array and push each emit into that array. Then, at any future time during that server execution, you can consult the array.
If, you want the emits stored across server invocations, then you would need to store them to some persistent storage (file, database, etc...).
// save incoming data from one particular message
var emitsForSomeMessage = [];
io.on("someMessage", function(data) {
// save the incoming data for future reference
emitsForSomeMessage.push(data);
});
Or, if you're trying to store all outgoing .emit() data, then you can override that method and save away what is sent.
var outgoingEmits = [];
(function() {
var oldEmit = io.emit;
io.emit = function(msg, data) {
outgoingEmits.push({msg: msg, data: data});
return oldEmit.apply(this, arguments);
};
})();
Since there are many different messages that may be sent or received, you can add your own logic to decide which messages are saved in the array or not.