I have an problem when i'm dealing with an big size array (> 50k, sometimes > 100k depends on the fromDate and toDate in query). This data is retrieved from an static query that i can't change so that i can not use offset or limit. Afterthat, i have to insert 50k data into an table in postgres with using sequelize. Morever, when i've done retriving the data i have to change some value in each item in array. Ex:
array.map(item => {
if (item.some_field) {
item.condition_field = some_data;
}
return item;
})
The problem is the process is taking too long, do you guys have any suggestions for my problem ?
Related
I am trying to pull a URL for an image in storage that is currently logged in the firebase real time database.
This is for a game of snap - there will be two cards on the screen (left image and right image) and when the two matches the user will click snap.
All of my image urls are stored in the following way:
Each one has a unique child called "index" - I also have another tree that is just a running count of each image record. So currently I am running a function that checks the total of the current count, then performs a random function to generate a random number, then performs a database query on the images tree using orderByChild and an equalTo that contains the random index number.
If I log the datasnap of this I can see a full node for one record (So index, score, url, user and their values) however if I try to just pull the URL I get returned a value of Null. I can, rather annoyingly, return the term "URL" seemingly at my leisure but I can't get the underlying value. I've wondered if this is due to it being a string and not a numeric but I can't find anything to suggest that is a problem.
Please bare in mind I've only been learning Javascript for about a week at max, so if I'm making obvious rookie errors that's probably why!
Below is a code snippet to show you what I mean:
var indRef = firebase.database().ref('index')
var imgRef = firebase.database().ref('images')
var leftImg = document.getElementById('leftImg')
var rightImg = document.getElementById('rightImg')
document.addEventListener('DOMContentLoaded', function(){
indRef.once('value')
.then(function(snapShot){
var indMax = snapShot.val()
return indMax;
})
.then(function(indMax){
var leftInd = Math.floor(Math.random()* indMax + 1)
imgRef.orderByChild('index').equalTo(leftInd).once('value', function(imageSnap){
var image = imageSnap.child('url').val();
leftImg.src=image;
})
})
})
When you execute a query against the Firebase Database, there will potentially be multiple results. So the snapshot contains a list of those results. Even if there is only a single result, the snapshot will contain a list of one result.
Your code needs to cater for that list, by looping over Snapshot.forEach():
imgRef.orderByChild('index').equalTo(leftInd).once('value', function(imageSnap){
imageSnap.forEach(function(child) {
var image = child.child('url').val();
leftImg.src=image;
})
})
This is the JSON I'm working with:
https://data.cityofnewyork.us/resource/xx67-kt59.json?$where=camis%20=%2230112340%22
I'd be dynamically making the queries using different data, so it'll possibly change.
What I'm essentially trying to do is to somehow organize the elements within this array into different arrays based on inspection_date.
So for each unique inspection_date value, those respective inspections would be put into its own collection.
If I knew the dates beforehand, I could easily iterate through each element and just push into an array.
Is there a way to dynamically create the arrays?
My end goal is to be able to display each group of inspections (based on inspection date) using Angular 5 on a webpage. I already have the site up and working and all of the requests being made.
So, I'm trying to eventually get to something like this. But of course, using whatever dates in the response from the request.
2016-10-03T00:00:00
List the inspections
2016-04-30T00:00:00
List the inspections
2016-04-12T00:00:00
List the inspections
Just for reference, here's the code I'm using:
ngOnInit() {
this.route.params.subscribe(params => {
this.title = +params['camis']; // (+) converts string 'id' to a number
this.q.getInpectionsPerCamis(this.title).subscribe((res) => {
this.inspectionList = res;
console.log(res);
});
// In a real app: dispatch action to load the details here.
});
}
I wish I could give you more info, but at this point, I'm just trying to get started.
I wrote this in jQuery just because it was faster for me, but it should translate fairly well to Angular (I just don't want to fiddle with an angular app right now)
Let me know if you have any questions.
$(function() {
let byDateObj = {};
$.ajax({
url: 'https://data.cityofnewyork.us/resource/xx67-kt59.json?$where=camis%20=%2230112340%22'
}).then(function(data) {
//probably do a check to make sure the data is an array, im gonna skip that
byDateObj = data.reduce(function(cum, cur) {
if (!cum.hasOwnProperty(cur.inspection_date)) cum[cur.inspection_date] = [];
//if the cumulative array doesn't have the inspection property already, add it as an empty array
cum[cur.inspection_date].push(cur);
//push to inspection_date array.
return cum;
//return cumulatie object
}, byDateObj);
//start with an empty object by default;
console.log(byDateObj);
}, console.error);
});
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
I have a 2d array representing rows in a database. I'm using officeJS to load and manipulate the data in Excel. I update, insert, and delete rows. The challenge I'm facing is that I need to figure out the changed rows (inserted, deleted or updated) so that I can update only those rows in the database. I'm sending one query for the updated and inserted rows and one query for the deleted rows. I'm able to do this using lodash for data with 5000 rows and 10 columns. I'd like to scale this to a much larger data set and I'm wondering if there are any alternatives to what I'm currently doing. Below is the code I'm using to find the difference.
insertedOrUpdatedRows = _.differenceWith(modifiedData, originalData, _.isEqual);
deletedRows = _.differenceWith(originalData, modifiedData, compareFunction);
function compareFunction(a, b) {
if(a[0] == b[0]) {
return true;
}
else
return false;
}
Sample data array
[ [1,data,data,data],
[2,data,data,data] ]
The first element is the primary key.
Because you have mentioned that your Javascript engine is crashing (which it should not, at 50,000 rows - so I would revisit the logic), I would recommend chunking out the data using Lodash's _.chunk function:
_.chunk(modifiedData, modifiedData.length/500).map({
...
...
});
Ok im using the following logic. Not sure why its crashing at 50K rows. OriginalData and ModifiedData are in the format of the sample 2D array mentioned above.
var originalDataStrings = [];
var modifiedDataStrings = [];
var insertedOrUpdatedRows;
originalData.forEach(function(row){
originalDataStrings.push(JSON.stringify(row));
});
modifiedData.forEach(function(row){
modifiedDataStrings.push(JSON.stringify(row));
})
insertedOrUpdatedRows = _.differenceWith(modifiedDataStrings, originalDataStrings, _.isEqual);
console.log(insertedOrUpdatedRows);
I'm building an API for my site (using Node.js and Mongoose) and I would like to incorporate pagination in it. My problem is the following: if the page size is for example 15 and I make a request for the first page so it sends me the first 15 items ordered by date of creation but then what if before I make the request for the second page, 15 new items are created in the database, the returned data will be the same as previously if I just use a skip on mongoose.
Is there a way to avoid doublons with mongoose? What I have at the moment is an "exclude" parameter in the query so it excludes all items already loaded but I'm thinking if there are lots of loaded items, the URL might be very long and I'm not sure that's a good thing...
Is there a better way to do this or do I have to just leave it with the risk of having doublons?
You can use mongoose-paginate-v2
And to prevent any duplicates from returned documents, you can pass (last document id & limit)
And your query should be like :
import { PaginateModel } from 'mongoose-paginate-v2';
constructor(
private readonly _paginateModel: PaginateModel<any>
) {}
const query = this._paginateModel
.find({ _id: { $lt: lastId } });
this._paginateModel.paginate(query, { limit: 10 });
You can use skip and limit function in which you can pass min and max value .
like in first page if you get 15 records values for min and max will be :
min = 0 and max = 15
for page 2
min = 15 and max = 30
Am working on a windows store javascript application. The application uses data from azure mobile services.
Consider the below code:
var itemTable = mobileService.getTable('item');
//item is the table name stored in the azure database
The code fetches the entire table item and saves it to a variable itemTable.
What code will return the no of rows present in itemTable??
What you're looking for is the includeTotalCount method on the table/query object (unfortunately it's missing from the documentation, I'll file a bug to the product team to have it fixed).
When you call read on the query object, it will return by default 50 (IIRC, the number may be different) elements from it, to prevent a naïve call from returning all elements in a very large table (thus either incurring the outbound bandwidth cost for reserved services, or hitting the quota for free ones). So getting all the elements in the table, and getting the length of the results may not be accurate.
If all you want is the number of elements in the table, you can use the code below: returning zero elements, and the total count.
var table = client.getTable('tableName');
table.take(0).includeTotalCount().read().then(function (results) {
var count = results.totalCount;
new Windows.UI.Popups.MessageDialog('Total count: ' + count).showAsync();
});
If you want to query some elements, and also include the total count (i.e., for paging), just add the appropriate take() and skip() calls, and also the includeTotalCount as well.
If anybody comes here and interested in how to get the totalCount only on C# (like me), then this is how you do it:
var table = MobileService.GetTable<T> ();
var query = table.Take(0).IncludeTotalCount();
IList<T> results = await query.ToListAsync ();
long count = ((ITotalCountProvider)results).TotalCount;
Credit goes to this blog post here
You need to execute read() on the table query and then get the length of the results.
var items, numItems;
itemTable.read().then(function(results) { items = results; numItems = items.length; });
If you are only showing a record count and not the entire results - you should just select the ID column to reduce the amount of data transmitted. I don't see a count() method available yet in the JS Query API to fill this need.
var itemTable = mobileService.getTable('item').select('itemID');