We Keep Coding

Pouch DB data is not inserted in sorted order

I am working on small form in electron and pouchdb.
When inserting a data into db, until the ninth doc it inserting perfectly, but on 10th doc it is inserting to second position of doc instead on inserting into last
before inserting the 10th doc
{
"total_rows": 9,
"offset": 0,
"rows": [
{ "id": "1", "key": "1", "value": {...} },
{ "id": "2", "key": "2", "value": {...} },
{ "id": "3", "key": "3", "value": {...} },
{ "id": "4", "key": "4", "value": {...} }
{ "id": "5", "key": "5", "value": {...} },
{ "id": "6", "key": "6", "value": {...} },
{ "id": "7", "key": "7", "value": {...} },
{ "id": "8", "key": "8", "value": {...} },
{ "id": "9", "key": "9", "value": {...} }
]
}
After inserting 10th doc
{
"total_rows": 10,
"offset": 0,
"rows": [
{ "id": "1", "key": "1", "value": {...} },
{ "id": "10", "key": "10", "value": {...} },
{ "id": "2", "key": "2", "value": {...} },
{ "id": "3", "key": "3", "value": {...} },
{ "id": "4", "key": "4", "value": {...} }
{ "id": "5", "key": "5", "value": {...} },
{ "id": "6", "key": "6", "value": {...} },
{ "id": "7", "key": "7", "value": {...} },
{ "id": "8", "key": "8", "value": {...} },
{ "id": "9", "key": "9", "value": {...} }
]
}
here attached the js code
// form submit
const form = document.getElementById("form1");
form.addEventListener("submit", dbsubmit);
function dbsubmit(e) {
e.preventDefault();
// getting values from form
var Sno = document.getElementById("number").value
var date = document.getElementById("date").value;
var Time = document.getElementById("time").value;
var Trip = document.querySelector('input[name="Trip"]:checked').value;
var TripType = document.querySelector('input[name="Type"]:checked').value;
// assigning form values to db table names
var doc = {
_id: Sno,
Date: date,
time: Time,
trip: Trip,
triptype: TripType,
};
// inserting to db
db.put(doc, function(err, response) {
if (err) {
return console.log(err);
} else {
console.log("Document created Successfully", response);
}
});
}
Inserting until ninth 9th doc it's perfectly inserting but when inserting the 10th doc it is inserting in second position.
The doc should be sorted by id and want to use it to view in another page.
I debugged but cannot find a solution, can anyone help me with the solution?
Thank you

You have an errant notion regarding document insertion, whether as if the order of insertion with respect to time matters which is to be expected from say an RDBMS like MS/SQL, or a belief that a string representation of a numeric value should be sorted according to the implied numerical value (which would be wild guessing by the db).
As explicitly stated here, here and here, and as specified by the CouchDB documentation 3.2.2.5. Collation Specification
Comparison of strings is done using ICU which implements the Unicode Collation Algorithm, giving a dictionary sorting of keys. This can give surprising results if you were expecting ASCII ordering.
Documents are not returned by order of insertion rather they are returned according to the collation rules applied to the document ids, and document ids are strings.
For example you expect this ordering:
row #
id
0
1
1
2
2
3
3
4
4
5
5
6
6
7
7
8
8
9
9
10
BUT that is not according to the collation rules, which yields
row #
id
0
1
1
10
2
2
3
3
4
4
5
5
6
6
7
7
8
8
9
9
Again, it cannot be emphasized enough that the order of insertion, much less an implied numeric value, does not affect the order of results. From an RDBMS perspective it is helpful to approach _all_docs results as being ordered by an index on _id which is sorted according to collation rules (again, for strings since all _ids are of type string).
The snippet below demonstrates this by creating 100 documents with ids ranging from 1 to 100.
// generate canned test documents with doc _id's in the range 1 to 100.
function getDocsToInstall() {
let docs = [];
for (let i = 1; i < 101; i++) {
// _id must be a string type because that's the way it is, period.
docs.push({
_id: i.toString()
});
}
return docs;
}
let db;
// init db instance
async function initDb() {
db = new PouchDB('test', {
adapter: 'memory'
});
await db.bulkDocs(getDocsToInstall());
}
initDb().then(async() => {
let options = {
limit: 25,
include_docs: false,
reduce: false
};
const result = await db.allDocs(options);
showResult(result);
});
function gel(id) {
return document.getElementById(id);
}
function cel(name) {
return document.createElement(name);
}
function addColumn(tr, value) {
const td = cel('td');
td.innerText = value;
tr.append(td);
}
function addRow(row, index) {
const tr = cel('tr');
[index, row.id].forEach(value => addColumn(tr, value));
gel('table').append(tr);
}
function showResult(result) {
const table = gel('table');
result.rows.forEach((row, index) => {
addRow(row, index);
});
}
th,
td {
padding: .5em;
text-align: left;
min-width: 4em;
}
table {
border-collapse: collapse;
}
<script src="https://cdn.jsdelivr.net/npm/pouchdb#7.1.1/dist/pouchdb.min.js"></script>
<script src="https://github.com/pouchdb/pouchdb/releases/download/7.1.1/pouchdb.memory.min.js"></script>
<table id="table" border="1">
<tr>
<th>row #</th>
<th>id</th>
</tr>
</table>
The must-get takeaway from the snippet is that the first 3 documents returned are
row #
id
0
1
1
10
2
100
As janl pointed out here if you need to access documents in terms of numerical ordering, then creating a view is a good option, which the following snippet demonstrates.
The snippet shows results from _all_docs and the my_index view side by side.
// generate canned test documents with doc _id's in the range 1 to 100.
function getDocsToInstall() {
let docs = [];
for (let i = 1; i < 101; i++) {
// _id must be a string type because that's the way it is, period.
docs.push({
_id: i.toString()
});
}
// add a design doc/view for numerically ordered documents
docs.push({
_id: '_design/my_ddoc',
views: {
my_index: {
map: function(doc) {
emit(Number(doc._id));
}.toString()
}
}
})
return docs;
}
let db;
// init db instance
async function initDb() {
db = new PouchDB('test', {
adapter: 'memory'
});
await db.bulkDocs(getDocsToInstall());
}
initDb().then(async() => {
let options = {
limit: 25,
include_docs: false,
reduce: false
};
const allDocs = await db.allDocs(options);
const view = await db.query('my_ddoc/my_index');
showResult(allDocs, view);
});
function gel(id) {
return document.getElementById(id);
}
function cel(name) {
return document.createElement(name);
}
function addColumn(tr, value) {
const td = cel('td');
td.innerText = value;
tr.append(td);
}
function addRow(index, allDocs, view) {
const tr = cel('tr');
[index, allDocs.rows[index].key, view.rows[index].key].forEach(value => addColumn(tr, value));
gel('table').append(tr);
}
// show allDocs and view results side by side
function showResult(allDocs, view) {
const table = gel('table');
for (let i = 0; i < allDocs.rows.length; i++) {
addRow(i, allDocs, view);
};
}
th,
td {
padding: .5em;
text-align: left;
min-width: 4em;
}
table {
border-collapse: collapse;
}
<script src="https://cdn.jsdelivr.net/npm/pouchdb#7.1.1/dist/pouchdb.min.js"></script>
<script src="https://github.com/pouchdb/pouchdb/releases/download/7.1.1/pouchdb.memory.min.js"></script>
<table id="table" border="1">
<tr>
<th>row #</th>
<th>_all_docs key</th>
<th>my_index key</th>
</tr>
</table>
Note that for _all_docs, key and id are the document _id. It's important to understand that _all_docs is in fact a built-in view which explains why key and id are present in _all_docs results.
If in fact you want documents ordered with respect to time, then add a creation timestamp to documents and build a view based on that timestamp, e.g. the map function might look like this if creationTime is an integer representing a point in time (milliseconds) in the Unix epoch.
function(doc) {
emit(doc.creationTime);
}

How to add values from multiple JSON object and store the value in another JSON object in Angular 6

I have two different JSON object. One object is empList and other one is holidayList.
I want to add hours from each JSON object.Andthe sum of hours should be pushed to sumHoursList JSON object.I am doing this using Angular6.
I am not getting exactly how to iterate this to get the required result.
Basically I want to add hours from both the datalist of empList , to that want to add hours from holiday list, and the sum value should
append in sumhourlist
Below is my code.
this.empList = [
{
'id': 1,
'name': 'Name1',
datalist: [
{
"date": 1,
"hours": 6
},
{
"date": 2,
"hours": 0
},
{
"date": 3,
"hours": 12
}
]
},
{
'id': 2,
'name': 'Name2',
datalist:[
{
"date": 1,
"hours": 0
},
{
"date": 2,
"hours": 8
},
{
"date": 3,
"hours": 0
}
]
},
];
this.holidayList=[
{
"date": 1,
"hours": 0
},
{
"date": 2,
"hours": 8
},
{
"date": 3,
"hours": 12
}
]
sumHoursList = [
{
"date": 1,
"hours": 6
},
{
"date": 2,
"hours": 16
},
{
"date": 3,
"hours": 24
}
]
Can anyone please help me how to do this.

If I understand you correctly, you need to sum the hour values of all entries with the same date and it's irrelevant in which source they are located.
Basically you could do it like this (not the most efficient way but to give you an idea):
// map sources into a unified data structure
const extractedArraysFromEmpList = emplist.map((entry)=>entry.datalist)
const sources:Array<Array<{
date:number,
hours:number
}>> = [...extractedArraysFromEmpList,holidaylist];
// now reduce the various sources into one summed array
const result = sources.reduce((reducer:Array<{date:number,hours:number}>, singleSourceArray)=>{
singleSourceArray.forEach((item:{date:number,hours:number}) => {
const existingEntry = result.find((summedItem)=>summedItem.date===item.date);
// if a entry with matching date already exists, sum hours
if(existingEntry) existingEntry.hours += item.hours;
// if no entry exists, add it.
else result.push({date:item.date,hours:item.hours})
}
}, [])
// result is sumHoursList
Note that code is untested. But it should give you an idea of how to solve this case.

This is how I understand the question. You want for each date add the hours together. With this loop you emp array can contain a dynamic number of datalists.
let sum = holidayList
empList.forEach((emp) => emp.datalist.forEach((obj, i) => sum[i].hours += obj.hours))
console.log(sum)
// output: [{ date: 1, hours: 6 }, { date: 2, hours: 16 }, { date: 3, hours: 24 }]
Please also note that these are not JSON objects. They are JavaScript objects.

Elasticsearch - find closest number when scoring results

I need a way to match the closest number of an elasticsearch document.
I'm wanting to use elastic search to filter quantifiable attributes and have been able to achieve hard limits using range queries accept that results that are outside of that result set are skipped. I would prefer to have the closest results to multiple filters match.
const query = {
query: {
bool: {
should: [
{
range: {
gte: 5,
lte: 15
}
},
{
range: {
gte: 1979,
lte: 1989
}
}
]
}
}
}
const results = await client.search({
index: 'test',
body: query
})
Say I had some documents that had year and sales. In the snippet is a little example of how it would be done in javascript. It runs through the entire list and calculates a score, then based on that score it sorts them, at no point are results filtered out, they are just organized by relevance.
const data = [
{ "item": "one", "year": 1980, "sales": 20 },
{ "item": "two", "year": 1982, "sales": 12 },
{ "item": "three", "year": 1986, "sales": 6 },
{ "item": "four", "year": 1989, "sales": 4 },
{ "item": "five", "year": 1991, "sales": 6 }
]
const add = (a, b) => a + b
const findClosestMatch = (filters, data) => {
const scored = data.map(item => ({
...item,
// add the score to a copy of the data
_score: calculateDifferenceScore(filters, item)
}))
// mutate the scored array by sorting it
scored.sort((a, b) => a._score.total - b._score.total)
return scored
}
const calculateDifferenceScore = (filters, item) => {
const result = Object.keys(filters).reduce((acc, x) => ({
...acc,
// calculate the absolute difference between the filter and data point
[x]: Math.abs(filters[x] - item[x])
}), {})
// sum the total diffences
result.total = Object.values(result).reduce(add)
return result
}
console.log(
findClosestMatch({ sales: 10, year: 1984 }, data)
)
<script src="https://codepen.io/synthet1c/pen/KyQQmL.js"></script>
I'm trying to achieve the same thing in elasticsearch but having no luck when using a function_score query. eg
const query = {
query: {
function_score: {
functions: [
{
linear: {
"year": {
origin: 1984,
},
"sales": {
origin: 10,
}
}
}
]
}
}
}
const results = await client.search({
index: 'test',
body: query
})
There is no text to search, I'm using it for filtering by numbers only, am I doing something wrong or is this not what elastic search is made for and are there any better alternatives?
Using the above every document still has a default score, and I have not been able to get any filter to apply any modifiers to the score.
Thanks for any help, I new to elasticsearch links to articles or areas of the documentation are appreciated!

You had the right idea, you're just missing a few fields in your query to make it work.
It should look like this:
{
"query": {
function_score: {
functions: [
{
linear: {
"year": {
origin: 1984,
scale: 1,
decay: 0.999
},
"sales": {
origin: 10,
scale: 1,
decay: 0.999
}
}
},
]
}
}
}
The scale field is mandatory as it tells elastic how to decay the score, without it the query just fails.
The decay field is not mandatory, however without it elastic does not really know how to calculate the new score to documents so it will end up giving a default score only to documents in the range of origin + scale which is not useful for us.
source docs.
I also recommend you limit the result size to 1 if you want the top scoring document, otherwise you'll have add a sort phase (either in elastic or in code).
EDIT: (AVOID NULLS)
You can add a filter above the functions like so:
{
"query": {
"function_score": {
"query": {
"bool": {
"must": [
{
"bool": {
"filter": [
{
"bool": {
"must": [
{
"exists": {
"field": "year"
}
},
{
"exists": {
"field": "sales"
}
},
]
}
}
]
}
},
{
"match_all": {}
}
]
}
},
"functions": [
{
"linear": {
"year": {
"origin": 1999,
"scale": 1,
"decay": 0.999
},
"sales": {
"origin": 50,
"scale": 1,
"decay": 0.999
}
}
}
]
}
}
}
Notice i have a little hack going on using match_all query, this is due to filter query setting the score to 0 so by using the match all query i reset it back to 1 for all matched documents.
This can also be achieved in a more "proper" way by altering the functions, a path i choose not to take.

Sorting array of object javascript / typescript on firestore timestamp

I have below array structure
[
{
"id": "8gFUT6neK2I91HIVkFfy",
"element": {
"id": "8gFUT6neK2I91HIVkFfy",
"archived": false,
"updatedOn": {
"seconds": 1538653447,
"nanoseconds": 836000000
}
},
"groupBy": "pr"
},
{
"id": "9jHfOD8ZIAOX4fE1KUQc",
"element": {
"id": "9jHfOD8ZIAOX4fE1KUQc",
"archiveDate": {
"seconds": 1539250407,
"nanoseconds": 62000000
},
"archived": false,
"updatedOn": {
"seconds": 1538655984,
"nanoseconds": 878000000
}
},
"groupBy": "pr"
},
{
"id": "CeNP27551idLysSJOd5H",
"element": {
"id": "CeNP27551idLysSJOd5H",
"archiveDate": {
"seconds": 1539248724,
"nanoseconds": 714000000
},
"archived": false,
"updatedOn": {
"seconds": 1538651075,
"nanoseconds": 235000000
}
},
"groupBy": "pr"
},
{
"id": "Epd2PVKyUeAmrzBT3ZHT",
"element": {
"id": "Epd2PVKyUeAmrzBT3ZHT",
"archiveDate": {
"seconds": 1539248726,
"nanoseconds": 226000000
},
"archived": false,
"updatedOn": {
"seconds": 1538740476,
"nanoseconds": 979000000
}
},
"groupBy": "pr"
}
]
and below code to sort
Sample JSfiddle
http://jsfiddle.net/68wvebpz/
let sortedData = this.arraydata.sort((a:any, b:any) => { return Number(new Date(b.element.date).getTime()) - Number(new Date(a.element.date).getTime()) })
This does not make any effect.

There are a few problems that we need to fix:
Your updatedOn object is not something that can be converted to a date. You need to do extra work.
JavaScript doesn't support nanoseconds, only milliseconds. You will therefore need to divide that number by a million.
By using getTime for the comparison, you're actually discarding the milliseconds - that function returns seconds.
To fix the first two, use this function:
function objToDate(obj) {
let result = new Date(0);
result.setSeconds(obj.seconds);
result.setMilliseconds(obj.nanoseconds/1000000);
console.log('With nano', result);
return result;
}
This creates a new date and then sets the seconds and milliseconds. This gives you dates in October 2018 when I use your test data.
Then, to compare them and fix the remaining problems, use this (much simpler) form:
let sortedData = data.sort((a:any, b:any) => {
let bd = objToDate(b.element.updatedOn);
let ad = objToDate(a.element.updatedOn);
return ad - bd
});
That should do it.
To reverse the sort order, just use the less-than operator:
return bd - ad

Turn your strings into dates, and then subtract them to get a value that is either negative, positive, or zero:
array.sort(function(a,b){
return new Date(b.date) - new Date(a.date);
});

Is it something like this:
var array = [
{id: 1, name:'name1', date: 'Mar 12 2012 10:00:00 AM'},
{id: 2, name:'name2', date: 'Mar 8 2012 08:00:00 AM'}
];
console.log(array.sort((a, b) => {
return new Date(a.date) - new Date(b.date)
}))

Javascript: group by with aggregation

I have a simple json list like the one below
{
"myList": [
{
"endOfPeriod": 1461362400000,
"rate": 0.03726378
},
{
"endOfPeriod": 1461535200000,
"rate": 0.03726378
},
{
"endOfPeriod": 1461967200000,
"rate": 0.03708314
},
{
"endOfPeriod": 1461708000000,
"rate": 0.03492851
},
{
"endOfPeriod": 1461794400000,
"rate": 0.03845068
},
{
"endOfPeriod": 1461621600000,
"rate": 0.03544827
}
]
}
Where endOfPeriod is a unix epoch timestamp. All the timestamps in the example belong to the same month (April 2016), but could be some other periods.
Assuming that I have already converted this json list into an array, and each unix timestamp into a DD.MM.YYYY date (I can keep them in unix timestamp too). Is there an efficient way to create a new array with the most recent rate for grouped by month/year?
I have to write code in Javascript.
For instance:
20.04.2016 / 0.33
21.04.2016 / 0.55
14.04.2016 / 0.88
02.05.2016 / 1.33
01.05.2016 / 5.44
New array must contain:
21.04.2016 / 0.55
02.05.2016 / 1.33
Thanks for your help.

If I understand correctly, you want to extract the most recent rate for each month. I would use lodash.
_.chain(arr)
.groupBy(function(item) {
var date = new Date(item.endOfPeriod);
return date.getFullYear() + '-' + date.getMonth();
})
.map(function(group) {
return _.maxBy(group, function(item) {
return item.endOfPeriod;
});
})
.value()
We start with a list of objects in the form:
{
"endOfPeriod" : 1464818400000,
"rate" : 0.05
}
The chain() function wraps the list into a lodash object.
Then, we group elements by year and month. After the groupBy(), we have the following structure (note that getMonth() is 0-based in Javascript, hence a value of 3 corresponds to April, and so on):
{
"2016-3" : [array of objects in April 2016],
"2016-4" : [array of objects in May 2016]
...
}
Then, for each group, we take the item with maximum endOfPeriod.
Finally, value() unwraps the lodash object back into a plain Javascript array.

Here is a result without using lodash. But for me it's better not to reinvent the wheel.
const myList = [
{
"endOfPeriod": 1461362400000,
"rate": 0.03726378
},
{
"endOfPeriod": 1461535200000,
"rate": 0.03726378
},
{
"endOfPeriod": 1461967200000,
"rate": 0.03708314
},
{
"endOfPeriod": 1461708000000,
"rate": 0.03492851
},
{
"endOfPeriod": 1461794400000,
"rate": 0.03845068
},
{
"endOfPeriod": 1461621600000,
"rate": 0.03544827
}
];
const res = myList.reduce((prev, current) => {
const date = new Date(current.endOfPeriod);
const month = date.getMonth();
const year = date.getFullYear();
const key = `${year}-${month}`;
if (prev[key] && prev[key].endOfPeriod < current.endOfPeriod) {
prev[key] = current;
} else {
prev[key] = current;
}
return prev;
}, {});
const finalResult = Object.keys(res).map((key) => {
return {
key: res[key].rate
}
});
console.log(finalResult);