We Keep Coding

node-redis/search query elements by geospatial position and additional indexes

I want to be able to query elements in a redis cache based on 3 different indexes. Those indexes would be:
A MAC address stored as a String.
A number.
A latitude and longitude(to be able to query spatially).
I have seen that Redis has support for multi indexing using redis search and native geospatial api.
so using nodejs and node-redis I have written the following index:
client.ft.create(
'idx:cits',
{
mid: {
type: SchemaFieldTypes.TEXT
},
timestamp: {
type: SchemaFieldTypes.NUMERIC,
sortable: true
},
position: {
type: SchemaFieldTypes.GEO
}
},
{
ON: 'HASH',
PREFIX: 'CITS'
}
)
Now, i would like to insert records on the database that include those 3 parameters plus an additional String that stores some payload. I have tried using
await client.hSet('CITS:19123123:0:0:00:00:5e:00:53:af', {
timestamp: 19123123,
position: {latitude:0, longitude:0},
mid: '00:00:5e:00:53:af',
message: 'payload'
})
But I get the following error:
throw new TypeError('Invalid argument type');
^
TypeError: Invalid argument type
So, i can't add the latitude and longitude that way, I also tried
using the module ngeohash and computing an 11 character wide geohash like so:
await client.hSet('CITS:19123123:0:0:00:00:5e:00:53:af', {
timestamp: 19123123,
position: geohash.encode(0, 0, 11),
mid: '00:00:5e:00:53:af',
message: 'payload'
})
And it does not give any error but when using redis search querys It does not find points near it.
Is it even possible what I am trying to do? If so, how would you input the data to the redis database?
Here is a minimal reproducible example (Im using "ngeohash": "^0.6.3" and "redis": "^4.5.0"):
const { createClient, SchemaFieldTypes } = require('redis')
const geohash = require('ngeohash')
const client = createClient()
async function start(client) {
await client.connect()
try {
// We only want to sort by these 3 values
await client.ft.create(
'idx:cits',
{
mid: {
type: SchemaFieldTypes.TEXT
},
timestamp: {
type: SchemaFieldTypes.NUMERIC,
sortable: true
},
position: {
type: SchemaFieldTypes.GEO
}
},
{
ON: 'HASH',
PREFIX: 'CITS'
}
)
} catch (e) {
if (e.message === 'Index already exists') {
console.log('Skipping index creation as it already exists.')
} else {
console.error(e)
process.exit(1)
}
}
await client.hSet('CITS:19123123:0:0:00:00:5e:00:53:af', {
timestamp: 19123123,
position: geohash.encode(0, 0, 11),
mid: '00:00:5e:00:53:af',
message: 'payload'
})
await client.hSet('CITS:19123123:0.001:0.001:ff:ff:ff:ff:ff:ff', {
timestamp: 19123123,
position: geohash.encode(0.001, 0.001, 11),
mid: 'ff:ff:ff:ff:ff:ff',
message: 'payload'
})
const results = await client.ft.search(
'idx:cits',
'#position:[0 0 10000 km]'
)
console.log(results)
await client.quit()
}
start(client)
Additionally, I would like to ask if there is maybe another type of database that better suits my needs. I have chosen redis because it offers low latency, and that is the biggest constraint in my environment(I will probably do more writes than reads per second). I only want it to act as a inmediate cache, as persistent data will be stored in another database that does not need to be fast.
Thank you.

You get the Invalid argument type error because Redis does not support nested fields in hashes.
"GEO allows geographic range queries against the value in this attribute. The value of the attribute must be a string containing a longitude (first) and latitude separated by a comma" (https://redis.io/commands/ft.create/)

how to query dynamodb GSI with multiple operators

Lets say i have an entity with this model:
{
id: 'apples',
createdAt: 'some date'
rate: 430,
side: 'SELL',
status: 'OPEN',
GSI1: 'SELL#OPEN#430'
GSI2: 'apples'
}
i want to query using the GSI attributes of GSI2 beign the hash and GSI1 being the range.
The query im looking for is get all apples(GSI2) where GSI1 begins with SELL#OPEN and >= SELL#OPEN#430 so basically im trying to get all apples being sold for 430 or greater and are open.
Please how do i go about this using dynamodb query?
what i have done is:
params = {
TableName: process.env.ORDERS_TABLE_NAME,
IndexName: "GSI2_GSI1",
KeyConditionExpression: `GSI2 = :item and ((begins_with(GSI2, :sideStatus) and >= :baseRate)`,
ExpressionAttributeValues: {
":item": `apple`,
":baseRate": `SELL#OPEN#${rate}`,
":sideStatus": "SELL#OPEN",
},
};
thanks

You can only operate on the Key attributes in the key condition expression. These parameters should do what you want because you have all the information in the GSI1 attribute
params = {
TableName: process.env.ORDERS_TABLE_NAME,
IndexName: "GSI2_GSI1",
KeyConditionExpression: 'GSI2 = :item and GSI1 BETWEEN :lower AND :upper',
ExpressionAttributeValues: {
":item": `apple`,
":lower": `SELL#OPEN#430`,
":upper": "SELL#OPEN#999", // you can probably also use "SELL#OPEN$" because $ is the character following # in ascii order
},
};
Note this assumes that your rate in the GSI1 attribute is left padded with 0s. You need the string to be sorted in the same order as the numbers so if the rate is 10 then you need to store SELL#OPEN#010. (Note you might need more leading 0s depending on the maximum rate.)

Facebook Messenger, Temporary send message failure when sending receipt

I want to send to user my receipt with dummy data.
I use this library which simplifies message sending to Facebook.
The structure of my payload is this:
var payload = {
template_type: 'receipt',
recipient_name: '#' + user.name + ' ' + user.surname,
order_number: (new Date).getTime(),
currency: 'USD',
payment_method: 'Наличными',
order_url: 'http://www.example.com',
timestamp: (new Date).getTime() + '',
elements: [
{
title: title,
subtitle: subtitle,
quantity: 1,
price: 20,
currency: 'USD',
image_url: image_url
}
],
address: {
street_1:"Nurly tau",
street_2:"",
city:"Almaty",
postal_code:"050000",
state:"KZ",
country:"KZ"
},
summary: {
subtotal: 20,
shipping_cost: 0,
total_tax: 0,
total_cost: 20
},
adjustments: []
};
I have just filled receipt fields with simple fake data. Also, Facebook tracks the uniqueness of order_numbers of all sent recepts.
When I try to send this receipt I receive an error message:
{ message: '(#1200) Temporary send message failure. Please try again later',
type: 'OAuthException',
code: 1200,
fbtrace_id: 'BHmHRCEQUC4' }
What does this error mean? Facebook's error messages are so enigmatic?

I just had the same problem and after some fiddling I figured it out! The problem is that when you construct the timestamp using (new Date).getTime() it returns the amount of miliseconds since epoch. However, Facebook requires it to be in seconds.

I had the same problem, after a lot of tries, I figured out that the problem is with the timestamp parameter passed with the JSON payload.
I have no clue about what it could be, but it worked for me removing that. (Maybe, the timestamp should be for a moment before the API call, I don't know).

Weird value returned for sum aggregation in Elasticsearch Javascript client

I'm trying to perform a sum aggregation on a numeric field (with type double) using the Javascript client for Elasticsearch.
Here's my code:
this.client.search({
index: "customers",
body: {
aggs: {
counts_in_range: {
filter: {
range: {
timestamp : {
gte : startDate,
lt : endDate
}
}
},
aggs: {
counts: {
sum : {
field : "price"
}
}
}
}
}
}
}).then(function (resp) {
cb(resp.aggregations, null);
}, function (err) {
cb(null, err);
});
Example document:
{
_index: "customers",
_type: "purchase",
_id: "98cb1066-057b-48e1-adff-eb32d9ed75a5",
_score: 1,
_source: {
timestamp: "2014-06-11T18:14:36+03:00",
itemId: 1,
price: 0.54
}
}
What I get back from the aggregation is a very long number e.g. 27549779928520990000 instead of a decimal number. The problem seems to be that in my document I store decimal numbers and not integers. If I store an integer in the price field the aggregation works just fine.
Not sure if this is a parsing issue with the Javascript client.

When you're first indexing a document, the type for each field is decided by elasticsearch if none is specified. In your case, elasticsearch thinks you're storing integers in your document, even if you will later store decimal numbers. So when it's computing the sum, it will try to work only with integers, but it came across to decimal numbers, hence the long number returned.
To avoid that from happening, map each field to a core value when creating a new type of document.

Linkedin API - get number of connections

I can successfully download LinkedIn connections, but now I am trying to get one extra object for the user - their number of connections. The documentation seems to suggest that I need "total" or "_total" but I have tried all combinations with no success. All I get is "undefined". Here's the code with examples of what I'm trying to get the total count:
IN.API.Connections("me")
.fields(["id", "firstName", "lastName", "mainAddress", "dateOfBirth", "phoneNumbers", "positions", "pictureUrl"])
.params({"count":500})
.error(displayError)
.result(function(result) {
document.write("1 ", result.values._total, "<br>");
document.write("2 ", result.values.total, "<br>");

The returned JSON is of the format:
{
"_count": #,
"_start": #,
"_total": #,
"values": [
{
...
},
{
...
}
]
}
So you would access the total number of connections returned via result._total.