We Keep Coding

node-redis/search query elements by geospatial position and additional indexes

I want to be able to query elements in a redis cache based on 3 different indexes. Those indexes would be:
A MAC address stored as a String.
A number.
A latitude and longitude(to be able to query spatially).
I have seen that Redis has support for multi indexing using redis search and native geospatial api.
so using nodejs and node-redis I have written the following index:
client.ft.create(
'idx:cits',
{
mid: {
type: SchemaFieldTypes.TEXT
},
timestamp: {
type: SchemaFieldTypes.NUMERIC,
sortable: true
},
position: {
type: SchemaFieldTypes.GEO
}
},
{
ON: 'HASH',
PREFIX: 'CITS'
}
)
Now, i would like to insert records on the database that include those 3 parameters plus an additional String that stores some payload. I have tried using
await client.hSet('CITS:19123123:0:0:00:00:5e:00:53:af', {
timestamp: 19123123,
position: {latitude:0, longitude:0},
mid: '00:00:5e:00:53:af',
message: 'payload'
})
But I get the following error:
throw new TypeError('Invalid argument type');
^
TypeError: Invalid argument type
So, i can't add the latitude and longitude that way, I also tried
using the module ngeohash and computing an 11 character wide geohash like so:
await client.hSet('CITS:19123123:0:0:00:00:5e:00:53:af', {
timestamp: 19123123,
position: geohash.encode(0, 0, 11),
mid: '00:00:5e:00:53:af',
message: 'payload'
})
And it does not give any error but when using redis search querys It does not find points near it.
Is it even possible what I am trying to do? If so, how would you input the data to the redis database?
Here is a minimal reproducible example (Im using "ngeohash": "^0.6.3" and "redis": "^4.5.0"):
const { createClient, SchemaFieldTypes } = require('redis')
const geohash = require('ngeohash')
const client = createClient()
async function start(client) {
await client.connect()
try {
// We only want to sort by these 3 values
await client.ft.create(
'idx:cits',
{
mid: {
type: SchemaFieldTypes.TEXT
},
timestamp: {
type: SchemaFieldTypes.NUMERIC,
sortable: true
},
position: {
type: SchemaFieldTypes.GEO
}
},
{
ON: 'HASH',
PREFIX: 'CITS'
}
)
} catch (e) {
if (e.message === 'Index already exists') {
console.log('Skipping index creation as it already exists.')
} else {
console.error(e)
process.exit(1)
}
}
await client.hSet('CITS:19123123:0:0:00:00:5e:00:53:af', {
timestamp: 19123123,
position: geohash.encode(0, 0, 11),
mid: '00:00:5e:00:53:af',
message: 'payload'
})
await client.hSet('CITS:19123123:0.001:0.001:ff:ff:ff:ff:ff:ff', {
timestamp: 19123123,
position: geohash.encode(0.001, 0.001, 11),
mid: 'ff:ff:ff:ff:ff:ff',
message: 'payload'
})
const results = await client.ft.search(
'idx:cits',
'#position:[0 0 10000 km]'
)
console.log(results)
await client.quit()
}
start(client)
Additionally, I would like to ask if there is maybe another type of database that better suits my needs. I have chosen redis because it offers low latency, and that is the biggest constraint in my environment(I will probably do more writes than reads per second). I only want it to act as a inmediate cache, as persistent data will be stored in another database that does not need to be fast.
Thank you.

You get the Invalid argument type error because Redis does not support nested fields in hashes.
"GEO allows geographic range queries against the value in this attribute. The value of the attribute must be a string containing a longitude (first) and latitude separated by a comma" (https://redis.io/commands/ft.create/)

Iterate over JSON data in JavaScript

I am working on some trains' open data feed and getting some JSON as a response from a server. I parse this JSON into a data variable and display it as seen below. However, I cannot find a way to iterate over the response to be able to manipulate each message. I want to iterate over each message and then use the data for a record in a SQL database. I cannot get to the point of accessing any individual message data.
How can I create a loop to iterate over each message and extract it's data?
[
{
SF_MSG: {
time: '1666370311000',
area_id: 'TD',
address: '0C',
msg_type: 'SF',
data: '1F'
}
},
{
CA_MSG: {
to: '4333',
time: '1666370311000',
area_id: 'WO',
msg_type: 'CA',
from: '4331',
descr: '2K60'
}
}, ...
]
Edit: using data.forEach(function(message) produces an output of the structure:
{ CA_MSG: { to: '6021', time: '1666372120000', area_id: 'CY', msg_type: 'CA', from: 'STIN', descr: '2Y73' } }
, however, how do I query this inner object, the names of the objects will differ depending on message type if this matters?

try this:
data = JSON.parse(yourJSONdata)
data.map((o, i)=>{
//o is the object, i is the index
// do your processing here
then at the end do
data[i]=processedobject
})

Getting only the insides of the source from the elastic search query

What im trying to do is to get only the source from the elastic search query in order to skip the processing on the javascript, so i could squize some more performance gains.Is it possible to do so?
So here is what i currenly do, i just get the data from the elastic search and iterate every one of those objects in order to construct an array that only contains the what's inside of _source:
const { body } = await this.elsticSearchService.search<any>({
index: this.configService.get('ELASTICSEARCH_INDEX'),
body: {
query: {
match_all: {},
},
size: 10000,
},
});
const hits = body.hits.hits;
return hits.map((item: any) => item._source);
So my question is, is there a way to only get the _source from the elastic search, in order
to skip the processing on the JavaScript?
So the object returned from the elastic search would look like this
[
0:{ "key": "value"}, // key,value from _source object
1:{ "key": "value"}, // key,value from _source object
2:{ "key": "value"}, // key,value from _source object
]
so without all of the other fields like hits, took etc...

It's not possible to change the structure of the response you get from the search call.
However, what you can do is to specify filter_path so you only get the _source content in the response and you wouldn't need to process it since you know you only have _source content
const { body } = await this.elsticSearchService.search<any>({
index: this.configService.get('ELASTICSEARCH_INDEX'),
filter_path: '**._source', <--- add this line
body: {
query: {
match_all: {},
},
size: 10000,
},
});

Axios sends an array of strings instead of an array of objects

Axios sends an array of strings instead of an array of objects. I have an array of objects that contains data about the event. I'm trying to send a request to the server via axios, but I get a array of strings insteenter image description heread of objects at the output
let data = {
title: 'Game',
subject: 'Some subject',
date: ['01/01/2021','01/01/2021'],
years: ['1970', '1970'],
address: 'None',
ages: [
{
title: 'Men',
weights: [{page: 0, title: '60'}]
}
]
};
api.Create({
params: data
}).then(function (response) {
console.log(response.data);
})
.catch(function (err) {
console.log(err);
});
api response

try to:
console.log(JSON.parse(response.data))
What you get back from the server is string. you need to parse it.
When you send data to and from a server, it is sent as a 'serialized' string, usually JSON format
That's how server responses work

It turned out to be an incorrect request. I used GET to pass the object, instead of POST, so it converts it to a string. I want to notice that there is an evil community on this site.

Send object as get parameter with Axios

The API expects the following GET request:
/resource?labels[team1]=corona&labels[team2]=virus
My issue is to generate this URL using the Axios params option.
I tried the following params structures and they both generate the wrong url:
{
labels: {
team1: 'corona',
team2: 'virus'
}
}
{
labels: [
team1: 'corona',
team2: 'virus'
]
}
I at least thought it would work with the string indexed array but that generates no get params at all.
So, can anyone tell me how to generate the desired URL?

The solution was to use the paramsSerializer with the same setup as in the axios readme. And I used the first params object from my post above..
paramsSerializer: (params) => {
return qs.stringify(params, { arrayFormat: 'brackets' });
}