Automatically stringifying object when inserting to a MySQL JSON column with knex - javascript

Let's jump straight to an example code:
create table test_json_table
(
data json not null
);
I can insert to the table like this:
const columns = { data: "{ some_json: 123 }" }; // notice that the data column is passed as string
await knex('test_json_table').insert(columns);
And get data from the table like this:
await knex('test_json_table').select();
// returns:
// [
// { data: { some_json: 123 } } // notice that the data is returned as parsed JavaScript object (not a string)
// ]
When inserting a row the JSON column needs to be passed as a serialised string. When retrieving the row, an already parsed object is returned.
This is creating quite a mess in the project. We are using TypeScript and would like to have the same type for inserts as for selects, but this makes it impossible. It'd be fine to either always have string or always object.
I found this topic being discussed at other places, so it looks like I am not alone in this (link, link). It seems like there is no way to convert the object to string automatically. Or I am missing something?
It'd be nice if knex provided a hook where we could manually serialise the object into string when inserting.
What would be the easiest way to achieve that? Is there any lightweight ORM with support for that? Or any other option?

You could try objection.js that allows you to declare certain columns to be marked as json attributes and those should be stringified automatically when inserting / updating their values https://vincit.github.io/objection.js/api/model/static-properties.html#static-jsonattributes
I haven't tried if it works with mysql though. I don't see any reason why it wouldn't.

I think the easiest way using jsonb data type. mysql json type
We prefer postgresql for this kind of problem at office, easier and solid database for your problem.

Well you could call your own function before inserting that converts all objects to string and call it every time before you insert.
You can probably wrap knex to do it automatically as well.

Related

JSON_MERGE_PATCH with null values (in Javascript)

As written in the docs, JSON_MERGE_PATCH will remove each value that is set to null, the following example will remove the header value from my settings json field
const data = JSON.stringify({header: null, otherdata: ...})
await connection.query(UPDATE shops SET JSON_MERGE_PATCH(settings, ?), data)
However what if I want to set the value to null, If I surround the header: 'null', with quotes, you can guess it: it enters 'null' as a string into my database.
Does anyone know if it's possible to have mysql update my json field with a null value?
As there doesn't seem to be a pure MySQL solution for this problem, you might be better off implementing this in JavaScript.
You'd implement this something like this:
Fetch all records you want to modify some ID
Use a solution like How can I merge properties of two JavaScript objects dynamically? to merge the objects
Update all records with the new value
An alternate approach could be to use JSON_SET for each object key you have:
UPDATE shops SET JSON_SET(settings, '$.header', null)
-- Then repeat for each json key you want to modify

JSON Array from String

I have an array of JSON plots which I store in MySQL. When I retrieve this information from MySQL it is given as one long string. How can I restore this back into an array of JSON objects using Javascript? I'm running this using NodeJS and MySQL package.
My data is returned like the following:
'[{"x":0,"y":0},{"x":1,y:1},{"x":2,"y":2}]'
What I would like to be able to do is use the data like:
var data = [{"x":0,"y":0},{"x":1,"y":1},{"x":2,"y":2}];
console.log(data[0].x);
I've had a try using JSON.parse and originally stored the data using JSON.stringify on the array, but it is not behaving as I would expect.
Are there any methods or packages available to handle this?
Edit: I realize now that this is not JSON but rather objects. Apologies for the wrong terminology here, but my problem still remains.
var data = new Function ('return ' + dataString)();

MongoDB - How to build query string with [Object] as a field

I'm building up a string in Meteor to drill down into my data from MongoDB.
My data looks like this:
Data
In my Meteor projects JavaScript I have built up the string like so:
const concentrationTier1 = MyCollection.findOne({_id: "85gh43tnb23v4"}).BILL.Sovereign.USD.Short.Low.High.N.ARGENTINA.IssueName00006.ARARGE5203E7;
console.log(concentrationTier1);
But now in my console it is returning the following:
Console
How would I add [Object] to my string to be able to display the next part of the data?
I have tried .[Object] .Object .0 and of course these didn't work.
Can any body help with this one?
Many thanks,
G
You would have to access that array element as in plain normal javascript, like this:
...IssueName00006.ARARGE5203E7[0].concentrationTier1
And the reason is your MongoDB query already returned a document, you're not querying in your database anymore so there's no need to use dot notation to access array elements.

pyodbc result set has data type embedded

I am querying Teradata using pyodbc and flask with the intention of charting the data using d3.
The result set has two columns, one is a decimal and the other an integer. When I pass the results to my html page and log the output I am getting something along these lines:
[(Decimal(& #39;-16.200000000& #39;), 5), (Decimal(& #39;-23.100000000& #39;), 12), (Decimal(& #39;500.300000000& #39;), 5)].
The embedded data type information is making it difficult to do anything with the result set. How can I get output to look like this instead?
[[-16.200000000, 5], [-23.100000000, 12], [500.300000000, 5]]
In other words I just want an array of arrays.
Steps I am following:
create a connection
create a cursor
execute the SQL
use fetchall() to store the rows in a variable
pass the variable to my html page using render_template
in javascript set variable equal to data passed in
var data={{dataset}};
console.log(data);
I have seen many flask examples where they take the result set and iterate through it to print the lines in the html but I want to use the resulting dataset as an input to my d3 code. Why is it displaying the data type? And why would it not also display the data type of the integer column?
I am guessing the issue is related to the row construct but I have tried creating a list instead and can't get rid of the data type information. I then get all sorts of errors due to the ampersands.
I think the better approach is as provided in this answer, as it is best to dumps the data of the query into a json object so you can easily retrieve it from the client side using JS, without the need to parse it using regex or other mechanism.
STEPS:
1 - Subclass the json.JSONEncoder:
class DecimalEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, decimal.Decimal):
return float(o) #Type conversion happens here
return super(DecimalEncoder, self).default(o)
2 - Pass the params to the template as a json object:
data = json.dumps({'pi': decimal.Decimal('3.14')}, cls=DecimalEncoder)
return render_template('mytemplate.html', dataset=data)
3 - Read back the result of the query in your JS:
var data = {{dataset}};
import re
processed = []
for piece in results:
processed.append([re.sub(".*?(-?\d+\.\d+).*", "\1", repr(piece[0])), piece[1]]
Something along those lines. It's displaying that data because it's of type Decimal (the teradata kind). I'm guessing repr() will give you the same results you're seeing in the log... then just regex it to what you want.

Dictionary Object confusion from jQuery to Django!

I'm attempting to send a dictionary from jQuery to Django using a getJSON call:
jQuery.getJSON(URL,JSONData,function(returnData){});
The JSONData object is formatted as follows:
JSONData = {
year:2010101,
name:"bob",
data:{
search:[jim,gordon],
register:[jim],
research:[dave],
}
}
This is put together programmatically but looks fine.
Once passed to Django the "year" and "name" objects are as expected. The data object however contains the following keys/values - "search[0]":"jim", "search[1]":"gordon","register[0]":"jim","research[0]":"dave", rather than the expected "search":(array of data), "register":(array of data), "research":(array of data).
Similar things happen if I use objects in place of the arrays.
Is this an issue with Django's interpretation of the object?
Any idea how I might correct this...cleanly?
EDIT:
I have now simplified the data to make testing easier:
JSONData = {
year:2010101,
name:"bob",
search:[jim,gordon],
register:[jim],
research:[dave],
}
request.GET is not an instance of a normal python dict, but of the django class QueryDict, that can deal with multiple values for one key. If you need multiple values for a key returned as a list you have to use getList!
EDIT: Also have a look at this jQuery parameter settings!

Categories