I am migrating few queries from Google BigQuery to MySQL and need help in replicating the below BigQuery Java script UDF to equivalent MySQL. I don't see any reference over the internet. Does MySQL support Java Script UDFs ?
The requirement here is to Split a JSON array into a simple array of string ( each string represents individual JSON string ).
CREATE OR REPLACE FUNCTION `<project>.<dataset>.json2array`(json STRING) RETURNS ARRAY<STRING> LANGUAGE js AS R"""
if (json) {
return JSON.parse(json).map(x=>JSON.stringify(x));
} else {
return [];
}
""";
No, MySQL does not support JavaScript stored functions. MySQL supports stored functions written in a procedural language. It also supports server-loadable functions compiled from C or C++, but these are less common.
MySQL doesn't have an ARRAY data type. The closest you can get in MySQL is a JSON data type, which may be a one-dimensional array of strings. If your JSON document is assured to be an array in that format, then you can simply do the following:
CREATE FUNCTION json2array(in_string TEXT) RETURNS JSON DETERMINISTIC
RETURN CAST(in_string AS JSON);
I'm not sure what the point of creating a stored function is in this case, since it only does what CAST() can do. So you might as well just call CAST() and skip creating the stored function.
Perhaps a good use of a stored function is to test the input to make sure it's a document with an array format (use JSON_TYPE()). For example:
CREATE FUNCTION json2array(in_string TEXT) RETURNS JSON DETERMINISTIC
RETURN IF(JSON_TYPE(in_string) = 'ARRAY', CAST(in_string AS JSON), JSON_ARRAY());
Related
Consider a list of numpy arrays:
arr = [np.linspace(a1,a2,11) for a1,a2 in [(1,10),(20,30)]]
nparr = np.array(arr)
I would like to serialize this to transmit to a Javascript REST client. The preferred approach is
Efficiently serialize into a binary-safe format and bake that into a Base64 encoded field in a JSON object
Transmit the JSON object over http
Receive the JSON object into javascript listener.
Base64 decode the field and deserialize into binary array using an efficient javascript deserialization library
I have done an initial investigation into apache arrow that has support in both languages.
Note: I tried the following:
convert to two dimensional numpy array
convert to pyarrow
Following happened
pyarr = pya.array(nparr)
ArrowInvalid Traceback (most recent call last)
<ipython-input-11-68eb3e5f578f> in <module>
----> 1 pyarr = pya.array(nparr)
ArrowInvalid: only handle 1-dimensional arrays
So pyarrow seems pretty limited in terms of the structures of the data it can serialize. I also am looking into the apache parquet format : but that seems to require actually writing to disk/filesystem?
Working code for those two technologies or possibly a different library/approach would be welcome.
Arrow is capable of serializing list of arrays of float. But I think it needs a little help if the list is multi dimension numpy array:
pa.array(
arr.tolist(),
pa.list_(pa.float64())
)
But given your use case, since all arrays have got the same length, I'd recomment using a Table instead an Array
schema = pa.schema(
[
pa.field(str(i), pa.float64())
for i in range(len(nparr))
]
)
table = pa.Table.from_arrays(
nparr,
schema=schema
)
On sql server : Out put : 0x5C8C8AAFE7AE37EA4EBDF8BFA01F82B8
SELECT HASHBYTES('MD5', convert(varchar,getdate(),112)+'mytest#+')
On JavaScript : Out put : 5c8c8aafe7ae37ea4ebdf8bfa01f82b8
//to get Md5 Hash bytes
vm.getMd5Hashbytes = function () {
var currentDate = moment().format('YYYYMMDD');
var md5Hash = md5.createHash(currentDate + 'mytest#+');
return md5Hash;
}
angular-md5 module
Q : Can you tell me why this difference ? SQL server shows 0x as prefix.Why ?
This is purely a formatting issue. Both versions are producing an identical sequence of bytes. SQL Server and node just have different conventions when it comes to presenting these bytes in a human readable format.
You can get similar formatting by specifically telling SQL Server how to format your binary data
declare #hashAsBinary varbinary(max)
declare #hashAsText char(32)
set #hashAsBinary = HASHBYTES('MD5', '20160818mytest#+')
set #hashAsText = LOWER(CONVERT(varchar(max), #hashAsBinary, 2))
select #hashAsText
Which outputs:
5c8c8aafe7ae37ea4ebdf8bfa01f82b8
See SQL Server converting varbinary to string
I am not sure how else to explain it but it will take more space than a comment allows for so I will post it as an answer.
Look at the source code that you are referencing. At the end (lines 210 and 212) you will see it converts the binary value to a hex string (and then to lower case which does not matter unless you opt for a string comparison at the end). End result = your JavaScript library returns a representation using the type string formatted as hex.
Your Sql function HASHBYTES on the other hand produces a varbinary typed result (which is a different type than string (varchar)).
So you have 2 different data types (each living on their own space as you have not pulled one to the other). You never mention where you are doing the comparison, ie: on the database or are you pulling from the database to script. Either way to do a comparison you need to convert one type so you are either comparing 2 strings types OR comparing two binary types. If you do not compare similar types you will get unexpected results or run time exceptions.
If you are comparing using strings AND in JavaScript then look at your library that you are referencing, it already has a call named wordToHex, copy and paste it and reuse it to convert your Sql result to a string and then do a string comparison (do not forget to compare case insensitive or also make it lower case).
Edit
WebApi is black box for me.It is a 3rd party service.I just need to send the security token as mentioned above.
Assuming that the type accepted by that web api is byt[] appending 0x to your string in javascript and then sending it to the web api should work as in the web api will then translate the incoming parameter as a byte array and execute the comparison using the correct types. As this is a black box there is no way to know for certain unless you either ask them if the accepted type is indeed a byte array or to test it.
Is there an already built tool in PHP or Javascript that parses MySQL queries, including JOINs.
What I need as a result is simply two arrays:
The first is an array of the Query Result fields. (result[])
The second is an array of the Query Parameters. (params[])
Example:
Select PersonID,firstName,lastName,countryName
FROM Person
LEFT JOIN Country ON Person.countryID=Country.countryID
Where firstName="Jonathan" and countryName="Canada"
Would result in the following:
result=['PersonID','firstName','lastName','countryName'];
params=['firstName','countryName'];
This library would probably help you as it parse the query and return every information as an array: php-sql-parser
There is also a javascript library: javascript sql parser
The parser returns an object that contains all the information about the sql query. it supports for now only select statement.
Hello i need help in parsing an object that is being return to java. I am running some automated tests using selenium. In my test i am using the framework to call javascript functions and what i am doing is assigning what is passed back from the javascript function to a java object. The event object passed from javascript looks like this
{
data={
isFailover=false,
baseClip={
expression=full,
isAd=true,
connectionBitrate=0,
baseURL=null,
contentCustomData={
fw: titleId=null,
fw: advertiserId=null,
fw: category=null
},
overlays=[],
ratings=[]
}
}
}
I have an object called "responseToMediaIsAd" defined as
private static Object responseToMediaIsAd;
The data response is being outputted (above) comes when i do this
System.out.println(responseToMediaIsAd);
So its great that i get data back:-) The only problem now is HOW do i parse thru this data. do i have to convert it to json then parse that data?... Can someone tell me how to read the value of (say isFailover, and isAd), ie need to know how to step through and get values
I need to know how to use Java to get to this data that Javascript is returning. Thank you
Well, you can use the Jackson JSON parser library, or if you know regex, you can painfully roll a custom one with some crafted queries.
In Mongodb (2.6.1), I need to query a document by _id using pure json (without using ObjectIds). As mentioned in the mongodb extended json, I was expecting db.collection.findOne({"_id": {"$oid": "51b6eab8cd794eb62bb3e131"}}) to work but it does not. It even throw the following exception.
Can't canonicalize query: BadValue unknown operator: $oid
Anyone knows how to do it?
The extended JSON syntax is intended as a "transfer" format so that if for example you are sending JSON ouput to a remote client there is still a way to determine the actual implemented type such as ObjectId, Date, Binary etc.
The only place AFIAK where this is implemented is within the C# driver which provides a json parser utility method which would take JSON with the extended syntax fields and then "cast" those into objects of the required type.
So in much the same way you can implement your own parser routine to do much the same thing, it is just a matter of testing the key values for something that represents the type of object specified in the key. Given a sample fragment:
{ "_id": { "$oid": "51b6eab8cd794eb62bb3e131" } }
In simplified form without recursively checking by depth:
data = JSON.parse( json );
for ( k in data ) {
if ( data[k].hasOwnProperty("$oid") )
data[k] = new ObjectId( data[k]["$oid"] );
// etc
}
So just because you may be using JavaScript it doesn't mean the "extended syntax" is valid as a query source, but you can as with other languages post-process the parsed JSON into the valid object notation required by that language and the query interface.
Similar "casting" is performed by some drivers on "string" values supplied against an _id field in order to cast to the correct object type required by the BSON wire protocol.