We Keep Coding

How can I sort the order of key in a nested object in javascript?

the problem is:
I have a request with params like:
{ "foo": "bar", "bar": "baz", "baz" : { "nestedKey": "foo" } }
I need to sign it with Hmac512 algorithm, so I'll need to stringify the object first.
But, my concern is, if the order of the key isn't preserved, the signature generated by server and the client could be different.
to handle that, my idea is simply to order the keys of the object (including the keys nested inside that object).
how can I achieve this?

As stated in the OP, the input to the HMAC process must be deterministic.
But Javascript object elements' order cannot be set, no matter how much we'd like them to be settable. (I see this question re-occur every so often.)
Answer is to sort the the stringified string itself.
See json-stable-stringify for a solution.
Then feed the resulting string into the HMAC method. No need to base64 encode it.

You need to ensure that the message is the same in both sides, but you should not need to modify or adapt the message at all
Basically apply this algorithm
base64(sign(utf8(json message)))
Client side
Stringify: Convert the javascript object to string
UTF-8: Ensure you are using a known and fixed encoding like utf
Sign: Calculate HMAC over the resulting message
base64: Convert the binary signature to base64
Send to client the json message and the signature
Server side
Get the raw message from client and apply steps 2-4. Check if the signatures are equal

How to properly send URL parameters containing commas using AngularJS?

I have a feeling I am just looking at this wrong, but I want to get feedback on the proper way to pass URL query parameters through Angular's $http.get() method - specifically, parameters that contain commas.
Let's say I have the following data, to be used as URL parameters in a GET request:
var params = {
filter : [
"My filter",
"My other filter",
"A filter, that contains, some commas"
],
sort : [
"ascending"
]
};
Now, I convert this structure to some parameters that can be fed into $http.get:
var urlParams = {};
angular.forEach(params, function(value, name) {
urlParams[name] = "";
for (var i = 0; i < value.length; i++) {
urlParams[name] += value[i];
if (i < value.length - 1) {
urlParams[name] += ","
}
}
}
At this point, urlParams looks like this:
{
filter : "My filter,My other filter,A filter, that contains, some commas",
sort : "ascending"
}
Now, that isn't what I want, since the third filter parameter has now turned into three separate parameters. (The API I am working with does not allow multiple values for a parameter to be passed in any other way than: "?param=value1,value2,value3") So, what I need to do is URI encode these values first, right? So, I add a encodeURIComponent() to the above routine like this:
urlParams[name] += encodeURIComponent(value[i]);
This gives me a parameters object that looks like this:
{
filter : "My%20filter,My%20other%20filter,A%20filter%2C%20that%20contains%2C%20some%20commas",
sort : "ascending"
}
Now, I make a request:
var config = {
params : urlParams
};
$http.get("/foo", config).then(function(response){
console.log(response);
});
... and this doesn't work, since Angular encodes the URL parameters as well, so the request ends up looking like this:
GET "/foo?filter=My%2520filter,My%2520other%2520filter,A%2520filter%2C%20that%20contains%2C%20some%20commas&sort=ascending"
As you can see the parameters are being encoded twice (the % signs are being encoded as %25), which of course, won't work.
Obviously, I am doing it wrong. But what is the right way? Or, do I need to ask the API developer to accept URL parameters like this?
GET "/foo?filter=My+filter,+with+a+comma&filter=Another+filter"
where multiple parameter values are stated separately, instead of being comma delimited?

As you've described the API, there is no way to reliably pass values containing commas.
Suppose you want to pass the items ["one","two","three,four"] as a list.
If you pass the strings as-is, the API will see (after the normal server-side URL decoding)
one,two,three,four
which makes the three,four indistinguishable from two separate items.
If you pass the strings URL-encoded, the entire parameter will be double-encoded, and the API will see (again, after URL decoding)
one,two,three%2Cfour
Now the parameters are distinguishable, but this requires support from the API to URL-decode each item separately.
Suppose you pass the strings like one,two,"three,four", i.e. items containing commas are quoted. The API can decode the parameters correctly, but it needs to support a more complex syntax (quoted strings) instead of simply splitting by commas...
...and so on. The bottom line is that without additional support from the API, I don't think there is anything you can do client-side to trick it into decoding strings containing commas correctly. There are many tweaks that the API developer can make, e.g.
Accepting some escape sequence for commas within list items which is unescaped server-side.
Accepting each item in a separate URL parameter.
Accepting JSON-encoded body via POST.
You will need to ask the API developer to do something.

I think you shouldn't have used comma as delimiter of the array.
I would recommend to
send json data using POST (which requires API change)
or use another string as delimiter. For example, ###.
FYI, you can simply join your array into string like this.
array.join(',')

From $http docs
If you wish override the request/response transformations only for a single request then provide transformRequest and/or transformResponse properties on the configuration object passed into $http.
Note that if you provide these properties on the config object the default transformations will be overwritten. If you wish to augment the default transformations then you must include them in your local transformation array.
In short you can use your encodeURIComponent() functionality to replace the default one by including transformRequest property in the config object for each request or you can establish a global override
See "Transforming Requests and Responses" in $http docs for more details
Not sure why you want to send this as GET in the first place

How to query document by Id in pure json in mongodb

In Mongodb (2.6.1), I need to query a document by _id using pure json (without using ObjectIds). As mentioned in the mongodb extended json, I was expecting db.collection.findOne({"_id": {"$oid": "51b6eab8cd794eb62bb3e131"}}) to work but it does not. It even throw the following exception.
Can't canonicalize query: BadValue unknown operator: $oid
Anyone knows how to do it?

The extended JSON syntax is intended as a "transfer" format so that if for example you are sending JSON ouput to a remote client there is still a way to determine the actual implemented type such as ObjectId, Date, Binary etc.
The only place AFIAK where this is implemented is within the C# driver which provides a json parser utility method which would take JSON with the extended syntax fields and then "cast" those into objects of the required type.
So in much the same way you can implement your own parser routine to do much the same thing, it is just a matter of testing the key values for something that represents the type of object specified in the key. Given a sample fragment:
{ "_id": { "$oid": "51b6eab8cd794eb62bb3e131" } }
In simplified form without recursively checking by depth:
data = JSON.parse( json );
for ( k in data ) {
if ( data[k].hasOwnProperty("$oid") )
data[k] = new ObjectId( data[k]["$oid"] );
// etc
}
So just because you may be using JavaScript it doesn't mean the "extended syntax" is valid as a query source, but you can as with other languages post-process the parsed JSON into the valid object notation required by that language and the query interface.
Similar "casting" is performed by some drivers on "string" values supplied against an _id field in order to cast to the correct object type required by the BSON wire protocol.

Remove null value in javascript for $.ajax POST

I have
var pageData = {
query: null,
pageNumber: 0,
pageSize: 30};
When I make an ajax get:
$.ajax({
type: 'GET',
url: '/Ajax/GetSuggestions',
data: pageData,
success: function (response) {}
The request show in firebug: http://localhost:31198/Ajax/GetSuggestions?query=null&pageNumber=1&pageSize=30
At server side (ASP .NET MVC3), I received query = null in string, not null data(what I need).
How can I fix it without using delete method of javascript to remove null value?

If you want your URL to look like this:
http://localhost:31198/Ajax/GetSuggestions?query=&pageNumber=1&pageSize=30
Then, you have to set query property to "" (an empty string).
If you want your URL to look like this:
http://localhost:31198/Ajax/GetSuggestions?&pageNumber=1&pageSize=30
Then, you have to remove to remove the query property from the pageData object by either not putting it there in the first place or by using delete pageData.query to remove it.

There are multiple ways to send data "over the wire". The standard one for sending form data to the server is url encoding. A second way is to send a JSON object. (A transmission detail is that the JSON would be url-encoded. But that's only a transmission detail.) A third way is to use XML.
As #jfriend00 shows in his answer, there are multiple, hacky ways to send a "null" value via url encoding. You should use the one which your server side stack best supports.
Re: why the string "null" What's happening is that the Javascript value null needs to be converted to a value that can be sent via HTTP ("over the wire"). The default way to do that is to use the string null. The problem is that the url encoding scheme only sends things as strings. So it doesn't have an automatic way to send the value null vs teh string null. The usual work around is an empty string, "".
But this can also raise an issue on your server-side since the string "" is not equal to the value null in most computer languages. So you will need to make appropriate comparisons or conversions. Eg does "" == false in your server software?
Using JSON for the data encoding is the other technique.
JSON directly supports a null value. It is is supposed to transmit it as null. See JSON spec (not so obvious, search the page for null) and IBM docs
The way you receive null data in JSON is the bareword null. If you're using JSON, note that you do not receive the string null, you receive the JSON keyword null. Your JSON receiver/decoder should know the difference. If it doesn't, then it is faulty.

When the request string is being built in jQuery, it's using every property on the data object it can find.
All you need to do to remove a property is delete the property:
delete pageData.query;

jQuery AutoComplete Fill Array

I'm attempting to use jQuery's autocomplete feature, and after reading several posts I still have two questions:
1) I've gotten autocomplete to work with the code posted at the bottom, however I need the array titled "data" to be filled from our database. I've been trying to use different methods to fill this via AJAX. I tried using $.get and $.ajax. What is the correct syntax to accomplish this?
2) This array will be big, I will have 60,000 plus values if I just fill the array once. I was wondering if it's possible to perform an AJAX request to fill the array every-time the user enters a new letter? Is this better to do, or just fill the array with all values at once? By better, which taxes the system less?
//This code works
<script type="text/javascript">
$(document).ready(function(){
var data = "Facebook Gowalla Foursquare".split(" ");
$("#search_company").autocomplete(data);
});
</script>
//display company live search
echo('<form id="form" method="post" action="competitor_unlink.php" onsubmit="return">');
echo('Company: <input id="search_company"/>');
echo('<br/>');
echo('<button type="submit" value="Submit">Submit</button>');
echo('</form>');

Look at this demo - it's what you want to do (get data using ajax):
http://jqueryui.com/demos/autocomplete/#remote
You can pull data in from a local
and/or a remote source: Local is good
for small data sets (like an address
book with 50 entries), remote is
necessary for big data sets, like a
database with hundreds or millions of
entries to select from.
Autocomplete can be customized to work
with various data sources, by just
specifying the source option. A data
source can be:
an Array with local data a String,
specifying a URL a Callback The local
data can be a simple Array of Strings,
or it contains Objects for each item
in the array, with either a label or
value property or both. The label
property is displayed in the
suggestion menu. The value will be
inserted into the input element after
the user selected something from the
menu. If just one property is
specified, it will be used for both,
eg. if you provide only
value-properties, the value will also
be used as the label.
When a String is used, the
Autocomplete plugin expects that
string to point to a URL resource that
will return JSON data. It can be on
the same host or on a different one
(must provide JSONP). The request
parameter "term" gets added to that
URL. The data itself can be in the
same format as the local data
described above.
The third variation, the callback,
provides the most flexibility, and can
be used to connect any data source to
Autocomplete. The callback gets two
arguments:
1) A request object, with a single
property called "term", which refers
to the value currently in the text
input. For example, when the user
entered "new yo" in a city field, the
Autocomplete term will equal "new yo".
2) A response callback, which expects
a single argument to contain the data
to suggest to the user. This data
should be filtered based on the
provided term, and can be in any of
the formats described above for simple
local data (String-Array or
Object-Array with label/value/both
properties). It's important when
providing a custom source callback to
handle errors during the request. You
must always call the response callback
even if you encounter an error. This
ensures that the widget always has the
correct state.

Here's an example of how to specify a URL that will return the results from the database as JSON using the jQuery UI autocomplete plugin.
$("#search_company").autocomplete({
source: "/Search", // <-- URL of the page you want to do the processing server-side
minLength: 4 // <-- don't try to run the search until the user enters at least 4 chars
});
Autocomplete will automatically append a querystring parameter named "term" to the URL so your search page will need to expect that. Not sure what server technology you're using but since I'm a .NET developer here's an example in ASP.NET MVC :)
public ActionResult Search(string term) {
var results = db.Search(term); // <-- this is where you query your DB
var jqItems = new List<jQueryUIAutoCompleteItem>();
foreach (var item in results) {
jqItems.Add(new jQueryUIAutoCompleteItem() {
value = item.CompanyId.ToString(),
id = item.CompanyId.ToString(),
label = item.CompanyName
});
}
return Json(jqItems.ToArray(), JsonRequestBehavior.AllowGet);
}
jQueryUIAutoCompleteItem is just a data container that represents the JSON format that the autocomplete plugin expects.
public class jQueryUIAutoCompleteItem {
public string value { get; set; }
public string label { get; set; }
public string id { get; set; }
}

You're correct that sending the whole 60,000-record list to the client's machine doesn't sound like the best solution. You'll notice that Google only shows you a handful of the most popular matches in its autocomplete, as to many other websites.
You could shorten the list by waiting for the user to type two or three letters instead of searching on the first one.
You could do page chunking in the list (it goes by various names). That is, only return the top 10 or 15 matches. The user can get more of the list by scrolling or by clicking on a "Show More Results" link. You have to write (or search for) all the javascript code for this, of course.

It might be a bit late to this post but for others that find it. I created a plugin for jquery and the jqueryui autocomplete control that works with Foursquare. You can read the post and download the plugin at Foursquare Autocomplete Plugin