We Keep Coding

How can I identify potential parameters within a path from a list of URLs?

I am trying to do a HAR to OpenAPI spec conversion, and as part of this process, I'd like to be able to identify which URLs can be consolidated and replaced with path parameters.
Whilst I understand it may not be 100% accurate because all APIs are different, it would be good to automate a good amount and then allow the user to manually override afterwards if necessary.
Example Input
[
"/api/v1/users",
"/avatars/jane.doe",
"/avatars/joe.bloggs",
"/sounds/bing",
"/sounds/bong",
"/api/v1/users/jane.doe/profile",
"/api/v1/users/joe.bloggs/profile",
"/api/v1/parent/one/child/one",
"/api/v1/parent/one/child/two",
"/api/v1/parent/two/child/one"
]
Expected Output
[
"/api/v1/users",
"/avatars/:param1",
"/sounds/:param1",
"/api/v1/users/:param1/profile",
"/api/v1/parent/:param1/child/:param2"
]
Bonus points for also naming the parameter based on the segment before the parameter.
e.g.
[
"/api/v1/users",
"/avatars/:avatar",
"/sounds/:sound",
"/api/v1/users/:user/profile",
"/api/v1/parent/:parent/child/:child"
]
I've tried using the following Python library but this also attempts to fit Regular Expressions into the URLs which is more than what I am looking for.
https://pypi.org/project/os-urlpattern/
Ideally the solution will be provided in Javascript/Typescript.

How to get table data as rows and columns from wikipedia api?

When I tried to get table data as json, I could find distinguishable children in json output of the following query:
https://en.wikipedia.org/w/api.php?action=parse&page=List_of_football_clubs_in_India&prop=wikitext&section=3&format=json
I want to get the rows and columns of this table (the text) :-
https://en.wikipedia.org/wiki/List_of_football_clubs_in_India#Assam
The JSON output seems complicated and I don't find a good way to extract text from it.
(I am doing this in Javascript (Node.js)
Please help..

I'm not sure, what you expect. Your API request to the page is actually returning the wikitext encapsulated into a JSON structure. However, the wikitext (where the table is part of) is not JSON, so you can not really interpret it as such.
I'm also not quite sure, what information you want to have. If you want to have the football clubs in the table, then your only bet is to parse the wikitext (you can also return the actual parsed HTML from the API to make it "easier") and go through the data yourself. However, this is probably an error prone and not fun task.
So, if you want to get all football clubs of india in a structured data format, I would probably better try Wikidata for that. It allows you to crunch structured data for the information you need (and also get you the links to Wikipedia articles, if the objects has a link to a Wikipedia page). In your use case, it's probably a good idea to try out the Wikidata Query service.
There you could issue a query like:
SELECT ?itemLabel ?sitelink WHERE {
?item wdt:P31 wd:Q476028;
wdt:P17 wd:Q668.
?sitelink schema:isPartOf <https://en.wikipedia.org/>;
schema:about ?item.
SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
}
which queries a list of all football clubs in India and returns you a list with the item label as well as the link to the english Wikipedia article:
https://query.wikidata.org/#SELECT%20%3FitemLabel%20%3Fsitelink%20WHERE%20%7B%0A%20%20%3Fitem%20wdt%3AP31%20wd%3AQ476028%3B%0A%20%20%20%20%20%20%20%20wdt%3AP17%20wd%3AQ668.%0A%20%20%3Fsitelink%20schema%3AisPartOf%20%3Chttps%3A%2F%2Fen.wikipedia.org%2F%3E%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20schema%3Aabout%20%3Fitem.%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%22.%20%7D%0A%7D

Assume that this is the res is the data that you get from the wiki
//This will get you the innermost part of the object which is the text you want
let wikiText = res.parse.wikitext['*'];
//This will strip out all the numbers and non-alphabet charater.
let pureText=wikitext.replace(/[^a-zA-Z\s]+/g, ' ');
The above code can give you clean access to the text; however, how you are going to separate the column and row is up to you.

This will slow down performance a bit (It seems, but I'm not sure if any other faster way exists).
This can be done by setting prop=text and then parsing the obtained HTML using JSDOM (comes with/for Node.js)

I know this question is old but there is an API for this. You can supply a page title and it will return the tables of your choice in JSON.

Recieving high volume data in client browser using Jquery / JavaScript

I am in the process of making a WordPress based application where a student can take the examination on his web-browser. The questions will be randomly selected and served from the question bank stored in a WordPress CMS.
In this regard following is important to share:
-each examination can have as many as 100 multiple choice questions.
-Each question can have images, each choice can have associated images.
-since the examination is time bound I can not send request to server every time the student completes his question.
My query is :
How do I send the questions from the server:
-should I send the whole question set in one go and then have the Java Script parse all the questions and choices parsed at the client side
or
-should the client repeatedly request the questions from server in the background in the chunks of say 5 question each, for example. If this is better approach I am not sure how do I implement this. Any pointers?, please.
Or is there a third approach which I am not aware of.
Please advise for any comments and solutions for the problem.
Thanks in advance.

Depends on user's selection,send appropriate JSON data to client and render it dynamivally,but if you want to use XML then lets talk about it:
I should mention that this comparison is really from the perspective of using them in a browser with JavaScript. It's not the way either data format has to be used, and there are plenty of good parsers which will change the details to make what I'm saying not quite valid.
JSON is both more compact and (in my view) more readable - in transmission it can be "faster" simply because less data is transferred.
In parsing, it depends on your parser. A parser turning the code (be it JSON or XML) into a data structure (like a map) may benefit from the strict nature of XML (XML Schemas disambiguate the data structure nicely) - however in JSON the type of an item (String/Number/Nested JSON Object) can be inferred syntactically, e.g:
myJSON = {"age" : 12,
"name" : "Danielle"}
The parser doesn't need to be intelligent enough to realise that 12 represents a number, (and Danielle is a string like any other). So in javascript we can do:
anObject = JSON.parse(myJSON);
anObject.age === 12 // True
anObject.name == "Danielle" // True
anObject.age === "12" // False
In XML we'd have to do something like the following:
<person>
<age>12</age>
<name>Danielle</name>
</person>
(as an aside, this illustrates the point that XML is rather more verbose; a concern for data transmission). To use this data, we'd run it through a parser, then we'd have to call something like:
myObject = parseThatXMLPlease();
thePeople = myObject.getChildren("person");
thePerson = thePeople[0];
thePerson.getChildren("name")[0].value() == "Danielle" // True
thePerson.getChildren("age")[0].value() == "12" // True
Actually, a good parser might well type the age for you (on the other hand, you might well not want it to). What's going on when we access this data is - instead of doing an attribute lookup like in the JSON example above - we're doing a map lookup on the key name. It might be more intuitive to form the XML like this:
<person name="Danielle" age="12" />
But we'd still have to do map lookups to access our data:
myObject = parseThatXMLPlease();
age = myObject.getChildren("person")[0].getAttr("age");

Why and when do we need to flatten JSON objects?

I am surprised that no one on StackOverflow asked this question before.
Looking through the JSON object documentation and a quick google search did not yield satisfactory results.
What's the advantage of it? How does it work?
Edit: To make it clear, take a look at this flatten/un-flatten example.
Fastest way to flatten / un-flatten nested JSON objects
Thank you.

There are many situations where you get JSON text that was automatically built by some library. Throughout the programming languages, there are many libraries that build JSON text (one example is here).
Whenever libraries add some additional object or array wrappings, you might want to get rid of them maybe because you send the JSON to the server and your code there crashes because it expects a primitive value instead of an object (or an array). Or, if your JSON is a server response, you don't want the resulting Javascript code having to differ between object/array or not object/array. In all these cases, flattening is helpful as it will save you time. You will have to implement lesser if/elses, and you can reliably expect your data structure to be as flat as possible.
The other approach to improve code for the scenario mentioned is to write the code in a maximal robust way so there is no way for it to crash by superfluous wrappings ever. So always expect some wrappers and get it's contents. Then, flattening is not needed.
You see, it depends on what is building the JSON and what is parsing it. The building may be out of your scope.
This leads also to data model questions. I've worked with XML code that needed to be parsed quiet a different way if there where 0 entries of some XY, or if there were >0 entries of some XY. Having a wrapper that is allowed to have 0 or more entries of some XY will make live easier. These are data model desicions.
In all cases where the JSON represents an object structure that I've combined manually, I expect it not to change. So flattening something I've designed in detail would be disturbing. Standard operations as far I've seen them do not need flattening (e.g. JSON.stringify(), json_encode() etc.)

Here's a simple scenario: In a web app you have an HTTP POST that is updating a complex relational object.
POST
update=1
&user.id=12345
&user.email=testmail#domain.tld
&user.profile.name=Mr. Test
&user.profile.age=42
&user.profile.friend.0.email=tom#domain.tld
&user.profile.friend.1.email=sally#domain.tld
&user.profile.friend.2.email=bob#domain.tld
&user.profile.skill.0.id=100
&user.profile.skill.0.name=javascript
&user.profile.skill.1.id=200
&user.profile.skill.1.name=piano
Everything is already in a flat structure, so why not have a simple one-to-one binding? If you had a list of constraints or security requirements that you needed to enforce you could validate them by searching directly on the sorted key list.
Flat structures are easier for people to understand and work with there's even some cross-over with database de-normalisation. It also allows for context specific security and constraints to be implemented in a readable, but more verbose way.
When showing a user's view in full you may want to hide the display of the primary key ID for the user's list of skills.
"user.profile.skill.#.id": { hidden: true, readonly: true }
But when looking directly at a skill (to possibly edit it as an administrator) you may want to see the ID.
"skill.id": { readonly: true }
If you were writing a user-centric/self-service type CMS application you'd get more users on board and able to contribute using a straightforward flat model (flat abstraction of the underlying nested relational model) than you would with just the nested model.
TLDR: Flat is easier to read than nested. While programmers can handle nested schemas, recursive parsing and processing; end-users and admins usually prefer that part abstracted away.

I realize this is a 5 year old question at this point, but I figured, I'd add my thoughts to it, in case someone runs into a similar use case and finds this useful.
One of the use cases why you would want to flatten a JSON object, is for dynamic template binding via Regular Expression (RegEx) string interpolation. Well wasn't that a mouthful 👀😅? It simply translates to "template filling a string without hardcoding".
Ok Imagine a scenario, you have a template string like this for an email:
Hello {{firstName}},
It is amazing you chose to join our site. We are happy to have you on board.
To get started, we would really love it if you can confirm your email address
by clicking on the link: {{confirm_url}}.
Welcome aboard
The Team!
Given the following JSON object in memory:
{
"user" : {
"prefix" : "Dr.",
"firstName" : "Awah",
"lastName" : "Teh",
"email" : "awah#superduperubercoolsite.com",
"address" : {
"street": "100 Main St",
"city" : "PleasantVille",
"state" : "NY",
"phone" : "+1-212-555-1212"
}
},
"meta" : {
"confirm_url" : "http://superduperubercoolsite.com/confirm/ABC123"
}
}
it seems super simple to do a Regular Expression replace like so (assuming our email template string was stored in a variable named template and the json object was stored in a variable called templateData:
template = template.replace(new RegExp('{{firstName}}', 'g'), templateData.user.firstName);
template = template.replace(new RegExp('{{confirm_url}}', 'g'), templateData.meta.confirm_url);
Easy right? --> Actually yes! How about this email had 10 templated fields, or you wanted to decouple the template from the code, by storing it in a separate system like SendGrid, where your cool head of marketing can access the template and make changes to the copy-language, without having to call someone from engineering to make changes to the code, test the code and redeploy to production (what a hassle).
This is exactly where flattening of the JSON comes save the day!
Now there are many ways to flatten JSON, I have attached a link to a codepen I wrote that has logic to flatten JSON (actually, I demonstrate two similar but different approaches in the methods flattenJSONIntoKVP and flattenJSONIntoRAW check 'em out!).
That said, there are other implementations out there, and it is worth remembering that the focus on this post is to discuss the WHY JSON flattening could be useful, not the HOW.
Moving on! Assume you flattened the JSON from above (using my implementation that results in key value pairs) to something like this:
[
{ "key": "user.prefix", "value": "Dr."},
{ "key": "user.firstName", "value": "Awah"},
{ "key": "user.lastName", "value": "Teh"},
{ "key": "user.email", "value": "awah#superduperubercoolsite.com"},
{ "key": "user.address.street", "value": "100 Main St"},
{ "key": "user.address.city", "value": "{PleasantVille"},
{ "key": "user.address.state", "value": "NY"},
{ "key": "user.address.phone", "value": "+1-212-555-1212"},
{ "key": "meta.confirm_url", "value": "http://superduperubercoolsite.com/confirm/ABC123"},
]
Now, my friend, you are cooking with GAS!
Why, cause now you can dynamically interpolate the template string with values from the JSON object without giving too much worry to the structure of the JSON (if it changes due to the application evolving, you don't have to also remember to come down here and change this interpolation code -- you simply just have to update the email template itself, which mind you, is on SendGrid [per this example]).
So how to do it you say?: Simple, iteratively. Let's assume that flattened from above was stored in a variable called flatJSON:
///Notice how I use Javascripts native string interpolation to create my RegExp
///Also note that I am replacing the dot (.) in my flattened JSON variable names with a double underscore (__), I only do this because my intended target is SendGrid, and I don't believe it likes dots in its template placeholders.
flatJSON.forEach(kvp=>template = template.replace(new RegExp(`{{${kvp.key.replace(/\./g, '__'}}}`, 'g'), kvp.value));
That's it, one line of code to replace possibly 10 or even hundreds or even thousands (ok.. maybe not thousands, but you get the point).
Ohh! almost forgot, we need to update our template string.
Notice how now, in our new templated string we can use a somewhat FQDN style variable to map back to our original JSON (Ideally if SendGrid supported dots in their template placeholders, this would look super sweet but alas, can't always win everything!😭.
Hello {{user__firstName}},
It is amazing you chose to join our site. We are happy to have you on board.
To get started, we would really love it if you can confirm your email address
by clicking on the link: {{meta__confirm_url}}.
Welcome aboard {{user__prefix}} {{user__lastName}}!
The Team!
Et Voila!
Just like that, we have accomplished some good here today; we have:
Answered the WHY of flattening JSON objects
We dibble-dabbled into the how, with the codepen example
And we even overviewed a use case where taking advantage of JSON flattening can help you write durable dynamic code, that evolves as your underlying object structures change -- and that doesn't require you to leverage the big bad ugly eval method (we can talk about big bad ugly eval on another post).

Search Functionality for a dynamic webpage

I have a webpage with multiple images loaded using a JSON File with schema something like -
[
{ "item" : "Lotus",
"images" : [ "images/lotus1.jpg", "images/lotus2.jpg", "images/lotus3.jpg" ]
},
{ "item" : "Tulip",
"images" : [ "images/tulip1.jpg", "images/tulip2.jpg", "images/tulip3.jpg" ]
}
....
]
If I want to add a search functionality on the page and dynamically show only the relevant images.
for eg. If a user wants to search for a particular flower and enters search text as "Tu", images with name having the search text should be displayed.
What technique, libraries or procedure I can follow? Is it possible to handle this only through front end or Does this percolate till the back end?
Any pointers are really appreciated!

I recommend you to read this page http://api.jquery.com/find/ and try

One way would be to display all the images on one page and let the isotope plugin take care of the filtering.
Here is a link to the docs on filtering.
As a bonus it comes with fancy CSS effects, which make sorting a real pleasure ;)
You can find the code which addresses filtering by typing in a search field here.
It basically works by adding the names of your images as class names, showing only the ones that match your search while hiding the other classes.
However, if you have to deal with large amounts of images, this might not be an ideal solution, as all the images are loaded in advance to make this work. In this case, server side handling of the search would be the way to go.
Edit
Here is another example on how to filter with a search field and isotope.

I think the best way to do this is with the back end. You should have your server return a json file with only relevant images, then your client side doesn't have to process that, and it could even be cached.
Otherwise client side it would just be an issue of looping through all the image "items", and searching for the query with .indexOf in the name. However, this would put a lot more work on your client side and probably make your site slower.

If you'd like to keep it all in the front-end, you can use underscore.js. There are ways to filter through a array/object, see: http://underscorejs.org/#filter. You would query using regex to match against all the item names, example:
var images = [{ "item" : "Lotus", "images" : [ "images/lotus1.jpg", "images/lotus2.jpg", "images/lotus3.jpg" ]},{ "item" : "Tulip","images" : [ "images/tulip1.jpg","images/tulip2.jpg", "images/tulip3.jpg" ]}];
var searchResults = _.filter(images, function(obj){ return /lot/i.test(obj.item) });
This would return an object, with only the lotus, and all of the images needed. See fiddle: http://jsfiddle.net/cbM5W/1/
This does use the underscore library, but you could implement this with vanilla JS.