My real world problem is: users of my mobile app type their city and I have to make sure it really exists, and that it is correctly written (caseinsensitive, so these are correct: New York, NEW york, new york. This is not correct: newyork)
There are online apis that work quite well (Google Geocode API for example) but:
After a very little amount of requests, you have to pay (2.500/day right now)
Users must be connected to the internet
That's why I tought that an offline-local solution would be better. There are many websites (like Maxmind) where you can download a list containing every city in the world. I could embed this huge txt/csv right inside my application and do a string search locally (it's a big file, ok, but not that big. It's just a onetime download of something like 30-40MB of uncompressed .txt)
I'm trying to avoid jQuery at all costs and I don't want to use any PHP/MySQL solutions (even if fulltext indexes could be handy), that's why I'm trying to do all this just using javascript.
Given a string as input, let's say "city3", what's the best/fastest way to check if it's inside an (external) huge list like:
city1,
city2,
city3,
city4,
[...]
After solving this (big) problem: if there are no exact matches, is there a way to search for the correct city without freezing the device for 10 minutes?
In the example before, lets say the user types "cit y3" or "cyty3" or "cìty3": can any js function tell him that he might be looking for "city3"? Is this kind of search too slow in this scenario?
Thanks
If speed is an issue then I would recommend loading the data into a JavaScript object and performing an in-memory search rather than repeatedly scanning a big blob of text in a file.
Try formatting the data into JSON with the city names as keys, that will give you good search performance.
A Workaround is creating a Database either SQL either noSQL, and Query this database through your JavaScript Code, using jquery Json functions.
Using a SQL Database ideal would be either MySQL either MariaDB An enhanced, drop-in replacement for MySQL.
In this solution you will probably need a Backend such as PHP to fetch the data from your Database convert them to JSON Format, and then get them through your JavaScript using jQUery Library , with the $.getJSON function
Using a noSQL Database ideal would be MongoDB.
In this solution you can fetch your data directly from javascript, also with the $.getJSON function.
Example for MongoDB Provided Here
if you dont want to use database i think you can do this:
-first , instead use one big file split it into several files. (you can write a script for this and use it just one time for split the big file). in each file put cities that starts with (example) aa , second file cityes that starts with ab.
-then for each city check first letters and then search inside that file.
For example if you need to search for city "Ahmedabad" it will search only in the files with cities that starts with Ah. Probably this is not the best solution ,at the end you got 421 file instead 1 , but reasearch will be faster.
Related
This is more of a architectural questions. An external platform had product and price information for let's say, books. There is an API available to get this information.
What I read is that it should be possible to create a function in Javascript and connect the Javascript to a page where you want to show the data on my own website. This would mean that for each page request an API-call is made. Since the requested information only changes once a day maximum this does not sound the most efficient solution.
Can someone advise a better solution? Something into the direction of a similar php or javascript function that does the request on the background, schedule an update and import the data into mysql? If so, what language would be most common.
I need the solution for a Joomla/php/mysql environment
Here's a simple idea - fetch and store results from the API (ones you think aren't gonna change in a day), either on disk, or in the database, and later use these stored results to retrieve what you otherwise would've fetched from the API.
Since storing anything in frontend JS across page reloads isn't easy, you need to make use of PHP for that. Based on what's given, you seem to have two ways of calling the API:
via the frontend JS (no-go)
via your PHP backend (good-to-go)
Now, you need to make sure your results are synced every (say) 24 hours.
Add a snippet to your PHP code that contains a variable $lastUpdated (or something similar), and assign it the "static" value of the current time (NOT using time()). Now, add a couple of statements to update the stored results if the current time is at least 24 hours greater than $lastUpdated, followed by updating $lastUpdated to current time.
This should give you what you need with one API call per day.
PS: I'm not an expert in PHP, but you can surely figure out the datetime stuff.
It sounds like you need a cache, and you're not the first person to run into that problem - so you probably don't need to reinvent the wheel and build your own.
Look into something like Redis. There's an article on it available here as well: https://www.compose.com/articles/api-caching-with-redis-and-nodejs/
Does anyone know how i can parse LinkedIn accounts? Or Any tool( not paid ).
For example:
I will look for "Software Engineer" from Dallas,TX.
Tool will automatically pick all candidates from linkedin or for example first 100 candidates, and store their First Name, Last Name , LinkedinLink and Experience in excel document? ( Or from specific company)
Is it should be done threw API, or there specific account which allow to do this? Or does anyone knows tools which will help to do this? Or Script?
I need to parse a large amount of candidates , 100+ maybe 1000+ and store them.
I have multiple thoughts about implementation but i feel that it 100% already implemented.
https://developer.linkedin.com/docs/rest-api
Use linked in APIs to fetch data and process it however you would like. I don't know how much of 'private' fields you can get access to but names seem to be there.
I use nodeJS to process excel data - xlsx is a very good option but it only allows synchronous execution so you would have to spawn another process. It also has filter function so you can do whatever you want with it.
The problem that I had faced with parsing large data into excel is that excel file is a compressed xml format so it takes a long time to parse both reading and writing. A faster option would be to create and read csv which excel can naturally do as well.
I guess i have a noob question here.
I have a huge Json file (30MB) that i would like to parse with a Jquery web app.
Now, the ideal would be to load it into local storage, regex what i want and show the results.
I would like it to start showing the results as soon as i type (google style) but it looks like every attempt i've made the app just hangs.
If i reduce the Json file to 1 MB then it works.
Does anybody know how to do that? maybe with an example that i can see?
thanks a lot!
I would recommend you to not use this method for search. Because even if you will manage to make your search quicker it will take very long time to download 30MB file.
What you could do is convert your data from JSON to SQL and do search by sending AJAX calls to the server from your javascript code.
For further optimizations you can send AJAX calls only after search term's length exceeds certain number of characters probably 3.
Sorry for my ignorance on the lack of knowledge I have on this subject however I cannot find an answer to my question anywhere.
So I have this MySQL table:
Feed_ID Vehicle_ID FullRegistration Colour FuelType Year Mileage Bodytype Doors Make Model Variant EngineSize Price PreviousPrice Transmission PictureRefs ServiceHistory PreviousOwners Description FourWheelDrive Options Comments New Used Site Origin V5 Condition ExDemo FranchiseApproved TradePrice TradePriceExtra ServiceHistoryText Cap_ID
As you can see each column will contain vehicle data.
I have displayed all of the results in the database using PDO onto my front end, all data is displayed in a listing style similar to Ebay.
Now I need to filter these results however I have noticed that many result filter systems are using JS.
Here are some examples so you get a better idea of what I am talking about:
http://www.autotrader.co.uk/search/used/cars/
http://www.motors.co.uk/search/car/
As you can see all the filters are using JS however I am having a problem understanding how JS is filtering the MySQL query?
I know this question might be a little broad but can someone show me an example of how JS can filter PDO results just like the examples I have shown?
Thanks
The first one uses what I suspect to be a combined method of Javascript and a server-side language (it's hard to prove, because I can't see the server-side code involved). For simplicity, I'll assume this server-side language is PHP, though it could easily not be.
Basically, all Javascript is doing on the first website is setting cookies and telling you to refresh the page. Once you refresh, PHP fetches the cookies that Javascript set and filters the results from the MySQL query based on those cookies.
Now, the second one is actually filtering using Javascript, yet at the same time still using PHP (again, it could be any server-side language).
This is a method called AJAX. It is a function built into Javascript which allows you to fetch another page from Javascript (aka. send and receive an HTTP request).
The reason this is useful is because once you've changed an option on that page, Javascript can send an HTTP query using AJAX to something like "http://www.motors.co.uk/search/getcarinfo.php?transmission=manual", allowing PHP to fetch a new dataset from MySQL and return it to Javascript (this probably isn't the API entry point that they use, but it has to be somewhere in their Javascript).
Once Javascript receives the response from that page (usually in JSON or XML form), it can modify HTML to update what's shown on the page.
To answer your question directly, Javascript doesn't filter the data. MySQL filters the data based on a PHP query, which returns its response to the Javascript. Then, Javascript just puts it on the screen.
I am new to the world of webdesign and already assigned myself with a very (at least for me) difficult task: I want to build a webpage, that sends a query string to the website of the German Railway (bahn.de) with the parameters I entered on my webpage.
My question now is, if there is a way to decipher the answer, the other webpage (bahn.de) is sending back in regard to my query string.
In my case there will be departure and arrival times, fares, line numbers, .... Is it possible to extract this information from the answer the bahn.de- page is sending?
First and foremost you need to determine what type of data-encoding the website is returning. Is it XML or perhaps JSON? Both formats can be dealt with using a server-side language such as PHP, but the extraction process may differ slightly.
In order to continue along your learning path (which is great, by the way!) you'll need to find out a bit more about what kind of data object the website is sending back from your query. There are plenty of great resources at the other end of a Google search that can teach you how to handle that incoming data once you know which format it is in.