Google not crawling links in AngularJS application - javascript

I have an AngularJS application that is injected into 3rd party sites. It injects dynamic content into a div on the 3rd party page. Google is successfully indexing this dynamic content but does not appear to be crawling links within the dynamic content. The links would look something like this in the dynamic content:
Link Here
I'm using query parameters for the links rather than an actual url structure like:
http://www.example.com/support/title/Example Title/titleId/12345
I have to use the query parameters as I don't want the 3rd party site to have to change their web server configuration to redirect unfound URLs.
When the link is clicked I use the $locationService to update the url in the browser and then my angular application responds accordingly. Mainly it shows just the relevant content based on the query params, sets the page title and meta description.
Many of the articles I have read use the route provider in angularJS and templates but I'm not sure why this would make a difference to the crawler?
I have read that google should view urls with query parameters as separate pages so I don't believe that should be the issue:
https://webmasters.googleblog.com/2008/09/dynamic-urls-vs-static-urls.html
The only things I have not tried are 1. providing a sitemap with the urls that have the query parameters and 2. adding static links from other pages to the dynamic links to help google discover those pages.
Any help, ideas or insights would be greatly appreciated.

This happens because google crawlers are not able to get the static html from your url since your pages are dynamically rendered with Javascript, you can achieve what you want using the following :
Since #! is deprecated, You can tell google that your pages are rendered with javascript by using the following tag in your header
<meta name="fragment" content="!">
On finding the above tag google bots will request your urls with the _escaped_fragment_ query parameter from your server like
http://www.example.com/?_escaped_fragment_=/support?title=Example Title&titleId=12345
Then you need to rebuild your original url from the _escaped_fragment_ on your server and it will look like this again
http://www.example.com/support?title=Example Title&titleId=12345
Then you will need to serve the static HTML to the crawler for that url.
You can do that using a headless browser to access the url. Phantom.js is a good option to render your page using the javascript and then give the contents into a file to create a HTML snapshot of your page. You can save the snapshot as well on your server for further crawling, so when google bots visit can you can directly serve the snapshot instead of re-rendering the page again.

The web crawler might be running at a higher priority than the AngularJS interpretation of your dynamic links as the web crawler loads the page. Using ng-href makes the dynamic link interpretation happen at a higher priority. Hope it works!

If you use urls with #
Nothing after the hash in the url gets sent to your server. Since Javascript frameworks originally used the hash as a routing mechanism, that's a main reason why Google created this protocol.
Change your urls to #! instead of just using #.
angular.module('myApp').config([
'$locationProvider',
function($locationProvider) {
$locationProvider.hashPrefix('!');
}
]);

This is how Google and bing handle the ajax calls.
The documentation is mentioned here.
The overview as mentioned in the docs is as follows
The crawler finds a pretty AJAX URL (that is, a URL containing a #! hash fragment). It then requests the content for this URL from your server in a slightly modified form. Your web server returns the content in the form of an HTML snapshot, which is then processed by the crawler. The search results will show the original URL.
Step by Step guide is shown in the docs.
Since the Angular JS is designed for the Client Side so you will need to configure your Web server to summon a headless html browser to access your web page and deliver a hashbang url which will be given to the special google URL.
If you use hashbang URL then you would need to instruct the angular application to use them instead of regular hash values
App.config(['$routeProvider', '$locationProvider', function($routes, $location) {
$location.hashPrefix('!');
$routes.when('/home',{
controller : 'IndexCtrl',
templateUrl : './pages/index.html'
});
as mentioned in the code example here
However if you do not wish to use hashtag url but still inform the google of the html content but still want to inform the google then you can use this meta tag as this
<meta name="fragment" content="!" />
and then configure the angular to use the htmlURL's
angular.module('HTML5ModeURLs', []).config(['$routeProvider', function($route) {
$route.html5Mode(true);
}]);
and then whichever method to be installed via module
var App = angular.module('App', ['HashBangURLs']);
//or
var App = angular.module('App', ['HTML5ModeURLs']);
Now you will need a headless browser to access the url
You can use phantom.js to download the contents of the page ,run the javascript and then give the contents into a temporary file.
Phantomrunner.js which takes any url as input,downloads and parses the html into DOM and then checks the data status.
Test each page by using the function defined here
SiteMap can also be made as well as shown in this example
The best feature is you can use search console of verify your site url using
Google search console
Full attribution goes to the website and the author mentioned in this site
.
UPDATE 1
Your crawler needs the pages as -
- com/
- com/category/
- com/category/page/
By default, however, Angular sets your pages up as such:
- com
- com/#/category
- com/#/page
Approach 1
Hash bang allows Angular to know which HTML elements to inject with JS which can be done as mentioned before but since it has been depericated hence the another solution would be the following
Configure the $locationProvider and set up the base for relative links
You can use the $locationProvider as mentioned in these docs and set the html5mode to true
$locationProvider.html5Mode(true);
This lets Angular change the routing and URLs of our pages without refreshing the page
Set the base and head of your document as <base href="/">
The $location service will automatically fallback to the hashbang method for browsers that do not support the HTML5 History API.
Full attribution goes to the page and the author
Also to mention there are also some other measures and tests that you can take care of as mentioned in this document

Related

How to reload page if html5mode is enabled in angularjs?

I had made a SPA in angularjs, when I run it on browser it showed me '#' in the address bar. So I enabled html5Mode. This successfully removed '#' from the url but when I try to reload the page, it gives 404: page not found error. I tried writing in .htaccess from this url:
htaccess redirect for Angular routes
Nothing change.
I also followed grunt answers but that was also not helpful.
Did you correctly enable html5 mode?
Take a look at the answer to a question regarding html5 mode/hashbang mode.
Did you add <base href="/"> to the head of your html?
You need to setup your server routes to return index file for all angular urls.
Usually you register your api at some special endpoint (as mysite.com/api/v1) and if url is not matching this pattern return index.html file with angular bootstrap logic.
This happens because without html5 mode all requests are sent to root server url (mysite.com/ even if link is mysite.com/#!/profile) but when you enable html5 mode, requests are sent as declared in angular so if you have '/' route with href to profile (/profile) and reload page from there (mysite.com/profile), request will be sent to mysite.com/profile and it will respond with 404 as it is setuped to return index file only for '/' url

How to extract title, image from others' blog posts and publish on own site

I'm planning to build a site where I can share my handpicked curated contents and I couldn't wrap my head around the basic idea of getting those data fed into my site without going through API.
I first thought maybe I should inspect the source HTML of the page I want to embed on my site and access it with something like $('div.post').find('img').attr('src').
But I can't imagine myself doing that every time so I guess there must be a better way.
It's what Google+ does with their post. Once you add a url link, after a second it pulls featured image and some text snippet from the linked page.
Many sites use the Open graph protocol to get the meta-title, meta-description, image etc. for any url.
For example open: view-source:https://blog.kissmetrics.com/open-graph-meta-tags/ and search for "Open Graph Protocol Meta".
They are contained in the page source. You will have to send a request to the URL you want to crawl from, and read the appropriate meta tags through Regular Expr / HTML Parsers.
You can't make this with javascript. You need a server-side script that downloads the page you need and then parse it with a DOM parser.
With PHP you can get the content of one URL with cURL.
See more:
http://php.net/manual/es/book.curl.php

Embed Angular App within another page?

I'm writing a javascript app which is called within another page I do not control. I'd like to be able to embed an Angular JS app, but how do I handle routing without modifying the URL? How would testing (ie. e2e) work in this scenario?
Edit: The app is a wordpress plugin which is overlayed on the Wordpress dashboard, therefore the URL and Page History cannot be modified. The app will be bound to a div which is embedded in the page, and overlayed on existing content using CSS.
I believe you can just inquire AngularJS and your app, and using hooks to provide necessary HTML. A few things to note though:
Because the URL cannot be changed, you should use router in legacy mode (using URL hash) instead of HTML5 mode.
Since you're running in generated URL, the templateURLs must be built dynamically.
For example, ouput a script tag for URL to your plugin folder:
<script>var MY_PLUGIN_BASE = '<?php echo plugins_url(); ?>'</script>
And then later define your route and templateURL using that constant:
$routeProvider
.when('/', {
templateUrl: MY_PLUGIN_BASE+'my-plugin/views/main.html',
controller: 'Main'
})
This works, but in general I will avoid using routes in such situations. This cause more work in the controllers and directives, but it is safer because there could be other client side MVC apps on the page.
The ng-app is attached to the root element of your view instead of body.
For more general embedded AngularJS app (in my experience: a bookmarklet), you need to:
Inject AngularJS if needed (check for window.angular), CSS and HTML.
Inject your app's JS file. Because only one ng-app will be auto bootstraped, bootstrap your app manually using angular.bootstrap
angular.bootstrap(document.getElementById('my-app'), ['MyApp'])
Use absolute URL for templateURL, and make sure that URL have CORS enabled.
Again, avoid using routes if possible. For the Wordpress plugin, we're pretty sure that there's no other app using hash for routing (Wordpress is using Backbone, but I don't see the routes), but in general there are already a MVC app on the page that handle routing.

Is it possible to have traditional anchor links in Angular.js applications

Angular.js routes create URLs such as these:
http://cooldomain.com:3000/#/search
http://cooldomain.com:3000/#/docs
In my docs url, I would like to have one long page with <a name="sdsds"> sections and a traditional table of content with anchor links so that the user can hop up and down the page
Conceptually the table of contents would produce lots of invalid URLs such as http://cooldomain.com:3000/#/docs#coolAPIFunction which of course wouldn't work because of the double hash
So- is it possible to use anchor links in Angular.js applications that have routes?
You could enable html5 pushstate and get rid of the # in your routes. You can do so by adding this to your .config
$locationProvider.html5Mode(true);
However, be aware that now there will not be a distinction between Angular routes vs. server requests. You'll have to config your server to deliver the appropriate static html file (e.g. index.html) for that url.

Facebook - multiple like buttons with one host page (GWT)

I want to add facebook like buttons to different pages and use different titles, descriptions and images.
The problem now is: Facebook uses meta tags from the header to determine this values, e.g.: . I use GWT and therefore I only have one host page (e.g. index.html) and different content is rendered in this page:
"www.myurl.com#blogpost:1" would load the blogpost with id "1". Therefore every blogpost would have the same title, description, image. I could change the metatags with javascript according to which blogpost is requested. But I guess javascript is not executed by the facebook parser. Is there a way to realize different like buttons with only one host page?
I now generate a special link for facebook. So if my GWT URL looks like "www.myurl.com#blogpost:1", I will generate the URL "www.myurl.com/fb/blogpost/1". Now I check in a Servlet Filter for URLs starting with "fb". If I find a Request with a URL like that, I just write out the Meta tags and a java script forward to my actual page: "www.myurl.com#blogpost:1". The facebook crawler just sees the meta tags and doesn't use the javascript forward.
Normal Users on the other hand are forwarded to the regular page. This works pretty good for me. Thanks CBroe for the hint.

Categories