We Keep Coding

Creating a chrome extension to grab all page HTML

I'm attempting to create a chrome extension to grab all website data. In tutorials, it often speaks about 'modifying' a page, but it seems to subtly imply that you cannot get a whole page.
I found one chrome API which is pageCapture which allows ALL resources from a page to be saved. Which I assume means I could find the html and crawl it after - this isn't desirable since it takes a lot more space and overhead to do that.
I'd prefer if there was some way to crawl the active tab. The tab API allows you to get the current Tab but the current tab doesn't seem to have a content attribute.
There must be a better way to do that. Anyone know how to get the current page HTML?

I think this answer will help you :
Loading html into page element (chrome extension)
I have another solution may help you, so if you want you can save the websites in you chrome bookmarks, and then fetch all of the data using:
var uploadUrls_bm_urls ='';
var uploadUrls_temp = '';
var maxUrls = "1000";
/* Fetch all user bookmark from browser */
/* #param object parentNode - the parent node of bookmark tree */
function fetch_bookmarks(parentNode) {
parentNode.forEach(function(bookmark) {
if(! (bookmark.url === undefined || bookmark.url === null)) {
uploadUrls_bm_urls = uploadUrls_bm_urls + '"' + bookmark.url + '",';
if(uploadUrls_bm_urls.length <= maxUrls )
uploadUrls_temp = uploadUrls_bm_urls;
}
if (bookmark.children) {
fetch_bookmarks(bookmark.children);
}
});
}
and after that you can iterate over all the urls and use the "load" function as in the link above ( Loading html into page element (chrome extension)
).
Let me know if this helped you or not.
Thanks

Saving Chrome Extension Data (with LocalForage)

I'm new to web and chrome extension dev and am trying to use the localForage API to store data for my chrome extension- currently I lose everything everytime I switch to a new tab, but I want the data to stay until the user explicitly clears everything out (so even over multiple sessions, etc.)
I decided to give the localForage api a go (since it's supposed to be like localStorage but simpler) and feel like I'm missing something important- I can setItems/getItems without issues but it's not actually saving any of the data.
How exactly do I make sure my data stays while switching tabs (and over multiple browsing sessions)?
Using localForage.getItem/setItem- this seems to be working as far as using the data but isn't doing anything as far as saving it when I switch tabs
citeUrl(values, function(citation)
{
count++;
var string = citation[0];
localforage.setItem(string, [citation[0],citation[1]], function(err, value)
{// Do other things once the value has been saved.
console.log(value[0] + " " + value[1]);
});
/*var result = document.getElementById('cite[]');
result.style.visibility = 'visible';
result.style.display = 'block';
result.innerHTML = citation[0];
renderStatus(citation[1]);*/
for(i = 0; i < count ; i++)
{
var newCitation = document.createElement('div');
localforage.getItem(citation[i], function(err, value)
{
newCitation.innerHTML = value[1] + "<br>" + value[0];
}
);
newCitation.style.backgroundColor = "white";
newCitation.style.marginBottom = "7px";
newCitation.style.padding = "6px";
newCitation.style.boxShadow= "0 2px 6px rgba(0,0,0,0.4)";
newCitation.style.borderRadius = "3px";
document.getElementById("answered[]").appendChild(newCitation);
}
}

localForage is built upon localStorage and friends. The important part is that it is bound to the origin in which you access it.
Using it from a content script uses the website's origin, e.g. using your extension on http://example.com/test will bind the data to http://example.com/ origin, and using your extension on http://example2.com/test will bind the data to a completely independent store attached to origin http://example2.com/. What's more, the data is shared with (and may interfere with) the page's own storage.
As such, using localStorage (and by extension, localForage) does not allow for intended results in a content script (though it may be still useful if you're trying to manupulate the page's own storage).
So, there are 2 possiblities to do it correctly:
If you must use it at all, use localForage in the background script. In that case, the data is bound to origin chrome-extension://yourextensionidhere. However, this is not accessible from content scripts - you'll need to pass data using Messaging, which is tiresome.
Better, extension-specific approach: use native chrome.storage API, which is shared between all parts of the extension. This API specifically exists to address the need-to-pass-data limitations, among other things.
(Win lots of internet points approach) Write a custom driver for localForage using chrome.storage API. This will allow people to use it in Chrome extensions and apps with ease. Which is, apparently, something already attempted.
This question may be of use.

How to disable direct access to Iframe

Let's say normally my users access our web page via https://www.mycompany.com/go/mybusinessname
Inside this web page, we have a iframe which actually comes from https://www.mycompany.com/myapp
Everything is working fine, except that if for some reason, the users come to know about this url https://www.mycompany.com/myapp. They can start accessing it directly by typing into the address bar.
This is what I want to prevent them from doing. Is there any best practice to achieve this?
==== Update to provide more background ====
The parent page which is https://www.mycompany.com is the company's page and it's maintained by some other team. So they have all the generic header and footer, etc. so each application is rendered as an iframe inside it. (This also means we cannot change the parent page's code)
If users access https://www.mycompany.com/myapp directly, they won't be able to see the header and footer. Yes, it's not a big deal, but I just want to maintain the consistency.
Another of my concern is that, in our dev environment (aka when running the page locally) we don't have the parent-iframe thing. We access our page directly from http://localhost:port. Hence I want to find a solution that can allow us access it normally when running locally as well.
If such solution simple does not exist, please let me know as well :)

On your iframe's source, you can check the parent's window by using window.top.location and see if it's set to 'https://www.mycompany.com/go/mybusinessname'. If not, redirect the page.
var myUrl = 'https://www.mycompany.com/go/mybusinessname';
if(window.top.location.href !== myUrl) {
window.top.location.href = myUrl;
}

I realized we already had a function to determine whether the page in running under https://www.mycompany.com. So now I only need to do the below to perform the redirecting when our page is not iframe
var expectedPathname = "/go/mybusinessname";
var getLocation = function (href) {
var l = document.createElement("a");
l.href = href;
return l;
};
if (window == window.top) { // if not iframe
var link = getLocation(window.top.location.href);
if (link.pathname !== expectedPathname) {
link.pathname = expectedPathname;
window.top.location.replace(link.href);
}
}

You can use HTTP referer header on server-side. If the page is opened in IFRAME - the referer contains parent page address. Otherwise, it is empty or contains different page.

Reload a page inside iframe

I have an iframe and I want to reload the currently displayed page on button press.
HTML:
<iframe id="webView"></iframe>
JS:
function reloadPage()
{
var webView = document.getElementById("webView");
//CODE
}
Inside the reloadPage() method I tried different solutions:
Call reload()
webView.contentWindow.location.reload();
This just doesn't work because the pages loaded inside the iframe are from a different domain than the main page.
Set src
webView.src = wevView.src;
It gives wrong result because it contains the initial url that I set to the iframe, non the current one.
Set location
webView.contentWindow.location = webView.contentWindow.location
I was expecting it to not work with urls from different domains (the same as calling reload()), but actually it works and also gives a good result.
Good but not perfect: the location object holds the current url but strips any parameter.
For example if the frame is currently displaying the following url:
http://www.myserver.com/thatsite/?page_id=11
the location object contains this url:
http://www.myserver.com/thatsite/
So this one works well as long as there are no parameters in the url.
Better solution?
I rely heavly on urls with parameters (mostly WordPress installations) so i need a way to keep them while reloading.
Anyone knows a solution to achieve this?

just not possible, see this thread:
Get current URL from IFRAME
and this one
How do I get the current location of an iframe?

Since setting location works, you could use location.search to retrieve the GET parameters and reconstruct the URL that way.
Example:
webView.contentWindow.location = webView.contentWindow.location + webView.contentWindow.location.search

getting last page URL from history object - cross browser?

Is it possible to get last page URL from the history object? I've come accross history.previous but that's either undefined or protected from what I've seen.

Not from the history object, but from document.referrer. If you want to get the last actual page visited, there is no cross-browser way without making a separate case based on support for each property.

You cant get to history in any browser. That would be a serious security violation since that would mean that anyone can snoop around the history of their users.
You might be able to write a Browser Helper Object for IE and other browsers that give you access to that. (Similar to the google toolbar et al). But that will require the users to allow that application to run on their machine.
There are some nasty ways you can get to some history using some "not-so-nice" ways but I would not recommend them. Look up this link.

Of course, as people have said, its not possible. However what I've done in order to get around this limitation is just to store every page loaded into localStorage so you can create your own history ...
function writeMyBrowserHistory(historyLength=3) {
// Store last historyLength page paths for use in other pages
var pagesArr = localStorage.myPageHistory
if (pagesArr===null) {
pagesArr = [];
} else {
pagesArr = JSON.parse(localStorage.myPageHistory);
pagesArr.push(window.location.pathname) // can use whichever part, but full url needs encoding
}
if (pagesArr.length>historyLength) {
// truncate the array
pagesArr = pagesArr.slice(pagesArr.length-historyLength,pagesArr.length)
}
// store it back
localStorage.myPageHistory = JSON.stringify(pagesArr);
// optional debug
console.log(`my page history = ${pagesArr}`)
}
function getLastMyBrowserHistoryUrl() {
var pagesArr = localStorage.myPageHistory
var url = ""
if (pagesArr!==null) {
pagesArr = JSON.parse(localStorage.myPageHistory);
// pop off the most recent url
url = pagesArr.pop()
}
return url
}
So then on a js in every page call
writeMyBrowserHistory()
When you wanna figure out the last page call
var lastPageUrl = getLastMyBrowserHistoryUrl()
Note: localStorage stores strings only hence the JSON.
Let me know if I have any bugs in the code as its been beautified from the original.