I use Qt Installer Framework to create an installer for my application. Before the installation starts, the controller script checks in the TargetDirectoryPage callback, whether the program is already installed on the system and if yes, then the installer should terminate without doing anything.
But I could not find a way to stop the installer from copying (and therefore overwriting existing) files like uninstaller.exe, uninstaller.ini, installer.dat and some more into the target dir and from requesting elevated rights for that.
To skip the installer pages and show the error message in the finish page I use the following function in the controller script based on an official example:
function cancelInstaller(message) {
console.log("Cancel installer: " + message);
// this key value pair sets the label text in the Finish page:
installer.setValue("FinishedText", "<font color='#e53935' size=3>" + message + "</font>");
// skip all installation pages except for the finish page
installer.setDefaultPageVisible(QInstaller.Introduction, false);
installer.setDefaultPageVisible(QInstaller.TargetDirectory, false);
installer.setDefaultPageVisible(QInstaller.ComponentSelection, false);
installer.setDefaultPageVisible(QInstaller.ReadyForInstallation, false);
installer.setDefaultPageVisible(QInstaller.StartMenuSelection, false);
installer.setDefaultPageVisible(QInstaller.PerformInstallation, false);
installer.setDefaultPageVisible(QInstaller.LicenseCheck, false);
// this key value pair is for telling components that installation is canceled:
installer.setValue("installerCanceled", true);
}
The value installerCanceled is later used in the components to skip the execution of beginInstallation, so that at least the component's data is not copied to the target dir.
I hope you can help me, I wasted too much time for that already. Thanks.
I'm trying to trigger some sort of Folder Selection Dialog, I have a working model with nodejs and the powershell but it only works when the server and client are on the same machine. I need the prompt to occur on the client side triggered from the browser. From what i understand I can not trigger Powershell from Chrome? So is there an alternative or am i just screwed?
My current Powershell script
{
param([string]$Description="Select Folder",[string]$RootFolder="Desktop")
[System.Reflection.Assembly]::LoadWithPartialName("System.windows.forms") |
Out-Null
$objForm = New-Object System.Windows.Forms.FolderBrowserDialog
$objForm.Rootfolder = $RootFolder
$objForm.Description = $Description
$Show = $objForm.ShowDialog()
If ($Show -eq "OK")
{
Return $objForm.SelectedPath
}
Else
{
Write-Error "Operation cancelled by user."
}
}
$folder = Select-FolderDialog # the variable contains user folder selection
write-host $folder
My javascript function
async function asyncfindDir() {
//executes powershell script
let promise = new Promise((resolve, reject) => {
const Shell = require('node-powershell');
const ps = new Shell({
executionPolicy: 'Bypass',
noProfile: true
});
ps.addCommand('./selectfolder.ps1');
ps.invoke()
.then(output => {
//console.log(output);
var shelloutput = output;
console.log (shelloutput + '^^from external script');
res.send(shelloutput);
})
.catch(err => {
console.log('please select a directory path')
//console.log('err');
});
});
};
Is there anyway to get that working locally?
Is there a trigger i'm not aware of to access that kind of dialog from the browser? I know i'm not the only person with this issue but i have yet to see a real solution.
Short answer: No.
Longer answer, is best illustrated by rephrasing your question with a different script name:
Using my browser, can I click on a link to visit a website, and have it run a random
PowerShell script called Delete_All_Files.ps1?
Answers why you will never be able to run a PowerShell script from a browser, on a remote machine, and why browsers will deliberately block you from doing it, because people usually don't want to have all their files deleted when they click on a random link in their email.
If you want to run PowerShell scripts on remote machines, then you should look into PSRemoting and Enter-PSSession.
#kuzimoto is right. If you just want to display a folder dialog box, there are easier ways to do that and Fine Uploader is an easier way.
Replying to your comment: If you want to specify a directory name, the reason you can't do it is because you are essentially asking:
Using my browser, can I click on a link to visit a website, and have
it run a script that will enumerate through all the files and folders
in my C:\ so that it can choose the folder C:\users\Justin
Miller\Desktop\SECRET FILES\?
The reason both operations do not work is because both operations require local computer access. i.e. local script execution access, and local directory knowledge access. Security-wize, we, in general, don't want to visit a random website and have it execute random code, or know what files/folders I have on my machine, which is why you won't be able to do what you want to try to do.
So I have a system that essentially enabled communication between two computers, and uses a WebRTC framework to achieve this:
"The Host": This is the control computer, and clients connect to this. They control the clients window.
"The Client": The is the user on the other end. They are having their window controlled by the server.
What I mean by control, is that the host can:
change CSS on the clients open window.
control the URL of an iframe on the clients open window
There are variations on these but essentially thats the amount of control there is.
When "the client" logs in, the host sends a web address to the client. This web address will then be displayed in an iframe, as such:
$('#iframe_id').attr("src", URL);
there is also the ability to send a new web address to the client, in the form of a message. The same code is used above in order to navigate to that URL.
The problem I am having is that on, roughly 1 in 4 computers the iframe doesn't actually load. It either displays a white screen, or it shows the little "page could not be displayed" icon:
I have been unable to reliably duplicate this bug
I have not seen a clear pattern between computers that can and cannot view the iframe content.
All clients are running google chrome, most on an apple powermac. The only semi-link I have made is that windows computers seem slightly more susceptible to it, but not in a way I can reproduce. Sometimes refreshing the page works...
Are there any known bugs that could possibly cause this to happen? I have read about iframe white flashes but I am confident it isn't that issue. I am confident it isn't a problem with jQuery loading because that produces issues before this and would be easy to spot.
Thanks so much.
Alex
edit: Ok so here is the code that is collecting data from the server. Upon inspection the data being received is correct.
conn.on('data', function(data) {
var data_array = JSON.parse(data);
console.log(data_array);
// initialisation
if(data_array.type=='init' && inititated === false) {
if(data_array.duration > 0) {
set_timeleft(data_array.duration); // how long is the exam? (minutes)
} else {
$('#connection_remainingtime').html('No limits');
}
$('#content_frame').attr("src", data_array.uri); // url to navigate to
//timestarted = data_array.start.replace(/ /g,''); // start time
ob = data_array.ob; // is it open book? Doesnt do anything really... why use it if it isnt open book?
snd = data_array.snd; // is sound allowed?
inititated = true;
}
}
It is definitele trying to make the iframe navigate somewhere as when the client launches the iframe changes - its trying to load something but failing.
EDIT: Update on this issue: It does actually work, just not with google forms. And again it isn't everybody's computers, it is only a few people. If they navigate elsewhere (http://www.bit-tech.net for example) then it works just fine.
** FURTHER UPDATE **: It seems on the ones that fail, there is an 'X-Frames-Origin' issue, in that its set the 'SAMEORIGIN'. I dont understand why some students would get this problem and some wouldn't... surely it depends upon the page you are navigating to, and if one person can get it all should be able to?
So the problem here was that the students were trying to load this behind a proxy server which has an issue with cookies. Although the site does not use cookies, the proxy does, and when the student had blocked "third party cookies" in their settings then the proxy was not allowing the site to load.
Simply allowed cookies and it worked :)
iframes are one of the last things to load in the DOM, so wrap your iframe dependent code in this:
document.getElementById('content_frame').onload = function() {...}
If that doesn't work then it's the document within the iframe. If you own the page inside the iframe then you have options. If not...setTimeout? Or window.onload...?
SNIPPET
conn.on('data', function(data) {
var data_array = JSON.parse(data);
console.log(data_array);
// initialisation
if (data_array.type == 'init' && inititated === false) {
if (data_array.duration > 0) {
set_timeleft(data_array.duration); // how long is the exam? (minutes)
} else {
$('#connection_remainingtime').html('No limits');
}
document.getElementById('content_frame').onload = function() {
$('#content_frame').attr("src", data_array.uri); // url to navigate to
//timestarted = data_array.start.replace(/ /g,''); // start time
ob = data_array.ob; // is it open book? Doesnt do anything really... why use it if it isnt open book?
snd = data_array.snd; // is sound allowed?
inititated = true;
}
}
}
I just updated a website I'm working on, unfortunately a few links I didn't create are popping up in one of the corners. I could really use a bit of guidance as to how I should go about cleaning the site and removing all of it.
The only files I uploaded to the server were a handful of bootstrap css files, the index.html, and one image for a background. I've run aVast on all these files but they're coming up clean, although I'm not sure if this is a sufficient enough scan.
All files were uploaded using filezilla FTP.
I've opened up inspect element in chrome while loading the webpage and under resources it shows files being loaded that aren't mine and that I can't locate on the server. To be specific, one is an image file whose URL points to acint(dot)net and the other is a script called aci.js, which is located at acint(dot)net/aci.js (code below)
(function(n){n(window,"undefined"===typeof window._acic?{}:window._acic,"undefined"===typeof window._aci_debug?!1:window._aci_debug)})(function(n,f,t){function k(){if(!(this instanceof k))return new k;this.version="0.0.9";this.urlHit="//www.acint.net/hit/";this.urlJump="//www.acint.net/jump/";this.uid="";this.config={dataProvider:"",allowExtLinksTrack:!0,customData:null}}var v=!!t,s={};"object"==typeof JSON&&"function"==typeof JSON.stringify?s.stringify=function(a){return JSON.stringify(a)}:function(){function a(a){function b(a){return 10>
a?"0"+a:a}if(a&&"object"==typeof a){if(a instanceof Date)return isFinite(a.valueOf())?a.getUTCFullYear()+"-"+b(a.getUTCMonth()+1)+"-"+b(a.getUTCDate())+"T"+b(a.getUTCHours())+":"+b(a.getUTCMinutes())+":"+b(a.getUTCSeconds())+"Z":null;if(a instanceof String||a instanceof Number||a instanceof Boolean)return a.valueOf()}return a}function c(a){e.lastIndex=0;return e.test(a)?'"'+a.replace(e,function(a){var b=h[a];return"string"===typeof b?b:"\\u"+("0000"+a.charCodeAt(0).toString(16)).slice(-4)})+'"':'"'+
a+'"'}function d(e,u){var h,r,f,k,n=b,p,m=u[e],m=a(m);"function"===typeof q&&(m=q.call(u,e,m));switch(typeof m){case "string":return c(m);case "number":return isFinite(m)?String(m):"null";case "boolean":case "null":return String(m);case "object":if(!m)return"null";b+=g;p=[];if("[object Array]"===Object.prototype.toString.apply(m)){k=m.length;for(h=0;h<k;h+=1)p[h]=d(h,m)||"null";f=0===p.length?"[]":b?"[\n"+b+p.join(",\n"+b)+"\n"+n+"]":"["+p.join(",")+"]";b=n;return f}if(q&&"object"===typeof q)for(k=
q.length,h=0;h<k;h+=1)r=q[h],"string"===typeof r&&(f=d(r,m))&&p.push(c(r)+(b?": ":":")+f);else for(r in m)Object.prototype.hasOwnProperty.call(m,r)&&(f=d(r,m))&&p.push(c(r)+(b?": ":":")+f);f=0===p.length?"{}":b?"{\n"+b+p.join(",\n"+b)+"\n"+n+"}":"{"+p.join(",")+"}";b=n;return f}}var e,b,g,h,q;"function"!==typeof s.stringify&&(e=/[\\\"\x00-\x1f\x7f-\x9f\u00ad\u0600-\u0604\u070f\u17b4\u17b5\u200c-\u200f\u2028-\u202f\u2060-\u206f\ufeff\ufff0-\uffff]/g,h={"\b":"\\b","\t":"\\t","\n":"\\n","\f":"\\f","\r":"\\r",
'"':'\\"',"\\":"\\\\"},s.stringify=function(a,c,e){var h;g=b="";if("number"===typeof e)for(h=0;h<e;h+=1)g+=" ";else"string"===typeof e&&(g=e);if((q=c)&&"function"!==typeof c&&("object"!==typeof c||"number"!==typeof c.length))throw Error("JSONStub.stringify");return d("",{"":a})})}();var e={generateUUID:function(){return"xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(/[xy]/g,function(a){var c=16*Math.random()|0;return("x"==a?c:c&3|8).toString(16)}).toLowerCase()},stringTrimLimit:function(a){a=a.replace(/\s+/gmi,
" ").replace(/^\s+|\s+$/g,"");1E3<a.length&&(a=a.substr(0,1E3));return a},getTimeZoneOffsetIso8601:function(){var a=function(a){return 10>a?"0"+a:a},c=(new Date).getTimezoneOffset();return(0<c?"-":"+")+a(Math.floor(Math.abs(c)/60))+":"+a(Math.abs(c)%60)},isObject:function(a){return"object"===typeof a&&"[object Object]"===Object.prototype.toString.call(a)},isArray:function(a){return"undefined"!==typeof Array.isArray?Array.isArray(a):"[object Array]"===Object.prototype.toString.call(a)},isEmptyObject:function(a){for(var c in a)return!1;
return!0},isString:function(a){return"string"===typeof a||a instanceof String},encodeUriParam:function(a){a=""+a;return encodeURIComponent instanceof Function?encodeURIComponent(a):escape(a)},implodeUriParams:function(a){var c,d=[];for(c in a)a.hasOwnProperty(c)&&d.push(c+"="+a[c]);return d.join("&")},makeRequestUri:function(a,c){return a+"?"+e.implodeUriParams(c)},makeRequestImage:function(a){(new Image).src=a},addEventListenerCrossBrowser:function(a,c,d){a.addEventListener?a.addEventListener(c,
d,!1):a.attachEvent?a.attachEvent("on"+c,d):"function"==typeof a["on"+c]&&(a["on"+c]=d)},addReadOnlyProperty:function(a,c,d){Object.defineProperty(a,c,{value:d,writeable:!1,enumerable:!0,configurable:!1})},fireEventCrossBrowser:function(a,c){var d,e=document;if(e.createEvent){var b=null;switch(c){case "click":case "dblclick":case "mousedown":case "mouseup":case "mouseover":case "mousemove":case "mouseout":case "mouseenter":case "mouseleave":b="MouseEvent";break;case "wheel":b="WheelEvent";break;case "load":case "unload":case "abort":case "error":case "select":case "resize":case "scroll":b=
"UIEvent";break;case "focus":case "focusin":case "focusout":case "blur":b="FocusEvent";break;case "beforeinput":case "input":b="InputEvent";break;case "keydown":case "keyup":b="KeyboardEvent";break;case "compositionstart":case "compositionupdate":case "compositionend":b="CompositionEvent"}try{d=e.createEvent(b)}catch(g){try{d=e.createEvent("HtmlEvents")}catch(h){try{d=e.createEvent("Event")}catch(q){throw Error("Cannot create event object for specified event: "+c);}}}d.initEvent(c,!0,!1);a.dispatchEvent(d)}else if(e.createEventObject)d=
e.createEventObject(),d.eventType=c,a.fireEvent("on"+c,d);else if("function"==typeof a["on"+c])a["on"+c]()},fireSameEventCrosBrowser:function(a,c,d){var e=document,b;if(e.createEvent)try{b=new d.constructor(c,d),a.dispatchEvent(b)}catch(g){b=e.createEvent("MouseEvent"),b.initMouseEvent(d.type,!0,!0,window,0,d.screenX,d.screenY,d.clientX,d.clientY,d.ctrlKey,d.altKey,d.shiftKey,d.metaKey,d.button,d.relatedTarget||null),a.dispatchEvent(b)}else if(e.createEventObject)b=e.createEventObject(d),b.eventType=
c,a.fireEvent("on"+c,b);else if("function"==typeof a["on"+c])a["on"+c]()},bindOnReady:function(a){function c(){e||(e=!0,a())}function d(){if(!e)try{h.doScroll("left"),c()}catch(a){setTimeout(d,10)}}var e=!1,b=window,g=document,h=g.documentElement;if(g.addEventListener)g.addEventListener("DOMContentLoaded",c,!1);else if(g.attachEvent){try{var q=null!=b.frameElement}catch(l){}h.doScroll&&!q&&d();g.attachEvent("onreadystatechange",function(){"complete"===g.readyState&&c()})}else if(b.addEventListener)b.addEventListener("load",
c,!1);else if(b.attachEvent)b.attachEvent("onload",c);else{var f=b.onload;b.onload=function(){f&&f();c()}}}};k.prototype.init=function(){if(!this.isAlreadyLoaded()){var a=this,c,d=document;this.uid=e.generateUUID();this.parseConfig();c=e.makeRequestUri(this.urlHit,this.collectDataOnInit());e.makeRequestImage(c);if(!0===this.config.allowExtLinksTrack){var f=function(b){var d=window,c=!1,f;b=b||d.event;var l=b.target||b.srcElement,k=l;if(!("tagName"in l&&"a"==l.tagName.toLowerCase())){for(c=l.parentNode;c;){if("tagName"in
c&&"a"==c.tagName.toLowerCase()){f=c;break}if("parentNode"in c)c=c.parentNode;else break}if(f)l=f;else return}if("href"in l&&/^(http:|https:|)\/\/.+/.test(l.href)&&!("hostname"in l&&l.hostname===d.location.hostname)){if(b.shiftKey||b.altKey||b.ctrlKey||b.metaKey)c=!0;c=!0;l.hasOwnProperty("_delayClick")?!1===l.hasOwnProperty("_canSkipDelay")?"preventDefault"in b?b.preventDefault():b.returnValue=!1:(delete l._canSkipDelay,delete l._delayClick):(d=e.makeRequestUri(a.urlJump,a.collectDataOnClick(b,l)),
e.makeRequestImage(d),!1===c&&(l._delayClick=!0,setTimeout(function(){l._canSkipDelay=!0;e.fireSameEventCrosBrowser(k,"click",b)},200),"preventDefault"in b?b.preventDefault():b.returnValue=!1))}};"interactive"==d.readyState||"complete"==d.readyState?e.addEventListenerCrossBrowser(d.body,"click",f):e.bindOnReady(function(){e.addEventListenerCrossBrowser(d.body,"click",f)})}}};k.prototype.isAlreadyLoaded=function(){if("object"==typeof n._acil&&"function"==typeof n._acil.isLoaded)return!0;"undefined"!==
typeof Object.defineProperty?(e.addReadOnlyProperty(n,"_acil",{}),e.addReadOnlyProperty(n._acil,"isLoaded",function(){return!0})):n._acil=function(){return{isLoaded:function(){return!0}}}();return!1};k.prototype.parseConfig=function(){if(e.isObject(f)&&!e.isEmptyObject(f)&&(f.hasOwnProperty("dataProvider")&&(this.config.dataProvider=isNaN(parseInt(f.dataProvider,10))?"":Math.abs(parseInt(f.dataProvider,10))),f.hasOwnProperty("allowExtLinksTrack")&&(this.config.allowExtLinksTrack=!0===f.allowExtLinksTrack),
f.hasOwnProperty("customData"))){var a=f.customData;if(!("undefined"===typeof a||null==a||"function"==typeof a||e.isString(a)&&0===a.length||e.isObject(a)&&e.isEmptyObject(a)||e.isArray(a)&&0===a.length)){try{if(e.isObject(a)||e.isArray(a)){if(a=s.stringify(a),"{}"===a||"[]"===a)a=null}else e.isString(a)||(a=a.toString(),0===a.length&&(a=null))}catch(c){if(a=null,!0===t)throw c;}this.config.customData=a}}};k.prototype.dataAddObligatoryParams=function(a){a.v=this.version;a.uid=this.uid;a.dp=this.config.dataProvider;
a.tz=e.encodeUriParam(e.getTimeZoneOffsetIso8601());a.nc=Math.random().toString().substr(2,8)};k.prototype.collectDataOnInit=function(){var a=document,c=window,d={u:e.encodeUriParam(c.location.href),r:e.encodeUriParam(a.referrer||""),rs:c.screen.width+"x"+c.screen.height,t:e.encodeUriParam(e.stringTrimLimit(a.title)),oE:+this.config.allowExtLinksTrack};null!==this.config.customData&&(d.cd=e.encodeUriParam(e.stringTrimLimit(this.config.customData)));c.parent!==c&&(d["if"]=e.encodeUriParam(c.location.href),
d.u=e.encodeUriParam(a.referrer||""),d.r="");this.dataAddObligatoryParams(d);return d};k.prototype.collectDataOnClick=function(a,c){var d=document,f=window,b={vP:"",c:"",r:"",u:"",aT:"",hT:0},g=a.target.ownerDocument||d,d=g.documentElement,g=g.body;b.vP=(f.innerWidth||d.clientWidth||g.clientWidth)+"x"+(f.innerHeight||d.clientHeight||g.clientHeight);null==a.pageX&&null!=a.clientX?b.c=a.clientX+(d&&d.scrollLeft||g&&g.scrollLeft||0)-(d&&d.clientLeft||g&&g.clientLeft||0)+"x"+(a.clientY+(d&&d.scrollTop||
g&&g.scrollTop||0)-(d&&d.clientTop||g&&g.clientTop||0)):null!==a.pageX&&(b.c=a.pageX+"x"+a.pageY);b.r=e.encodeUriParam(f.location.href);b.u=e.encodeUriParam(c.href);var f=c.childNodes,h=0,k=0,l=0;if(1<=f.length){for(d=0;d<f.length;d++)g=f[d],3!==g.nodeType&&1!==g.nodeType||1!==g.nodeType||(k+=1,"tagName"in f[d]&&"img"===f[d].tagName.toLowerCase()&&(h+=1,l=d));1==h&&h==k?b.hT=1:0<k&&(b.hT=2)}switch(b.hT){case 1:b.aT=f[l].getAttribute("alt")||"";break;case 2:case 0:b.aT=c.innerHTML?c.innerHTML.toString().replace(/<\/?[^>]+>/gmi,
""):""}0<b.aT.length&&(b.aT=e.stringTrimLimit(b.aT),""!==b.aT&&(b.aT=/^(\S+(\s|)){1,10}/gmi.exec(b.aT)[0].replace(/^\s+|\s+$/g,"")));b.aT=e.encodeUriParam(b.aT);this.dataAddObligatoryParams(b);return b};try{(new k).init()}catch(w){if(!0===v)throw w;}});
Any help would be really appreciated!
Some security weakness in your website is allowing someone to inject malicious scripts and/or code.
Disable the web server while you work on resolving this issue.
Change your FTP password.
If you can, switch to SFTP and use a certificate for authentication instead of a password.
Set the firewall to only accept FTP connections from your address, if that IP address is static (will not change on you), or from a range of IP addresses that you may be assigned by your ISP if you can determine such a range.
Remove all of your website code and re-upload from source control.
If you have a database behind your code:
Review your code for SQL Injection attack opportunities
Manually review all text columns in all tables for anything that looks like JavaScript.
Note: Since aVast did not turn up problems, the above steps are likely to get you back to where you need to be. If the problem persists, you may need to perform a complete reinstall of your server.
If you disable your extensions does it still show up? It is possible , if it isn't within the source code itself (like if the source code on the website is the same as on your computer) the script is an ads/monetizing script added by an extension like Freecorder. This is especially suspicious of the extension adding the script is an adware such as RelevantKnowledge, Yontoo, Iminent or iNTERNETTurbo.
If disabling your extensions/using a different browser makes no difference, some webhosts such as 1freehosting will add a script to your page for analytical purposes to check if people still visit your site and/or give you a graph of how many visits you've gotten. The script allows them to count the calls to the server for that page. This is not within your control - the webhost itself adds that code after the end html tag.
If neither it is possible your FTP has been accessed and you should change your password.
My wordpress websites were becoming slow and I've found out the same code was infecting my websites.
The malware javascript is loaded from http://www.acint.net/aci.js and because of it it takes about 20 seconds to fully load the page.
The malware creating a gif file which contains some script, it's located in "wp-content/themes/(your theme name)/images" and it's called logos.gif or logos2.gif - if you go to that location with your FTP client and sort files by most recently created it should be on the top of the list. I think it's created every time a home page is opened.
Make sure your wordpress is up to date (make files and database backup before updating) - that should remove the malware, also after that remove the logos.gif file.
https://sitecheck.sucuri.net/ - that's a cool free scanner to check if your website is infected.
I use this plugin for signing in but it didn't help in that case:
https://wordpress.org/plugins/are-you-robot-recaptcha/
I've installed this Firewall plugin so hopefully it'll make the website more secure:
https://wordpress.org/plugins/wp-simple-firewall/
I wrote a PhantomJS app to crawl over a site I built and check for a JavaScript file to be included. The JavaScript is similar to Google where some inline code loads in another JS file. The app looks for that other JS file which is why I used Phantom.
What's the expected result?
The console output should read through a ton of URLs and then tell if the script is loaded or not.
What's really happening?
The console output will read as expected for about 50 requests and then just start spitting out this error:
2013-02-21T10:01:23 [FATAL] QEventDispatcherUNIXPrivate(): Can not continue without a thread pipe
QEventDispatcherUNIXPrivate(): Unable to create thread pipe: Too many open files
This is the block of code that opens a page and searches for the script include:
page.open(url, function (status) {
console.log(YELLOW, url, status, CLEAR);
var found = page.evaluate(function () {
if (document.querySelectorAll("script[src='***']").length) {
return true;
} else { return false; }
});
if (found) {
console.log(GREEN, 'JavaScript found on', url, CLEAR);
} else {
console.log(RED, 'JavaScript not found on', url, CLEAR);
}
self.crawledURLs[url] = true;
self.crawlURLs(self.getAllLinks(page), depth-1);
});
The crawledURLs object is just an object of urls that I've already crawled. The crawlURLs function just goes through the links from the getAllLinks function and calls the open function on all links that have the base domain of the domain that the crawler started on.
Edit
I modified the last block of the code to be as follows, but still have the same issue. I have added page.close() to the file.
if (!found) {
console.log(RED, 'JavaScript not found on', url, CLEAR);
}
self.crawledURLs[url] = true;
var links = self.getAllLinks(page);
page.close();
self.crawlURLs(links, depth-1);
From the documentation:
Due to some technical limitations, the web page object might not be completely garbage collected. This is often encountered when the same object is used over and over again.
The solution is to explicitly call close() of the web page object (i.e. page in many cases) at the right time.
Some included examples, such as follow.js, demonstrate multiple page objects with explicit close.
Open Files Limit.
Even with closing files properly, you might still run into this error.
After scouring the internets I discovered that you need to increase your limit of the number of files a single process is allowed to have open. In my case, I was generating PDFs with hundreds to thousands of pages.
There are different ways to adjust this setting based on the system you are running but here is what worked for me on an Ubuntu server:
Add the following to the end of /etc/security/limits.conf:
# Sets the open file maximum here.
# Generating large PDFs hits the default ceiling (1024) quickly.
* hard nofile 65535
* soft nofile 65535
root hard nofile 65535 # Need these two lines because the wildcards (above)
root soft nofile 65535 # are not applied to the root user as well.
A good reference for the ulimit command can be found here.
I hope that puts some people on the right track.
I had this error come up while running multiple threads in my ruby program.
I was running phantomjs with Capybara-poltergeist and each thread was visiting a page opening up the same CSV file and writing to it.
I was able to fix it by using the Mutex class.
lock = Mutex.new
lock.synchronize do
CSV.open("reservations.csv", "w") do |file|
file << ["Status","Name","Res-Code","LS-Num","Check-in","Check-out","Talk-URL"]
$status.length.times do |i|
file << [$status[i],$guest_name[i],$reservation_code[i],$listing_number[i],$check_in[i],$check_out[i], $talk_url[i]]
end
end
puts "#{user.email} PAGE NUMBER ##{p+1} WRITTEN TO CSV"
end
end