I am a web scraping newbie. I have identified a JS function on a server, whose I/O I need to track. It's an obscured encryption function, and I need to know what values it takes for input and what it outputs. How can I get this done?
So far, I've opened up the web page in firebug, identified where the function is called and then set a breakpoint. However, when I refresh the page and clear my cache, the browser ignores my breakpoint and loads the page as if there were no breakpoints on it.
You can use console.log in your function to load data to console. But it's better to use it any debug state:
if (app.debug) {
console.log(data)
}
Where app for example is your namespace, debug is flag for logging and data is your function/application data.
More info about console.
About clear cache: Firebug and Chrome Dev Tools remove all breakpoints from file if file was changed.
Related
I am debugging a Chrome extension off github. (https://github.com/artemave/translate_onhover)
I want to see the debug() messages that are in the code: like this one...
--------From contentscript.js------------
const hit_text_node = getExactTextNode(text_nodes, e)
if (!hit_text_node) {
debug('hit between lines')
return ''
}
I have tried:
In Chrome on a web page, I tried right-click verbose, but they don't appear.
I do see console.log('jim103',word) from devtools from chrome://extensions background.js
I also tried putting the word
debugger;
in both files, but it just caused a breakpoint in background.js, not the debug message I seek...
I know these two files are back/front separated... That's all I know.
JavaScript does not have a built-in function called debug, which tells us that it's a custom one. We could also figure it out by trying to step-into that function during debugging (with F11 key). This leads us to trying to find its definition – and it's at the top of the file: const debug = require('debug')('transover'). The fact that the argument of require here is not a path, but just a name, tells us that this is a package – go to package.json and there it is: "debug": "^4.1.1". By default, NPM packages are downloaded from npmjs.com, let's head there and find the package – https://www.npmjs.com/package/debug. Its docs tell us that in the browser environment its behaviour is controlled by localStorage https://www.npmjs.com/package/debug#browser-support. So all you have to do is type localStorage.debug = 'transover*' In the console for a particular site and reload the page. Debug messages will start appearing in the console.
-В Иванов
There are two outputs, back/front background.js comes out in DevTools, but front=contentscript.js:debug() stuff comes out on the page.
Do right-click->Inspect and look on the right side.
Does anyone know of a way to save the console.log output in Chrome to a file? Or how to copy the text out of the console?
Say you are running a few hours of functional tests and you've got thousands of lines of console.log output in Chrome. How do you save it or export it?
Good news
Chrome dev tools now allows you to save the console output to a file natively
Open the console
Right-click
Select "save as.."
Chrome Developer instructions here.
I needed to do the same thing and this is the solution I found:
Enable logging from the command line using the flags:
--enable-logging --v=1
This logs everything Chrome does internally, but it also logs all the console.log() messages as well. The log file is called chrome_debug.log and is located in the User Data Directory which can be overridden by supplying --user-data-dir=PATH (more info here).
Filter the log file you get for lines with CONSOLE(\d+).
Note that console logs do not appear with --incognito.
I have found a great and easy way for this.
In the console - right click on the console logged object
Click on 'Store as global variable'
See the name of the new variable - e.g. it is variableName1
Type in the console: JSON.stringify(variableName1)
Copy the variable string content: e.g. {"a":1,"b":2,"c":3}
Go to some JSON online editor:
e.g. https://jsoneditoronline.org/
There is an open-source javascript plugin that does just that, but for any browser - debugout.js
Debugout.js records and save console.logs so your application can access them. Full disclosure, I wrote it. It formats different types appropriately, can handle nested objects and arrays, and can optionally put a timestamp next to each log. You can also toggle live-logging in one place, and without having to remove all your logging statements.
For better log file (without the Chrome-debug nonsense) use:
--enable-logging --log-level=0
instead of
--v=1 which is just too much info.
It will still provide the errors and warnings like you would typically see in the Chrome console.
update May 18, 2020: Actually, I think this is no longer true. I couldn't find the console messages within whatever this logging level is.
This may or may not be helpful but on Windows you can read the console log using Event Tracing for Windows
http://msdn.microsoft.com/en-us/library/ms751538.aspx
Our integration tests are run in .NET so I use this method to add the console log to our test output. I've made a sample console project to demonstrate here: https://github.com/jkells/chrome-trace
--enable-logging --v=1 doesn't seem to work on the latest version of Chrome.
For Google Chrome Version 84.0.4147.105 and higher,
just right click and click 'Save as' and 'Save'
then, txt file will be saved
A lot of good answers but why not just use JSON.stringify(your_variable) ? Then take the contents via copy and paste (remove outer quotes). I posted this same answer also at: How to save the output of a console.log(object) to a file?
There is another open-source tool which allows you to save all console.log output in a file on your server - JS LogFlush (plug!).
JS LogFlush is an integrated JavaScript logging solution which include:
cross-browser UI-less replacement of console.log - on client side.
log storage system - on server side.
Demo
If you're running an Apache server on your localhost (don't do this on a production server), you can also post the results to a script instead of writing it to console.
So instead of console.log, you can write:
JSONP('http://localhost/save.php', {fn: 'filename.txt', data: json});
Then save.php can do this
<?php
$fn = $_REQUEST['fn'];
$data = $_REQUEST['data'];
file_put_contents("path/$fn", $data);
Right-click directly on the logged value you want to copy
In the right-click menu, select "Store as global variable"
You'll see the value saved as something like "temp1" on the next line in the console
In the console, type copy(temp1) and hit return (replace temp1 with the variable name from the previous step). Now the logged value is copied to your clipboard.
Paste the values to wherever you want
This is especially good as an approach if you don't want to mess with changing flags/settings in Chrome and don't want to deal with JSON stringifying and parsing etc.
Update: I just found this explanation of what I suggested with images that's easier to follow https://scottwhittaker.net/chrome-devtools/2016/02/29/chrome-devtools-copy-object.html
These days it's very easy - right click any item displayed in the console log and select save as and save the whole log output to a file on your computer.
On Linux (at least) you can set CHROME_LOG_FILE in the environment to have chrome write a log of the Console activity to the named file each time it runs. The log is overwritten every time chrome starts. This way, if you have an automated session that runs chrome, you don't have a to change the way chrome is started, and the log is there after the session ends.
export CHROME_LOG_FILE=chrome.log
the other solutions in this thread weren't working on my mac. Here's a logger that saves a string representation intermittently using ajax. use it with console.save instead of console.log
var logFileString="";
var maxLogLength=1024*128;
console.save=function(){
var logArgs={};
for(var i=0; i<arguments.length; i++) logArgs['arg'+i]=arguments[i];
console.log(logArgs);
// keep a string representation of every log
logFileString+=JSON.stringify(logArgs,null,2)+'\n';
// save the string representation when it gets big
if(logFileString.length>maxLogLength){
// send a copy in case race conditions change it mid-save
saveLog(logFileString);
logFileString="";
}
};
depending on what you need, you can save that string or just console.log it and copy and paste. here's an ajax for you in case you want to save it:
function saveLog(data){
// do some ajax stuff with data.
var xhttp = new XMLHttpRequest();
xhttp.onreadystatechange = function(){
if (this.readyState == 4 && this.status == 200) {}
}
xhttp.open("POST", 'saveLog.php', true);
xhttp.send(data);
}
the saveLog.php should append the data to a log file somewhere. I didn't need that part so I'm not including it here. :)
https://www.google.com/search?q=php+append+to+log
This answer might seem specifically related, but specifically for Network Log, you can visit the following link.
The reason I've post this answer is because in my case, the console.log printed a long truncated text so I couldn't get the value from the console. I solved by getting the api response I was printing directly from the network log.
chrome://net-export/
There you may see a similar windows to this, just press the Start Logging to Disk button and that's it:
Create a batch file using below command and save it as ChromeDebug.bat in your desktop.
start chrome --enable-logging --v=1
Close all other Chrome tabs and windows.
Double click ChromeDebug.bat file which will open Chrome and a command prompt with Chrome icon in taskbar.
All the web application logs will be stored in below path.
Run the below path in Run command to open chrome log file
%LocalAppData%\Google\Chrome\User Data\chrome_debug.log
I want to create a custom profiler for Javascript as a Chrome DevTools Extension. To do so, I'd have to instrument all Javascript code of a website (parse to AST, inject hooks, generate new source). This should've been easily possible using chrome.devtools.inspectedWindow.reload() and its parameter preprocessorScript described here: https://developer.chrome.com/extensions/devtools_inspectedWindow.
Unfortunately, this feature has been removed (https://bugs.chromium.org/p/chromium/issues/detail?id=438626) because nobody was using it.
Do you know of any other way I could achieve the same thing with a Chrome Extension? Is there any other way I can replace an incoming Javascript source with a changed version? This question is very specific to Chrome Extensions (and maybe extensions to other browsers), I'm asking this as a last resort before going a different route (e.g. dedicated app).
Use the Chrome Debugging Protocol.
First, use DOMDebugger.setInstrumentationBreakpoint with eventName: "scriptFirstStatement" as a parameter to add a break-point to the first statement of each script.
Second, in the Debugger Domain, there is an event called scriptParsed. Listen to it and if called, use Debugger.setScriptSource to change the source.
Finally, call Debugger.resume each time after you edited a source file with setScriptSource.
Example in semi-pseudo-code:
// Prevent code being executed
cdp.sendCommand("DOMDebugger.setInstrumentationBreakpoint", {
eventName: "scriptFirstStatement"
});
// Enable Debugger domain to receive its events
cdp.sendCommand("Debugger.enable");
cdp.addListener("message", (event, method, params) => {
// Script is ready to be edited
if (method === "Debugger.scriptParsed") {
cdp.sendCommand("Debugger.setScriptSource", {
scriptId: params.scriptId,
scriptSource: `console.log("edited script ${params.url}");`
}, (err, msg) => {
// After editing, resume code execution.
cdg.sendCommand("Debugger.resume");
});
}
});
The implementation above is not ideal. It should probably listen to the breakpoint event, get to the script using the associated event data, edit the script and then resume. Listening to scriptParsed and then resuming the debugger are two things that shouldn't be together, it could create problems. It makes for a simpler example, though.
On HTTP you can use the chrome.webRequest API to redirect requests for JS code to data URLs containing the processed JavaScript code.
However, this won't work for inline script tags. It also won't work on HTTPS, since the data URLs are considered unsafe. And data URLs are can't be longer than 2MB in Chrome, so you won't be able to redirect to large JS files.
If the exact order of execution of each script isn't important you could cancel the script requests and then later send a message with the script content to the page. This would make it work on HTTPS.
To address both issues you could redirect the HTML page itself to a data URL, in order to gain more control. That has a few negative consequences though:
Can't reload page because URL is fixed to data URL
Need to add or update <base> tag to make sure stylesheet/image URLs go to the correct URL
Breaks ajax requests that require cookies/authentication (not sure if this can be fixed)
No support for localStorage on data URLs
Not sure if this works: in order to fix #1 and #4 you could consider setting up an HTML page within your Chrome extension and then using that as the base page instead of a data URL.
Another idea that may or may not work: Use chrome.debugger to modify the source code.
Pretty straight-forward, I'm developing an MVC5 application and have noticed (lately) that my Browser appears to be caching the JavaScript code I have on the view within #section Scripts { }.
Currently I am developing with Chrome and I have tried CTRL+F5 & CTRL+SHFT+R which reloads the page, but the alert() I uncommented within the javascript code is still rendering as commented. I also tried going to my localhost through Incognito Mode as well as other Browsers (Firefox, IE) and am getting the same behavior. This is my /Home/Index.cshtml View, which is the default View which loads when the application starts. I have also tried adding some extra HTML text into the page and again the new code is not taking effect/showing.
My current Chrome version is Version 41.0.2272.118 m if anyone has any ideas what might be going on?
UPDATE:
I have gone under the Developer Tools => General Settings in Chrome and checked [X] Disable cache (while DevTools is open) and then repeatedly (with DevTools still open) tried CTRL+SHFT+R and CTRL+F5 with the same results of before where my changes are not taking effect.
UPDATE 2:
With DevTools open I have also held the Refresh button down and tried Normal/Hard/and Empty Cache & Hard Reload options all with the same result. For simplicity of testing I added an alert in the below to dispaly as soon as the page loads (and currently no alert comes up):
$(document).ready(function () {
alert("Test");
// Other Code/Functions -- No Error showing in Console
});
If you are using Bundling from MVC, you have two options to disable caching:
Use BundleTable.EnableOptimizations. This instructs the bundling to minify and optimize your bundle even while debugging. It generates a hash in the process, based on the content of the script, so your customers browsers can cache this file for a long time. It will generate a whole different hash the next time your file changes, so your customers can see your changes. The downside is that your script will become unreadable and you won't be able to debug it, so this might not be your best option.
Use System.Web.Optimization.BundleTable.Bundles.ResolveBundleUrl("url", true) to resolve your script's URL, the second parameter (true) is requiring a hash to be generated with the URL, thus, preventing caching from your browser when you change the file. This is exactly the same hash generated in the first option, but without minifying.
I created a small demo showing that the second option prevents caching from happening, the trick is getting the hash generated from your script's content without minifying your script.
I created a script file called myscript.js with this content:
$(document).ready(function () {
alert('a');
});
Then I added this to my BundleConfig.cs:
// PLEASE NOTE this is **NOT** a ScriptBundle
bundles.Add(new Bundle("~/bundles/myscripts").Include(
"~/Scripts/myscript*"));
If you add a ScriptBundle, you will get a minified response again, since ScriptBundle is just a Bundle using JsMinify transformation (source). That's why we just use Bundle.
Now you can just add your script using this method to resolve the script URL with the hash appendend. You can use the Script.Render
#Scripts.Render(System.Web.Optimization.BundleTable.Bundles.ResolveBundleUrl("~/bundles/myscripts", true))
Or the script tag:
<script src="#System.Web.Optimization.BundleTable.Bundles.ResolveBundleUrl("~/bundles/myscripts", true)"></script>
Either way will generate a URL with a hash to prevent caching:
After editing my file:
You might want to add a no_cache variable after your script url like:
<script src="js/stg/Stg.js?nocache=#random_number"></script>
If you manage to put a random number to the place i indicated, the browser will automatically download the latest version of the script after an F5
A quick trick that solves this problem consists of opening the script file in a new tab, then refresh it on this page.
If you happen to have Chrome dev tools open it will even refresh it there.
From dev tool you can even easily right click-open in new tab the script.
The main idea is to run a random page on Internet Explorer and get javascript errors and logs.
Is there a way to recover javascript console logs and execution error
from a random web page without accessing the F12 tool on Internet
Explorer?
I found that with Chrome based browser, you can get it on your AppData file log by adding --enable-logging --v=1 args when launching.
Any solution with any language are welcome.
Thank you for your answer.
NOTE :
random page on Internet Explorer means that I do not have the access on the source code.
Basic solution to this would be:
1. Use Exception Handling to catch the errors.
2. Log errors in a Global Array
3. Log the errors in a file using Blob and URL.createObjectURL. All recent browsers support this.
Have you considered using a Bookmarklet that:
Overrides window.console.log and window.console.error (to intercept messages)
Logs incoming messages somewhere using createObjectURL?
Or you could use something like firebuglite and auto-enable it like this:
<script type="text/javascript" src="https://getfirebug.com/firebug-lite.js">
{
overrideConsole: false,
startInNewWindow: true,
startOpened: true,
enableTrace: true
}
</script>
More instructions are here: http://getfirebug.com/firebuglite
If the F12 tool is not of your interest, then what about the Event Viewer? Open Event Viewer from Control Panel -> System and Security -> Administrative Tools -> Event Viewer. Then select the log Applications and Services Logs\Internet Explorer.
By default no events are being logged for Internet Explorer, to enable them create a new DWORD registry value named Feature_Enable_Compat_Logging under the following registry key:
HKCU\SOFTWARE\Microsoft\Windows\Internet Explorer\Main \FeatureControl
and set the registry value to 1.
Check the logs you get to see if it's what you're looking for.
One idea would be to write a browser extension which listens for window.onerror and writes to a file. Definitely not as elegant as the Chrome solution, but it would work fairly well.
Using local proxy might be a good one-time solution.
Charles web debugging proxy app has nice UI and it allows to replace any response with local resource.
So basically you'll need:
Download one any of the js files used on target page
add any code you wish to saved version
set up Charles to serve you your local version instead of remote one
You might try Fiddler. It's got its own logging and has amazing inspection power. It won't capture IE specific errors, since it's at a different layer, but it will definitely get you any code that's coming over the wire.