I'm looking for a tool that would list all the network information that a html dom (javascript etc as well) page loads. Exeactly what chrome shows on the developer tools/Network tab.
I've tried the Chromium-browser under ubutu to export this data but was not able to get the networking stats.
So what I would need is to parse a request preferably thru the command line (ubuntu/debian) which would return the networking stats.
There is network monitor at Firefox dev tools. Take a look at mdn page where this feature is explained. It is possible to export HAR information from the Network panel by right-clicking and selecting "Save all as HAR". HAR is a network request archive format used by many performance and request analysis tools.
You can also use firebug addon which has also his own network monitor and extract data by using the netexport firebug extension.
For automated process (that will open the browser, then firebug, then the page and then will export data), you probably need the auto-tool used for testing firefox which is called mozmill
Related
I am trying to automate some online media news downloads. Since not all web pages follow the DOM of their structure, I open web pages with Chrome or Firefox and take screenshots and then send them to an OCR of my own development. The problem is that it brings a lot of junk information from the advertising and the page. For this reason, I thought about making selenium open the browser's reading view and extract much cleaner information.
The readability library that Mozilla uses in Firefox is open source and available here: https://github.com/mozilla/readability
You can fetch the web contents in selenium and pass out all through this process to get the text output without needing an OCR step at all.
While trying to extract information from a web page, I tend to use the console in the web developer tools to write some simple javascript to get the DOM elements and capture the data I need.
I tend to treat the web developer console as an REPL and I would like to save those code snippets for possible re-use in the future. Spark repl has a very useful command :save which saves all the commands from the session to a file so I can replay it later.
Is there such a feature in firefox web developer console to save the commands in the current console session?
I couldn't find any mention of such thing in the user documents. My google search and stack overflow search also turned empty. Only thing I've come close to is Scratchpad, which can be enabled from the web developer tools settings, but it doesn't seem to fit my need.
What I want to do
Make a simple socket connection to a server on the browser. I want to not send any header information with the socket connection.
The Problem
It looks like I am unable to make a socket connection with javascript that does not send header data (Is there a way to do a tcp connection to an IP with javascript?).
I thought maybe I could make a connection with a chrome extension, however it looks like the socket API is only available for chrome apps (Google Chrome Socket API in extensions).
I am thinking that I might need to make a native application that will make socket connections through requests made by the browser using Native Messaging.
Is there anyway I can achieve this or am I out of luck?
Raw socket connections through the browser are wrapped up in security concerns. Users can be easily manipulated to allow things to run that shouldn't.
TCP and UDP Socket API
W3C Editor's Draft 20 January 2016
is located here.
http://raw-sockets.sysapps.org/
Mozilla's API information here: https://developer.mozilla.org/en-US/docs/Archive/B2G_OS/API/TCPSocket "This API is available on Firefox OS for privileged or certified applications only."
If you work with raw TCP connections. I would suggest
(1) downloading PHP onto the local computer. PHP has a developer web host build in so you can run whatever application you want on PHP using the browser as your GUI.
(2) download node.js.
You are not out of luck you just need to achieve it with the understanding that you are working outside the box for normal browser based scripting created from security concerns, and that means the user/client needs to install something manually.
If you must use chrome browser on the client side, you will need to make an -extension- correction webapp. You can as a developer make one that you can use on your own computers.
https://developer.chrome.com/extensions/getstarted
https://developer.chrome.com/apps/first_app
Load the extension#
Extensions that you download from the Chrome Web
Store are packaged up as .crx files, which is great for distribution,
but not so great for development. Recognizing this, Chrome gives you a
quick way of loading up your working directory for testing. Let's do
that now.
Visit chrome://extensions in your browser (or open up the Chrome menu
by clicking the icon to the far right of the Omnibox: The menu's icon
is three horizontal bars. and select Extensions under the Tools menu
to get to the same place).
Ensure that the Developer mode checkbox in the top right-hand corner
is checked.
Click Load unpacked extension… to pop up a file-selection dialog.
Navigate to the directory in which your extension files live, and
select it.
Alternatively, you can drag and drop the directory where your
extension files live onto chrome://extensions in your browser to load
it.
If the extension is valid, it'll be loaded up and active right away!
If it's invalid, an error message will be displayed at the top of the
page. Correct the error, and try again.
This insures that non developers don't load an extension which does not comply with the normal security concerns.
Communicating between with the script on the web page to the extension.
Can be done with message passing ... https://developer.chrome.com/extensions/messaging
The extension can add content directly to the web page which is available to the script on the web page. If for example the extension replaced the web cam image with a static image when the webcam script reads what it believes is the webcam it gets the static image instead, which explains why I look like an alien from space on the webcam. Although I did not create an extension to do that, I merely modified an existing extension to replace the function that gets the webcam image with a function to get a static image.
You can use SignalR, it is javascript library (JQuery Plugin) and it enables you to open web sockets from the browser to a server. Please check the following links:
https://blog.3d-logic.com/2015/03/29/signalr-on-the-wire-an-informal-description-of-the-signalr-protocol/
http://blog.teamtreehouse.com/an-introduction-to-websockets
https://github.com/SignalR/SignalR
Is there a way to find out the destination of the information or track where it goes when form instance information is created from document.submit?
I have to reverse engineer this application that is taking form submissions but I am having difficulty trying to find out where it's going.
If you are using a modern web browser, like Chrome, you can use the network developer tools within to help you.
You can access the Chrome developer tools with the F12 key and also Ctrl+Shift+i and then go to the network tab. network tools example here. There you will be able to click on individual requests and see the request headers and response data, along with any other relevant data.
We are trying to figure out how something works on the web (for web scraping/automation) and one of the web pages we are working on issues a popup to do some of the work. One of our most commonly used debug tools is the Chrome network tab in Developer Tools, hit "record" do some work, and then examine what was done and then replicate the work done "offline".
However the Developer Tools (in Chrome, Safari and Firefox - all work the same) do not follow requests across a popup, even if you hit "record".
Is there some configuration value I'm missing, or some way to record all network events? We can't use tcpdump/wireshark for this because it's all done over SSL. One option we've considered is a man-in-the-middle https proxy, but I can't find anything pre-written so we'd have to create one ourselves.
I don't know of any way to follow the requests across pop-ups, as each window has its own Web Inspector, however you can use Fiddler to inspect HTTPS requests. It will MITM, and subsequently throw a certificate error, which should allow you to inspect all requests in the order that they happened.
You can use Charles Web Debugging Proxy, which is an app that lets you see all the traffic and even replace some responses with your own. Of course that may break HTTPS so you have to accept the certificate errors, but that's usually a minor problem. It works on Win, Mac and even Linux.
The object inspector cannot inspect what isn't in the current page. Therefore, you will need to open the inspector inside the popup url with same parameters in order to see what it does.
As a tool, you can use a web sniffer to see exactly which url were called during the process.