so I want to create cross-platform app that will be invisible overlay over whole screen. When user interacts with whatever is under, I want to catch the event (mouse location, mouse click etc) so I can have info of what user attempted and where, and then "pass" the event through. Also, I might wanna display something in overlay as result of the event.
Coming from .NET, I understand how I would hook into mouse events. I also understand that it is possible to simulate user clicks with Mouse class in .NET Core. However I was wondering if this is possible with electron, as I was checking it for cross-platform solution.
I checked some topics (like this click-through discussion) that mostly deal with passing events through overlay, but I am not sure those do this catching-then-passing. So is there something like that and is it actually possible with only electron/nodeJS? Or would I have to use additional libs, and would they be different for Win/Linux/Mac?
I checked at:
Overlay Electron and
electron-overlay-window
but both seem complicated with attaching to game and following the window respectively. I was hoping it is possible and simple somehow.
IMHO, it's is possible, but will require a ton of OS-specific code and possibly some native modules.
I suggest to start with https://www.electronjs.org/docs/api/browser-window#wingetnativewindowhandle and https://www.electronjs.org/docs/api/browser-window#winsetignoremouseeventsignore-options
Related
I am writing a chrome extension that records your actions like ( mouse click, keyboard keyup ). The idea of the extension is to help me and my colleagues to reduce the boring testing of our web based project. I made it to record the events and store it on the dev server as mysql so i can use or share to them. But the problem is replaying the saved actions.
So how if there is a way to force mouse move, mouse click events. Can it be done from flash,java or something like that.
PS. The project is Extjs but i want to make the extension useful for developer using other frameworks and publish it.
Consider using Selenium for this. It has support for many languages, and you can script your whole test with it. You can for example set it to click on an element, wait for something to happen or fill text boxes.
Imagine some random website controlling your mouse ... not cool, is it? (That's why you cant force mousemove via javascript)
However, you can trigger clicks on elements. To achieve that, you need to save the event(mouse-over|out/(dbl)click/whatever) and the according element (in the eventfunction: this). That should be sufficient to simulate theworkflow.
jQuery-Example:
$('#item').click();
$('#item').trigger('click');
vanilla javascript:
document.querySelector("#item").click();
Beatports new interface has solved a major problem I was looking for the solution too.
Namely, it keeps a "player" interface at the moment and you can browser to different parts of the site (also changing the url) without reloading or interrupting the player.
I cannot for the life of me understand how they have done this, can any of you guys figure it out?!
Many thanks for any replies I get
Looks like they are just using AJAX to load new content but have taken care to make it work and look pretty seamless. You can get better insight into what events are attached to what elements via the Visual Events bookmarklet. Once you find the code that triggers the event, you can run the obfuscated javascript through JSBeautifier to examine it more closely.
Specifically, it looks like they're adding click handlers to all anchor tags, passing off the event if it was triggered with a middle click or modified with a keyboard key, otherwise passing it to a dynamic loader which handles state and other specific conditions like multiple clicks. The seamlessness of it comes from the way they deal with URLs making every page bookmarkable and the browser history so the back and forward buttons work as you would expect on a "normal" site.
How I do monitor all the clicks and inputs on my page? I would ideally want to make a video of what the user does on my app; the way they uses my app.
How do I go about this?
Any links or demos would be really great.
Edit : What I have in mind is:
log all the events into a cookie , and send it to the server in frequent intervals.
then startup a Firefox session and trigger these events on the page . and capture it as a video.
Edit: Basically I have an app, and I want to see what the user does on the app, so I will monitor their clicks and inputs (that's pretty much what they do on the click), and then trigger those events back on my web app. So I will pretty be able to replicate their actions on my page. Now the question is, how do I capture all the necessary events, how do I serialize and trigger these events for later?
Not sure that you can make a 'video' unless you tracked every single movement of the mouse, which would extremely unfriendly, probably take a lot of processing time and raise ethical questions.
If you want to know how a user interacts with your site so that you can improve it (presumably) then you need to get into web analytics.
There are many options. My personal favourite is Google Analytics
With JavaScript you can create what's known as a heatmap of use clicks:
http://tympanus.net/codrops/2010/02/08/a-jquery-heat-map/
You can't capture a video of the users desktop session using JavaScript.
I remember seeing ClickHeat (or a similar variant) a long time ago. Had no experience with it, so I can't help with pros and cons, and it doesn't answer your question entirely (don't think it does keystrokes, although maybe I'm wrong, like I said I don't really know).
Might not be a total solution, but should be a good starting point.
In a similar vein it wouldn't be hard to capture mouse position every so often along with a timestamp and recreate a given user's actions. Generating a general/average movement is much more difficult with this type of data, but it's useful on the single-unit basis. So too with keystrokes, they're all just events.
I am writing an application in JavaScript (with JQuery). The application has a lot of functionality that the user needs to access quickly and possibly at the same time (you could think of it as a game, even though it's not a game) so I've set it up to use the keyboard. I've got a prototype of the system working (in Chrome, at least), but there is a problem in that when I press, for example, Ctrl-T, it opens a new tab. I would like to use this as part of my application to, for example, toggle some setting. In general, I would like to disable the Ctrl-key functionality of the browser. This goes for Alt and Shift, too. In a different way, I would also like to disable the browser's usage of the Tab key. Basically, I want my application to have complete control over the keyboard while it's being used (if they want to navigate away, they will still be able to do so with the mouse). I will add the caveat that I don't care about some of the less-common keys (F1-F12, Windows/Mac-Command, Menu, anything right of the Enter key) because my application mainly targets laptops and many don't have those keys, so they won't be used.
My question is, how can I disable this browser functionality with JavaScript/JQuery/DOM?
I understand that taking over browser functionality in this way this sounds very authoritarian, but it's actually quite necessary. From the prototype and some target-audience test users (my family) everyone has agreed that this was a good idea.
I've also considered moving my application out of the browser and writing it in some application language (probably Java), but I want to find out if this kind of functionality is possible in JavaScript before making that kind of switch. This is prototyping time, though, so every idea is still up for consideration!
Basically, you return false at the right time. See this: How to create Ctrl+Key shortcuts in Javascript.
I should add and say sometimes this doesn't work - it's up to the browser to allow this.
Also, it is sometimes possible to change e.which to another key, but again, this wouldn't always work.
I'm having trouble understanding conceptually what I should do while trying to make my first large Javascript web application.
Depending on which tab a user has selected, I show different content inside a container. The content is more than just text and uses different Javascript functions and events. I am using the Yahoo! UI Library's "TabView" implementation, but the way that this issue should be handled would probably apply to other Tab approaches.
What I was thinking of doing was basically the following:
Create separate modules for each tab (e.g. MYAPP.modules.tabCalendar and MYAPP.modules.tabJournal). When the user clicks on a different tab (or navigates with browser buttons to a previous tab state), I could call MYAPP.modules[oldModule].disable() and MYAPP.modules[newModules].enable(). These functions would subscribe or unsubscribe their custom events (for example, a general click handler attached to the container).
An alternate approach to dealing with events might be to have a single global click handler. If the click is inside the container, then determine which tab is currently selected and send the click event to MYAPP.modules[currentTab].onClick().
Or, the global click handler could fire a Custom Event to which the modules have subscribed since page load, and each module's onClick events will run and determine whether or not they should do anything.
There seem to be a lot of options, but I've been having trouble finding resources that talk about the best ways to do things like this. Am I on the right path? How should I handle this?
Use the events already built into TabView to queue your JS to do things.
http://developer.yahoo.com/yui/tabview/#handlingevents
For tab changes you'll be told the previous/next tabs selected and such which should be more than enough for your JS to figure out what it should do. If you want to write a translation layer that'll look at the events and do something based on it that's fine but it's not strictly necessary.
I'm a bit fuzzy on the problem.
Yes, you should modularize your code.
Have each module setup event handlers on the elements in their respective container.
That's it. YUI TabView handles the tab switching so you don't need to enable/disable anything.