3
I was wondering if you could tell the difference between java.awt.Robot mouse click...
I need to know if it was the Robot that clicked or if it was the mouse.
1 answer
4
I’m not entirely sure, but the class java.awt.Robot must enter click events in the operating system queue. So much so that you can use it to click on other programs, drag-and-drop on the desktop or anything like that. With this, the only way to differentiate this would be if you were to dig into the bowels of the operating system, and therefore do not expect the code to be portable.
But can’t you still try using just java code? Well, I tried and I came to the conclusion that it doesn’t, at least not with 100% certainty. Here’s the code of what I tried:
import java.awt.AWTException;
import java.awt.EventQueue;
import java.awt.MouseInfo;
import java.awt.Robot;
import java.awt.event.ActionEvent;
import java.awt.event.InputEvent;
import java.awt.event.MouseEvent;
import java.awt.event.MouseListener;
import java.awt.event.MouseMotionListener;
import java.awt.event.MouseWheelEvent;
import java.awt.event.MouseWheelListener;
import java.lang.reflect.InvocationTargetException;
import java.util.concurrent.atomic.AtomicBoolean;
import javax.swing.JButton;
import javax.swing.JFrame;
/**
* @author Victor
*/
public class DetectRobot {
public static void main(String[] args) {
EventQueue.invokeLater(DetectRobot::go);
}
private static void go() {
try {
Mouser m = new Mouser();
Robot r = new Robot();
JFrame jf = new JFrame("Teste");
JButton jb = new JButton("Clique aqui");
jb.addActionListener(DetectRobot::clicked);
jb.addMouseListener(m);
jb.addMouseWheelListener(m);
jb.addMouseMotionListener(m);
jf.add(jb);
jf.setBounds(10, 10, 250, 250);
jf.setDefaultCloseOperation(JFrame.DISPOSE_ON_CLOSE);
jf.setVisible(true);
Thread t = new Thread(() -> clicker(jf, r));
t.start();
} catch (AWTException e) {
throw new RuntimeException(e);
}
}
private static void clicker(JFrame jf, Robot r) {
Thread t = Thread.currentThread();
try {
while (!Thread.interrupted()) {
Thread.sleep(5000);
System.out.println("Robo clicando");
EventQueue.invokeAndWait(() -> {
if (jf.isDisplayable()) {
r.mouseMove(150, 150);
r.mousePress(InputEvent.BUTTON1_DOWN_MASK);
} else {
t.interrupt();
}
});
Thread.sleep(50);
EventQueue.invokeAndWait(() -> {
r.mouseRelease(InputEvent.BUTTON1_DOWN_MASK);
if (!jf.isDisplayable()) t.interrupt();
});
Thread.sleep(50);
System.out.println("Robo terminou");
}
} catch (InterruptedException e) {
// Ignora e deixa a Thread morrer naturalmente.
} catch (InvocationTargetException ex) {
ex.printStackTrace();
}
}
private static void clicked(ActionEvent e) {
log(e);
}
// Para simplificar a saída do log. Não precisamos mostrar no System.out qual foi o JButton clicado, apenas os eventos.
private static void log(Object x) {
String a = x.toString();
int i = a.indexOf(" on javax.swing.JButton[");
System.out.println(a.substring(0, i));
}
private static class Mouser implements MouseListener, MouseMotionListener, MouseWheelListener {
@Override
public void mouseClicked(MouseEvent e) {
log(e);
}
@Override
public void mousePressed(MouseEvent e) {
log(e);
}
@Override
public void mouseReleased(MouseEvent e) {
log(e);
}
@Override
public void mouseEntered(MouseEvent e) {
log(e);
}
@Override
public void mouseExited(MouseEvent e) {
log(e);
}
@Override
public void mouseDragged(MouseEvent e) {
log(e);
}
@Override
public void mouseMoved(MouseEvent e) {
log(e);
}
@Override
public void mouseWheelMoved(MouseWheelEvent e) {
log(e);
}
}
}
That is the result:
Robo clicando
java.awt.event.MouseEvent[MOUSE_ENTERED,(132,109),absolute(150,150),button=0,clickCount=0]
java.awt.event.MouseEvent[MOUSE_MOVED,(132,109),absolute(150,150),clickCount=0]
java.awt.event.MouseEvent[MOUSE_PRESSED,(132,109),absolute(150,150),button=1,modifiers=Button1,extModifiers=Button1,clickCount=1]
java.awt.event.ActionEvent[ACTION_PERFORMED,cmd=Clique aqui,when=1424373236982,modifiers=Button1]
java.awt.event.MouseEvent[MOUSE_RELEASED,(132,109),absolute(150,150),button=1,modifiers=Button1,clickCount=1]
java.awt.event.MouseEvent[MOUSE_CLICKED,(132,109),absolute(150,150),button=1,modifiers=Button1,clickCount=1]
Robo terminou
Robo clicando
java.awt.event.MouseEvent[MOUSE_PRESSED,(132,109),absolute(150,150),button=1,modifiers=Button1,extModifiers=Button1,clickCount=1]
java.awt.event.ActionEvent[ACTION_PERFORMED,cmd=Clique aqui,when=1424373242084,modifiers=Button1]
java.awt.event.MouseEvent[MOUSE_RELEASED,(132,109),absolute(150,150),button=1,modifiers=Button1,clickCount=1]
java.awt.event.MouseEvent[MOUSE_CLICKED,(132,109),absolute(150,150),button=1,modifiers=Button1,clickCount=1]
Robo terminou
Robo clicando
java.awt.event.MouseEvent[MOUSE_PRESSED,(132,109),absolute(150,150),button=1,modifiers=Button1,extModifiers=Button1,clickCount=1]
java.awt.event.ActionEvent[ACTION_PERFORMED,cmd=Clique aqui,when=1424373247185,modifiers=Button1]
java.awt.event.MouseEvent[MOUSE_RELEASED,(132,109),absolute(150,150),button=1,modifiers=Button1,clickCount=1]
java.awt.event.MouseEvent[MOUSE_CLICKED,(132,109),absolute(150,150),button=1,modifiers=Button1,clickCount=1]
Robo terminou
java.awt.event.MouseEvent[MOUSE_MOVED,(132,108),absolute(150,149),clickCount=0]
... Mais um monte de eventos MOUSE_MOVED disparados praticamente ao mesmo tempo, a medida que eu ia movendo o mouse ...
java.awt.event.MouseEvent[MOUSE_MOVED,(192,21),absolute(210,62),clickCount=0]
java.awt.event.MouseEvent[MOUSE_PRESSED,(192,21),absolute(210,62),button=1,modifiers=Button1,extModifiers=Button1,clickCount=1]
java.awt.event.ActionEvent[ACTION_PERFORMED,cmd=Clique aqui,when=1424373250075,modifiers=Button1]
java.awt.event.MouseEvent[MOUSE_RELEASED,(192,21),absolute(210,62),button=1,modifiers=Button1,clickCount=1]
java.awt.event.MouseEvent[MOUSE_CLICKED,(192,21),absolute(210,62),button=1,modifiers=Button1,clickCount=1]
java.awt.event.MouseEvent[MOUSE_MOVED,(191,25),absolute(209,66),clickCount=0]
... Mais um monte de eventos MOUSE_MOVED disparados praticamente ao mesmo tempo, a medida que eu ia movendo o mouse ...
java.awt.event.MouseEvent[MOUSE_MOVED,(159,67),absolute(177,108),clickCount=0]
java.awt.event.MouseEvent[MOUSE_PRESSED,(159,67),absolute(177,108),button=1,modifiers=Button1,extModifiers=Button1,clickCount=1]
java.awt.event.ActionEvent[ACTION_PERFORMED,cmd=Clique aqui,when=1424373250629,modifiers=Button1]
java.awt.event.MouseEvent[MOUSE_RELEASED,(159,67),absolute(177,108),button=1,modifiers=Button1,clickCount=1]
java.awt.event.MouseEvent[MOUSE_CLICKED,(159,67),absolute(177,108),button=1,modifiers=Button1,clickCount=1]
java.awt.event.MouseEvent[MOUSE_PRESSED,(159,67),absolute(177,108),button=1,modifiers=Button1,extModifiers=Button1,clickCount=2]
java.awt.event.ActionEvent[ACTION_PERFORMED,cmd=Clique aqui,when=1424373251059,modifiers=Button1]
java.awt.event.MouseEvent[MOUSE_RELEASED,(159,67),absolute(177,108),button=1,modifiers=Button1,clickCount=2]
java.awt.event.MouseEvent[MOUSE_CLICKED,(159,67),absolute(177,108),button=1,modifiers=Button1,clickCount=2]
java.awt.event.MouseEvent[MOUSE_PRESSED,(159,67),absolute(177,108),button=1,modifiers=Button1,extModifiers=Button1,clickCount=3]
java.awt.event.ActionEvent[ACTION_PERFORMED,cmd=Clique aqui,when=1424373251309,modifiers=Button1]
java.awt.event.MouseEvent[MOUSE_RELEASED,(159,67),absolute(177,108),button=1,modifiers=Button1,clickCount=3]
java.awt.event.MouseEvent[MOUSE_CLICKED,(159,67),absolute(177,108),button=1,modifiers=Button1,clickCount=3]
java.awt.event.MouseEvent[MOUSE_MOVED,(160,66),absolute(178,107),clickCount=0]
... Mais um monte de eventos MOUSE_MOVED disparados praticamente ao mesmo tempo, a medida que eu ia movendo o mouse ...
java.awt.event.MouseEvent[MOUSE_MOVED,(228,0),absolute(246,41),clickCount=0]
java.awt.event.MouseEvent[MOUSE_EXITED,(230,-1),absolute(248,40),button=0,clickCount=0]
Robo clicando
java.awt.event.MouseEvent[MOUSE_ENTERED,(132,109),absolute(150,150),button=0,modifiers=Button1,extModifiers=Button1,clickCount=0]
java.awt.event.MouseEvent[MOUSE_MOVED,(132,109),absolute(150,150),clickCount=0]
java.awt.event.MouseEvent[MOUSE_RELEASED,(132,109),absolute(150,150),button=1,modifiers=Button1,clickCount=3]
Robo terminou
java.awt.event.MouseEvent[MOUSE_MOVED,(131,109),absolute(149,150),clickCount=0]
... Mais um monte de eventos MOUSE_MOVED disparados praticamente ao mesmo tempo, a medida que eu ia movendo o mouse ...
java.awt.event.MouseEvent[MOUSE_MOVED,(198,1),absolute(216,42),clickCount=0]
java.awt.event.MouseEvent[MOUSE_EXITED,(199,-1),absolute(217,40),button=0,clickCount=0]
Note that there is no simple way to distinguish events from Robot of user events. The best I think you can do is to check if the mouse is "jumping" and not "walking". To do this, you can track the mouse position using java.awt.MouseInfo.getPointerInfo().getLocation(), which provides the mouse position on the graphic device where it is and java.awt.MouseInfo.getPointerInfo().getDevice() to tell which is the graphic device it is on (important if there is more than one monitor). With this, it is not difficult to detect if the mouse is "jumping" and not "walking" and if it clicks right after jumping, it was probably a Robot. But obviously this does not directly solve the problem as it can be bypassed by a Robot that makes the mouse walk and not jump and does not solve the case of the mouse being clicked without being moved.
It is also possible to search for other typical behavior patterns of robots, as always click on the exact same position, always move the mouse with the same pattern, always have the same interval or duration of clicks or anything like that. But just by using java, without going into the bowels of the operating system, I don’t think that what you want is possible with 100% warranty.
The main problem here is that the intention of the Robot is exactly to impersonate the user using the computer. That is, one of its purposes is exactly to be indistinguishable from that of a user operating the system, but what you need is to distinguish them exactly!
However, if you have access to the Robot (or to be more precise, the class that controls the Robot), then the problem becomes easy. Note those System.out.println("Robo clicando"); and System.out.println("Robo terminou");, if its controller of the Robot let your application know somehow that it will click, then your application will know that the click that next came from Robot and not of the user.
-
At kernel level, I know that you can know the pid tid of the process that is using an input or output operation at a given time, so it would not be possible to search who was controlling this device at that time ?
-
@Isvaldofernandes I don’t know, but if you can elaborate an answer on top of that, go ahead, I’d love to know that. Maybe you can "smell a robot" if you detect that some unexpected process has messed with the mouse at the operating system level. :)
-
Thank you for commenting, must have given work to make this comment... but I want when the robot click do a different action when it is mouse click
-
@I added two paragraphs at the end of my reply.
-
I already managed to fix, I just changed the preferences, I think I got it, the robot class edited ta working, now I have to make the system to detect...
Browser other questions tagged java
You are not signed in. Login or sign up in order to post.
what level are you talking about, a bot that controls the mouse, or a program that sends requests automatically ? , can describe where you want to use this and for what purpose ?
– Isvaldo Fernandes
Perhaps it would have to start from some principles that only a user would do, a robot just takes the DOM element by code and clicks on it, already a user hovers over it (Hover, move) and gives the click, a validation of combined events.
– Leonardo Bosquett
What is the reason for this verification? Perhaps there is a better alternative.
– André Ribeiro
Your question originally had "Java" in the title and "javascript" in the tag. As you are talking about Robot, which has no sense in javascript, I edited your question and arranged the tag to avoid confusing other users.
– Victor Stafusa
I want when the robot click do a different action than when it is mouse click
– pauloabr