Making Humanistic Mouse Clicks with Java Robot Class

Preface: Java's Robot class allows you to send mouse and keyboard signals to the operating system through APIs designed for indirect interaction with the PC. By using this Java class, we can perform actions like scrolling, clicking, and typing without actually doing so.

Problem: While experimenting with the Robot class for an Arduino gamepad I was making, I discovered a problem with clicking the virtual mouse. Let's look at the code.

This is how it is explained in Java examples, and how other users have taken to using the mouse click:

Robot robot = new Robot();
    
    robot.mousePress(InputEvent.BUTTON1_MASK);
    robot.mouseRelease(InputEvent.BUTTON1_MASK);

The button is pressed and immediately released. Even adding a delay by way of a Thread.Sleep will still result in the same problem. What if I have a function that detects an event and presses the mouse button? This certainly works for that case, but what if I am holding the mouse button down? Instead, this code will simulate a set of rapid clicks for as long as the button is held. That's not the behavior I wanted. Furthermore, the right click code was verbatim with the mask changed to account for the different button. Right clicking did not work at all since it opened and close any context menus far too fast to be useable.

Solution: Using some extra state information, we can track whether or not the mouse button should be held down, or if it should be released. Assuming the event triggers occur often enough, the user should not notice whether the mouse is only clicked or held for any period of time.

int currentLeftClickState = 0;
...
if(trigger == 0 && currentLeftClickState == 1){
robot.mouseRelease(InputEvent.BUTTON1_DOWN_MASK);
this.currentLeftClickState = 0;
}
else if(trigger == 1 && currentLeftClickState == 0){
robot.mousePress(InputEvent.BUTTON1_DOWN_MASK);
this.currentLeftClickState = 1;
}

In my example, I declare a simple flag using an int that will represent the current state.  When I receive the literal 'trigger' my function code will validate the trigger against the current state. Only if there is a difference between them, do we need to change the mouse behavior.

This code allows you to now select windows, drag objects, open context menus correctly, and even select multiple lines. Cool stuff right?

To see the project this code was written for, please check out my GITHUB.

Long Term Test Result Analysis with TUSK

This is Tusk. Tusk was made to remember your test result and do analysis for you.

I came up with Tusk to solve a problem frequently associated with testing. Automation allows us to generate testing results more quickly, but more often we are looking at the one off results and discarding the historical data. What about that test that seems to fail every other week? It may have been mentioned between coworkers, but no one really knows its failure pattern, how it coordinates to product versions, etc. Tusk was designed to parse test results, store them in a database, and perform analysis on the database while improving debugging and testing workflow.

(Tusk Rest API)

I've since been drawn away, interested to work on other projects, but I did make enough progress that it is very close to a minimum viable product and can be functionally tested. Further, Tusk is a really good example of JSON serialization/deserialization using Go/Golang. Check it out at my GitHub.

Keeping Track of Skill Practice Hours (A JSON, JAVA Project)

I've been wanting to write program that allowed me to keep track of my practice hours for various computer related skills. Some years ago, I read a book by Malcom Gladwell that examined experts of various fields from athletics to music. Each of these experts had practiced their craft for at least an estimated10,000 hours. Eventually, I will surely reach this number with crafts like programming, but it would be hard to say when if I wasn't actually keeping track. So I wrote an interactive program to do just that.

I called it 'Keeper', and it's freely available to download on my GitHub if you want to check it out. 

How it Works: Keeper uses what I called 'trackers' to record events for specified skills. In the source, you'll notice that I used mostly computer languages as the skills you could track. This is a limitation of the example code, but in practice, you would not be limited to the number of trackable skills currently available. Once you've entered data, you could perform analysis and get the following data:

1. What year did I practice X the most/least
2. What time of day did I practice X the most/least
3. Which skill did I practice the most/least
4. etc

I really enjoyed this project, but when I showed it to a friend, they asked why I wouldn't just use Excel to do the same thing. I didn't have a good answer, so I have since switched over to using Excel. The project still serves as a couple good examples of using JSON to serialize, deserialize data.

Playing with Computers at Work (I/O vs Compute)

I used computer science at work and it was fun.

A coworker and I were talking just the other day about a situation where our code would need to enumerate every open file for a given process. We had a binary tree structure in memory that maintained the list for our process as well as the option of retrieving the information via '/proc'. Since we needed to touch every element in our tree in memory, that's easily identifiable as O(n), but what was unclear for our now group of four deliberators was whether the enumerating of the /proc/[pid]/fd file would be just as fast, faster, or slower. Ultimately, a couple of them settled on the fact that it would be the same speed since you still needed to enumerate every file descriptor, and it would be just as fast to make the system call to retrieve the proc info. I disagreed. My argument was that compute is always going to be faster than I/O, and even though it's a system call to a pseudo-filesystem, it makes sense to crawl the tree instead of getting it from proc. To prove my suspicion, I created the following experiment:

1. Run a program that opens and holds 2**x file descriptors.
2. Run a script to build a tree of 2**x levels in memory.
3. Using the same script, walk the tree using a depth first search. (Time this function)
4. Using the same script, attempt to read the list of FDs from /proc/[pid]/fd. (Time this function)
5. Compare the result of the function timers.

Results: At 10 levels, the tree enumeration took 0 milliseconds, while the I/O required about 55 milliseconds to complete. When I increased the size of the tree to 16 levels, the time required to walk the structure increased to around 25 milliseconds. Compute was faster.

Notes:
1. This is not bleeding edge accurate. The python package 'psutil' is essentially a piece of middleware that could influence the time it takes to return the list of file descriptors, but I think still accurately simulates what enumerating the proc file would require.
2. The accuracy of the time.time() value used for the experiment is two decimal places on my test machine.

You can find the experiment files HERE.

Prototyping with Leap Motion

If you have a Mac and plan on using Python with the Leap, let me save you some trouble and just tell you:

Install Homebrew straight away.

First, some back story. I wanted to start using Python first to prototype with the Leap controller and get a feel for how the frames and gestures are handled. Just setting up the environment turned out to be a headache due to how OS X handles its own Python installation. Several path and dependency problems ensued, but to save you the frustration yourself, install Homebrew if you haven't already and let it manage your packages instead of OS X.

After you have downloaded Homebrew and Python using the simple 'brew' cli commands, you're going to need to use 'install_name_tool' to modify the LeapPython.so library dependency locations. Follow this tutorial.

Once you're finished with those steps, import it into your Python project or Python path and you should be ready to start programming with the Leap Motion!

I spent far too much time trying to get these dependencies resolved on my mac tonight so I didn't get as far as I would have liked with the programming, but I did upload a very simple example that builds on the 'Hello World' tutorial posted by Leap. Will probably get more into it tomorrow when I have a little more time. See you then!

Reading Magnetic Strips with Square Reader

To start off, I want to admit that I haven't gotten this to work just yet. However, before I get into what I found, I wanted to explain my goals and what I have discovered so far.

My hope was to use a Square card reader in conjunction with my macbook and some simple software to read magnetic strips on the various cards in my wallet. No nefarious motives, just for fun and my own edification. I was able to write a Java program that would interface with the macbook's default audio-in data line, but the macbook has some mechanism that handles which device you get to read from at that interface address. Basically, if it doesn't recognize the device (Square card reader in my case) it thinks you want audio from the internal microphone and so the mixer sends you data from the microphone. Now, if you plug in a headset, you get data from the headset, but at the same device address. It was at this point I realized I would have to do a little more digging on how to beat the mac at its trickery.

Turns out the macbook wasn't at fault at all. After some careful reading, I discovered that the Java APIs for getting audio are some what limited. This may not be the case for all environments, it is just what I have found(through StackOverflow and the like) to be the case for macs and the Java audio APIs.

Don't lose hope just yet! Someone has actually already accomplished this task with a nice graphical application named MagRead. Check it out here: MagRead

If I end up continuing this project, it looks like I'll need to go with a programming language less restrictive than Java. For now, my source for reading from the default audio device is up on my github at: Audio Reader Example

*Disclaimer: I do not own this image. The image was taken from www.cnet.com. If you own this image and do not want me using it, let me know and I can replace it.