Game Player - Step 1 - Transfer BufferedImage to C

When we ended the last post we had our screen captured and stored in a Java BufferedImage object. The problem we face now is that we want to do all our reasoning inside C using OpenCV, not in Java, so we need a way to gain access to the image data from C. This is where JNI and our native methods come into play.

So far we have only defined one native method that prints a line to the screen letting us know the library has been loaded successfully. In reality we don't need to print this message as an exception will be thrown if the library doesn't load but we'll leave it in for now just for the added reassurance. We will also define another native method but this time we will pass in our newly acquired image with it. There is just one small task we want to do before hand though...



The BufferedImage class provides a lot of extra functionality that we simply don't need when we are working on the data in C, OpenCV will have all the functions we need and more! So let's just pull out the minimum data we need and pass that in instead. This is really easy to do using one of those extra methods I mentioned, just add this line:

int[] pixelData = screen.getRGB(0, 0, screen.getWidth(), screen.getHeight(), null, 0, screen.getWidth());

after the ImageIO.write(screen, "PNG", new File("Screen.png")); line.

You can read the API for BufferedImage.getRGB() to see what each of these parameters means but suffice to say it returns a int array full of the image data and we've passed in parameters that give us the entire image rather than just a small part of it. We could, if we wanted, just extract the game board from the image and pass that into C which is probably faster than passing the entire image but I'm looking to do all the image manipulation in C as a learning exercise so we'll stick with the whole thing for now.

We now have our image data inside the int array pixelData, lets have a quick look at it before we pass it into C. For those that aren't aware a common way to store image data, and the way the Java BufferedImage class does it, is to have the amount of Red, Green and Blue in a single pixel stored as a value between 0 and 255, by mixing these values we can make any colour we want much like mixing paints. Java is also kind enough to store the Alpha, basically the solidity of a colour or the transparency, which is stored in the same manner.

As I am sure you are well aware the number 255 can be represented with 8 bits aka 1 byte which means our 4 values ARGB (A for Alpha rememeber;)) take up 32 bits or 4 bytes and wouldn't you know that 1 int in Java = 32bits or 4 bytes so all the data for each pixel can be represented with a single int! I swear, you think these Java guys had planned this! And that is exactly what we have now, an array of ints where each one represents in order, left to right, top to bottom, the data of each and every pixel in our image. Amazing eh?!

Now let's see what it contains. The following code will print out the size of our array and display the information about the first pixel in it:

System.out.println("Array size from Java: " + pixelData.length);
System.out.println("pixelData[0] A = " + ((pixelData[0] >> 24) & 0xFF));
System.out.println("pixelData[0] R = " + ((pixelData[0] >> 16) & 0xFF));
System.out.println("pixelData[0] G = " + ((pixelData[0] >> 8) & 0xFF));
System.out.println("pixelData[0] B = " + (pixelData[0] & 0xFF));

You might not have used the >> operator in Java before, it is the bit shift operator and it moves the bits of a variable in the direction give by the amount of spaces given (the opposite way being <<). So remember our image data is represented by a 4 byte int per pixel which is 32 bits, the order is ARGB so A is the left most 8 bits aka left most byte, R the 2nd left most byte or 9th to 16th bits and so forth. To display their individual values we need to get just the bits we want to see otherwise we'd get some strange number which is represented by the entire 4 bytes combined. We do this by shifting the bits of interest into the first* 8 bit positions and then zeroing off the rest. So for Alpha, the furthest to the left, we move right by 24 to settle everything in place (if we moved by 32 we'd have dropped them all off the end!) for Red we move 16 bits, Green 8 and Blue we don't move at all. We then do a bitwise AND with FF (hexidcemal for 255) which ensures we are only going to see the bits we have moved and then display the value.

*I've used the word first here which is something I am wary of when talking about bits since some people may regard the left most bit as the first bit but I definitely prefer classes the right most bit as the first bit. Whenever I talk about the first bit this is the one I'll mean, but I'll normally call it the right most bit if it isn't clear. 1001 1101 <----Right most bit is the first bit, even if it is the last one you just read.

So you run that and you should get a series of values which will vary depending on what is in the top left of your screen. We'll play about with checking different values on the screen later, for now lets just crack on with getting our data into C.

Same routine as the last post, first we'll write our declaration of the native method in Java. Put this below the other one:

static public native int[] calcMove(int[] pixelData);

You can see we are passing in an array of ints, that's our pixel data, and we are also return an array of ints, my current thinking is it return the X and Y location of where to click as a small array (X in int[0] and Y in int[1]). This may change as we go on but we'll stick with it for now.

Go back into your C project and add the following into your gameplayer_GamePlayer.h header file:

/*
 * Class:     gameplayer_GamePlayer
 * Method:    calcMove
 * Signature: ([I)[I
 */
JNIEXPORT jintArray JNICALL Java_gameplayer_GamePlayer_calcMove
  (JNIEnv *, jclass, jintArray);

You can of course generate this using the javah program as before but for one class I wouldn't bother unless you are getting errors and want to rule it out as the source. You can see in the comments in this code that the signature is ([I)[I the [ sign means an array and the I means int. In the return type and the method arguments we have the jintArray type which is used to show it uses int arrays from Java. I'll talk about these types in the future at some point or at least give you a link to a place you can find out about them but I want to get this part covered so let's continue. Now into the main C file add this:

/*
 * Class:     gameplayer_GamePlayer
 * Method:    calcMove
 * Signature: ([I)[I
 */
JNIEXPORT jintArray JNICALL Java_gameplayer_GamePlayer_calcMove
  (JNIEnv *env, jclass obj, jintArray data){
    using namespace std;
    jsize len = env->GetArrayLength(data);
    jint* elem = env->GetIntArrayElements(data, 0);
    printf("Array size from C: %d\n", len);
    cout << "pixelData[0] A = " << ((elem[0] >> 24) & 0xFF) << "\n";
    cout << "pixelData[0] R = " << ((elem[0] >> 16) & 0xFF) << "\n";
    cout << "pixelData[0] G = " << ((elem[0] >> 8) & 0xFF) << "\n";
    cout << "pixelData[0] B = " << (elem[0] & 0xFF) << "\n";
    fflush(stdout);
    env->ReleaseIntArrayElements(data, elem, 0);
}

Here we see the same new parts we saw in the header file and our main code. We can't use the int array straight away, we need to call a couple of JNI methods to get the data in the format C expects. First GetArrayLenfth(data) gets us the size of the array and then GetIntArray(data,0) gives us a pointer to the start of the int array's location in memory. We can then use this pointer just as we would a normal array in Java and that is just what we do. The next section of code does exactly the same thing as we done in Java but this time from C.

Don't be fooled by the << operator being used between the cout, the quote marks and the parentheses, it isn't doing any bit shifting, it is just sending our data to be printed to the screen. However the >> between the elem[0] and the numbers is, don't worry if you're not sure how to tell which is which, the compiler knows and with a bit of practice you'll soon get it. Finally we release our int array so that C can grab the memory back.

Compile that then head back to Java and add a call to the new native method to your main method, just below the last call to System.out.println() where all the info about the pixel is printed out like so:

calcMove(pixelData);

Don't worry about setting a variable from our methods return value (remember we said it returned an int array?) since we don't actually return anything from it!

Compile, run and you should get the following output:

Output from Java then C both on the same program

Your figures may vary but the important thing is that the information you get from Java matches what you are getting from C, if not something has went wrong.

That's it for today, we've got our image data in C so now we can start our manipulating, woohoo!

Here is the source to the Java program in case you need it or it wasn't clear above, the C is really just what was posted earlier:

package gameplayer;

import java.awt.Rectangle;
import java.awt.Robot;
import java.awt.Toolkit;
import java.awt.image.BufferedImage;
import java.io.File;
import javax.imageio.ImageIO;

/**
 * Application to play games, using OpenCV library for analysis via JNI
 * @author RyanfaeScotland http://workingwithcomputervision.blogspot.com
 */
public class GamePlayer {

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
        try {
            System.loadLibrary("GamePlayer"); // Loads our native library into memory
            printMsg(); // Calls the printMsg() method
            Robot rob = new Robot();
            Rectangle rect = new Rectangle(Toolkit.getDefaultToolkit().getScreenSize());
            BufferedImage screen = rob.createScreenCapture(rect);
            ImageIO.write(screen, "PNG", new File("Screen.png"));
            int[] pixelData = screen.getRGB(0, 0, screen.getWidth(), screen.getHeight(), null, 0, screen.getWidth());
            System.out.println("Array size from Java: " + pixelData.length);
            System.out.println("pixelData[0] A = " + ((pixelData[0] >> 24) & 0xFF));
            System.out.println("pixelData[0] R = " + ((pixelData[0] >> 16) & 0xFF));
            System.out.println("pixelData[0] G = " + ((pixelData[0] >> 8) & 0xFF));
            System.out.println("pixelData[0] B = " + (pixelData[0] & 0xFF));
            calcMove(pixelData);
        } catch (Exception e) {
            e.printStackTrace();
            System.exit(-1);
        }
    }
    static public native void printMsg(); // Prints a message from C
    static public native int[] calcMove(int[] pixelData);
}