Xuggler RTSP Source Code

I get asked for this quite a lot these days and for a spell I thought I'd deleted it but I found it again so before I lose it again here it is:

Game Player - Step 1.5 - Tidying Up The Code

We have done a lot of work over the last few posts and looked at some interesting features of OpenCV.

As it stands there is a lot of this stuff, although handy for debugging, we don't need in our final system. Things like the calls to printf and cout slow the system down unnecessarily so we can remove them. We can also take this opportunity to separate the majority of the code into its own method that completes the task of getting the game state. This will make the flow of the call to C easier to see and allow us to easily replace ways of completing a task (such as finding the colour of a game piece in an image) easier which is handy if we want to try out new algorithms.

Game Player - Step 1 - Representing the Game Board

I'd say we are just about finished Step 1 now which looking back at the first post for this project I said should "Grab an image of the game board." Well we've definitely done that and we've also separated the game board out from the rest of the image to make the rest of the processing easier and faster but there is still a few things to take care of before I can officially declare us on Step 2.

Although we can see the game board the computer still can't really see it. It's kind of like in the last post when we had worked out the purple areas but still had to do a little processing for the computer to be able to make sense of it. What we need to do is represent the game board in a way the computer can understand it and that is what this post, and the end of Step 1, is going to be all about.

Our captured game board

Game Player - Step 1 - Finding the Game Board

I seem to be spending a great amount of time writing blog posts and not a huge amount of time actually progressing with the program. I think this is because a lot of my time so far has been spent capturing screen shots for the setting up portion of the system so now that that is over with things should proceed a bit more briskly.

So we've got our image data in C and we also have the ability to generate a window, let's combine the two so we can see our image from C and that way we can be certain that we do in fact have our image data and not just a jumble of non-sense.

Problems Setting up 64 bit OpenCV

If you've been following the Game Player project I've been blogging about you'll know I has some trouble getting the 64 bit libraries of OpenCV to link with my project and had to stick with the 32 bit ones. Here I'll go into more details about the problem and hopefully find a solution.

First a little bit of background information on my system:

• Windows 7 Home Premium 64 bit
• OpenCV 2.4.2 pre-compiled binaries
• Netbeans 7.0.1
• Java 1.6 (not sure of bitness)
• MinGW-32 C Compiler with gcc version 4.5.2

Game Player - Step 1 - Setting Up and Using OpenCV

This is the part I hate the most because it can be fraught with errors and you can end up spending masses of time on what is essentially just setting up the environment. I don't mind spending ages on a problem when it is a problem in development or trying to work out how to solve a tricky algorithm issue but spending ages on just setting something up drives me mad. Hopefully it will go smoothly this time...

Game Player - Step 1 - Transfer BufferedImage to C

When we ended the last post we had our screen captured and stored in a Java BufferedImage object. The problem we face now is that we want to do all our reasoning inside C using OpenCV, not in Java, so we need a way to gain access to the image data from C. This is where JNI and our native methods come into play.

So far we have only defined one native method that prints a line to the screen letting us know the library has been loaded successfully. In reality we don't need to print this message as an exception will be thrown if the library doesn't load but we'll leave it in for now just for the added reassurance. We will also define another native method but this time we will pass in our newly acquired image with it. There is just one small task we want to do before hand though...

Game Player - Step 1 - Capturing the Board

Right, Step 1 in our Game Player program is capturing the game board.

This is actually an incredibly easy step so before we do that let's do a little setting up of our system's architecture. Yuk, I hate phrases like that, over complicated computing science terms are a pain in the butt and will not feature in this blog where avoidable. I'd much rather say let's get all the bits of our system in place...

As mentioned we are going to use OpenCV for all our reasoning purposes so we need a way of accessing OpenCV from Java. In order to do this we make a native library that we can call from within our Java code using the Java Native Interface (JNI). What is a native library you ask? It is code written in another language, often C, that can be called from other programs. All those .dll files in your Windows system or all those .so files in your Linux distribution are examples of native libraries and are what we are about to make. Don't worry, it's not as scary as it sounds!

The Game Player Project

I've decided my C and C++ skills really need to be improved since more and more of my work appears to depend on them. So in order to improve them I've decided to take on a project in my spare time that covers both programming in C and C++, OpenCV, Java and JNI. These are all pretty useful tools to use together so hopefully these entries will be of help to others as well.

The project is going to be called 'Game Player' and the aim is to develop an automated system, based on computer vision techniques, that can be used to play games. I make a point of emphasising the computer vision techniques part because there are already lots of automated game players out there that use data held in the computer's memory to play games and that's not what I'm looking to do. I want to build a system that can play a game based purely on what it can 'see' in front of it, the same a regular player does.

Fixes in OpenCV and RTSP

We've finally got it going!

I'll get onto what was wrong in a second but first I want to share this line with you:

    CvCapture *camera = cvCreateFileCapture("rtsp://admin:admin@192.168.1.128/ch1-s1?tcp"); 

As you can see it is the same as my camera connection url in the previous post except I've added ?tcp to the end. I've found that this forces the rtsp connection to run using the tcp protocol instead of the udp protocol which is useful if you can't afford to have any packet loss. This gives us a decent picture but still performed very poorly, freezing often and dropping the frame rate down to something terrible.

Issues with OpenCV and RTSP

For months now I've been fighting with OpenCV to try and get a decent image coming through my internet camera and it's really beginning to drive me mad. I've got a reasonable little program to test the connection which looks like this:

/* 
 * File:   main.cpp
 * Author: Ryan
 *
 * Created on October 27, 2011, 2:23 PM
 */
#include <cstdlib>
#include "opencv2/core/core.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include "opencv2/highgui/highgui.hpp"
#include "opencv2/contrib/contrib.hpp"
#include "opencv2/features2d/features2d.hpp"
#include "opencv2/legacy/legacy.hpp"
#include "opencv2/video/tracking.hpp"
#include <stdio.h> 

using namespace std;
using namespace cv;

int main() {
// CvCapture *camera = cvCreateFileCapture("/home/ryan/NetBeansProjects/OpenRTSP/video-H264-1");
// CvCapture *camera = cvCreateFileCapture("stream_fifo");    
// CvCapture *camera = cvCreateFileCapture("Wildlife.wmv");
   CvCapture *camera = cvCreateFileCapture("rtsp://admin:admin@192.168.1.128/ch1-s1"); //purple cam /?tcp
// CvCapture *camera=cvCreateFileCapture("http://192.168.1.5/image.jpg"); //black cam
// CvCapture *camera = cvCreateCameraCapture(0); //built in cam
// cvWaitKey(10000); 
    if (camera == NULL) {
        printf("camera is null, aborting...");
        return -1;
    }
    printf("camera is not null\n");
    fflush(stdout);
    cvNamedWindow("img");
    int i = 0;
    while (cvWaitKey(100) != 27) {
 // CvCapture *camera=cvCreateFileCapture("http://192.168.1.5/image.jpg");
        IplImage *img = cvQueryFrame(camera);
            if (img == NULL) break;
        cvShowImage("img", img);
// cvReleaseCapture(&camera); 
// printf("Image: %i\n", ++i);
        }
    cvReleaseCapture(&camera);
    return 0;
}

This is where it began...

Years from now, when I'm a computer vision master, I'm going to look back at this post and think "Wow, this post was pretty pointless."