One approach that worked well for us when we tried to do this for an automated labyrinth project was to only operate on one color channel and template match an ideal ball while updating the template based on lighting conditions. Our project was slower moving though so the ball didn't deform quite as much. Our webcam was terrible so there was probably more visual distortion than I seem to recall.
Edit: also in our case we used a region of interest that moved with the ball position to cut down on the computation.
You might have good luck with the background removal techniques because your background is stationary. In our case the camera was stationary but the board constantly moved so that tactic was not very useful in our case.
Edit: also something to consider is using a kalman filter to help predict and smooth out noisy predictions. We did that and it helped considerably.
IIRC Obscura were selling that for $100k+. If the current business model doesn't pan out then I guess GoCardless could always pivot into that market instead :-)
Oh wow, that would be an amazing addition to the GoCardless project. I look forward to playing their open source code, but first I need a pool table :)
Huge pool fan here:
Put this in a mobile device that I can mount from any angle and record all my games anywhere - I would pay a lot more than $0.99 for it!
Very similar to my undergraduate research project based on tracking bubbles moving through the air. I also started with the Hough transform but found that things broke down quickly when the targets were moving at speed, then switched to segementation and morphological operations which worked really nicely when set up correctly.
Great application - looking forward to the future posts :)
They could project a light on the table based on expected ball movement, to show bounces off of the bumpers or other balls. That would be tremendously cool as a learning aid.
The black proved slightly harder to track than the yellow and red balls, as it didn't have a particularly distinctive hue. The first step towards tracking the black was filtering out everything except the green surface, which is demonstrated in the video. Unfortunately we ran out of time before getting the black tracked.
I'm wondering whether you could apply an edge-detect filter to your input frames and combine it with the hue-based detection. I'd assume that irrespective of the ball color, an edge detect + posterize would give you a video frame with clear spots where the balls are, which should be relatively easy to find. I remember from the image analysis courses I took in college that spot detection is a very common operation in automatic analysis of medical images, and hence extensively researched. After you know where each ball is you could convert the frame to YUV and use the UV (chroma) channels to detect the ball colors.
Edge detection algorithms are fairly slow and noisy from what I found. To do this in realtime you have to have fast algorithms or cut down on the information to process over.
Background subtraction works really well for these kinds of problems. You should give that a go, even better if you can make your background based on a rolling average of the image pixels.
As for the speed issue, your can often get massive gains by restricting the colour classification and blob detection to areas of the image which have changed since the previous frame.
This is cool! I'm sure that their is a huge business opportunity here too (I don't watch sports but sure that their must be entire channels dedicated to snooker/pool etc) you could make games a lot more exciting to watch with VR replays of interesting shots etc (check similar companies for virtual sportscasting for America's cup and off-road racing :)
It's already been done I'm afraid — certainly on snooker coverage in the UK, they frequently do VR demonstrations of possible alternative shots, VR views of different angles on the table, etc.
How about an app that tracks the ball positions and gives advice on the "leave?" Knowing how to optimize for your own fallibility and minimize positional advantages for your opponent if you miss and maximize them for when your aim is good is actually the most important part of the game.
So it learns your strengths/weaknesses then when you're playing uses this information to evaluate each possible shot based on the predicted final position of the ball and how useful this would be to the opponent and your chances of success...
One possible difficulty would be the learning phase - it would have to be able to know your intentions when you take a shot, which would be difficult to guess automatically. I believe it might be billiards where players have to say which ball they're going for - you might have to build that in...
This is very, VERY cool... but I can't help but think that an Arduino and some color sensors might be easier. Or some small RFID stickers and basic readers as long as the stickers didn't mess with the weight/spin of the balls, since color sensors might have problems with striped balls depending on spin.
Actually, figuring out how to do this without a webcam sounds like a pretty snazzy hurricane project.
There are very stable linux drivers out there for it and you can get up to 120fps. For a past project, my team was able to get up past 60FPS using this camera.
I've been meaning to do this for chess, which seems like it would be much easier than pool. The checkerboard pattern of the board is rather simple to distinguish, and you don't need to visually distinguish individual pieces, just what squares the pieces are on.
I don't mind manually entering the chosen promotion. (I've seen someone promote to a knight once, but only because he was being a jackass. Promoting to Queen would have been checkmate.)
It wouldn't be handy but it'd be fucking cool. If you know the velocity and trajectory of the ball then on an unobstructed shot you could instantly display the updated position of the ball after it has been struck. The difficulty would be gauging things like back spin which aren't (I'm guessing) observable by the camera, but which have a drastic impact on where the balls end up.
Yes, predictions can greatly improve recognition performance. Not only can you focus the bulk of your computations on the expected region of interest, but you can also potentially reason about appearance (e.g., if you know the frame-rate/integration time of the camera and expected speed of the ball, you could get estimates about how much blur to expect). You could also better handle difficult cases, such as when balls strike each other (e.g., simple conservation of momentum estimates should greatly constraint possible hypotheses for ball positions).
Edit: also in our case we used a region of interest that moved with the ball position to cut down on the computation.