How to Build a Computer Vision Based Color Detection Feature With OpenCV
OpenCV is an open-source library focused on machine learning and computer vision. As these fields have been progressing incredibly fast over the last few years, it’s good to have such a library at hand. It’s under a BSD license which means businesses and independent developers can utilize and modify its code.
Since we already have a certain background with OpenCV, we’ve decided to play with it a little bit to show you what it’s capable of in terms of mobile development:
Table of contents
In this tutorial, we’re going to consider how to implement functionality allowing to change the color of objects using a smartphone’s camera.
What you’re getting at the end
Getting Started With OpenCV SDK
The first step is to set up OpenCV to your Xcode project. There are tons of tutorials like this one on how to do this.
Before we start, there are several things you should know:
- It’s an image processing library.
- Mat — The Basic Image Container.
- Getting the image from the real world a device transforms it into digital values. OpenCV records numerical values for each image point.
- Any image is a matrix containing all the intensity values of pixel points.
Now a couple of things about the project itself:
- We’ll be using Swift classes for implementation of UI and Objective-C classes to implement detection.
- You can find out how to use Objective-C classes in Swift right here.
Step 1: Setting Everything Up
The first thing we should do is add the OpenCVDetector Objective-C class that’ll be working with the library.
In the class below we’re reducing namespaces to work with OpenCV library more smoothly.
using namespace cv;
using namespace std;
OpenCV has a CvVideoCamera class which is basically a wrapper around AVFoundation. So we provide some of the AVFoundation camera options as properties.
@implementation OpenCVDetector
- (instancetype)initWithCameraView:(UIView *)view scale:(CGFloat)scale preset:(AVCaptureSessionPreset) preset type:(OpenCVDetectorType) type { self = [super init];
if (self) {
self.videoCamera = [[CvVideoCamera alloc] initWithParentView: view];
self.videoCamera.defaultAVCaptureSessionPreset = preset;
self.videoCamera.defaultAVCaptureVideoOrientation = AVCaptureVideoOrientationPortrait;
self.videoCamera.defaultFPS = 30;
self.videoCamera.delegate = self;
self.scale = scale;
OpenCVDetector must implement the CvVideoCameraDelegate protocol and has to be set as a delegate to the video camera.
- (void)startCapture {
[self.videoCamera start];
}
- (void)stopCapture {
[self.videoCamera stop];
}
Then, create a CameraViewController Swift class.
class CameraViewController: UIViewController {
lazy var detecor = OpenCVDetector(cameraView: view, scale: 1, preset: .vga640x480, type: .back)
override func viewDidLoad() {
super.viewDidLoad()
detecor.startCapture()
}
This way, we initialize the camera and get self-view as a target for rendering every frame.
What Are Color Spaces?
As I’ve already mentioned, we’re going to change the color of an object. In my case, it’s a red spinner, which is quite easy to identify.
RGB is the most commonly used color space. Its acronym stands for Red Green Blue. In plain English, these three colors are mixed together in various ways to reproduce any color you may need.
Talking technical language, each of the colors can take a value between 0 and 255. Meaning (0,0,0) is black and (255,255,255) is white.
An important thing to mention here is that OpenCV reads images in BGRA format by default.
Step 3: Segmenting Out the Detected Red-Colored Object
In the previous step, we generated a mask to determine the area in the frame matching the detected color. We refine this mask and then use it for segmenting out the red spinner from the frame.
The code below illustrates how it’s done.
cv::Size blurSize(8,8);
blur(mask1, mask1, blurSize);
threshold(mask1, mask1, 50, 255, THRESH_BINARY);
Mat kernel = Mat::ones(3,3, CV_32F);
morphologyEx(mask1, mask1, cv::MORPH_OPEN, kernel);
morphologyEx(mask1, mask1, cv::MORPH_DILATE, kernel);
// creating an inverted mask to segment out the object from the frame
bitwise_not(mask1, mask2);
Mat res1, res2, final_output;
// Segmenting the object out of the frame using bitwise and with the inverted mask
bitwise_and(img, img, res1, mask2);
Step 4: Create a New Background for the Object
We create a new background from our HSV image with another H-value color. Then, we enlarge saturation and brightness S-V-values.
Mat background = Mat(hsv.rows, hsv.cols, hsv.type(), Scalar(fillingHSVColor[0], NAN, NAN));
for (int i = 0; i < background.cols; i++) {
for (int j = 0; j < background.rows; j++) {
CvPoint point = cvPoint(i, j);
background.at<Vec3b>(point).val[1] = MIN(hsv.at<Vec3b>(point).val[1] + 50, 255);
background.at<Vec3b>(point).val[2] = MIN(hsv.at<Vec3b>(point).val[2] + 50, 255);
}
}
cvtColor(background, background, COLOR_HSV2BGR);
Step 5: Generating the Final Output
Finally, we can replace the pixel values of the detected red-colored area with matching pixel values of the background to generate an augmented output.
Love AR? Check this open-source face detection library with overlaying we’ve recently developed.
To do so, we’re going to use bitwiseand operation to create an image with pixel values matching the detected area and equal to the background’s pixel values. Then, we should add the output to the image (res1) from which we’ve segmented out the red spinner.
// creating image showing static background frame pixels only for the masked area
bitwise_and(background, background, res2, mask1);
// Generating the final augmented output.
cv::add(res1, res2, final_output);
Here’s how it works on practice
That’s It!
We’ve built an application able to detect the color of an object and then change it in one tap. Go and play with it on your own, it’s open-source! Here’s a link to our GitHub profile with the library.
Comments