OpenCV
Overview
This article shows how to convert OpenNI2 video frames to OpenCV Mat format. Once you have frame in Mat format, you can leverage all OpenCV functions.
Expect Output

Tutorial - C++
After connect to camera and start each sensor (color, depth and IR). Read the frame into
VideoFrameRef
VideoFrameRef colorFrame;
VideoFrameRef depthFrame;
VideoFrameRef irFrame;
color.readFrame(&colorFrame);
depth.readFrame(&depthFrame);
ir.readFrame(&irFrame);
Convert
VideoFrameRef
tocv::Mat
// Color frame
cv::Mat colorMat = cv::Mat(colorFrame.getHeight(), colorFrame.getWidth(), CV_8UC3, (void *)colorFrame.getData());
// Depth frame
cv::Mat depthMat = cv::Mat(depthFrame.getHeight(), depthFrame.getWidth(), CV_16UC1, (void *)depthFrame.getData());
// IR frame
cv::Mat irMat = cv::Mat(irFrame.getHeight(), irFrame.getWidth(), CV_16UC1, (void *)irFrame.getData());
For better visualization, you can do the following conversion
// For color frame
cv::cvtColor(colorMat, colorMat, cv::COLOR_BGR2RGB);
// For depth frame
depthMat.convertTo(depthMat, CV_8U, 255.0 / 1024.0);
cv::applyColorMap(depthMat, depthMat, cv::COLORMAP_JET);
// For IR frame
irMat.convertTo(irMat, CV_8U, 255.0 / 1024.0);
Tutorial - Python
After connect to camera and start each sensor (color, depth and IR). Read the video frames
colorFrame = colorStream.read_frame()
depthFrame = depthStream.read_frame()
irFrame = irStream.read_frame()
Convert video frame to OpenCV format which is numpy array
# Color frame
colorMat = np.fromstring(colorFrame.get_buffer_as_uint8(), dtype=np.uint8).reshape(480, 640, 3)
# Depth frame
depthMat = np.fromstring(depthFrame.get_buffer_as_uint16(), dtype=np.uint16).reshape(480, 640)
# IR frame
irMat = np.fromstring(irFrame.get_buffer_as_uint16(), dtype=np.uint16).reshape(480, 640)
For better visualization, you can do the following conversion
# For color frame
colorMat = cv2.cvtColor(colorMat, cv2.COLOR_BGR2RGB)
# For depth frame
depthMat = np.uint8(depthMat.astype(float) * 255 / 1024)
depthMat = cv2.applyColorMap(depthMat, cv2.COLORMAP_JET)
# For IR frame
irMat = np.uint8(irMat.astype(float) * 255 / 1024)
Full code
Last updated