The freenect2 module provides a Python interface to the libfreenect2 library. The libfreenect2 library provides a library allowing depth and RGB data to be extracted from a Kinect for Windows v2 (K4W2) device.
libfreenect2 must be installed prior to building this module. See the installation instructions provided by libfreenect2. The following is a brief summary of the required actions:
$ git clone https://github.com/OpenKinect/libfreenect2 ... $ cd libfreenect2; mkdir build; cd build $ cmake -DCMAKE_INSTALL_PREFIX=$HOME/.local .. && make all install ... $ export PKG_CONFIG_PATH=$HOME/.local/lib/pkgconfig $ pip install --user freenect2pkg-config errors¶
The libfreenect2 library uses pkg-config to record where it is installed on the system. If you install libfreenect2 manually as outlined above you will need to set the PKG_CONFIG_PATH environment variable.
Simple usage¶The library is intended to provide an easy to use Pythonic interface to libfreenect2. As an example of library usage, here is a simple infrared (IR) camera which can be used to capture an image from the Kinect’s build in IR camera:
"""
Simple IR camera using freenect2. Saves captured IR image
to output.jpg.
"""
# Import parts of freenect2 we're going to use
from freenect2 import Device, FrameType
# We use numpy to process the raw IR frame
import numpy as np
# We use the Pillow library for saving the captured image
from PIL import Image
# Open default device
device = Device()
# Start the device
with device.running():
# For each received frame...
for type_, frame in device:
# ...stop only when we get an IR frame
if type_ is FrameType.Ir:
break
# Outside of the 'with' block, the device has been stopped again
# The received IR frame is in the range 0 -> 65535. Normalise the
# range to 0 -> 1 and take square root as a simple form of gamma
# correction.
ir_image = frame.to_array()
ir_image /= ir_image.max()
ir_image = np.sqrt(ir_image)
# Use Pillow to save the IR image.
Image.fromarray(256 * ir_image).convert('L').save('output.jpg')
The library also supports extracting real-world 3D co-ordinates from the depth maps and saving libpcl-compatible PCD files:
"""
Capture a single RGB and depth frame and save them to output.pcd in
the libpcl PCD format. View the resulting cloud with:
pcl_viewer output.pcd
"""
from freenect2 import Device, FrameType
import numpy as np
# Open the default device and capture a color and depth frame.
device = Device()
frames = {}
with device.running():
for type_, frame in device:
frames[type_] = frame
if FrameType.Color in frames and FrameType.Depth in frames:
break
# Use the factory calibration to undistort the depth frame and register the RGB
# frame onto it.
rgb, depth = frames[FrameType.Color], frames[FrameType.Depth]
undistorted, registered, big_depth = device.registration.apply(
rgb, depth, with_big_depth=True)
# Combine the depth and RGB data together into a single point cloud.
with open('output.pcd', 'wb') as fobj:
device.registration.write_pcd(fobj, undistorted, registered)
with open('output_big.pcd', 'wb') as fobj:
device.registration.write_big_pcd(fobj, big_depth, rgb)
Reference¶
freenect2.NoDeviceError¶
Raised by Device when there is no default device to open.
freenect2.NoFrameReceivedError¶
With the default frame listener this is raised when no frame has been received from the device within a set time.
freenect2.Device(c_object=None)¶
Control a single device.
If called with no arguments, the default device is opened.
color_camera_params¶
(ColorCameraParams or None) A structure describing the RGB camera factory calibration. Before the start() is called, this is None since the device only reports calibration when capture begins.
ir_camera_params¶
(IrCameraParams or None) A structure describing the IR camera factory calibration. Before the start() is called, this is None since the device only reports calibration when capture begins.
close()¶
Close the device and free any associated resources.
color_frame_listener¶
A callable called whenever a new color frame arrives from the device. The callable should take two positional arguments, the frame type (an instance of FrameType) and the frame itself (an instance of Frame).
get_next_frame(timeout=None)¶
Get the next frame type and frame from the device.
Parameters: timeout (number or None) – If not-None, a positive number of seconds to wait for a frame before raising aNoFrameReceivedError exception. Returns: A FrameType, Frame tuple representing the type of received frame and the frame itself.
ir_and_depth_frame_listener¶
A callable called whenever a new IR or depth frame arrives from the device. The callable should take two positional arguments, the frame type (an instance of FrameType) and the frame itself (an instance of Frame).
registration¶
An instance of Registration which can be used to undistort the raw depth image and register the RGB image with it.
running(*args, **kwargs)¶
A context manager which can be used to ensure that the device’s streams are stopped. Any arguments are passed to start().
from freenect2 import Device
# Open default device
device = Device()
# Start depth and color frames
with device.running():
# ... frame listener callbacks will be called ...
# Device is now stopped
start(frame_listener=None)¶
Start depth, IR and RGB streams.
stop()¶
Stop any running streams.
freenect2.FrameType¶
Available types of frames.
Color = 1¶
1920x1080. BGRX or RGBX
Depth = 4¶
512x424 float, unit – millimeter. Non-positive, NaN, and infinity are invalid or missing data
Ir = 2¶
512x424 float. Range is [0.0, 65535.0]
freenect2.FrameFormat¶
Pixel format.
BGRX = 4¶
4 bytes of B, G, R, and unused per pixel
Float = 2¶
A 4-byte float per pixel
Gray = 6¶
1 byte of gray per pixel
Invalid = 0¶
Invalid format.
RGBX = 5¶
4 bytes of R, G, B, and unused per pixel
Raw = 1¶
Raw bitstream. ‘bytes_per_pixel’ defines the number of bytes
freenect2.Frame(frame_ref)¶
A single frame received from the device.
These should not be constructed directly since they are usually created by the freenect2 library itself. However you may need to construct “blank” frames for use with Registration. In which case, you should use the Frame.create() class method.
bytes_per_pixel¶
Number of bytes in a pixel. If format is FrameFormat.Raw, this is the buffer size.
create(width, height, bytes_per_pixel)¶
Create a blank frame with the specified width, height and bytes per pixel. Memory for the frame is automatically allocated. No other attributes are initialised.
data¶
A buffer object pointing to the raw memory contents of the frame.
exposure¶
From 0.5 (very bright) to ~60.0 (fully covered)
format¶
Byte format. Informative only, doesn’t indicate errors. An instance of FrameFormat.
gain¶
From 1.0 (bright) to 1.5 (covered)
gamma¶
From 1.0 (bright) to 6.4 (covered)
height¶
Number of lines in the frame
sequence¶
Increasing frame sequence number
status¶
zero if ok; non-zero for errors
timestamp¶
Unit – roughly or exactly 0.1 millisecond
to_array()¶
Convert the image to a numpy array instance.
The memory is not copied so be careful performing any operations which modify the contents of the frame.
to_image()¶
Convert the Frame to a PIL Image instance.
width¶
Length of a line (in pixels)
freenect2.Registration(depth_p, rgb_p)¶
Information required to undistort raw depth frames and register RGB frames onto depth.
Do not construct this directly. Instead use the Device.registration attribute.
apply(rgb, depth, enable_filter=True, with_big_depth=False)¶
Take an RGB and Depth image and return tuple with the undistorted depth image and color image rectified onto depth.
Parameters:Frame) – RGB frame received from deviceFrame) – Depth frame received from deviceA Frame pair representing the undistorted depth and registered RGB frames.
get_big_points_xyz_array(big_depth)¶
Like get_points_xyz_array() but operates on the “big” depth map which can be returned from apply().
Frame) – big 1920x1082 frame returned from apply(). Returns: A 1082x1920x3 array of 3D points. The last dimension corresponding to x, y and z.
get_points_xyz(undistorted, rows, cols)¶
Retrieve real-world co-ordinates corresponding to points in the undistorted depth image. Units are millimetres.
Parameters:Frame) – the undistorted depth frameA three element tuple containing numpy arrays for the x-, y- and z-co-ordinates of the points. Each array has the same shape as rows.
get_points_xyz_array(undistorted)¶
Return a 3D array of x, y, z points for each point in an undistorted frame. Invalid points are Nan-ed.
Parameters: undistorted (Frame) – the undistorted depth frame Returns: A 424x512x3 array of 3D points. The last dimension corresponding to x, y and z.
write_big_pcd(file_object, big_depth, rgb=None)¶
Write depth map and (optionally) RGB data to libpcl-compatible PCD format file. Works like write_pcd() except that it works on the “big” depth map which can be returned from apply(). If the RGB frame is present, each point is coloured according to the image otherwise the points are left uncoloured.
Note
Under Python 3 the file object must be opened in binary mode.
Parameters:write_pcd(file_object, undistorted, registered=None)¶
Write depth map and (optionally) RGB data to libpcl-compatible PCD format file. If the registered RGB frame is present, each point is coloured according to the image otherwise the points are left uncoloured.
Note
Under Python 3 the file object must be opened in binary mode.
Parameters:freenect2.IrCameraParams¶
IR/depth camera intrinsic calibration.
fx¶
Focal length for x-axis (pixels)
fy¶
Focal length for y-axis (pixels)
cx¶
Principal point for x-axis (pixels)
cy¶
Principal point for y-axis (pixels)
k1¶
Radial distortion co-efficient, 1st-order
k2¶
Radial distortion co-efficient, 2nd-order
k3¶
Radial distortion co-efficient, 3rd-order
p1¶
Tangential distortion co-efficient
p2¶
Tangential distortion co-efficient
freenect2.ColorCameraParams¶
Color camera intrinsic calibration.
fx¶
Focal length for x-axis (pixels)
fy¶
Focal length for y-axis (pixels)
cx¶
Principal point for x-axis (pixels)
cy¶
Principal point for y-axis (pixels)
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4