Custom camera

Most cameras will be able to interface with the system using OpenCV. However, if this does not work for your camera, or you want to use the camera SDK of your camera for more control over camera settings, you can write an extension containing a CameraConfig subclass, allowing you to select your camera implementation in the workspace settings. Use the following command to create the extension:

LinuxWindows

# ensure that your python venv is sourced
incar create_pkg --name [package_name] --cameras [path]

# ensure that your python venv is sourced
incar create_pkg --name [package_name] --cameras [path]

Ensure that the package name is unique to avoid conflicts with other extensions. path is the location where the extension will be created.

Creating the Camera Implementation

Success

prior to starting, ensure that you manage to read the camera frames from the camera with a minimum example of the SDK.

The camera implementation is relatively straight-forward, it will consist of a subclass of CameraConfig containing the configuration and a subclass of Camera for the implementation of the camera. For example, the oak1 camera is implemented as follows:

from incar.extensions.cameras import Camera, CameraConfig
from dataclasses import dataclass

@CameraConfig.register_subclass("oak1")
@dataclass
class Oak1Config(CameraConfig):
    device_id: str = ""
    fps: int = 30

    def build_camera(self) -> "Oak1Camera":
        return Oak1Camera(self) 

class Oak1Camera(Camera):
    def __init__(self, conf: Oak1Config):
        import depthai as dai       # only import here to avoid importing when camera is not used anyways
        device_info = dai.DeviceInfo(conf.device_id)
        device = dai.Device(device_info)
        self.pipeline = dai.Pipeline(device)
        cam = self.pipeline.create(dai.node.Camera).build()
        self.video_queue = cam.requestOutput((conf.width, conf.height), fps=conf.fps).createOutputQueue()
        self.pipeline.start()

    def next_frame(self):
        videoIn = self.video_queue.get()
        frame = videoIn.getCvFrame() # in BGR
        frame[:,:,[0,2]] = frame[:,:,[2,0]] # to RGB
        return frame

Notes:

• The Oak1Config class has to implement a function build_camera(self) that returns a Camera object.
• The Oak1Config already inherits the following parameters: width, height, frame_type. frame_type can be RGB, DEPTH or RGBD.
• The Oak1Camera class has to implement a function next_frame(self) that returns the next frame as a numpy array. The function is allowed to be blocking.

Using the Camera Implementation

Create an extension containing the camera implementation. Install the extension using pip.

Now, the camera can be used just as any other camera type already included in the Incar Skill System. For example, the camera implementation above can be used by including the following in your workspace_config.json:

workspace_config.json

{
    "cameras": {
        "camera_label": {
            "type": "oak1",
            "device_id": "1844301051E042F500",
            "fps": 60
        }
    },

    ...
}

The type value is the same as defined in the @CameraConfig.register_subclass() wrapper, and all other available fields are the fields as defined in the CameraConfig subclass you created. In this example, that is device_id, fps and the inherited fields width, height, frame_type. If a field is not defined, it will use the standard value as defined in the CameraConfig (sub)class.

After reloading the workspace, a camera feature called camera_label will show in the GUI.

Depth cameras

A camera with frame_type = "DEPTH" works the same as any other camera. The next_frame function still has to return a numpy array with 3 channels as output (you can just duplicate the values to each channel).

An RGBD camera will create two features, the [key] as defined in the workspace_config, and a [key]_depth feature. Now, the next_frame feature has to return a tuple of two numpy arrays, both with 3 channels as output. For reference, here is how the realsense cameras implement the function:

def next_frame(self):
    frames = self.pipeline.wait_for_frames()
    if self.frame_type == FrameType.RGB or self.frame_type == FrameType.RGBD:
        rgb_frame = np.asarray(frames.get_color_frame().get_data(), np.uint8)
    if self.frame_type == FrameType.DEPTH or self.frame_type == FrameType.RGBD:
        depth_frame = np.asanyarray(self.colorizer.colorize(frames.get_depth_frame()).get_data())

    if self.frame_type == FrameType.RGBD:
        return (rgb_frame, depth_frame)
    if self.frame_type == FrameType.RGB:
        return rgb_frame
    if self.frame_type == FrameType.DEPTH:
        return depth_frame