Using Python to Record a Movie from an EasyN IP Camera with OpenCV

Posted:12/19/2014 2:17PM

Using Python to Record a Movie from an EasyN IP Camera with OpenCV

Mike Mclain discusses how to record a movie from a EasyN IP Camera using Python and OpenCV

Preface:

A few days ago I posted an article discussing how to use Python to capture JPEG images from a EasyN IP Camera, and today I have decided to post a minor extension to this article, regarding how to expand the functionality of the developed EasyN Python class in order to record a movie from the capture JPEG images.

Conversely, before I begin further discussion on this topic, it is worth mentioning that OpenCV, Python Imaging Library (PIL), and Numpy Python packages are required in order for this method to work, and while the underlying procedure required to install these packages will not be discussed; however, in the event you are having trouble getting OpenCV to install, I recommend obtaining a free copy of the Enthought Python Distribution (EPD) since such packages are natively included within the EPD distribution.

Nevertheless, with this being said, it is important to recognize that, because additional python dependencies are required in order to create a movie from the obtained JPEG images, the separation between the original EasyN IP camera class and the new movie recording class is essential in allowing lower end devices (like embedded Linux distributions with Python) the ability to obtain simplistic JPEG information while, at the same time, allowing higher end devices the ability to record movies, thus increasing the overall flexibility of the developed EasyN Python module.

The Code Differential:

Towards this end, my first step in developing this movie recording class was to create a new python class named EasyNMovie that inherited the, previously developed, EasyN class, like so:

# import OpenCV
import cv2
# import Numpy
import numpy as np
# import Python Imaging Library
from PIL import Image
# import the previously created EasyN Class
from EasyN import EasyN
from StringIO import StringIO

# A class to communicate with the EasyN IP Camera with cv2 movie recording support
class EasyNMovie(EasyN):

    def __init__(
        self,
        ip,
        port,
        user,
        password,
        resolution=EasyN.Resolutions["640x480"],
        frame_rate=EasyN.Rate["20"],
        debug=True
    ):
        EasyN.__init__(
                self,
                ip,
                port,
                user,
                password,
                resolution,
                frame_rate,
                debug
        )

in which, the only task predominantly achieved here was the initialization of the inherited EasyN class.

Likewise, because frame rate is rather important, at least when it comes to recording a movie, and the encoding scheme utilized by the EasyN IP Camera to define what the current frame rate is, feels rather convoluted, I decided to define an inverse Python dictionary, like so:

# Inverse Frame rate, a way to quickly map frame Rate constants to a numerical value
# Note the "Full" value is likely not 25, but is MIPS Based, so look into this someday
I_Rate = {
    0: 25,
    1: 20,
    3: 15,
    6: 10,
    11: 5,
    12: 4,
    13: 3,
    14: 2,
    15: 1,
    17: 1.0/2.0,
    19: 1.0/3.0,
    21: 1.0/4.0,
    23: 1.0/5.0
}

that is based upon the Rate dictionary constants, defined within the EasyN class, in order to allow the end-user the ability to only utilize EasyN IP camera string values for the selection of a given frame rate.

Next, because the JPEG format obtained within the EasyN Python class is not in the correct format utilized by the cv2 movie recording module, a function needed to be created in order to convert this image information into the correct format (ideally without having to save or read anything from local disk storage).

While this process might sound relatively straightforward; however, I discovered that the JPEG image information obtained from the EasyN IP camera was sometimes slightly malformed by the camera (relative to the cv2 expected format) and would often times produce the following error message:

Corrupt JPEG data: 1 extraneous bytes before marker 0xd9

upon conversion within the cv2 module.

Likewise, upon further investigation of this particular error, I discovered that the EasyN IP camera was occasionally appending additional null bytes before the ending JPEG 0xd9 image marker; however, attempts to remove these extraneous bytes proved to be rather difficult because of the innate inability in differentiating between a legitimate null byte versus an actual extraneous one, thus such determinations would ultimately require creating a JPEG image decoder in order to consistently and correctly fix these errors.

Conversely, because such tasks are not trivial and the Python Imaging Library is capable of resolving these errors, I opted to utilize the Python Imaging Library module rather than create my own JPEG decoder; however, if the inclusion of this module is problematic within your intended application, it is possible to recompile the cv2 module to ignore, or actually throw a python catchable assert for these particular errors (which makes it easier to correctly remove the null bytes) , or to create a JPEG image decoder that can accurately determine if an extraneous byte exists, so it can be removed prior to processing within the cv2 module (noting that none of these topics will be discussed within this particular article).

Likewise, with this particular problem resolved, the conversion between the obtained JPEG image format and the expected cv2 image format, can be achieved using both Numpy and the cv2 module and results in the following function:

# A static function to convert a JPEG stream into a cv2 accepted input format
# color_mode >0 is RGB else BW only
@staticmethod
def _jpeg_stream_to_cv2_array(
    stream,
    color_mode=1
):
    # in order to stop the Corrupt JPEG data: 1 extraneous bytes before marker 0xd9 error from cv2 without
    # Recompiling the cv2 module
    # do some re-encoding with pil to ensure a cv2 accepted JPEG format
    stream_as_string_io = StringIO(stream)
    stream_as_pil = Image.open(stream_as_string_io)
    output_string = StringIO()
    stream_as_pil.save(output_string, format="JPEG")
    new_jpeg_stream = output_string.getvalue()
    output_string.close()
    stream_as_string_io.close()
    stream = new_jpeg_stream

    #  Convert the JPEG stream into a numpy array format
    numpy_array_stream = np.asarray(bytearray(stream), dtype=np.uint8)

    # Decode the numpy as a cv2 image object
    cv2_casted_image = cv2.imdecode(numpy_array_stream, color_mode)

    return cv2_casted_image

Now, with the problem of format conversion resolved, a relatively simplistic recording function:

# This function records a movie from JPEG frames over a given amount of time
# movie_format = -1 allows the user to select via UI the encoding format
# time is in seconds
# Warning fractional frame rates are not supported by all codecs
def record(
    self,
    filename,
    record_time=60,
    movie_format=cv2.cv.CV_FOURCC(*'mp4v'),
    fps=None
):

    # If no fps is given, look up the EasyN frame rate and use this value
    if fps is None:
        # Based on testing, it appears that the fps defined by the EasyN
        # is actually half the transmitted rate --- a possible Nyquist misnomer? --- So half this value
        fps = EasyNMovie.I_Rate[self.frame_rate]/2.0


    # Calculate how many frames are required to produce a movie of a given length of time
    frames_to_capture = int(round(fps*record_time))


    # pull a image from the camera to get size information
    cam_image = self._get_stream_jpeg()

    # Cast camera stream into cv2 stream
    c2v_cam_image = EasyNMovie._jpeg_stream_to_cv2_array(cam_image)


    height, width, layer = (None,None,None)

    # BW images only have a 2d shape while color has 3d shape
    # So act accordingly
    if len(c2v_cam_image.shape) == 3:
        height, width, layer = c2v_cam_image.shape
    else:
        height, width = c2v_cam_image.shape

    # Define our cv2 movie recording object
    video = cv2.VideoWriter(filename, movie_format, fps, (width, height))

    # Define our frame index for time tracking
    frame_index = 0

    # loop until we have all frames
    while frame_index < frames_to_capture:
        # get a image from the EasyN
        cam_image = self._get_stream_jpeg()
        # cast the image into cv2 format
        c2v_cam_image = self._jpeg_stream_to_cv2_array(cam_image)
        # write the image to the cv2 movie
        video.write(c2v_cam_image)
        # increase the index by 1
        frame_index += 1

    # shutdown our cv2 movie stream
    cv2.destroyAllWindows()
    # release the cv2 movie object
    video.release()

can be created, and (now that the EasyNMovie recording class is functional) a movie can be recorded from the EasyN IP camera, like so:

Demo = EasyNMovie({CAM IP}, {CAM PORT},"{CAM USER}","{CAM PASSWORD}")
Demo.record("Movie2.avi",record_time=60*5)
Demo.close()

and, in my case, resulted in the following video being created:

Concluding Remarks:

While this class seems to work reasonably well in obtaining and recording information from the easy and IP camera (although the half fps latency observed is rather curious and does deserve some further investigation); however, the inherent flexibility of the cv2 module (predominantly because of the intrinsic nature of video codecs) does imply that some minor tweaking will be required depending upon the end-user application, thus this is not truly a catchall solution, but rather a boilerplate template (of sorts) that will likely need some minor modifications prior to utilization within a given application.

Enjoy!

Comments:

comments powered by Disqus