Result Summary

Timvideos is trying to develop an opensource video conferencing solution. The organisation has both a hardware component (HDMI2USB) and software component (like Gstreamer, flumotion). I was working on the hardware side of the org specifically on HDMI2USB fpga firmware. HDMI2SUSB is basically a solution based on FPGA to compress HD video stream input from a source like video camera and output the compressed stream via USB, hence the name HDMI2USB.

GOALS

My  project was on "Optimisation of MPEG Core". In the original firmware, the output stream from USB was about 10-11 fps which is very low and not suitable for video recording, the primary purpose of HDMI2USB. The aim of my project was to improve this frame rate. I had proposed to improve it to 60 fps @ 720p. As the reason for a slow frame rate was not fully understood, I built a lot of firmware to get the "status" of the HDMI2USB which later on became my second project, to create a "Debug Infrastructure" for HDMI2USB.

Final Outcome

Finally I was able to improve the frame rate to 30 fps (which is the minimum required for recording) and set a system which allows developers to easily output debug data via CDC or UART Port.

How did the frame rate improve?

The title of my project "Optimisation of MPEG Core" isn't a very apt title. The work I did tried to optimize the whole firmware. The original firmware has an image buffer which stores a frame before being compress by the JPEG core. The state machine of the image buffer was such that no processing was done when a frame was being read and no frames were read when processing was done. This was the bottleneck which was causing a slow frame rate. To remove this bottleneck I pipelined the read and write cycles of the image buffer. This reduced the number of frames which were being dropped and hence improve the frame rate.

Debug Interface

The debug interface is a new feature added to the HDMI2USB firmware. Unlike programming language which can be easily debugged using printf statement, systems on hardware like FPGA require elaborate methods. The debug interface which has been implemented allows data to be output via CDC port which was part of the original firmware and UART port which has been added to allow more options to developers and users incase CDC fails. Also a program has been included which reads bytes from fpga and presents them to users in a human readable format.

Heart Beat Feature

Sometimes when the image is static, it is difficult to tell whether the fpga firmware is working or it has hung. To remove this ambiguity, a heart beat feature was added which is basically a small block of pixel in the bottom right corner of the frame pulsing at a constant rate. This feature can be turned on/off using a hardware switch and via the CDC control port.

Community Outcomes

The Timvideos developers’ community has promising outcomes after completion of this project. The biggest being the HDMI2USB can be used for recording. Now that HDMI2USB firmware streams at 30 fps, it can be used for recording at 30 fps which is the minimum frame rate required for video recording. Also, I may not have been able to meet the 60 fps as I mentioned in my proposal but I have successfully revealed some issues in the HDMI2USB firmware which need to be dealt with, most importantly the bandwidth of FX2 chip.

There are certain other features which can be implemented but have not been implemented due to lack of time like reducing subsampling to 4:2:0 and implementing a triple buffer in the image buffer. This may or may not change the frame rate but it will be interesting to see the outcomes if they are implemented. The debug infrastructure that has been setup will allow future developers to easily debug the hardware and also provide an easy way of troubleshooting for future users. Addition of UART allows multiple debug/control option incase cypress fx2 fails.  As my project dealt more with optimisation which involves getting a good understanding of the system as a whole, I have gained good knowledge about the fpga firmware. This will be useful for future developers as I am in a position to be able to guide them regarding specific issues. Also thanks to the exposure I received due to this project, I have started a Open Source Hardware Developers Group in my university where soon I will be introducing TimVideos. Hopefully this will increase the number of developers for HDMI2USB.

Replicating Results

All the code relevant to the project has been merged into the HDMI2USB repository.

Also check out the how to use section in the following blog posts:

• UART
• Heart Beat
• Debug Interface
• To check the frame rate  

Bugs/To Do

• Support 1080p via USB: I successfully added 1080p test pattern (Hidef snow) to the hdmi2usb. It was streaming out of the HDMI port but not out of USB. Here is the video.
• Add Control Port via UART: I was able to expose some features of control port to UART but not all. I will try to finish after GSOC.
• Automatic Encoding Quality Control: It has been noticed that the encoding quality depends highly on the type of image. So there can be a auto encoding qulaity control like the one in youtube (it changes the resolution though) which allows more compression if the frame rate is slow.

Documentation

Developers Guide has been updated to reflect new changes.

Learnings

This project has been a great learning curve for me. My knowledge about fpga and related stuff has increased exponentially. The mentors are great and supportive. Working for this org was a great introduction to Open Source Development. I would highly recommend the org to anyone who is new and wants to get dirty with FOSS.