NMSL - User contributions [en-ca]

Network and Multimedia Systems Lab (NMSL)

2018-02-22T04:07:12Z

Kcalagar: /* Concluded Projects */

__NOTOC__

'''Welcome to the Network Systems Lab (NSL) at SFU!'''

We are interested in the broad areas of multimedia networking and multimedia systems. We develop algorithms and systems to efficiently distribute multimedia content to large-scale user communities over wired and wireless networks. The Network Systems Lab is led by [http://www.cs.sfu.ca/~mhefeeda/ Dr. Mohamed Hefeeda.] and it is located in the TASC1 building, room 8210.

We hold regular [[group meeting]] for discussion and brainstorming.

Our current research interests include mobile multimedia, immersive and 3D video streaming, and cloud support for mobile and multimedia systems. Brief description and links to currently active projects are given below.

== '''Next Generation Video''' ==

With massive investments in the virtual reality (VR) hardware sector, companies encounter the challenge of providing VR content. The current solution of installing and operating VR camera rigs is expensive and not scalable. This project provides novel algorithms and methods to automatically convert standard broadcast 2D video streams to 3D and immersive VR streams of high quality. It also provides algorithms for adaptively streaming such complex multimedia content over the Internet to heterogeneous receivers.

* '''[[Immersive_Videos|Immersive Videos]]'''

* '''[[Hyperspectral Imaging]]'''

== '''Mobile Multimedia''' ==

We are designing algorithms to optimize video streaming in mobile wireless networks from different perspectives, including energy consumption of mobile receivers, quality of the videos delivered, and efficient utilization of the wireless bandwidth.

* '''[[hybridStreaming|Hybrid Multicast-Unicast Streaming over Mobile Networks]]'''

== '''ISP and CDN Traffic Management''' ==

We develop algorithms and systems for future ISP and CDN architectures. This includes resource management of ISP-managed CDNs (often called Telco-CDN). We develop stateless and efficient protocols and tools to support multicast traffic engineering in the ISP network.

* '''[[telcoCDN| Resource Management in Telco-CDNs]]'''

* '''[[Multicast in Carrier-Grade Networks| Multicast in Carrier-Grade Networks]]'''

= '''''Concluded Projects''''' =

== '''Industrial Automation as a Cloud Service''' ==

We are developing algorithms and systems to enable offering the whole stack of industrial automation systems from the cloud.

* '''[[cloudAutomation| Industrial Automation as a Cloud Service]]'''

== '''Mobile Multimedia''' ==

* '''[[mobileTV|Mobile TV Networks]]'''

* '''[[wimax|Multimedia Streaming over WiMAX Networks]]'''

== '''Peer-to-Peer Content Distribution''' ==

We are exploring the applicability of the P2P paradigm to build cost-effective content distribution systems. Problems such as sender selection, adaptive object replication, and content caching are being studied. We are also developing models to analyze the new characteristics of the P2P traffic and the impact of these characteristics on the cache replacement policies and object replication strategies.
Furthermore, we are devising analytic models to study the dynamics of the P2P system capacity and the impact of various parameters on it.

* '''[[pCDN|pCDN: Peer-assisted Content Distribution Network]]'''

* '''[[Modeling and Caching of P2P Traffic]]'''

* '''[[Scalable Multimedia Streaming]]'''

== '''Online Networked Games''' ==

We are designing various algorithms to improve the performance of online games.

* '''[[Minimizing Round-Trip Time in Online Games]]'''

* '''[[Minimizing Energy Consumption for Online Games on Mobile Phones]]'''

== '''Network Security''' ==

We are exploring network monitoring techniques to detect and thwart intrusion and denial-of-service attacks in their early stages by observing unusual traffic patterns injected by such attacks. We are studying the security of multimedia streaming systems that employ multi-layer and fine-grain scalable video streams.

* '''[[Security of Scalable Multimedia Streams]]'''

* '''[[Security of the SIP protocol]]'''

* '''[[Detecting DoS Attacks and Service Violations in QoS-enabled Networks]]'''

== '''Wireless Sensor Networks''' ==

We are developing coverage and connectivity maintenance protocols that consider probabilistic (i.e., more realistic) sensing and communication models. We are also designing protocols that provide controllable degrees of coverage (k-coverage).

* '''[[Probabilistic Coverage and Connectivity]]'''

* '''[[K-Coverage and its Application to Forest Fire Detection]]'''

Network and Multimedia Systems Lab (NMSL)

2018-02-22T04:06:55Z

Kcalagar: /* ISP and CDN Traffic Management */

Network and Multimedia Systems Lab (NMSL)

2018-02-22T04:06:34Z

Kcalagar: /* Mobile Multimedia */

2018-02-21T22:18:20Z

Kcalagar: Created page with "This project addresses different challenges in the full pipeline of capturing, transmitting, and identifying remote scenes using Hyperspectral images (HSI). == People == * ..."

Network and Multimedia Systems Lab (NMSL)

2018-02-21T21:51:06Z

Kcalagar: /* Next Generation Video */

__NOTOC__

'''Welcome to the Network Systems Lab (NSL) at SFU!'''

We are interested in the broad areas of multimedia networking and multimedia systems. We develop algorithms and systems to efficiently distribute multimedia content to large-scale user communities over wired and wireless networks. The Network Systems Lab is led by [http://www.cs.sfu.ca/~mhefeeda/ Dr. Mohamed Hefeeda.] and it is located in the TASC1 building, room 8210.

We hold regular [[group meeting]] for discussion and brainstorming.

Our current research interests include mobile multimedia, immersive and 3D video streaming, and cloud support for mobile and multimedia systems. Brief description and links to currently active projects are given below.

== '''Next Generation Video''' ==

With massive investments in the virtual reality (VR) hardware sector, companies encounter the challenge of providing VR content. The current solution of installing and operating VR camera rigs is expensive and not scalable. This project provides novel algorithms and methods to automatically convert standard broadcast 2D video streams to 3D and immersive VR streams of high quality. It also provides algorithms for adaptively streaming such complex multimedia content over the Internet to heterogeneous receivers.

* '''[[Immersive Videos]]'''
* '''[[Hyperspectral Imaging]]'''

== '''Mobile Multimedia''' ==

We are designing algorithms to optimize video streaming in mobile wireless networks from different perspectives, including energy consumption of mobile receivers, quality of the videos delivered, and efficient utilization of the wireless bandwidth.

* '''[[hybridStreaming|Hybrid Multicast-Unicast Streaming over Mobile Networks]]'''

== '''ISP and CDN Traffic Management''' ==

We develop algorithms and systems for future ISP and CDN architectures. This includes resource management of ISP-managed CDNs (often called Telco-CDN). We develop stateless and efficient protocols and tools to support multicast traffic engineering in the ISP network.

* '''[[telcoCDN| Resource Management in Telco-CDNs]]'''

* '''[[Multicast in Carrier-Grade Networks| Multicast in Carrier-Grade Networks]]'''

= '''''Concluded Projects''''' =

== '''Industrial Automation as a Cloud Service''' ==

We are developing algorithms and systems to enable offering the whole stack of industrial automation systems from the cloud.

* '''[[cloudAutomation| Industrial Automation as a Cloud Service]]'''

== '''Mobile Multimedia''' ==

* '''[[mobileTV|Mobile TV Networks]]'''

* '''[[wimax|Multimedia Streaming over WiMAX Networks]]'''

== '''Peer-to-Peer Content Distribution''' ==

We are exploring the applicability of the P2P paradigm to build cost-effective content distribution systems. Problems such as sender selection, adaptive object replication, and content caching are being studied. We are also developing models to analyze the new characteristics of the P2P traffic and the impact of these characteristics on the cache replacement policies and object replication strategies.
Furthermore, we are devising analytic models to study the dynamics of the P2P system capacity and the impact of various parameters on it.

* '''[[pCDN|pCDN: Peer-assisted Content Distribution Network]]'''

* '''[[Modeling and Caching of P2P Traffic]]'''

* '''[[Scalable Multimedia Streaming]]'''

== '''Online Networked Games''' ==

We are designing various algorithms to improve the performance of online games.

* '''[[Minimizing Round-Trip Time in Online Games]]'''

* '''[[Minimizing Energy Consumption for Online Games on Mobile Phones]]'''

== '''Network Security''' ==

We are exploring network monitoring techniques to detect and thwart intrusion and denial-of-service attacks in their early stages by observing unusual traffic patterns injected by such attacks. We are studying the security of multimedia streaming systems that employ multi-layer and fine-grain scalable video streams.

* '''[[Security of Scalable Multimedia Streams]]'''

* '''[[Security of the SIP protocol]]'''

* '''[[Detecting DoS Attacks and Service Violations in QoS-enabled Networks]]'''

== '''Wireless Sensor Networks''' ==

We are developing coverage and connectivity maintenance protocols that consider probabilistic (i.e., more realistic) sensing and communication models. We are also designing protocols that provide controllable degrees of coverage (k-coverage).

* '''[[Probabilistic Coverage and Connectivity]]'''

* '''[[K-Coverage and its Application to Forest Fire Detection]]'''

Network and Multimedia Systems Lab (NMSL)

2018-02-21T21:50:16Z

Kcalagar: /* Next Generation Video */

__NOTOC__

'''Welcome to the Network Systems Lab (NSL) at SFU!'''

We are interested in the broad areas of multimedia networking and multimedia systems. We develop algorithms and systems to efficiently distribute multimedia content to large-scale user communities over wired and wireless networks. The Network Systems Lab is led by [http://www.cs.sfu.ca/~mhefeeda/ Dr. Mohamed Hefeeda.] and it is located in the TASC1 building, room 8210.

We hold regular [[group meeting]] for discussion and brainstorming.

Our current research interests include mobile multimedia, immersive and 3D video streaming, and cloud support for mobile and multimedia systems. Brief description and links to currently active projects are given below.

== '''Next Generation Video''' ==

With massive investments in the virtual reality (VR) hardware sector, companies encounter the challenge of providing VR content. The current solution of installing and operating VR camera rigs is expensive and not scalable. This project provides novel algorithms and methods to automatically convert standard broadcast 2D video streams to 3D and immersive VR streams of high quality. It also provides algorithms for adaptively streaming such complex multimedia content over the Internet to heterogeneous receivers.

* '''[[Next Generation Video]]'''
* '''[[Hyperspectral Imaging]]'''

== '''Mobile Multimedia''' ==

We are designing algorithms to optimize video streaming in mobile wireless networks from different perspectives, including energy consumption of mobile receivers, quality of the videos delivered, and efficient utilization of the wireless bandwidth.

* '''[[hybridStreaming|Hybrid Multicast-Unicast Streaming over Mobile Networks]]'''

== '''ISP and CDN Traffic Management''' ==

We develop algorithms and systems for future ISP and CDN architectures. This includes resource management of ISP-managed CDNs (often called Telco-CDN). We develop stateless and efficient protocols and tools to support multicast traffic engineering in the ISP network.

* '''[[telcoCDN| Resource Management in Telco-CDNs]]'''

* '''[[Multicast in Carrier-Grade Networks| Multicast in Carrier-Grade Networks]]'''

= '''''Concluded Projects''''' =

== '''Industrial Automation as a Cloud Service''' ==

We are developing algorithms and systems to enable offering the whole stack of industrial automation systems from the cloud.

* '''[[cloudAutomation| Industrial Automation as a Cloud Service]]'''

== '''Mobile Multimedia''' ==

* '''[[mobileTV|Mobile TV Networks]]'''

* '''[[wimax|Multimedia Streaming over WiMAX Networks]]'''

== '''Peer-to-Peer Content Distribution''' ==

We are exploring the applicability of the P2P paradigm to build cost-effective content distribution systems. Problems such as sender selection, adaptive object replication, and content caching are being studied. We are also developing models to analyze the new characteristics of the P2P traffic and the impact of these characteristics on the cache replacement policies and object replication strategies.
Furthermore, we are devising analytic models to study the dynamics of the P2P system capacity and the impact of various parameters on it.

* '''[[pCDN|pCDN: Peer-assisted Content Distribution Network]]'''

* '''[[Modeling and Caching of P2P Traffic]]'''

* '''[[Scalable Multimedia Streaming]]'''

== '''Online Networked Games''' ==

We are designing various algorithms to improve the performance of online games.

* '''[[Minimizing Round-Trip Time in Online Games]]'''

* '''[[Minimizing Energy Consumption for Online Games on Mobile Phones]]'''

== '''Network Security''' ==

We are exploring network monitoring techniques to detect and thwart intrusion and denial-of-service attacks in their early stages by observing unusual traffic patterns injected by such attacks. We are studying the security of multimedia streaming systems that employ multi-layer and fine-grain scalable video streams.

* '''[[Security of Scalable Multimedia Streams]]'''

* '''[[Security of the SIP protocol]]'''

* '''[[Detecting DoS Attacks and Service Violations in QoS-enabled Networks]]'''

== '''Wireless Sensor Networks''' ==

We are developing coverage and connectivity maintenance protocols that consider probabilistic (i.e., more realistic) sensing and communication models. We are also designing protocols that provide controllable degrees of coverage (k-coverage).

* '''[[Probabilistic Coverage and Connectivity]]'''

* '''[[K-Coverage and its Application to Forest Fire Detection]]'''

Network and Multimedia Systems Lab (NMSL)

2018-02-21T21:48:20Z

Kcalagar: /* Next Generation Video */

__NOTOC__

'''Welcome to the Network Systems Lab (NSL) at SFU!'''

We are interested in the broad areas of multimedia networking and multimedia systems. We develop algorithms and systems to efficiently distribute multimedia content to large-scale user communities over wired and wireless networks. The Network Systems Lab is led by [http://www.cs.sfu.ca/~mhefeeda/ Dr. Mohamed Hefeeda.] and it is located in the TASC1 building, room 8210.

We hold regular [[group meeting]] for discussion and brainstorming.

Our current research interests include mobile multimedia, immersive and 3D video streaming, and cloud support for mobile and multimedia systems. Brief description and links to currently active projects are given below.

== '''Next Generation Video''' ==

With massive investments in the virtual reality (VR) hardware sector, companies encounter the challenge of providing VR content. The current solution of installing and operating VR camera rigs is expensive and not scalable. This project provides novel algorithms and methods to automatically convert standard broadcast 2D video streams to 3D and immersive VR streams of high quality. It also provides algorithms for adaptively streaming such complex multimedia content over the Internet to heterogeneous receivers.

* '''[[Immersive Videos]]'''
* '''[[Hyperspectral Imaging]]'''

== '''Mobile Multimedia''' ==

We are designing algorithms to optimize video streaming in mobile wireless networks from different perspectives, including energy consumption of mobile receivers, quality of the videos delivered, and efficient utilization of the wireless bandwidth.

* '''[[hybridStreaming|Hybrid Multicast-Unicast Streaming over Mobile Networks]]'''

== '''ISP and CDN Traffic Management''' ==

We develop algorithms and systems for future ISP and CDN architectures. This includes resource management of ISP-managed CDNs (often called Telco-CDN). We develop stateless and efficient protocols and tools to support multicast traffic engineering in the ISP network.

* '''[[telcoCDN| Resource Management in Telco-CDNs]]'''

* '''[[Multicast in Carrier-Grade Networks| Multicast in Carrier-Grade Networks]]'''

= '''''Concluded Projects''''' =

== '''Industrial Automation as a Cloud Service''' ==

We are developing algorithms and systems to enable offering the whole stack of industrial automation systems from the cloud.

* '''[[cloudAutomation| Industrial Automation as a Cloud Service]]'''

== '''Mobile Multimedia''' ==

* '''[[mobileTV|Mobile TV Networks]]'''

* '''[[wimax|Multimedia Streaming over WiMAX Networks]]'''

== '''Peer-to-Peer Content Distribution''' ==

We are exploring the applicability of the P2P paradigm to build cost-effective content distribution systems. Problems such as sender selection, adaptive object replication, and content caching are being studied. We are also developing models to analyze the new characteristics of the P2P traffic and the impact of these characteristics on the cache replacement policies and object replication strategies.
Furthermore, we are devising analytic models to study the dynamics of the P2P system capacity and the impact of various parameters on it.

* '''[[pCDN|pCDN: Peer-assisted Content Distribution Network]]'''

* '''[[Modeling and Caching of P2P Traffic]]'''

* '''[[Scalable Multimedia Streaming]]'''

== '''Online Networked Games''' ==

We are designing various algorithms to improve the performance of online games.

* '''[[Minimizing Round-Trip Time in Online Games]]'''

* '''[[Minimizing Energy Consumption for Online Games on Mobile Phones]]'''

== '''Network Security''' ==

We are exploring network monitoring techniques to detect and thwart intrusion and denial-of-service attacks in their early stages by observing unusual traffic patterns injected by such attacks. We are studying the security of multimedia streaming systems that employ multi-layer and fine-grain scalable video streams.

* '''[[Security of Scalable Multimedia Streams]]'''

* '''[[Security of the SIP protocol]]'''

* '''[[Detecting DoS Attacks and Service Violations in QoS-enabled Networks]]'''

== '''Wireless Sensor Networks''' ==

We are developing coverage and connectivity maintenance protocols that consider probabilistic (i.e., more realistic) sensing and communication models. We are also designing protocols that provide controllable degrees of coverage (k-coverage).

* '''[[Probabilistic Coverage and Connectivity]]'''

* '''[[K-Coverage and its Application to Forest Fire Detection]]'''

2D to 3D Video Conversion

2018-02-21T19:43:58Z

Kcalagar: /* Publications */

== People ==
* Kiana Calagari
* Mohamed Hefeeda

== Abstract ==

A wide spread adoption of 3D displays is hindered by the lack of content that matches the user expectations. Producing 3D videos is far more costly and time-consuming than regular 2D videos, which makes it challenging and thus rarely attempted, especially for live events, such as soccer games. In this project we develop a high-quality automated 2D-to-3D conversion method for soccer videos. Our method is data driven, relying on a reference database of 3D videos. Our key insight is that we use computer generated depth from current computer sports games for creating a synthetic 3D database.

== Details ==

The figure below shows an overview of our conversion system. We infer depth from a database of synthetically generated high-quality depths, collected
from video games. We then perform the conversion by transferring the depth gradient field from the database and reconstructing depth using Poisson reconstruction. In order to maintain sharp and accurate object boundaries, we create object masks and modify the Poisson equation on object boundaries. Finally, using the 2D frames and their estimated depth the left and right stereo pairs are rendered using a stereo-warping technique.

<center>
{| border="0"
|[[Image:Overview_N.png|center|The proposed 2D-to-3D conversion system|300px]]
|-
|align="center" width="200pt"|The proposed 2D-to-3D conversion system
|}
</center>

The main components of of our depth gradient based conversion technique are as follows:

* ''' Visual Search: ''' For each frame of the query video we identify the K most similar frames to it based on GIST and color.

* ''' Block-based Matching: ''' Using the K candidate frames we construct a matching image which is similar to the query frame and provides a mapping between the candidates and the query frame. To construct this matching image, we divide the query frame into small blocks and compare each block against all possible blocks in the K candidates. The block with the smallest Euclidean distance is chosen as the corresponding block.

* ''' Poisson Reconstruction: ''' We copy the corresponding depth gradients from the matched image to the query frame and reconstruct the depth values from the copied depth gradients using the Poisson equation.

* ''' Object Boundary Cuts: ''' In order to maintain sharp and accurate object boundaries, we create object masks, detect their edges through Canny edge detector, and disconnect pixels from the object boundaries by not allowing them to use an object boundary pixel as a valid neighbor.

* ''' Smoothing: ''' We add smoothness constraints to the Poisson reconstruction by enforcing the higher-order depth derivatives to be zero.

<center>
{| border="0"
|[[Image:DepthEstimation_N.png|center|The main components of our depth gradient based conversion|800px]]
|-
|align="center" width="200pt"|The main components of our depth gradient based conversion
|}
</center>

<center>
{| border="0"
|[[Image:Depth_Phases_NN.jpg|center|Depth_Phases|800px]]
|-
|align="left" width="200pt"|The effect of each step in our depth estimation technique: (a) Query, (b) A subset of its K candidates, (c) Created matched image, (d) Object boundary cuts, (e) Depth estimation using Poisson reconstruction, (f) Gradient refinement and Poisson reconstruction, (g) Depth with object boundary cuts, (h) Final depth estimation with smoothness, and (i) The zoomed and amplified version of the yellow block in h.
|}
</center>

The following figure shows some results of our depth estimation technique. Note how we can handle a wide variety of video shots, including different camera
views.

<center>
{| border="0"
|[[Image:DGCSoccer2.jpg|center|Results|800px]]
|-
|align="center" width="200pt"|Depth estimation for a wide variety of soccer shots using our method
|}
</center>

== Publications ==

K. Calagari, M. Elgharib, P. Didyk, A. Kaspar, W. Matusik, and M. Hefeeda, “Gradient-based 2D-to-3D Conversion for Soccer Videos”, In Proc. of the ACM Multimedia conference (MM’15), p 331-340, October 2015.

K. Calagari, M. Elgharib, P. Didyk, A. Kaspar, W. Matusik, and M. Hefeeda, "Data Driven 2-D-to-3-D Video Conversion for Soccer", IEEE Transactions on Multimedia (TMM), Vol. 20, Issue 3, p 605-619, 2018

Immersive Content Generation from Standard 2D Videos

2018-02-21T19:43:48Z

Kcalagar: /* Publications */

== People ==
* Kiana Calagari
* Mohamed Hefeeda

== Abstract ==

The aim of this project is to create compelling immersive videos suitable for VR (virtual reality) devices using only standard 2D videos. The focus of the work is on field sports such as soccer, hockey, basketball, etc. Currently the only way to create immersive content is by using multiple cameras and 360 camera rigs. This means that in addition to the already existing standard 2D cameras around the field, an expensive infrastructure should be added and managed in order to shoot and generate immersive content. In this project, however, we propose a more favorable alternative in which we can utilize the content of the already existing standard 2D cameras around the field to generate an immersive video.

<center>
{| border="0"
|[[Image:field.png|center|The aim is to create compelling immersive videos using only standard 2D videos.|500px]]
|-
|align="center" width="500pt"|The aim is to create compelling immersive videos using only standard 2D videos.
|}
</center>

== Details ==

''' Setup '''

We assume a setup in which we have at least 3 cameras as follows. Note that such camera setup is a practical setup in capturing and broadcasting field sports and the following cameras usually exist.

# The main camera, located in the middle of the field. This camera is a rotating camera capturing wide views and following the ball around the field. It is usually the main camera used for broadcasting games, and most of the feed that audience view comes from this camera.
# A camera on the right side of the field which covers the players on the right that might be missing in the main camera. This camera doesn't necessarily have to be rotating.
# A camera on the left side of the field which covers the players on the left that might be missing in the main camera. This camera doesn't necessarily have to be rotating.

''' Process '''

The main steps for generating an immersive video from 2D cameras around the field can be described as follows:

* '''Generating a still panorama using the motion of the main camera:''' The viewing angle in regular sports videos is usually not wide enough for an immersive experience. In order to improve the sense of presence, a wider viewing angle is needed. As a result,we increase the viewing angle by utilizing the camera rotation, and generating a panorama image which includes the static parts of the scene. The camera rotation is transformed to a wider viewing angle by aligning the frames using image registration techniques, and applying median filtering.
* '''Removing parallax between all video feeds:''' In a regular sports production the cameras are usually placed meters away from each other, causing a huge amount of parallax between them. By estimating the 3D pixel positions and the relative camera parameters, we warp each video feed to the position of the main camera to remove such parallax.
* '''Overlaying frames on the panorama:''' To seamlessly blend the copied parts with the background, we use Poisson blending. For each frame, we first overlay the main feed. Players missing from the main feed are then identified and copied from the left and right video feeds.

<center>
{| border="0"
|[[Image:Method.png|center|The main steps of our technique, and their main components.|400px]]
|-
|align="center" width="400pt"|The main steps of our technique.
|}
</center>

The following figure shows examples of final panoramas generated by our technique. The blue arrows indicate the missing players that were copied from the left and right feeds.

<center>
{| border="0"
|[[Image:SamplePanoramas.jpg|center|Examples of final panoramas generated by our technique for different games: basketball (top), hockey (middle), and
volleyball (bottom). The blue arrows indicate the players that have been copied from the left or right feeds.|900px]]
|-
|align="center" width="900pt"|Examples of final panoramas generated by our technique for different games: basketball (top), hockey (middle), and
volleyball (bottom). The blue arrows indicate the players that have been copied from the left or right feeds.
|}
</center>

== Publications ==

K. Calagari, M. Elgharib, S. Shirmohammadi, and M. Hefeeda, “Sports VR Content Generation from Regular Camera Feeds”, In Proc. of the ACM Multimedia conference (MM’17), p 699-707, October 2017.