Private:3DV Remote Rendering - Revision history

Aah10 at 18:26, 8 April 2011

2011-04-08T18:26:10Z

Aah10 at 23:40, 7 April 2011

2011-04-07T23:40:24Z

Aah10 at 23:15, 7 April 2011

2011-04-07T23:15:36Z

Aah10 at 23:03, 7 April 2011

2011-04-07T23:03:44Z

Aah10 at 22:37, 7 April 2011

2011-04-07T22:37:14Z

Aah10 at 20:57, 7 April 2011

2011-04-07T20:57:53Z

Aah10 at 20:40, 7 April 2011

2011-04-07T20:40:23Z

Aah10 at 20:24, 7 April 2011

2011-04-07T20:24:19Z

Aah10 at 19:51, 7 April 2011

2011-04-07T19:51:56Z

Mspenard at 15:15, 25 January 2011

2011-01-25T15:15:14Z

@@ Line 46: / Line 46: @@
 * A recent survey published in the Proceedings of the IEEE state that video-plus-depth (V+D) coding approach is preferred for scalability over multiple view coding (MVC), but it might be too computationally demanding for terminal devices as it requires view rendering and hence makes the device less power efficient. This statement needs to be verified by actual implementation and measurements.
 * If we reduce the depth maps resolution by down-sampling to reduce the bitrate of the transmitted stream, we will need to perform depth enhancement operations after reconstruction at the receiver side. These operations can be computationally expensive and may drain the mobile receivers power.
-* When considering which is more important in order to implement a prioritization technique, we are also faced with a dilemma. Having a high resolution texture stream is required for backward compatibility in case the device is only 2D capable. However, having a high quality depth map is very important to avoid shape/surface deformations after reconstructing the 3D scene.
+* When considering which is more important, texture stream or depth stream, in order to implement a prioritization technique, we are also faced with a dilemma. Having a high resolution texture stream is required for backward compatibility in case the device is only 2D capable. However, having a high quality depth map is very important to avoid shape/surface deformations after reconstructing the 3D scene.
 * Trying to divide the task of rendering the 3D video between the client and the server is also not trivial. For example, attempting to generate part of the view that will be rendered at the client-side on the server will not be possible because we do not know which viewpoint the client will render based on user input. Moreover, this defeats the goal that needs to be achieved which is sending the client only two neighboring views and delegating the rendering of intermediate views to it in order to reduce viewpoint change latency.
 * Attempting to utilize the GPU to speedup the view synthesis process also has its challenges. One main issue that may hinder achieving significant speedups is the 3D warping process. That mapping process between pixels in the reference view and pixels in the target view is not a one-to-one mapping process. Several pixels may be mapped to the same pixel location in the target view causing competition that needs to be resolved based on which pixel represents an object in the foreground and which one represents the background. Thus, attempting to perform 3D warping of the pixels in parallel will exhibit shared resource contention. How much effect does this has on achievable speedups will depend on the amount of contention and needs to be determined.

@@ Line 43: / Line 43: @@
 === Readings and Thoughts ===
-* A recent survey published in the Proceedings of the IEEE state that video-plus-depth (V+D) coding approach is preferred for scalability, but it might be too computationally demanding for terminal devices as it requires view rendering and hence makes the device less power efficient. This statement needs to be verified by actual implementation and measurements.
+* Almost all of the autostereoscopic mobile 3D displays available at the moment are 2-view displays. The number of different views in autostereoscopic displays come at a price of reduced spatial resolution and lowered brightness. In the case of small screen, battery-driven mobile devices, the tradeoff between the number of views and spatial resolution is very important. Since mobile devices are normally watched by a single observer only, two independent views are considered sufficient for satisfactory 3D perception and a good compromise with respect to spatial resolution.
 * If we reduce the depth maps resolution by down-sampling to reduce the bitrate of the transmitted stream, we will need to perform depth enhancement operations after reconstruction at the receiver side. These operations can be computationally expensive and may drain the mobile receivers power.
 * When considering which is more important in order to implement a prioritization technique, we are also faced with a dilemma. Having a high resolution texture stream is required for backward compatibility in case the device is only 2D capable. However, having a high quality depth map is very important to avoid shape/surface deformations after reconstructing the 3D scene.

@@ Line 41: / Line 41: @@
-=== Thoughts ===
+=== Readings and Thoughts ===
 * A recent survey published in the Proceedings of the IEEE state that video-plus-depth (V+D) coding approach is preferred for scalability, but it might be too computationally demanding for terminal devices as it requires view rendering and hence makes the device less power efficient. This statement needs to be verified by actual implementation and measurements.
@@ Line 51: / Line 51: @@
 * Scalable asymmetric coding of stereoscopic video can be achieved using SVC. Asymmetric encoding exploits the binocular suppression theory which states that the HVS tolerates lack of high-frequency components in one of the views. Hence, one of the views may be presented at a lower quality without degrading the 3-D video perception. It is possible to obtain spatial and/or quality scalable right and left views if they are simulcast coded using the SVC standard. Either both views are encoded using SVC, or one view is SVC encoded and the other is encoded with H.264/AVC. Thus, when available bandwidth is more that maximum rate of the video, the quality of the scalable bitstream dominates the perceived quality. Otherwise, the non-scalable bitstream becomes the high-quality pair of the asymmetry. The concept of asymmetric coding can be extended to multiple views where every other view is scalable coded using SVC for example.
 * In the case of multiple views plus depth (MVD) coding, view scalability and temporal scalability are achievable if MVC is used, and spatial scalability and quality scalability are achievable if SVC is used. The coding rate of the view and the corresponding depth maps can be modified during streaming to adapt to the dynamic network conditions (depth maps generally require 15-20% of video bitrate to produce acceptable results).
+* The paper by Hewage ''et al.'' uses subjective testing to measure the effect of compression artifacts and frame losses on the perceived quality (namely, overall image quality and depth perception) of the reconstructed stereoscopic video. Their results show that the VQM objective metric can be mapped so that it correlates strongly with both the overall viewer perception of image quality and depth perception. They conclude that VQM is a good model prediction for both the perceived overall image quality and depth of 3D video, whether the channel is error free or exhibits packet losses.
@@ Line 63: / Line 63: @@
 === References ===
 * [http://www.abiresearch.com/press/3579-Mobile+Form+Factor+Will+Bring+3D+to+Mainstream+Market Mobile Form Factor Will Bring 3D to Mainstream Market]

@@ Line 47: / Line 47: @@
 * When considering which is more important in order to implement a prioritization technique, we are also faced with a dilemma. Having a high resolution texture stream is required for backward compatibility in case the device is only 2D capable. However, having a high quality depth map is very important to avoid shape/surface deformations after reconstructing the 3D scene.
 * Trying to divide the task of rendering the 3D video between the client and the server is also not trivial. For example, attempting to generate part of the view that will be rendered at the client-side on the server will not be possible because we do not know which viewpoint the client will render based on user input. Moreover, this defeats the goal that needs to be achieved which is sending the client only two neighboring views and delegating the rendering of intermediate views to it in order to reduce viewpoint change latency.
-* Attempting to utilize the GPU to speedup the view synthesis process also has its challenges. One main issue that may hinder achieving significant speedups is the 3D warping process. That mapping process between pixels in the reference view and pixels in the target view is not a one-to-one mapping process. Several pixels may be mapped to the same pixel location in the target view causing competition that needs to be resolved based on which pixel represents an object in the foreground and which one represents the background. Thus, attempting to perform 3D warping of the pixels in parallel will exhibit shared resource contention. How much effect does this has on achievable speedups needs to be determined.
+* Attempting to utilize the GPU to speedup the view synthesis process also has its challenges. One main issue that may hinder achieving significant speedups is the 3D warping process. That mapping process between pixels in the reference view and pixels in the target view is not a one-to-one mapping process. Several pixels may be mapped to the same pixel location in the target view causing competition that needs to be resolved based on which pixel represents an object in the foreground and which one represents the background. Thus, attempting to perform 3D warping of the pixels in parallel will exhibit shared resource contention. How much effect does this has on achievable speedups will depend on the amount of contention and needs to be determined.
 <!-- * One possible solution that reduces the effect of this contention would be to perform rectification of the two views first to reduce the warping process to a horizontal shift process based on the depth value. Since we have two reference views, a left view and a right view, we -->
 * Scalable asymmetric coding of stereoscopic video can be achieved using SVC. Asymmetric encoding exploits the binocular suppression theory which states that the HVS tolerates lack of high-frequency components in one of the views. Hence, one of the views may be presented at a lower quality without degrading the 3-D video perception. It is possible to obtain spatial and/or quality scalable right and left views if they are simulcast coded using the SVC standard. Either both views are encoded using SVC, or one view is SVC encoded and the other is encoded with H.264/AVC. Thus, when available bandwidth is more that maximum rate of the video, the quality of the scalable bitstream dominates the perceived quality. Otherwise, the non-scalable bitstream becomes the high-quality pair of the asymmetry. The concept of asymmetric coding can be extended to multiple views where every other view is scalable coded using SVC for example.

@@ Line 43: / Line 43: @@
 === Thoughts ===
 * If we reduce the depth maps resolution by down-sampling to reduce the bitrate of the transmitted stream, we will need to perform depth enhancement operations after reconstruction at the receiver side. These operations can be computationally expensive and may drain the mobile receivers power.
 * When considering which is more important in order to implement a prioritization technique, we are also faced with a dilemma. Having a high resolution texture stream is required for backward compatibility in case the device is only 2D capable. However, having a high quality depth map is very important to avoid shape/surface deformations after reconstructing the 3D scene.
-* Trying to divide the task of rendering the 3D video between the client and the server is also not trivial. For example, attempting to generate part of the view that will be rendered on the client side will not be possible because we do not know which viewpoint the client will render based on user input. Moreover, this defeats the goal that needs to be achieved which is sending the client only two neighboring views and delegating the rendering of intermediate views to it in order to reduce viewpoint change latency.
+* Trying to divide the task of rendering the 3D video between the client and the server is also not trivial. For example, attempting to generate part of the view that will be rendered at the client-side on the server will not be possible because we do not know which viewpoint the client will render based on user input. Moreover, this defeats the goal that needs to be achieved which is sending the client only two neighboring views and delegating the rendering of intermediate views to it in order to reduce viewpoint change latency.
 * Attempting to utilize the GPU to speedup the view synthesis process also has its challenges. One main issue that may hinder achieving significant speedups is the 3D warping process. That mapping process between pixels in the reference view and pixels in the target view is not a one-to-one mapping process. Several pixels may be mapped to the same pixel location in the target view causing competition that needs to be resolved based on which pixel represents an object in the foreground and which one represents the background. Thus, attempting to perform 3D warping of the pixels in parallel will exhibit shared resource contention. How much effect does this has on achievable speedups needs to be determined.
 <!-- * One possible solution that reduces the effect of this contention would be to perform rectification of the two views first to reduce the warping process to a horizontal shift process based on the depth value. Since we have two reference views, a left view and a right view, we -->

← Older revision		Revision as of 20:24, 7 April 2011
Line 48:		Line 48:
	* Attempting to utilize the GPU to speedup the view synthesis process also has its challenges. One main issue that may hinder achieving significant speedups is the 3D warping process. That mapping process between pixels in the reference view and pixels in the target view is not a one-to-one mapping process. Several pixels may be mapped to the same pixel location in the target view causing competition that needs to be resolved based on which pixel represents an object in the foreground and which one represents the background. Thus, attempting to perform 3D warping of the pixels in parallel will exhibit shared resource contention. How much effect does this has on achievable speedups needs to be determined.		* Attempting to utilize the GPU to speedup the view synthesis process also has its challenges. One main issue that may hinder achieving significant speedups is the 3D warping process. That mapping process between pixels in the reference view and pixels in the target view is not a one-to-one mapping process. Several pixels may be mapped to the same pixel location in the target view causing competition that needs to be resolved based on which pixel represents an object in the foreground and which one represents the background. Thus, attempting to perform 3D warping of the pixels in parallel will exhibit shared resource contention. How much effect does this has on achievable speedups needs to be determined.
	<!-- * One possible solution that reduces the effect of this contention would be to perform rectification of the two views first to reduce the warping process to a horizontal shift process based on the depth value. Since we have two reference views, a left view and a right view, we -->		<!-- * One possible solution that reduces the effect of this contention would be to perform rectification of the two views first to reduce the warping process to a horizontal shift process based on the depth value. Since we have two reference views, a left view and a right view, we -->
		+	* Scalable asymmetric coding of stereoscopic video can be achieved using SVC. Asymmetric encoding exploits the binocular suppression theory which states that the HVS tolerates lack of high-frequency components in one of the views. Hence, one of
		+	the views may be presented at a lower quality without degrading the 3-D video perception. It is possible to obtain spatial and/or quality scalable right and left views if they are simulcast coded using the SVC standard. Either both views are encoded using SVC, or one view is SVC encoded and the other is encoded with H.264/AVC. Thus, when available bandwidth is more that maximum rate of the video, the quality of the scalable bitstream dominates the perceived quality. Otherwise, the non-scalable bitstream becomes the high-quality pair of the asymmetry.
		+

← Older revision		Revision as of 19:51, 7 April 2011
Line 39:		Line 39:
	* How will view synthesis and associated operations (e.g. 3D warping and hole filling) at the receiver-side affect the power consumption of the device?		* How will view synthesis and associated operations (e.g. 3D warping and hole filling) at the receiver-side affect the power consumption of the device?
	* Should we only focus on reducing the amount of data needed to be transmitted, as antennas consume a significant amount of power? Or should we experiment and see how those 2 variables are tied together, given that decoding 3D videos is a much more resource intensive process?		* Should we only focus on reducing the amount of data needed to be transmitted, as antennas consume a significant amount of power? Or should we experiment and see how those 2 variables are tied together, given that decoding 3D videos is a much more resource intensive process?
		+
		+
		+	=== Thoughts ===
		+
		+	* If we reduce the depth maps resolution by down-sampling to reduce the bitrate of the transmitted stream, we will need to perform depth enhancement operations after reconstruction at the receiver side. These operations can be computationally expensive and may drain the mobile receivers power.
		+	* When considering which is more important in order to implement a prioritization technique, we are also faced with a dilemma. Having a high resolution texture stream is required for backward compatibility in case the device is only 2D capable. However, having a high quality depth map is very important to avoid shape/surface deformations after reconstructing the 3D scene.
		+	* Trying to divide the task of rendering the 3D video between the client and the server is also not trivial. For example, attempting to generate part of the view that will be rendered on the client side will not be possible because we do not know which viewpoint the client will render based on user input. Moreover, this defeats the goal that needs to be achieved which is sending the client only two neighboring views and delegating the rendering of intermediate views to it in order to reduce viewpoint change latency.
		+	* Attempting to utilize the GPU to speedup the view synthesis process also has its challenges. One main issue that may hinder achieving significant speedups is the 3D warping process. That mapping process between pixels in the reference view and pixels in the target view is not a one-to-one mapping process. Several pixels may be mapped to the same pixel location in the target view causing competition that needs to be resolved based on which pixel represents an object in the foreground and which one represents the background. Thus, attempting to perform 3D warping of the pixels in parallel will exhibit shared resource contention. How much effect does this has on achievable speedups needs to be determined.
		+	<!-- * One possible solution that reduces the effect of this contention would be to perform rectification of the two views first to reduce the warping process to a horizontal shift process based on the depth value. Since we have two reference views, a left view and a right view, we -->

@@ Line 12: / Line 12: @@
 Transmission is to be carried via unicast over an unreliable wireless channel. A feedback channel would be necessary between the receiver and the proxy. This channel would be utilized to send information about current/desired viewpoint, buffer status, and network conditions, in addition statistics about mobile device itself (e.g. current battery level, screen resolution, expected amount of power for processing, etc.). Such feedback channel is crucial in order to have a fully-adaptive algorithm, that can quickly adapt to any change in these parameters.
-Because of the limited wireless bandwidth, we need efficient and adaptive compression of transmitted views/layers. In addition, an unequal error protection (UEP) technique will be required to overcome the unreliable nature of the wireless channel. Multiple description coding (MDC) has already been used in some experiments, and the results seem quite promising.
+Because of the limited wireless bandwidth, we need efficient and adaptive compression of transmitted views/layers. In addition, an unequal error protection (UEP) technique will be required to overcome the unreliable nature of the wireless channel. Multiple description coding (MDC) has already been used in some experiments, and the results seem quite promising for both multiview video coding, and video plus depth.
 It is assumed that the mobile receiver has a display capable of rendering at least two views.
 Interfacing with mobile receiver's display may be an issue since this will only be possible through a predefined driver API. Whether or not these APIs will be exposed and which 3D image format they expect will vary from one device to the other.
-We believe that such autostereoscopic displays will probably come with their own IP hardware to perform rendering operations such as 3D warping, hole filling, etc. Whether or not we would have control over this process is still unknown.
+We believe that such autostereoscopic displays will probably come with their own IP hardware to perform rendering operations such as 3D warping, hole filling, etc. Whether or not we would have control over this process is still unknown.  In the mean time, it is possible to experiment and send data that can be rendered in 2D, just like most of the experiments we have read so far.  This would enable us to establish the feasibility of our scheme, and benchmark it against previous works.
 Based on receiver feedback, the adaptation proxy is responsible for selecting the best views to send, perform rate adaptation based on current network conditions, and encode them quickly and efficiently. We can utilize RD-optimization techniques for rate adaptation.