Private:progress-diab - Revision history

Kdiab: /* Spring 2015 (RA) */

2015-01-30T01:56:35Z

Spring 2015 (RA)

Kdiab: /* January 12 */

2015-01-13T08:01:16Z

January 12

Kdiab: /* January 12 */

2015-01-13T08:00:56Z

January 12

Kdiab: /* Spring 2015 (RA) */

2015-01-13T08:00:33Z

Spring 2015 (RA)

Kdiab: /* Fall 2014 (RA) */

2015-01-13T07:44:43Z

Fall 2014 (RA)

Kdiab: /* October 02 */

2014-12-16T03:51:50Z

October 02

Kdiab: /* October 02 */

2014-10-02T22:33:24Z

October 02

Kdiab: /* October 02 */

2014-10-02T22:31:58Z

October 02

Kdiab at 21:04, 2 October 2014

2014-10-02T21:04:27Z

Kdiab: /* Summer 2014 (RA) */

2014-08-17T00:34:16Z

Summer 2014 (RA)

← Older revision		Revision as of 01:56, 30 January 2015
Line 2:		Line 2:

	* '''Courses:''' NA		* '''Courses:''' NA
		+
		+	=== January 29 ===
		+	* 3D Streaming System
		+	** Three centralized algorithms for storage management.
		+	** Some assumptions on video library and system resources.
		+	** Tried different orderings and assumptions to avoid eviction and make the analysis easier.
		+	** No combination of ordering or assumption help in avoid eviction
		+	** The simple algorithm is optimal (and correct) but not realistic
		+	** The approx. algorithm is polynomial but may be incorrect (with no eviction).
		+	** The approx. algorithm is correct but exponential (with eviction).
		+

	=== January 12 ===		=== January 12 ===

@@ Line 11: / Line 11: @@
 *** calculation of replication factor as described in the report
 ** To be discussed in the meeting (from my side):
-***The current model has one issue: when the replication factor > 1, at most one replica is processed not stored. For example, is replication factor is five, we can have 4 stored versions and 1 processed one. (The resources are not exceeded). We can solve this using non-linear constraint, but this would complicated things?
+***The current model has one issue: when the replication factor > 1, at most one replica is processed not stored. For example, is replication factor is five, we can have 4 stored versions and 1 processed one. (The resources are not exceeded). We can solve this using non-linear constraint, but this would complicate things?
 *** The multi-site data center is still in progress.
 *** The distributed algorithm would work with the new model, but need to add the "streaming servers" semantic to it.

@@ Line 10: / Line 10: @@
 *** static assignment of streaming servers, and
 *** calculation of replication factor as described in the report
-** To be discussed in the meeting:
+** To be discussed in the meeting (from my side):
 ***The current model has one issue: when the replication factor > 1, at most one replica is processed not stored. For example, is replication factor is five, we can have 4 stored versions and 1 processed one. (The resources are not exceeded). We can solve this using non-linear constraint, but this would complicated things?
 *** The multi-site data center is still in progress.

← Older revision		Revision as of 08:00, 13 January 2015
Line 1:		Line 1:
	= Spring 2015 (RA) =		= Spring 2015 (RA) =
		+
		+	* '''Courses:''' NA
		+
		+	=== January 12 ===
		+	* 3D Streaming System
		+	** Model modification to reflect the demand, IO bandwidth, and network bandwidth. Also, the processing times and capacities are more accurate now.
		+	** The replication factor depends on the version demands, watching probability and streaming servers.
		+	** I considered a model where the streaming servers are learned during the optimization given the replication factor, the model is slow even using a small data set. I believe that learning the replication factor from the optimization would result in the same slow optimization, so I went back to:
		+	*** static assignment of streaming servers, and
		+	*** calculation of replication factor as described in the report
		+	** To be discussed in the meeting:
		+	***The current model has one issue: when the replication factor > 1, at most one replica is processed not stored. For example, is replication factor is five, we can have 4 stored versions and 1 processed one. (The resources are not exceeded). We can solve this using non-linear constraint, but this would complicated things?
		+	*** The multi-site data center is still in progress.
		+	*** The distributed algorithm would work with the new model, but need to add the "streaming servers" semantic to it.

	= Fall 2014 (RA) =		= Fall 2014 (RA) =

← Older revision		Revision as of 07:44, 13 January 2015
Line 1:		Line 1:
		+	= Spring 2015 (RA) =
		+
	= Fall 2014 (RA) =		= Fall 2014 (RA) =
	* '''Courses:'''		* '''Courses:'''

@@ Line 3: / Line 3: @@
 ** CMPT-726: Machine Learning
 ** CMPT-880: Cloud Computing
 === October 02 ===

@@ Line 11: / Line 11: @@
 **** We have a distributed in-memory key-value store which stores the whole system state. This one is replicated to a disk periodically. Each manager updates this key-value store periodically, so we don't lost information. Specifically, manager's local memory acts as a L1 cache for this key-value store.
 *** Storage System: as before, couple of SSDs and HDDs besides the original videos.
-*** Streaming servers: machines are responsible of the actual streaming from the "Storage System" to the end user. The "streaming manager" selects a "streaming server" and negotiates with it to stream the requested segment. Once agreed, the "streaming server" starts the actual streaming. To reduce the overhead of streaming server<->manager communication, thy negotiate to stream X segments, not just one segment.
+*** Streaming servers: machines are responsible of the actual streaming from the "Storage System" to the end user. The "streaming manager" selects a "streaming server" and negotiates with it to stream the requested segment. Once agreed, the "streaming server" starts the actual streaming. To reduce the overhead of streaming server<->manager communication, thy negotiate to stream X segments, not just one segment. The buffer management part is not clear yet. But, the "streaming server" itself has a memory replacement scheme (e.g. LRU) but evicts the segments to its local disk. Also, each segment fetched from the storage system is replicated to K nearest streaming servers local disks, they will help in recovery and peer-assisted streaming. For example, when a "streaming server" detects that it is loaded (bandwidth or memory), it redirects its streaming responsibility to a nearest streaming server, and updates the manager.
 *** Haertbeat-based status update, we have tested this technique before, and was fine to the task.
 ** I defined recovery routines for each module:

@@ Line 5: / Line 5: @@
 === October 02 ===
-* 3D Streaming
+* 3D Streaming System
 * Courses
 ** Machine Learning: submitted an assignment (coding + math questions) this Monday.
 ** Cloud Computing: regular material readings (this week: MapReduce, GFS paper, and two students paper + Spark streaming paper)
 = Summer 2014 (RA) =

← Older revision		Revision as of 21:04, 2 October 2014
Line 1:		Line 1:
		+	= Fall 2014 (RA) =
		+	* '''Courses:'''
		+	** CMPT-726: Machine Learning
		+	** CMPT-880: Cloud Computing
		+
		+	=== October 02 ===
		+	* 3D Streaming
		+
		+	* Courses
		+	** Machine Learning: submitted an assignment (coding + math questions) this Monday.
		+	** Cloud Computing: regular material readings (this week: MapReduce, GFS paper, and two students paper + Spark streaming paper)
		+
		+
		+
	= Summer 2014 (RA) =		= Summer 2014 (RA) =
	* '''Courses:''' NA		* '''Courses:''' NA

@@ Line 1: / Line 1: @@
 = Summer 2014 (RA) =
 * '''Courses:''' NA
 === June 23 ===