Private:progress-dameh - Revision history

Tdameh at 01:19, 27 January 2012

2012-01-27T01:19:08Z

2011-11-11T00:13:31Z

2011-11-01T23:29:55Z

2011-10-26T06:02:34Z

2011-10-26T05:58:06Z

2011-10-12T09:40:53Z

2011-10-12T09:24:50Z

2011-10-12T09:20:16Z

2011-10-12T09:20:01Z

2011-10-12T09:18:54Z

← Older revision		Revision as of 01:19, 27 January 2012
Line 1:		Line 1:
−	= Fall 2011 (RA) =	+	= Fall 2011 and Spring 2012 (RA) =
		+
		+
		+	=== Jan 27 ===
		+	* Implementing my method on cluster, report to be sent soon.
		+
		+
		+	=== Dec 23 ===
		+	* Finished the study of different methods, the analysis and the implementation on a single machine.

← Older revision		Revision as of 00:13, 11 November 2011
Line 1:		Line 1:
	= Fall 2011 (RA) =		= Fall 2011 (RA) =
		+
		+
		+	=== Nov 9 ===
		+	* Scaled the Spectral Hashing.
		+	* Working on the US Census Data (1990) [number of features (dimensions) = 69 ]
		+	* run experiments on small DataSet (4k) using the 3 methods LSH Random Projection, LSH Spectral Hashing and Hilbert Curve
		+	* now running on large DataSet (1M) using LSH only (as H-curve doesn't work on large Scale)
		+	* experiment results to be added to the report

@@ Line 7: / Line 7: @@
 ** The concept of distance becomes less precise as the number of dimensions grows.
 ** There are other techniques to cluster the high-dimensional data (no usage of gram matrix) such as Subspace Clustering, Projected Clustering and Correlation Clustering.
-* I am looking for other real DataSets with low dimension (~100)
+* I start working on different real DataSets with low dimension (~100)

@@ Line 3: / Line 3: @@
 === Oct 26 ===
-* I am facing problems working on very high dimensional (and sparse) data (d~100000) [curse of dimensionality]
+* I am facing problems working on very high dimensional data (d~100000) [curse of dimensionality]
-** I can't use small data sets (1k points) to compare different methods (Hilbert curve fails for high dimensional but LSH still works)
+** I can't use small dataSets (1k points) to compare different methods (Hilbert curve fails for high dimensional but LSH still works)
-** The concept of distance becomes less precise as the number of dimensions grows
+** The concept of distance becomes less precise as the number of dimensions grows.
 ** There are other techniques to cluster the high-dimensional data (no usage of gram matrix) such as Subspace Clustering, Projected Clustering and Correlation Clustering.
 * I am looking for other real DataSets with low dimension (~100)

← Older revision		Revision as of 05:58, 26 October 2011
Line 2:		Line 2:


		+	=== Oct 26 ===
		+	* I am facing problems working on very high dimensional (and sparse) data (d~100000) [curse of dimensionality]
		+	** I can't use small data sets (1k points) to compare different methods (Hilbert curve fails for high dimensional but LSH still works)
		+	** The concept of distance becomes less precise as the number of dimensions grows
		+	** There are other techniques to cluster the high-dimensional data (no usage of gram matrix) such as Subspace Clustering, Projected Clustering and Correlation Clustering.
		+	* I am looking for other real DataSets with low dimension (~100)
		+
		+
	=== Oct 12 ===		=== Oct 12 ===
	* Process Very-High dimensional and Very-Large Scale real DataSets		* Process Very-High dimensional and Very-Large Scale real DataSets

@@ Line 4: / Line 4: @@
 === Oct 12 ===
 * Process Very-High dimensional and Very-Large Scale real DataSets
-** I chose document clustering, where each document i is commonly represented by a term-frequency vector xi = [x1i, x2i, ..., xDi], where D is the total number of keywords in the given document corpus, and xji is the number of occurrences of keyword j in document i. Keywords correspond to the words that remain after the pre-processing of stop-words removal and words stemming. Below N is the total number of documents.
+** I use document clustering, where each document i is commonly represented by a term-frequency vector xi = [x1i, x2i, ..., xDi], where D is the total number of keywords in the given document corpus, and xji is the number of occurrences of keyword j in document i. Keywords correspond to the words that remain after the pre-processing of stop-words removal and words stemming. Below N is the total number of documents.
 * DataSets: source: http://archive.ics.uci.edu/ml/datasets/Bag+of+Words:
 ** NYTimes news articles:

@@ Line 4: / Line 4: @@
 === Oct 12 ===
 * Process Very-High dimensional and Very-Large Scale real DataSets
-** I chose document clustering, where each document i is commonly represented by a term-frequency vector xi = [x1i, x2i, ..., xDi], where D is the total number of keywords in the given document corpus, and xji is the number of occurrences of keyword j in document i , below N is the total number of documents.
+** I chose document clustering, where each document i is commonly represented by a term-frequency vector xi = [x1i, x2i, ..., xDi], where D is the total number of keywords in the given document corpus, and xji is the number of occurrences of keyword j in document i. Keywords correspond to the words that remain after the pre-processing of stop-words removal and words stemming. Below N is the total number of documents.
-* DataSets (http://archive.ics.uci.edu/ml/datasets/Bag+of+Words):
+* DataSets: source: http://archive.ics.uci.edu/ml/datasets/Bag+of+Words:
 ** NYTimes news articles:
 *** orig source: ldc.upenn.edu
 *** N=300000
 *** D=102660
+** PubMed abstracts: (PubMed comprises more than 21 million citations for biomedical literature from MEDLINE, life science journals, and online books)
 *** orig source: www.pubmed.gov
 *** N=8200000
 *** D=141043
+* Currently I process 1M documents from PubMed with the dimesion 141043, total size of this DataSet on HD is about 250GB.
 * To Do: compare the performance using two different LSH families.

@@ Line 18: / Line 18: @@
 * currently I process 1M documents from PubMEd with the dimesion 141043, total size of this DataSet on HD about 250GB.
 * To Do: compare the performance using two different LSH families.
 === Spet 28 ===

← Older revision		Revision as of 09:18, 12 October 2011
Line 1:		Line 1:
	= Fall 2011 (RA) =		= Fall 2011 (RA) =
		+
		+
		+	=== Oct 12 ===
		+	* Process Very-High dimensional and Very-Large Scale real DataSets
		+	** I chose document clustering, where each document i is commonly represented by a term-frequency vector xi = [x1i, x2i, ..., xDi], where D is the total number of keywords in the given document corpus, and xji is the number of occurrences of keyword j in document i , below N is the total number of documents.
		+	* DataSets (http://archive.ics.uci.edu/ml/datasets/Bag+of+Words):
		+	** NYTimes news articles:
		+	*** orig source: ldc.upenn.edu
		+	*** N=300000
		+	*** D=102660
		+
		+	** PubMed abstracts:
		+	*** orig source: www.pubmed.gov
		+	*** N=8200000
		+	*** D=141043
		+
		+	* currently I process 1M documents from PubMEd with the dimesion 141043, total size of this DataSet on HD about 250GB.