Private:progress-hegazy

Sep 24-28

Accomplished (this week)
- I registered the paper abstract in IPDPS.
- I ported the paper to LaTeX and worked on the abstract, introduction, related work and problem definition.
- Ahmed's code now works with most TCP clients and servers.
Issues
- We are running the experiments now (Friday night) and time is tight for the IPDPS submission.
Next Steps
- We will finish as much experiments as we can Friday and Saturday.
- I should have the paper (almost) ready Saturday night.

Accomplished (this week)
- I had a couple of meetings with Abdullah; he came up with some file access logs from a benchmark
- We ran the partial autocorrelation function on it to get the Markov order.
- I up with a rough state definition based on the requested blocks (or rather "chunks") and storage status. I also came up with a rough definition for the reward/penalty function based on the same info.
Issues
- Progress is a bit slow, but not too bad.
Next Steps
- Abdullah is to get more logs from other real HPC applications
- I will continue to define the Markov Decision Process.

Accomplished (this week)
- I had a couple of meetings with Ahmed where I explained to him in more details what experiments to run.
- Ahmed prepared the testbed based on new cloud instances. He is very close to getting the results.
- I had a Skype call with Ali to explain to him almost the same thing as Ahmed, though Ali is a little behind Ahmed.
Issues
- Progress is a bit too slow; we could have easily obtained results this week, but we didn't.
Next Steps
- Ahmed should come up with the results soon.
- More experiments (this time, on random delay) are to be conducted.
- Ali should be able to run the same experiments soon (just to get familiar; nothing new to accomplish).

Accomplished (this week)
- Ahmed bought and successfully installed the Infiniband cards.
- Benchmark experiments were run for Infiniband.
Issues
- Ahmed's implementation of SDP doesn't support asynchronous calls.
- Ahmed couldn't run PCIe and Infiniband experiments from the same machine (library conflict between his SDP and Infiniband SDP)
- Time is tight for the IPDPS submission.
Next Steps
- Ahmed will implement asynchronous call support in order to run his application-based evaluation and compare it to Infiniband.
- I will register the paper's abstract to IPDPS over the weekend (deadline is Monday, Sept 24th).
- Although the time is tight, Ahmed will try to finish his asynchronous calls support and the application-based evaluation by the end of next week. This gives us a couple of days to polish up the paper.

Accomplished (this week)
- Abdullah tried to obtain HPC I/O logs off of the internet. When that was not possible, he tried a couple of benchmarks to to generate some I/O logs as a placeholder until we obtain logs from real HPC applications.
- During our meeting, we tried to analyze the data, e.g., to verify the Markov chain order using partial autocorrelation functions. I noticed the data was so fine-grained (time in nanoseconds and block size in KB).
Issues
- None
Next Steps
- Abdullah to work on the coarser representation of the log (e.g., 1 second and MBs).
- I will go into one more level of details for the general methodology.

Accomplished (this week)
- I have come up with the delay compensation results and updated the working document.
- I have met with Ahmed AbdelSadek and explained the problem and experiments to him.
- Ali finished his first experiment, which is nothing other than making sure Modbus is working fine for him on LabVIEW.
Issues
- Ahmed was supposed to finish his first task by the end of the week, but he was "discussing some issues with Dr. Mohamed regarding [his] research".
- Communication rate with Ali is a bit slow.
Next Steps
- I may have to work on the delay experiments myself until Ahmed is up to speed.
- Ahmed will try to accomplish the tasks rolled over from this week.
- I will have conference call with Ali to explain to him the next steps.

Accomplished (this week)
- I registered the paper abstract in IPDPS.
- I ported the paper to LaTeX and worked on the abstract, introduction, related work and problem definition.
- Ahmed's code now works with most TCP clients and servers.
Issues
- We are running the experiments now (Friday night) and time is tight for the IPDPS submission.
Next Steps
- We will finish as much experiments as we can Friday and Saturday.
- I should have the paper (almost) ready Saturday night.

Accomplished (this week)
- I had a couple of meetings with Abdullah; he came up with some file access logs from a benchmark
- We ran the partial autocorrelation function on it to get the Markov order.
- I up with a rough state definition based on the requested blocks (or rather "chunks") and storage status. I also came up with a rough definition for the reward/penalty function based on the same info.
Issues
- Progress is a bit slow, but not too bad.
Next Steps
- Abdullah is to get more logs from other real HPC applications
- I will continue to define the Markov Decision Process.

Accomplished (this week)
- I had a couple of meetings with Ahmed where I explained to him in more details what experiments to run.
- Ahmed prepared the testbed based on new cloud instances. He is very close to getting the results.
- I had a Skype call with Ali to explain to him almost the same thing as Ahmed, though Ali is a little behind Ahmed.
Issues
- Progress is a bit too slow; we could have easily obtained results this week, but we didn't.
Next Steps
- Ahmed should come up with the results soon.
- More experiments (this time, on random delay) are to be conducted.
- Ali should be able to run the same experiments soon (just to get familiar; nothing new to accomplish).

Accomplished (this week)
- Ahmed came up with results on application-level testing using Apache-wget
- I have finished up writing the IPDPS paper and sent it for Dr. Hefeeda to review.
- I have addressed at least 80% of Dr. Hefeeda's comments and submitted the paper to IPDPS
Issues
- None
Next Steps
- Ahmed is working on updating his report and writing his thesis
- I have some leftover comments that I will need to address in the paper.

Accomplished (this week)
- Abdullah came up with I/O logs from the benchmark
- During a one-on-one, I ran partial auto-correlation function (PACF) on the log, and the result is a first-order Markov Chain
Issues
- None
Next Steps
- Abdullah will come up with logs with the same format from HPC applications.
- We will start working on defining our state space, action space, and reward/penalty function.

Accomplished (this week)
- Ahmed successfully ran the delay experiments and obtained results.
- I asked for additional experiments for a different scan cycle, which he ran on the spot! and gave me the results.
- The results show robustness for all data points (up to 6.4 seconds of round-trip delay)
Issues
- None
Next Steps
- Ahmed will run an experiment for dual redundancy
- I am about to decide on the delay estimator to feed Smith Predictor (for variable delay cases).