HPDC 2013 计算机体系结构相关国际会议 review评审意见

HPDC 2013 计算机体系结构相关国际会议 review评审意见 （信息供参考学习）

我由Eurosys'2013转投HPDC'2013的一篇与多级缓存替换算法研究相关的论文，录用结果是reject。供大家学习经验教训吧！

Review of pap146s1 by Reviewer 1

top

Does this
paper address an important issue?
This paper presents two algorithms that implement an adaptive multi-level exclusive cache, able to dynamically adapt their replacement and placement strategies according to some access patterns detected in real-time. This
is a valuable contribution in a context where the democratization of large scale resources brought by the emergence of clouds built on top of data centres also faces us with the challenge of effectively managing large amounts of I/O buffer caches and multi-level
cache hierarchies.

The ReDARC takes an original path and adopts reuse distance as a means of locality measure instead of the widely adopted recency, used by LRU based approaches. By dividing accessed blocks into small and large reuse distance sets the algorithm is able to guarantee
that all SRD block are always cached in memory by means of dynamically balancing the size of the two sets according to the two access patterns. This approach might by quite useful for the community since it addresses a long studied challenge of multi-level
hierarchy caches, that of the weakened locality.

The second algorithm allows lower caches in the hierarchy to push blocks to higher one and used some hints from the previous algorithm to set blocks' locations. Again, this has a practical relevance for the cache environment since it provides an interesting
approach to deal with data redundancy.


Does this
paper present convincing results?
The authors make a thorough evaluation of their solutions with both synthetic and real-life traces by means of simulation. While the analysis and the results presented are rather convincing, what I think is missing is an
evaluation with a real system. This is mentioned as future work, but I think it would clear the impact of the two algorithms in a real scenario. For instance, think at a private cloud (using some open source cloud kit like OpenNebula or Nimbus) setup in a
datacenter where multi-level cache hierarchies could bring a benefit both on providers and users side. A discussion on the impact of these approaches on throughput, QoS and costs would be rather interesting.


Is this
paper sufficiently well executed?
I found the paper an interesting reading, with the potential of bringing relevant contributions on improving applications' performance in large data centers. As opposed to static approaches the proposal can achieve a high
aggregate hit ratio, low average response time, and low inter cache traffic.

The paper is rather well written and instructive. Some minor spelling (more likely Latex word splitting) errors should be fixed. For instance "algorithm-s" at the end of section 5.4.1 or the title of section 5.4.2

Review of pap146s1 by Reviewer 2

top

Does this
paper address an important issue?
This paper proposes a new method for managing multi-level exclusive caches. The first part is to use reuse distance as a metric for determining if data should be kept in cache (tracked through a ghost cache). The second
part is to avoid demote operations between caches and instead decide at the lower level cache if the block should be moved or copied to the upper level cache when it is requested.

This paper is well done, but IÕm suspect on its applicability for HPDC. Exclusive caches assume no file sharing, and the authors only tackle single applications with a single client. Neither of these assumptions is realistic in the scale-out design of modern
data centers.


Does this
paper present convincing results?
The scope and assumptions of this paper limit its applicability to modern storage systems.


Is this
paper sufficiently well executed?
This paper is very well written and executed. I applaud the step by step methodology, by first evaluating single level cache performance and then moving on to the multi-level caches.

The related work and motivation are also nicely described, although I feel like the authors never fully justified their claim that reuse distance is the best metric. A nice intuitive answer would have helped the reader understand why the results are better
in the experiments.

The main problem with the paper is one of scope. While the authors claim that simply turning off exclusivity is easy and not worth examining, I donÕt buy it. Modern storage systems are completely non-exclusive, with many clients requesting the same information
(boot image, hot file) or at least pounding away at a single cache with competing demands on a single cache. Some of the papers referenced in the related work do not handle this either, but many of them are 5-10 years old, and scale-out is now the norm.

Also, focusing on read is fine, but would like to see a discussion of how write affects the general design and what general things would change in the presence of writes.

Regarding related work, the authors point out the high overhead of many of the previous multi-level caching papers, but to track the reuse distance for a large shared file server would be almost tantamount to serializing their requests, introducing large overhead.
The authors should explain how this metric can be tracked for parallel requests to a file server in an efficient manner.

Review of pap146s1 by Reviewer 3

top

Does this
paper address an important issue?
The paper proposes a new caching techniques for multi-level exclusive caching. The problem is important because of the importance of cache performance for execution time on most applications. On the other hand, it is a well
mined area and the improvements can only be incremental.

The paper proposes a technique called REAL that combines a single cache policy called ReDARC with a global caching technique called ALACA. Both ReDARC and ALACA are introduced in this paper and proved through extensive simulations that they are more efficient
than previous solutions.


Does this
paper present convincing results?
The caching algorithms introduced in the paper are evaluated experimentally via simulations. The simulator has been developed by other researchers and used for some years now. The workloads used are also from the research
community and mostly collected from real-life applications.

The experiments evaluate ReDARC, which is a single-level cache policy, against other single-level cache policies and show that it performs as well as the best of them (Figure 6). However, when used with the ALACA strategy for global caching, the resulting REAL
techniques yields consistently better results in all three performance metrics considered: hit rate, average response time, and inter-cache traffic.

The results are thus convincing, and while the performance of the single-level cache policy is not impressive by itself, its contribution to the multi-level caching technique is consequential.


Is this
paper sufficiently well executed?
The paper is well written and the experiments seem solid. The Motivation section addresses well the design space that the paper focuses on.

Review of pap146s1 by Reviewer 4

top

Does this
paper address an important issue?
This paper proposed a new algorithm (a.k.a. REAL) to improve the response time of a multi-level cache. REAL has two sub algorithms: ReDARC and ALACA. ReDARC is to capture locality information on a single cache level. ALACA
coordinates all the information on each level (obtained from ReDARC) and achieves better overall response time.

This problem has been extensively studied, since it is critical to the overall performance, especially for the large-scale data-intensive systems. The authors claim that the proposed algorithm outperforms the other state-of-the-art ones.


Does this
paper present convincing results?
There are a few issues with the experiments. (1) All experiments are assuming a single client and a single storage server. This is unacceptable if the proposed work is targeting ¡°most client/server cluster¡±: how could
we know the scalability of the algorithm? (2) The cache hierarchy only has two layers (client cache and server cache), which I think is oversimplified. The presented model has n layers, and setting n=2 sounds like a base case, rather than a general one. (3)
Section 5.4.1 spends three full paragraphs and three plots to compare the performance of ReDARC algorithm to others. However the proposed ReDARC algorithm barely outperforms the existing LIRS algorithm in all the three benchmarks. This is somewhat disappointing,
and then the authors comment this by ¡°the main contribution of this paper is to improve the performance of multi-level cache systems¡±. So I assume the authors do not consider the ReDARC algorithm as a (main) contribution. However, in both abstract and introduction,
ReDARC is clearly highlighted as a proposed approach to better capture locality information. So here comes the question: is ReDARC a contribution of this paper, or just a building block of the other proposed ALACA algorithm? (4) In Section 5.4.2, six benchmarks
were tested upon the proposed REAL (ReDARC+ALACA) algorithm. I agree that the average response time (in Figure 8) makes more sense than the hit rate (in Figure 7) for a multi-level cache. However, when discussing the results of average response time, the authors
only focus on the ¡°Websearch1¡± benchmark, which seems to be the easiest one to explain. For example, in ¡°Financial1¡± benchmark, ¡°REAL¡± is worse than ¡°gloLRU¡± in some cases. It is OK that the algorithm is not perfect for all cases, but it would be more
useful if the authors could explain why that happens.


Is this
paper sufficiently well executed?
As for technical soundness, I have three questions. (1) The proposed algorithms involve block migration between different queues and some loops to maintain the queue¡¯s property. So what is the time/space complexity of the
algorithms? (2)Why should the S and EL queues have the doubled size of the actual cache? (3) How do you come up with the adjust_push equation in Figure 4, i.e. PUSH += (1 - PUSH) * PUSH * delta? Is this from some literature? Or did you prove this an optimal
solution? Or is this just from empirical data? Have you tried other adjust_push equations?

The presentation of this paper needs improvement. (1) The definitions of ¡°higher-level cache¡± and ¡°lower-level cache¡± should be given at the very beginning of the paper, rather than clearly depicted in Figure 5 on Page 7. Even in Figure 5, the naming is
still confusing: why is a *higher* level cache called ¡°cache_1¡±, and a *lower* level cache called ¡°cache_n¡±? (2) Section 3 ¨C Design Motivation, is almost a repetition of Section 2 with very little ¡°motivation¡±. And the writing is poor in this section
with a lot of grammar mistakes. (3) Section 4.2 has a couple of grammar errors. (4) I would not put Section 5.1 in the section of performance evaluation. The presented model seems more like the assumption, or model, based on which the proposed algorithms would
be applied. That is, this model is not specific to the experiments, but the theoretical basis. (5) Still in Section 5.1, I agree that minimizing Equation [2] is more important than Equation [1]. But then the authors turned to talk about minimizing Equation
[3], and did not explain the relationship between Equations [2] and [3]. Well I could guess the reasoning being something like ¡°t_i = SUM(Seq_i + D_i)¡±. Nevertheless, this should be explicitly presented, rather than asking the readers to wonder.

Furthermore, it is not clear how this work is connected to the HPDC community, as it does not present results in either multi/many-core computing, or cluster computing contexts.

Review of pap146s1 by Reviewer 5

top

Does this
paper address an important issue?
The paper addresses the problem of caching in multi-level cache hierarchies; although some algorithms exist, there can be room for improvement.

The paper claims two contributions: an algorithm for a single cache maintenance, called ReDARC (for Reuse Distance based Adaptive Replacement Caching); and an multi-level caching algorithm called ALACA (for Adaptive Level-Aware Caching Algorithm) that uses
probabilistic pushing based on individual cache indicators as given by ReDARC.

The writing lacks clarity and makes both algorithms hard to understand and assess technical merit. As far as I could tell, the first contribution is doubtful as the ReDARC algorithm appears to be very close to LIRS [16].

The second algorithm seems to be a variant of probPROMOTE (in [11]) that uses LRU stack depth instead of time difference between least-recently and most-recently elements in the LRU list. This might still be a contribution, albeit minor. The criticism directed
at probPROMOTE, that it requires timestamp recording and periodic hint transferring, is hard to understand given that ALACA relies on hint transferring with each miss; in any case, there are no experiments to measure this purported high overhead and compare
with the ALACA overhead.


Does this
paper present convincing results?
Not quite. I found it troubling that the ReDARC algorithm is so similar to LIRS, yet there is no discussion or even intuition given of why it should perform better than it. The description is rather unclear, and would have
been harder still to understand if not for similarity to LIRS. There are parts where it seems outright wrong: for example Figure 2 says "Delete from S; move X to the SRD set", but SRD elements can only be found in S! I think a state transition diagram would
have helped increase clarity.

Same with ALACA; it is very similar to [11] without saying why it should do better; is it because the individual caches are managed by ReDARC as opposed to other algorithms?

The experimental section compares REAL (ReDARC+ALACA) against Promote-LRU but not also against Promote-ARC, although the latter is shown to be better in [11].


Is this
paper sufficiently well executed?
The closeness of the algorithm to work published by others, combined with the poor writing style that avoids explaining clearly the relation to them, make me doubt that there is any new result here (although there might
be one).

Since the experimentation was done through simulation, it would have been nice to also show the performance of OPT-LBand OPT-UB (offline algorithms) from [11].

Review of pap146s1 by Reviewer 6

top

Does this
paper address an important issue?
The paper presents the design and evaluation of an adaptive multi-level exclusive cache policy for client-server systems. In particular, the proposal consists of two (interacting) components: (i) an adaptive reuse distance
based replacement policy, and (ii) an adaptive multi-level exclusive placement policy. The former dynamically balances a small and a large reuse distance set, which includes resident and non-resident blocks. The latter uses a hint-based mechanism to drive
placement decisions across different cache levels.

The design of efficient cache hierarchies has received a lot of attention in the past decades (especially in the context of processor and chip multi-processor design). The authors focus on cache hierarchies for client-server systems and workloads, and identify
some directions for improvement. The discussion and the evaluation are sound and thorough. In general, this work improves the understanding of cache hierarchies for client-server systems. The evaluation is done through simulation; however, if implemented this
paper can have practical relevance.


Does this
paper present convincing results?
The evaluation section is well organized and discussed, and the results presented are therefore convincing.

In particular, the experimental section has the following strengths: (i) the authors use a variety of traces with different characteristics; (ii) the authors present an exhaustive comparison with 4 single-level and 6 multi-level (inclusive and exclusive) caching
algorithms; (iii) the metrics used in the evaluation are sound and well-motivated; (iv) the discussion of the results is clear.

One aspect which is missing in the experimental section is a discussion of the overhead due to storing and managing the meta-data needed by the proposed caching policies (and a comparison with the other considered caching schemes). For example: what is the
overhead of the ghost cache? What is the implementation cost and the runtime overhead of the proposed schemes?


Is this
paper sufficiently well executed?
In general, I believe that the paper is well executed. The state of the art is well referenced and discussed. The design choices are well motivated and discussed. The evaluation section is well organized.

The presentation is in general clear. However, the paper contains some typos and need proofreading. For example: - Page 3, column 2 “relatively small space overhead that …” - Page 4, column 1 “send” -> “sent”, “usefulness”?, “demoting blocks to lower caches
meanwhile achieves” - Page 5, column 2: “hint messages that indicate data blocks should stay in which level cache”