HPC and Cloud Computing

Author: 
Philippe Olivier Alexandre Navaux, Eduardo Roloff, Emmanuell Diaz Carreño, Jimmy Valverde Sanchez
Focus Area: 

The area of our position is not directly related to the focus themes above, but it is related to the HPC communitty and the benefits and dificulties to adopt the Cloud as an environment to execute HPC applications.

Who stands to benefit and how : 

General HPC users, with most benefits of users that do not have access to large facilities and coul benefit from the elasticitity and pay-per-use model of cloud computing.

Position Paper: 

The Cloud Computing model has arrived as a viable alternative to clusters and supercomputers that traditionally have been used to execute High-Performance Computing (HPC) applications. By using the cloud, it is possible to configure experimental or production environments without upfront investments. Moreover, due to the elasticity characteristics of the model, it is possible to have access to a virtually unlimited amount of resources. These two are the main keys of the cloud computing paradigm that attract the attention of the HPC community.

However, the cloud is still not ready to substitute traditional HPC environments for all kinds of user needs. In the following text, we will point out some of the main issues of the model. One big problem of using the cloud is its heterogeneity and the virtualization of the cloud resources. Today, we do not have a cloud manager that is HPC oriented. For this reason, the allocation of machines in the cloud is not predictable in terms of network topology and resource isolation. The virtualization adds an extra layer that causes performance loss and could introduce noise in the execution as well. To address these problems, we need to implement a resource allocation strategy that takes into account the characteristics of HPC applications to provide an HPC cloud environment.

Additionally, it is necessary to explore the possibility of lightweight virtualization as an alternative to decrease the overhead of current virtualization technologies. The performance bottlenecks are also an issue for effective HPC in the Cloud. The biggest challenge in the cloud is the network performance, because normally the cloud providers use commodity interconnections that do not provide adequate performance for HPC applications. The disk I/O performance is also a key issue for HPC that presents performance losses in cloud environments. The main reasons for these two issues are the resource sharing normally used in the cloud, where several virtual machines are allocated on the same physical hardware. It is important to note that the CPU isolation technology is much more developed than the virtualization of network and disk I/O. The CPU performance in the cloud is close to the performance obtained in the physical processor. It is necessary to create virtualization techniques for network and disk I/O similar to the ones used in CPU virtualization. Another possibility is to create physical machines designed to be used for HPC in the cloud, with more network interfaces and disk controllers that could be allocated with more isolation levels. Clearly this strategy needs deeper studies to understand its economic viability. On the user side, it is possible to explore the development of applications and runtime libraries that could benefit from the cloud characteristics. Most importantly, the benefits of elasticity need to be explored, because the applications could use this capability to reduce the execution costs or to improve the overall performance. Also, HPC runtimes could be extended to map the application needs to provide a better match to the cloud environments. As a concluding remark, we understand that cloud computing has a tremendous potential to be used for HPC, but it is not yet fully compatible with all types of HPC applications.

EU EUBrasilCloudFORUM is funded by the European Commission under the Cooperation Programme, Horizon 2020 grant agreement No 689495. Este projeto é resultante da 3a Chamada Coordenada BR-UE em Tecnologias da Informação e Comunicação (TIC), anunciada pelo Ministério da Ciência, Tecnologia e Inovação (MCTI), no âmbito to acordo de subvenção Número 689495. Privacy Policy | Disclaimer Terms of use