Multi Attribute D-S Evidence Theory Based OCC for Shared-State Scheduling in Large Scale Cluster

Libo He, Zhenping Qiang, Wei Zhou, Shaowen Yao

Abstract


With the growth of big data problems, nowadays the size of cloud-scale computing clusters is growing rapidly to run complicated parallel processing jobs. To full utilize cluster resources, the cluster management system is being challenged by the scaling cloud size and the often more complicated application requirements. Omega scheduling software provides a flexible and scalable shared-state scheduling architecture for large scale cluster scheduling. One of its key ideas is using an optimistic concurrency control (OCC) algorithm to let parallel schedulers concurrently make decisions. However, there are few studies exploring to extend OCC for a shared-state scheduling architecture. Furthermore, most of the traditional’ shared-state scheduling architectures also use the same OCCs as Omega does. In this paper, we present a multi attributeDempster–Shafer (D-S) evidence theory based OCC for shared-state scheduling. This OCC adaptsthe multi attribute D-S evidence theory to help making conflict decisions for some scheduling transactions. Experiments’ results show that our method can obtain in some respects more optimized scheduling results compared to coarse-grained conflict detection of Omega.


Keywords


large scale cluster scheduling; multi attribute D-S evidence theory; optimistic concurrency control; Shared-state scheduling

Full Text:

PDF



International Journal of Online Engineering (iJOE).ISSN: 1861-2121
Creative Commons License
Indexing:
Web of Science ESCI logo Engineering Information logo INSPEC logo DBLP logo ELSEVIER Scopus logo EBSCO logo Ulrich's logoGoogle Scholar logo Microsoft® Academic Search