Job Performance Optimization Method Based on Data Balance in the Wireless Sensor Networks
DOI:
https://doi.org/10.3991/ijoe.v13i12.7882Keywords:
wireless sensor networks, MapReduce, hash function, data skew, heterogeneity-awareAbstract
In the wireless sensor network, the representative MapReduce computing model based on data center has been widely used in large-scale data processing. In the data transmission phase, the wireless sensor network system uses the hash method to distribute data for each Reduce task based on the number of Reduce tasks. This data partitioning method based on the hash function results in non-uniform distribution of the output data in the data transmission phase and further leads to skewing of the input data in the Reduce task. Data skew will result in load imbalance in the Reduce phase and causes the system performance to degrade. In order to eliminate the data skew problem in the Reduce phase, this paper presents a load balancing method, which consists of two parts: the virtual partitioning method based on the consistent hashing and the heterogeneity-aware loads balancing (HLB) algorithm. The experimental results show that the proposed method can eliminate the data skew in the Reduce phase and distribute the load equitably for each Reduce task. In addition, the method produces less system overhead.
Downloads
Published
How to Cite
Issue
Section
License
The submitting author warrants that the submission is original and that she/he is the author of the submission together with the named co-authors; to the extend the submission incorporates text passages, figures, data or other material from the work of others, the submitting author has obtained any necessary permission.
Articles in this journal are published under the Creative Commons Attribution Licence (CC-BY What does this mean?). This is to get more legal certainty about what readers can do with published articles, and thus a wider dissemination and archiving, which in turn makes publishing with this journal more valuable for you, the authors.
By submitting an article the author grants to this journal the non-exclusive right to publish it. The author retains the copyright and the publishing rights for his article without any restrictions.