Till KTH:s startsida Till KTH:s startsida

Meteor vs Pig

Pig (http://pig.apache.org/) is a platform for analyzing large data sets that consists of a high-level language and its compiler produces sequences of Map-Reduce programs, which are then executed on top of the Hadoop execution engine.

Meteor (http://bit.ly/QnX7T7) is a similar system, which is part of the Stratophere (http://stratosphere.eu/) stack. A Meteor program is transformed into a DAG of jobs which are run on the Nephele execution engine respectively.

Your task is to compare these systems, based on benchmarks similar to  PigMix and implement them both in PIg and Meteor. The evaluation should include a comparison of the expressivness of these systems as well as a presenentation of the performance both on the high-level and the lower programming level (Map-Reduce and PACT).

The objectives of this project are the following:

  • Get familiar with Stratosphere/PACTs
  • Learn how to use Pig and Meteor
  • Develop benchmark applications for Meteor, based on PigMix
  • Design and execute performance comparison experiments

Supervisor: Vasiliki Kalavri

Lärare Johan Montelius skapade sidan 14 september 2012

Vasiliki Kalavri redigerade 14 september 2012

Pig, or Pig Latin, is a programming language for MapReduce specifications. Its main target is the Hadoop execution engine but could also be us (http://pig.apache.org/) is a platform for analyzing large data sets that consists of a high-level language and its compiler produces sequences of Map-Reduce programs, which are then executed ion other execution environments. Meteor is a similar system but the target is the Stratophertop of the Hadoop execution engine.¶

Meteor (http://bit.ly/QnX7T6) is a similar system, which is part of the Stratophere (http://stratosphere.eu/) stack. A Meteor program is transformed into a DAG of jobs which are run on the Nephele execution engine respectively.

Your task is to compare these systems, based on benchmarks similar to  PigMix and implement them both in PIg and Meteor. The evaluation should include a comparison of the expressivness of these systems as well as a presenentation of the performance both on the high-level and the lower programming level (Map-Reduce and PACT).

The objectives of this project are the following:¶


* Get familiar with Stratosphere/PACTs
* Learn how to use Pig and Meteor
* Develop benchmark applications for Meteor, based on PigMix
* Design and execute performance comparison experiments

Supervisor: Vasiliki Kalavri

Johan Montelius flyttade sidan från Implementation of Distributed Systems (ID2219) 27 augusti 2013

Johan Montelius flyttade sidan från HT 2012 TDISM 27 augusti 2013

Johan Montelius flyttade sidan från Implementation of Distributed Systems (ID2219) 27 augusti 2013