Storm Hackathon Getting Started

Getting Started

This will help you get started with Storm. At the end of this exercise, you should be able to submit topologies to the Storm cluster.

Prerequisites

  1. A sane shell (Cygwin may or may not work)
  2. git
  3. A JDK
  4. Maven
  5. A fork of the storm-hackathon repo
  6. A team name (this will be used to partition your topologies and data from other teams)

Setting up your Environment

Here, we’ll clone your forked repository, build the example topologies and submit one to the Storm cluster.

export TEAM_NAME=<my-team-name>
git clone https://github.com/<your github.com account>/storm-hackathon
cd storm-hackathon
mvn package
./tools/submit org.hackreduce.storm.example.ExclamationTopology cluster

If all of these commands were successful, you should now see your topology running in the Storm UI. It should appear as <my-team-name>-Exclaim.

You may now kill this example topology:

./tools/kill <my-team-name>-Exclaim

Note that if you don’t specify a team name (through the TEAM_NAME environment variable), the helper code will pick one at random for you. Set the TEAM_NAME environment variable to get a constant prefix.

Hack!

Congradulations, you are now ready to create your own topologies and submit them in the same way: simply change the name of the main class when submitting.

Other example topologies are available in your forked repository. You can start from one of those or from scratch.

In order to submit a topology prefixed with your team’s name, use the org.hackreduce.storm.HackReduceStormSubmitter utility class or simply make sure you name your topologies accordingly.

Storm VM

If you would like to run a mini Storm cluster in a Vagrant vm, follow these instructions:

  1. Install VirtualBox
  2. Install Vagrant

  3. Initialize Vagrant project (this will create a Vagrantfile)

  4. Add the following line to your Vagrantfile 

    config.vm.network :private_network, ip: "192.168.101.11"

  5. Start VM 

    vagrant up

  6. open storm ui http://192.168.101.11:8082

  7. configure storm submitter to point to the local cluster 

    mkdir -p ~/.storm
echo ‘nimbus.host: "192.168.101.11"’ > ~/.storm/storm.yaml

    • to point back to the softlayer cluster, change nimbus.host to cluster-7-master.sl.hackreduce.net

The vagrant box was built with the following software versions:

  • Vagrant 1.1.0
  • VirtualBox 4.1.23

Resources

Learning Storm

Spouts, Bolts, etc.

Polyglot Storm

Presentations