The Angriest Developer

RIAK node is not reachable! 

This has happened a few times when setting up RIAK clusters and by doing a little googling it looks like somewhat common problem around RIAK users, especially new ones. So I decided to do a short blog post hopefully providing a little more understanding on why this might have happened to you :)

This is the scenario:

You've setup a new RIAK cluster on your preferred method and notice that the nodes are happily up but have no idea on each other.

root@riak-test-cluster-10-8-2-5:~# riak-admin ring-status
Attempting to restart script through sudo -H -u riak
================================== Claimant ===================================
Claimant:  'riak@10.8.2.5'
Status:     up
Ring Ready: true

============================== Ownership Handoff ==============================
No pending changes.

============================== Unreachable Nodes ==============================
All nodes are up and reachable

Sounds good when all nodes are up and reachable, but looks bad when there is just one of'em :) Your first option is to go and do the ping, if for no other reason, just to see the pong reply and have a smile:

root@riak-test-cluster-10-8-2-5:~# riak ping
Attempting to restart script through sudo -H -u riak
pong

Voilá! RIAK seems happy in this sense. Let's try to join another member into the cluster:

root@riak-test-cluster-10-8-2-5:~:~# riak-admin cluster join riak@10.8.2.6
Attempting to restart script through sudo -H -u riak
Node riak@10.8.2.6 is not reachable!

Oh noes :( So this is why you're here, let's get busy solving the problem! First of all: check your firewall. The checklist of doing that can be found here in the RIAK documentation:

The documentation states that following ports are needed to be opened:

• epmd listener: TCP:4369
• handoff_port listener: TCP:8099
• range of ports specified in app.config

• web_port: TCP:8098
• pb_port: TCP:8087

So make sure this is indeed what you have in place. If you're using iptables, start from there. If you're more open about yourself and rely on, say, Amazon security groups, that's your next target. What ever you do, check that the ports are both open, being listened and reachable between nodes by using your favorite approach (and yes, use of telnet is fine, just delete it if it wasn't part of your install base ;)).

Depending on if any changes were made, give it another go.

Still no luck? Let's do the next step and go dumpster diving. Enter the world of tcpdump. Your usage may vary, but since this node has pretty much no other traffic than our connection to the ssh server in port 22, this does the trick (note the 'not' before port number or you'll immediately regret this decision):

root@riak-test-cluster-10-8-2-5:~# tcpdump -ni eth0 port not 22 &
[1] 31478
Whoa! What's this. First it looks like everything is there's traffic to port 4369 which goes fine, but what is this traffic back to 57459 that fails? This isn't listed in the port range! Halp! So we end up with the inevitable:


Now if you're like me, you'll go back to the page where the ports were listened and consider writing an angry email to Basho stating they've ruined your afternoon by not documenting their stuff properly, which is when you'll probably notice the talk about port mapping and app config before the ports were being listed.

Oh yes, enter the world of port mapping. The way RIAK behaves is much like any other program that requires portmapper. If no limits are set for the port range to use, it simply sends a port number 0 to the operating system, which in general returns the next available port and uses that. This is not a problem if your nodes have no trouble communicating through basically ports between 0-65536, but usually the security people start twitching if you even suggest this being the case. That's where the basho's app.config file steps in:

{ kernel, [
            {inet_dist_listen_min, 6000},
            {inet_dist_listen_max, 7999}
          ]},

There we go. You can restrict the port range to something reasonable yet big enough range for RIAK to do its job. Do this, set your firewall accordingly, restart your node and I think things turn better in no time :)

They didn't? Perhaps you'll need to write that email before, but just before you do and if this is an option for you: allow the whole port range (TCP) and try again. If this works out for you, you'll probably have a rule missing or not applied somewhere. And if you don't, it's time to, well bash the basho...

I'll get my coat.

3 kommenttia: