Don't create a sub class of Put

Don't create a sub class of org.apache.hadoop.hbase.client.Put class unless you deliver the class to all HBase region servers. Otherwise, you will see an error message "java.io.IOException: IPC server unable to read call parameters: Error in readFields". What I experienced is described below.

I wrote a sub class of Put because I just wanted to avoid calling Bytes.toBytes(String) for every constructor/method arguments. To be concrete, I wrote the following.

import java.util.List;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.RowLock;
import org.apache.hadoop.hbase.util.Bytes;

public class JPut extends Put
{
    public JPut()
    {
    }

    public JPut(byte[] row)
    {
        super(row);
    }

    public JPut(String row)
    {
        super(toBytes(row));
    }

    public JPut(Put put)
    {
        super(put);
    }

    public JPut(byte[] row, RowLock rowLock)
    {
        super(row, rowLock);
    }

    public JPut(String row, RowLock rowLock)
    {
        super(toBytes(row), rowLock);
    }

    public JPut(byte[] row, long ts)
    {
        super(row, ts);
    }

    public JPut(String row, long ts)
    {
        super(toBytes(row), ts);
    }

    public JPut(byte[] row, long ts, RowLock rowLock)
    {
        super(row, ts, rowLock);
    }

    public JPut(String row, long ts, RowLock rowLock)
    {
        super(toBytes(row), ts, rowLock);
    }

    @Override
    public JPut add(byte[] family, byte[] qualifier, byte[] value)
    {
        return (JPut)super.add(family, qualifier, value);
    }

    public JPut add(String family, String qualifier, byte[] value)
    {
        return (JPut)super.add(toBytes(family), toBytes(qualifier), value);
    }

    @Override
    public JPut add(byte[] family, byte[] qualifier, long ts, byte[] value)
    {
        return (JPut)super.add(family, qualifier, ts, value);
    }

    public JPut add(String family, String qualifier, long ts, byte[] value)
    {
        return (JPut)super.add(toBytes(family), toBytes(qualifier), ts, value);
    }

    public List<KeyValue> get(String family, String qualifier)
    {
        return super.get(toBytes(family), toBytes(qualifier));
    }

    public boolean has(String family, String qualifier, byte[] value)
    {
        return super.has(toBytes(family), toBytes(qualifier), value);
    }

    public boolean has(String family, String qualifier, long ts, byte[] value)
    {
        return super.has(toBytes(family), toBytes(qualifier), ts, value);
    }

    public boolean has(String family, String qualifier, long ts)
    {
        return super.has(toBytes(family), toBytes(qualifier), ts);
    }

    public boolean has(String family, String qualifier)
    {
        return super.has(toBytes(family), toBytes(qualifier));
    }

    private static byte[] toBytes(String string)
    {
        if (string == null)
        {
            return null;
        }
        else
        {
            return Bytes.toBytes(string);
        }
    }
}

Then, I prepared an instance of the class and passed it to HTableInterface.put(Put) method like below.

String rowKey = ...;
String family = ...;
String qualifier = ...;
byte[] value = ...;

JPut jput = new JPut(rowKey);
jput.add(family, qualifier, value);

HTableInterface table = ...;

table.put(jput);

But, this caused "java.io.IOException: IPC server unable to read call parameters: Error in readFields". After grepping the source tree of HBase, I found "Error in readFields" was emitted by HbaseObjectWritable.java that runs on the client side. However, on the other hand, "IPC server unable to read call parameters" was found in HBaseServer.java that runs on the server side. So, I checked a log file of an HBase region server and found a stack trace like below.

2012-06-19 22:29:39,930 WARN org.apache.hadoop.ipc.HBaseServer: Unable to read call parameters for client 192.168.62.133
java.io.IOException: Error in readFields
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:655)
        at org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:125)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1248)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1177)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:713)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:505)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:480)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Error in readFields
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:655)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:565)
        at org.apache.hadoop.hbase.client.MultiAction.readFields(MultiAction.java:116)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:652)
        ... 9 more
Caused by: java.io.IOException: Can't find class JPut
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:644)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:565)
        at org.apache.hadoop.hbase.client.Action.readFields(Action.java:101)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:652)
        ... 12 more
Caused by: java.lang.ClassNotFoundException: JPut
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.getClassByName(HbaseObjectWritable.java:699)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:641)
        ... 15 more

The log said that the class 'JPut' was not found. I went back to the source code 'HBaseServer.java' and confirmed that it does reflection.

Writable param;
try {
  param = ReflectionUtils.newInstance(paramClass, conf);//read param
  param.readFields(dis);
} catch (Throwable t) {
  LOG.warn("Unable to read call parameters for client " +
           getHostAddress(), t);
  final Call readParamsFailedCall =
    new Call(id, null, this, responder, callSize);
  ByteArrayOutputStream responseBuffer = new ByteArrayOutputStream();

  setupResponse(responseBuffer, readParamsFailedCall, Status.FATAL, null,
      t.getClass().getName(),
      "IPC server unable to read call parameters: " + t.getMessage());
  responder.doRespond(readParamsFailedCall);
  return;
}

Finally I understood that I should not have used a sub class of 'Put'.