<?xml version="1.0" encoding="UTF-8"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" "http://forrest.apache.org/dtd/document-v20.dtd">

<document>
  <header>
    <title>Load and Store Interfaces</title>
  </header>
  <body>
 
 <!-- ==================================================================== --> 
  <section>
  <title>Set Up</title>
  
<p>The HCatLoader and HCatStorer interfaces are used with Pig scripts to read
and write data in HCatalog-managed tables. No HCatalog-specific setup is
required for these interfaces.</p>

</section>

<!-- ==================================================================== -->
<section>
  <title>HCatLoader</title>

<p>HCatLoader is used with Pig scripts to read data from HCatalog-managed tables.</p>

<section>
  <title>Usage</title>

<p>HCatLoader is accessed via a Pig load statement.</p>
<source>
A = LOAD 'tablename' USING org.apache.hive.hcatalog.pig.HCatLoader(); 
</source>

<p><strong>Assumptions</strong></p>

<p>You must specify the table name in single quotes: LOAD 'tablename'.
If you are using a non-default database you must specify your input as
'dbname.tablename'. If you are using Pig 0.9.2 or earlier, you must create
your database and table prior to running the Pig script. Beginning with
Pig 0.10 you can issue these create commands in Pig using the SQL command.</p>

<p>The Hive metastore lets you create tables without specifying a database;
if you created tables this way, then the database name is 'default' and is not
required when specifying the table for HCatLoader.</p>

<p>If the table is partitioned, you can indicate which partitions to scan by
immediately following the load statement with a partition filter statement
(see <strong>Load Examples</strong> below).</p>
</section>

 <!-- ==================================================================== --> 
<section> 
  <title>HCatalog Data Types</title>

<p>Restrictions apply to the types of columns HCatLoader can read from HCatalog-managed tables.</p>
<p>HCatLoader can read <em><strong>only</strong></em> the data types listed in the table below.
The table shows how Pig will interpret the HCatalog data type.</p>
  <table>
    <tr>
        <th>
           <p class="center">Primitives</p>
        </th>
        <th>
           <p class="center"> </p>
        </th>
    </tr>
    <tr>
        <td>
           <p class="cell"><strong>HCatalog Data Type</strong></p>
        </td>
        <td>
           <p class="cell"><strong>Pig Data Type</strong></p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">int</p>
        </td>
        <td>
           <p class="cell">int</p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">long</p>
        </td>
        <td>
           <p class="cell">long</p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">float</p>
        </td>
        <td>
           <p class="cell">float</p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">double</p>
        </td>
        <td>
           <p class="cell">double</p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">string</p>
        </td>
        <td>
           <p class="cell">chararray</p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">boolean</p>
        </td>
        <td>
           <p class="cell">boolean</p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">binary</p>
        </td>
        <td>
           <p class="cell">bytearray</p>
        </td>
    </tr>
</table>
<table>
    <tr>
        <th>
           <p class="center">Complex Types</p>
        </th>
        <th>
           <p class="center"></p>
        </th>
    </tr>
    <tr>
        <td>
           <p class="cell"><strong>HCatalog Data Type</strong></p>
        </td>
        <td>
           <p class="cell"><strong>Pig Data Type</strong></p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">map<br/>(key type should be string)</p>
        </td>
        <td>
           <p class="cell">map </p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">List<em>&lt;any type&gt;</em> </p>
        </td>
        <td>
           <p class="cell">bag </p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">struct<em>&lt;any type fields&gt;</em> </p>
        </td>
        <td>
           <p class="cell">tuple </p>
        </td>
    </tr>
 </table>

<p><br/></p>

</section> 

 <!-- ==================================================================== --> 
<section> 
  <title>Running Pig with HCatalog</title>

<p>Pig does not automatically pick up HCatalog jars. To bring in the necessary
jars, you can either use a flag in the <code>pig</code> command or set the
environment variables PIG_CLASSPATH and PIG_OPTS as described below.</p>

<p><strong>The -useHCatalog Flag</strong></p>
<p>To bring in the appropriate jars for working with HCatalog,
simply include the following flag:</p>

<source>
pig -useHCatalog
</source>

<p><strong>Jars and Configuration Files</strong></p>

<p>For Pig commands that omit <code>-useHCatalog</code>, you need to
tell Pig where to find your HCatalog jars and the Hive jars used by the
HCatalog client. To do this, you must define the environment variable
PIG_CLASSPATH with the appropriate jars.</p>

<p>HCatalog can tell you the jars it needs. In order to do this
it needs to know where Hadoop and Hive are installed.
Also, you need to tell Pig the URI
for your metastore, in the PIG_OPTS variable.</p>

<p>In the case where you have installed Hadoop and HCatalog via tar,
you can do this:</p>

<source>
export HADOOP_HOME=&lt;path_to_hadoop_install&gt;

export HCAT_HOME=&lt;path_to_hcat_install&gt;

export HIVE_HOME=&lt;path_to_hive_install&gt;

export PIG_CLASSPATH=$HCAT_HOME/share/hcatalog/hcatalog-*.jar:\
$HIVE_HOME/lib/hive-metastore-*.jar:$HIVE_HOME/lib/libthrift-*.jar:\
$HIVE_HOME/lib/hive-exec-*.jar:$HIVE_HOME/lib/libfb303-*.jar:\
$HIVE_HOME/lib/jdo2-api-*-ec.jar:$HIVE_HOME/conf:$HADOOP_HOME/conf:\
$HIVE_HOME/lib/slf4j-api-*.jar

export PIG_OPTS=-Dhive.metastore.uris=thrift://&lt;hostname&gt;:&lt;port&gt;
</source>

<p>Or you can pass the jars in your command line:</p>

<source>
&lt;path_to_pig_install&gt;/bin/pig -Dpig.additional.jars=\
$HCAT_HOME/share/hcatalog/hcatalog-*.jar:\
$HIVE_HOME/lib/hive-metastore-*.jar:$HIVE_HOME/lib/libthrift-*.jar:\
$HIVE_HOME/lib/hive-exec-*.jar:$HIVE_HOME/lib/libfb303-*.jar:\
$HIVE_HOME/lib/jdo2-api-*-ec.jar:$HIVE_HOME/lib/slf4j-api-*.jar  &lt;script.pig&gt;
</source>

<p>The version number found in each filepath will be substituted for <code>*</code>. For example, HCatalog release 0.5.0 uses these jars and conf files:</p>

<ul>
    <li><code>$HCAT_HOME/share/hcatalog/hcatalog-core-0.5.0.jar</code></li>
    <li><code>$HCAT_HOME/share/hcatalog/hcatalog-pig-adapter-0.5.0.jar</code></li>
    <li><code>$HIVE_HOME/lib/hive-metastore-0.10.0.jar</code></li>
    <li><code>$HIVE_HOME/lib/libthrift-0.7.0.jar</code></li>
    <li><code>$HIVE_HOME/lib/hive-exec-0.10.0.jar</code></li>
    <li><code>$HIVE_HOME/lib/libfb303-0.7.0.jar</code></li>
    <li><code>$HIVE_HOME/lib/jdo2-api-2.3-ec.jar</code></li>
    <li><code>$HIVE_HOME/conf</code></li>
    <li><code>$HADOOP_HOME/conf</code></li>
    <li><code>$HIVE_HOME/lib/slf4j-api-1.6.1.jar</code></li>
</ul>

<p><strong>Authentication</strong></p>
<table>
    <tr>
    <td><p>If you are using a secure cluster and a failure results in a message
like "2010-11-03 16:17:28,225 WARN hive.metastore ... - Unable to connect
metastore with URI thrift://..." in /tmp/&lt;username&gt;/hive.log, then make
sure you have run "kinit &lt;username&gt;@FOO.COM" to get a Kerberos ticket
and to be able to authenticate to the HCatalog server.</p></td>
    </tr>
</table>


</section>

 <!-- ==================================================================== --> 
<section> 
  <title>Load Examples</title>

<p>This load statement will load all partitions of the specified table.</p>
<source>
/* myscript.pig */
A = LOAD 'tablename' USING org.apache.hive.hcatalog.pig.HCatLoader(); 
...
...
</source>
<p>If only some partitions of the specified table are needed, include a
partition filter statement <strong><em>immediately</em></strong> following the
load statement in the data flow. (In the script, however, a filter statement
might not immediately follow its load statement.) The filter statement can
include conditions on partition as well as non-partition columns.</p>
<source>
/* myscript.pig */
A = LOAD 'tablename' USING  org.apache.hive.hcatalog.pig.HCatLoader();

-- date is a partition column; age is not
B = filter A by date == '20100819' and age &lt; 30; 

-- both date and country are partition columns
C = filter A by date == '20100819' and country == 'US'; 
...
...
</source>

<p>To scan a whole table, for example:</p>

<source>
a = load 'student_data' using org.apache.hive.hcatalog.pig.HCatLoader();
b = foreach a generate name, age;
</source>

<p>Notice that the schema is automatically provided to Pig; there's no need to
declare name and age as fields, as if you were loading from a file.</p>

<p>To scan a single partition of the table web_logs partitioned by the column
datestamp, for example:</p>

<source>
a = load 'web_logs' using org.apache.hive.hcatalog.pig.HCatLoader();
b = filter a by datestamp == '20110924';
</source>

<p>Pig will push the datestamp filter shown here to HCatalog, so that HCatalog knows to just scan the partition where
datestamp = '20110924'. You can combine this filter with others via 'and':</p>

<source>
a = load 'web_logs' using org.apache.hive.hcatalog.pig.HCatLoader();
b = filter a by datestamp == '20110924' and user is not null;
</source>

<p>Pig will split the above filter, pushing the datestamp portion to HCatalog
and retaining the '<code>user is not null</code>' part to apply itself.
You can also give a more complex filter to retrieve a set of partitions.</p>

<p><strong>Filter Operators</strong></p>

<p>A filter can contain the operators 'and', 'or', '()', '==', '!=', '&lt;', '&gt;', '&lt;='
and '&gt;='.</p>

<p>For example:</p>

<source>
a = load 'web_logs' using org.apache.hive.hcatalog.pig.HCatLoader();
b = filter a by datestamp &gt; '20110924';
</source>

<p>A complex filter can have various combinations of operators, such as:</p>

<source>
a = load 'web_logs' using org.apache.hive.hcatalog.pig.HCatLoader();
b = filter a by datestamp == '20110924' or datestamp == '20110925';
</source>

<p>These two examples have the same effect:</p>

<source>
a = load 'web_logs' using org.apache.hive.hcatalog.pig.HCatLoader();
b = filter a by datestamp &gt;= '20110924' and datestamp &lt;= '20110925';
</source>

<source>
a = load 'web_logs' using org.apache.hive.hcatalog.pig.HCatLoader();
b = filter a by datestamp &lt;= '20110925' and datestamp &gt;= '20110924';
</source>

</section> 
 
</section> 

<!-- ==================================================================== -->
<section>
  <title>HCatStorer</title>

<p>HCatStorer is used with Pig scripts to write data to HCatalog-managed tables.</p>

<section>
  <title>Usage</title>

<p>HCatStorer is accessed via a Pig store statement.</p>

<source>
A = LOAD ...
B = FOREACH A ...
...
...
my_processed_data = ...

STORE my_processed_data INTO 'tablename'
   USING org.apache.hive.hcatalog.pig.HCatStorer();
</source>

<p><strong>Assumptions</strong></p>

<p>You must specify the table name in single quotes: LOAD 'tablename'. Both the database and table must be created prior to running your Pig script. If you are using a non-default database you must specify your input as 'dbname.tablename'. If you are using Pig 0.9.2 or earlier, you must create your database and table prior to running the Pig script. Beginning with Pig 0.10 you can issue these create commands in Pig using the SQL command. </p>
<p>The Hive metastore lets you create tables without specifying a database; if you created
tables this way, then the database name is 'default' and you do not need to specify the
database name in the store statement. </p>
<p>For the USING clause, you can have a string argument that represents key/value pairs
for partition. This is a mandatory argument when you are writing to a partitioned table
and the partition column is not in the output column. The values for partition keys
should <em>NOT</em> be quoted.</p>
<p>If partition columns are present in data they need not be specified as a STORE argument. Instead HCatalog will use these values to place records in the appropriate partition(s). It is valid to specify some partition keys in the STORE statement and have other partition keys in the data.</p>
<p></p>
<p></p>

</section> 
<section> 
  <title>Store Examples</title>

<p>You can write to a non-partitioned table simply by using HCatStorer.  The contents of the table will be overwritten:</p>

<source>store z into 'web_data' using org.apache.hive.hcatalog.pig.HCatStorer();</source>

<p>To add one new partition to a partitioned table, specify the partition value in the store function. Pay careful
attention to the quoting, as the whole string must be single quoted and separated with an equals sign:</p>

<source>store z into 'web_data' using org.apache.hive.hcatalog.pig.HCatStorer('datestamp=20110924');</source>

<p>To write into multiple partitions at once, make sure that the partition column is present in your data, then call
HCatStorer with no argument:</p>

<source>store z into 'web_data' using org.apache.hive.hcatalog.pig.HCatStorer(); 
  -- datestamp must be a field in the relation z</source>

</section>

 <!-- ==================================================================== --> 
<section>
  <title>HCatalog Data Types</title>

<p>Restrictions apply to the types of columns HCatStorer can write to HCatalog-managed tables.</p>
<p>HCatStorer can write <em><strong>only</strong></em> the data types listed in the table.
The table shows how Pig will interpret the HCatalog data type.</p>

  <table>
    <tr>
        <th>
           <p class="center">Primitives</p>
        </th>
        <th>
           <p class="center"> </p>
        </th>
    </tr>
    <tr>
        <td>
           <p class="cell"><strong>Pig Data Type</strong></p>
        </td>
        <td>
           <p class="cell"><strong>HCatalog Data Type</strong></p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">int</p>
        </td>
        <td>
           <p class="cell">int</p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">long</p>
        </td>
        <td>
           <p class="cell">long</p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">float</p>
        </td>
        <td>
           <p class="cell">float</p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">double</p>
        </td>
        <td>
           <p class="cell">double</p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">chararray</p>
        </td>
        <td>
           <p class="cell">string</p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">boolean</p>
        </td>
        <td>
           <p class="cell">boolean</p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">bytearray</p>
        </td>
        <td>
           <p class="cell">binary</p>
        </td>
    </tr>
</table>
<table>
    <tr>
        <th>
           <p class="center">Complex Types</p>
        </th>
        <th>
           <p class="center"> </p>
        </th>
    </tr>
    <tr>
        <td>
           <p class="cell"><strong>Pig Data Type</strong></p>
        </td>
        <td>
           <p class="cell"><strong>HCatalog Data Type</strong></p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">map</p>
        </td>
        <td>
           <p class="cell">map<br/>(key type should be string)</p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">bag </p>
        </td>
        <td>
           <p class="cell">List<em>&lt;any type&gt;</em> </p>
        </td>
    </tr>
    <tr>
        <td>
           <p class="cell">tuple </p>
        </td>
        <td>
           <p class="cell">struct<em>&lt;any type fields&gt;</em> </p>
        </td>
    </tr>
 </table>

<p><br/></p>

</section>

</section>

  </body>
</document>
