Hi! Let's get started with the Interacting with Hadoop Distributed File System (HDFS) lesson.
In this lesson, you will learn how to interact with data stored in HDFS using Cloudera Hue,
Hadoop file system command-line interface, and WebHDFS and HttpFS protocols.
Hue runs in a browser and provides an easy-to-use interface to several applications to support
interaction with Hadoop and HDFS. Hue is automatically installed and configured
on the Oracle BDA. It runs on port 8888 of the ResourceManager node which is node03 by
default. Hue is also automatically installed and configured
on the Oracle Big Data Lite VM. It runs on port 8888.
You can use Hue to perform any of the following tasks:
Query Hive data stores using SQL-like queries called HiveQL, which are implicitly converted
into MapReduce or Spark jobs. Create, load, and delete Hive tables. Create, edit, and
submit workflows using the Oozie dashboard. Work with HDFS files and directories. Create,
submit, and monitor Spark and MapReduce jobs. Manage users and groups.
Before you can start Hue on the Oracle Big Data Lite VM, you need to make sure that the
required services are started. Double-click the Start/Stop Services icon on your VM's Desktop.
In the Start/Stop Oracle Big Data Lite Services terminal window, make sure both HDFS and Hue
services are started. If you will be using Hive or Impala too, you also need to start
both the Hive and Impala services. To start a service, simply highlight the service and
then use the spacebar to select it. You can also click a service to start it. When you
are done, click OK.
To start Hue in your Oracle Big Data Lite VM, click the browser's icon on the toolbar
to display your web browser.
On the web browser toolbar, click Hue. Hue is displayed. If you are prompted for a userid
and a password, enter oracle and welcome1 respectively. Remember, you can find this
information in the Start Here page which you can access by clicking the Start Here icon
on your VM desktop.
You can use Hue to browse for documents, files, tables, indexes, and more. Click the menu
in the top-left corner of the window.
In the Browsers section, click Files.
The list of directories and files in the current /user/oracle HDFS directory is displayed.
In this example, we navigated to the /user/oracle/moviework/data HDFS directory. This directory contains the
output files that were generated after running a MapReduce job. Let's look at the content
of the part-00001 file. Double-click the file name.
The content of the file is displayed.
Next, you will learn how to access data in HDFS using the Hadoop client command-line
interface. HDFS provides a command-line interface, named
file system or FS shell, that enables you to interact with the data stored in HDFS.
The syntax of this command set is similar to that of other shells.
You can invoke FS shell by using the hadoop fs command followed by some arguments. You
must add the hadoop fs prefix to all commands as shown on the screen. If you enter the command
hadoop fs by itself, the complete list of the available arguments or commands is displayed.
For a complete list of the FS shell commands, access the FileSystem shell site using the
url shown at the bottom of the screen.
Scroll down the FS shell page to display detailed information and examples on the available
commands. In this example, we are looking at the get command syntax, examples, and available options.
In the Oracle Big Data Lite VM, you issue FS shell commands using a terminal window.
To access a terminal window, click the Terminal icon on the toolbar.
A terminal window is displayed. You enter the Hadoop commands at the $ prompt.
The table lists some of the FS Shell commands. We will go over some examples in this lesson.
To display the contents of a directory in a local file system such as linux, you issue
the ls -l command. To display the contents of an HDFS directory,
you use the hadoop fs -ls command.
To display the version of Apache Hadoop on your machine, use the hadoop version command.
You can use the hadoop fs -help command to display the syntax and options for all of
the available commands.
To display help on specific commands, enter hadoop fs -help followed by the command or
commands. In this example, we are displaying help on both the get and put commands.
To display the contents of a specific HDFS directory, enter hadoop fs -ls followed by
the name of the directory. In this example, we are looking at the contents of the moviework
directory.
You can create a new HDFS directory using the hadoop fs -mkdir command followed by the
name of the directory. In this example, we are creating a new directory named training.
Next, we use the hadoop fs -ls command to confirm the creation of the training directory.
In this example, we use the linux pwd or print working directory command to identify the
current working directory which is practice_commands. Next, we list the contents of this directory
which contains a list of scripts. In the next few examples, we will use the lab_05_01.txt file.
You can use the put command to copy one of more files from the local file system to the
destination file system such as HDFS. In this example, we copy the lab_05_01.txt file from
the practice_commands current local file system directory to the training HDFS directory that
we created earlier. Next, we list the contents of the current directory. Finally, the contents
of the training HDFS directory to confirm the copy operation.
You can use the get command to copy files from HDFS to the local file system. In this
example, we are copying the lab_05_01.txt file from the training HDFS directory to the
current local file system directory. We name the copied file copy-lab_05_01.txt. Next,
we do an ls -l to confirm the successful copy operation.
You can use the rm command to delete an HDFS directory. The -r option deletes the directory
and any content under it recursively. In this example, we delete the training HDFS directory.
A confirmation message is displayed.
Next, you will learn how to access HDFS data by using the curl command with the WebHDFS
and HttpFS protocols.
If you do not have a Hadoop client installed, then you cannot use the hadoop fs ls command.
In that case, you can interact with HDFS by using either WebHDFS or HttpFS REST protocols.
REST is a popular way to access services in a language-agnostic way and Hadoop supports
it. curl is a tool to transfer data from or to
a server by using one of the supported protocols including WebHDFS and HttpFS.
To use WebHDFS to list the contents of a directory, you use the curl command with the LISTSTATUS
operation. In the syntax, you provide the host name, port #, the keywords webhdfs/v1,
the path of the directory for which you want to list its contents, and the LISTSTATUS operation.
In this example, we use the Oracle Big Data Lite VM host name port # and the /user/oracle
path. The contents of the /user/oracle directory is displayed in JSON format. This is the same
information as that displayed using the hadoop fs -ls command.
This example submits an HTTP PUT request to create a new directory named my_dir under
the /user/oracle directory. The owner of this directory is user oracle. You specify the
host name, port #, path, the keywords webhdfs/v1, the path where to create this directory, and
mkdirs as the operation. The -i option specified with the command Include the HTTP-header in
the output.
In this example, we list the contents of the current local file system directory. We will
use the test1.txt file in the next few examples.
In this example, we upload the test1.txt file to the my_dir HDFS directory. If my_dir does
not exist, it will be created.
In this example, we remove the my_dir3 HDFS directory. We use the &recursive option to
remove the directory and its contents, if any.
In this example, we create an HDFS directory and load the local test1.txt file to the new
directory using HttpFS. If the directory does not exist, it will be created. HttpFS uses
the default port 14000.
You can access this WebHDFS REST API documentation page using the url at the bottom of the screen.
This page contains the complete list of HTTP GET, PUT, and DELETE operations that we used
in this lesson.
You can access the Hadoop HDFS over HTTP Documentation Sets page using the url at the bottom of the screen.
In this lesson, you should have learned how to interact with data stored in HDFS using
Cloudera Hue, Hadoop file system command-line interface, WebHDFS, and HttpFS.
Thanks for watching!
Learn more at: docs.oracle.com/en/bigdata