Caffe Install Notes

I have spent all too much time trying to get Caffe running on a CentOS based cluster that I use. I was hoping this would be a straightforward process. Suffice it to say it has not been. Not even close. None of the problems I encountered were particularly challenging to solve. The complication came from the fact that I ran into one hurdle after another. I should note however that installing Caffe on my personal machine which runs Linux Mint 17.1 went smoothly.

I'm writing this post as a record of the problems I encountered and the solutions I used. Unfortunately because I wasn't expecting to need to write this I may be missing details. If you notice something missing please feel free to leave a comment and I will update this document. Similarly, if you know of better ways to solve any of these problems please feel free to share. I will not likely test the solutions my self unless I need to reinstall for some reason so your comments would be largely for posterity. One thing to note is that some if not many of the problems I encountered may be rather specific too the cluster I'm using. My apologies if the following doesn't address the problem you are experiencing.

The first problem I ran into was with protobuf. The problem was related to the members of a unions being defined as constant in src/google/protobuf/util/internal/datapiece.h. Specifically the union defining the types i32_, i64_, u32_, u64_, double_, float_, bool_, and str_. This problem appears to be fairly common according to a quick Google search and the fix is as simple as removing the const keyword. However the error itself can be a bit misleading as it doesn't lead one specifically to the offending lines.

The next problem I encountered was related to glog. Specifically it requires a newer version of autotools than was installed on the system I'm using. To solve this I performed a user space install of autoconf and automake. It proved tricky to install autoconf for reasons I still don't understand. Thanks to the cluster admins I was able to finally do so using these instructions. Automake simply required downloading the tarball, configuring for my home directory, and then performing make followed by make install. Unfortunately glog was still not happy! The included make file hardcoded aclocal-1.14 and the upgraded autotools gave me autotools-1.15. Bah! Executing autoreconf -ivf fixed that issue though. I am not completely certain this is a good solution however as I have not used autoreconf before.

The next hurdle was with gflags. Evidently it requires cmake which was not already installed. Downloading and installing cmake resolved this problem. One thing to note when performing a user space install of cmake is that you need to use the --prefix flag when calling the boostrap script to indicate that you want cmake installed in your home directory. I also found that I had to compile with -fPIC, so the complete cmake command I ended up using was CXXFLAGS="-fPIC" cmake -DCMAKE_INSTALL_PREFIX=~ .. in a build subdirectory of the repository.

I also had to install OpenCV. I didn't run into any trouble here. But a word of warning for those that have never built it before -- it takes a very long time!

Next up was leveldb. In this case I just cloned the github repository and ran make in it. From there caffe needs to be told where to find the header files and shared objects that were built. I told it as much by appending to the INCLUDE_DIRS and LIBRARY_DIRS lists respectively in caffe/Makefile.config. The headers are in the include subdirectory of the repository while the libraries will be placed at the root of the repository.

From there I found I needed lmdb. This library is developed under OpenLDAP. At the time of this writing they offer a github repository with just the lmdb code so I cloned it and built the library. From there I updated the INCLUDE_DIRS and LIBRARY_DIRS lists in caffe/Makefile.config to point to libraries/liblmdb within the repository.

Next I had to install the Google snappy library. In this case I had to get the tarball from the Google code repository not the github repository. For reasons I don't know or really care about it seems that build files are missing from the gihub repository.

The last problem I ran into was related to Atlas. I had previously performed a user space install of it but did not add the resulting lib directory to my LD_LIBRARY_PATH and LIBRARY_PATH environment variables. Doing so allowed me to finally able to execute make all in the Caffe repository and have it complete without errors. It took a while though, in part because I was using a single thread since I never knew what it would stumble over next. As such I advise throwing more threads at it by instead using the command make all -jX where X is the number of threads you want it to use.

One final note on the installation. Don't forget to add the appropriate atlas, leveldb, and liblmdb directories to LD_LIBRARY_PATH in your .bashrc.

At this point I'm really hoping it was worth the effort to install Caffe. Comparatively the Theano and Pylearn2 installations were so much easier on this same system.

Linux Mint 17.1, Nvidia, CUDA, and cuDNN

I recently replaced a Titan X, which was on loan, with a GTX 980. After messing with drivers for nearly a day I was able to get my dual monitor setup running again. Unfortunately whatever i did freaked out Theano yielding the error:

dustin@Cortex ~ $ ipython
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
Type "copyright", "credits" or "license" for more information.

IPython 1.2.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import theano
WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu0 \
  is not available (error: Unable to get the number of gpus available: \
  unknown error) 

I tried upgrading the Nvidia driver to 346.59 and CUDA from 6.5 to 7.0 with no luck. So I decided to start fresh since I had been wanting to upgrade from Linux Mint 17 to 17.1 anyway. Following are the steps I used to get my system running Theano again. I have not replicated these results so hopefully I am not overlooking any major steps.

Nvidia Driver

I used the xorg-edgers PPA to install Nvidia drivers 346.59 as described on Noobs Lab. In short, add the new repository:

sudo add-apt-repository ppa:xorg-edgers/ppa

Update to get the list of available packages:

sudo apt-get update

Install 346:

sudo apt-get install nvidia-346 nvidia-settings

I have a dual monitor setup that require I execute nvidia-settings to Enable Xinerama via the X Server Display Configuration page.

CUDA

I found some helpful instructions for installing CUDA. In short, start by installing the GNU Compiler Collection tools with:

sudo apt-get install build-essential

Download the Nvidia CUDA 7.0 DEB. Though I'm running Linux Mint 17.1 I used the Ubuntu 14.04 Network DEB. Install it:

sudo dpkg -i cuda-repo-ubuntu1404_6.5-14_amd64.deb

Update to get the list of packages:

sudo apt-get update

Install CUDA:

sudo apt-get install cuda

Finally alter your .bashrc to add CUDA to your PATH and LD_LIBRARY_PATH environment variables. Theano will also want to know where CUDA is located so now is a good time to setup the CUDA_ROOT environment variables as well.

export PATH=/usr/local/cuda-7.0/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-7.0/lib64:$LD_LIBRARY_PATH
export CUDA_ROOT=/usr/local/cuda-7.0

cuDNN

Installing cuDNN for use with Theano can be found on the cuDNN page of deeplearning.net. I used the first method currently suggested on that page which is to copy *.h to $CUDA_ROOT/includes and *.so* to $CUDA_ROOT/lib64.

Installing RL-Glue and ALE without Root Access

In 2012 Bellemare, Naddaf, Veness, and Bowling [1] introduced the Arcade Learning Environment (ALE) for developing and evaluating general AI methods. It interfaces with an Atari 2600 simulator called Stella. They motivate their use of the the Atari 2600 because it permits access to hundreds of "game environments, each one different, interesting, and designed to be a challenge for human players." ALE also interfaces with RL-Glue, a collection of tools for developing and evaluating Reinforcement Learning agents.

RL-Glue is a two part system; a language agnostic part, referred to as the core, and a language specific part, referred to as the codec. There are multiple codecs supporting development in C/C++, Java, Lisp, Matlab, and Python.

The following discusses installing RL-Glue and ALE. First I cover installing RL-Glue and then ALE.

Installing RL-Glue

The instructions for installing the RL-Glue core and Python codec are drawn from the technical manuals for versions 3.04 and 2.0 and assume versions 3.04 and 2.02 respectively. However, looking over older versions of the manual it appears these instructions have not changed much, if at all, which means they may also work for future versions with the correct changes in the commands. I leave it as an exercise to the reader to determine what changes are needed. (Don't you love it when an author does that?)

Installing RL-Glue in the user space as opposed to system wide requires compiling the core code from source. To do so, execute the following commands.

RL-Glue Core

1. Download RL-Glue Core

$ cd ~ && wget http://rl-glue-ext.googlecode.com/files/rlglue-3.04.tar.gz

2. Unpack RL-Glue Core

$ tar -xvzf rlglue-3.04.tar.gz

3. Make a directory into which RL-Glue will be installed:

$ mkdir ~/rlglue

4. Configure RL-Glue Core
At this point it is necessary to identify the location wherein you wish to install the core as you must tell this to the configure script. I made a directory called rlglue in my home directory resulting in the following command:

$ cd rlglue-3.04 && ./configure --prefix=<rlglue>

where <rlglue> is the absolute path to the directory into which you want RL-glue installed.
5. Build and install RL-Glue Core

$ make && make install

6. Update your PATH
Because the core has been installed in a non-standard location it is necessary to inform the system of it's location. This merely entails updating your PATH environment variable to include rlglue/bin, i.e.:

$ export PATH=~/rlglue/bin:$PATH

Note that you can make this change in your .bashrc to avoid doing it every time you open a new terminal.

At this point executing rl_glue should result in:

RL-Glue Version 3.04, Build 909
RL-Glue is listening for connections on port=4096

This indicates that RL-Glue is waiting for programs managing the agent, environment, and experiment to connect. For now you can type ctrl-c to exit the program.

Python Codec

1. Download the Python Codec

$ cd ~ && wget http://rl-glue-ext.googlecode.com/files/python-codec-2.02.tar.gz

2. Unpack the codec

$ tar -xvzf python-codec-2.02.tar.gz

3. Update your PYTHONPATH
This is another step that is only necessary because we're installing into a non-standard location.

$ export PYTHONPATH=~/python-codec/src

Note that this is another command that can be placed into your .bashrc to avoid executing it every time you open a new terminal.

Installing ALE

Installing ALE takes a little more effort. This is due to the fact that ALE does not supply a configure script to build a makefile specific for your system. These instruction are for installing ALE 0.4.3 and may not extend to newer versions well so your mileage may vary. As before, execute the following commands.

1. Download ALE

$ wget http://www.arcadelearningenvironment.org/wp-content/uploads/2014/01/ale_0.4.3.zip

2. Unpack ALE

$ unzip ale_0.4.3.zip

3. Select Makefile
ALE supports Linux, OSX, and Windows and as such a makefile is supplied for each platform. Installing ALE requires making a makefile from one of these. I advise copying the one you want as opposed to renaming it:

$ cd ale_0.4.3/ale_0_4/ && cp makefile.unix makefile

Update 2014-04-15: Frédéric Bastien notes that this step can be avoided by supplying the name of the preferred makefile to make on the command line as follows:

$ make -f makefile.unix

Still be sure to change your working directory to ale_0.4.3/ale_0_4 as the following commands assume that context.

4. Enable RL-Glue support
RL-Glue support in ALE is disabled by default. To enable it edit the makefile and change the line:

USE_RLGLUE := 0

to

USE_RLGLUE := 1

It is also necessary to inform ALE where the RL-Glue headers are located. This can be done by changing the line:

INCLUDES := -Isrc/controllers -Isrc/os_dependent -I/usr/include -Isrc/environment

to

INCLUDES :=-Isrc/controllers -Isrc/os_dependent -I/usr/include -Isrc/environment -I<rlgluedir>/include

where <rlgluedir> indicates the directory into which rlglue was installed earlier.

Similarly it is necessary to inform ALE where the RL-Glue libraries are installed. This is done by changing the line:

LIBS_RLGLUE := -lrlutils -lrlgluenetdev

to

LIBS_RLGLUE := -L<rlgluedir>/lib -lrlutils -lrlgluenetdev

Update 2014-04-15: Frédéric Bastien notes that one can override these variables on the command line. For example:

$ make USE_RLGLUE=1

sets USE_RLGLUE to 1. However it is unclear how to append to variables via the command line so this may only work for the USE_RLGLUE variable without restating the existing variable value.

5. Build ALE

$ make

6. Update LD_LIBRARY_PATH
Because the RL-Glue libraries have been installed in a non-standard location it is necessary to tell ALE where to find them. This is done using the LD_LIBRARY_PATH environment variable as follows:

$ export LD_LIBRARY_PATH=<rlgluedir>/lib:$LD_LIBRARY_PATH

Note that this is another command you can add to your .bashrc to avoid needing to execute the command every time you open a new terminal.

7. Update your PATH
As before it is necessary to update your path to inform the system of the location of the ALE binaries since they are installed in a non-standard location.

$ export PATH=~/ale_0.4.3/ale_0_4:$PATH

Note that this is yet another command you will have to execute every time you open a terminal unless you add this command to your .bashrc.

At this point you can execute ale which should result in:

A.L.E: Arcade Learning Environment (version 0.4)
[Powered by Stella]
Use -help for help screen.
Warning: couldn't load settings file: ./stellarc
No ROM File specified or the ROM file was not found.

Disregard the warnings, they are simply saying that ALE was unable to find your stella configuration and that a game ROM was not specified. This is to be expected since you did not specify the locations for them.

Conclusion

This covers installing RL-Glue and ALE. I have not discussed anything beyond basic testing as such material can be found in the documentation for the tools.

In a future post, or possibly more than one, I will outline the processes for making and executing agents, environments, and experiments.

References

[1] Bellemare, Marc G., et al. "The arcade learning environment: An evaluation platform for general agents." arXiv preprint arXiv:1207.4708 (2012).