Caffe

源码编译安装

Ubuntu

1. 安装依赖库

apt-get install libboost-all-dev liblmdb-dev libatlas-base-dev libhdf5-serial-dev libleveldb-dev libsnappy-dev

1. 安装 python 所需包

python 环境中只安装 numpy

for req in $(cat requirements.txt) pydot; do pip install $req; done

1. cmake 编译

cmake -DBUILD_docs=OFF \
    -DBUILD_python=ON \
    -Dpython_version=3 \
    -DPYTHON_LIBRARIES=/anaconda3/envs/caffe-1.0/lib \
    -DPYTHON_INCLUDE_DIR=/anaconda3/envs/caffe-1.0/include/python3.8 \
    -DPYTHON_EXECUTABLE=/anaconda3/envs/caffe-1.0/bin/python ..

1. 安装

make all -j$(nproc)

make pycaffe -j$(nproc)

1. 测试

make runtest -j$(nproc)

# 测试 python 接口
python -c "import sys;sys.path.append('/workspace/local/caffe-1.0/python');import caffe;from caffe import layers as L;print(caffe.__version__, caffe.__file__)"

1. 添加到 python 搜索路径

cd caffe/python
pwd > /anaconda3/envs/caffe-1.0/lib/python3.8/site-packages/caffe.pth

# 测试 python 接口
python -c "import caffe;from caffe import layers as L;print(caffe.__version__, caffe.__file__)"

使用 python 接口添加自定义 layer

前提

• 编译 caffe 时，设置WITH_PYTHON_LAYER := 1

示例1：自定义批量 resize layer

实现功能：从一个文件夹中批量读取图片并且resize为固定尺寸

1. 定义 caffe layer

## MyPythonLayer.py

# -*- coding: utf-8 -*-

import sys
sys.path.insert(0,".../caffe/python")

import caffe
import yaml, glob
from random import shuffle
import cv2
import logging
logging.basicConfig(level=logging.INFO)

class myPythonLayer(caffe.Layer):
    """
    reshape images

    继承 caffe.Layer 的四个基本方法setup, reshape, forward, backward
    """
    def setup(self, bottom, top):
        """
        setup中主要添加类的初始化过程，而当类被调用时，则会调用方法forward
        """

        # 参数有多个对，但并不会解析多个对，提前 split
        params_str = self.param_str.split(',')

        # yaml.load(str)时，str必须以空格间隔
        # 比如”‘batch_size’: 50”, ‘batch_size’: 与 50 以空格间隔，反之会报错
        params = [yaml.load(item) for item in params_str]
        print params

        # 参数的顺序很重要，网络定义的 prototxt 的参数的顺序要根据这个来
        self.source = params[0]['source_dir']   # 图片的根目录
        self.target_size = params[1]['target_size'] # resize的目标尺寸大小
        self.batch_size = params[2]['batch_size']   # 批大小

        self.batch_loader = BatchLoader(source_dir=self.source, target_size=self.target_size)
        print 'Parameter batch_size:{}\n' \
              'source_dir:{}\n' \
              'target_size:{}'.format(self.batch_size, self.source, self.target_size)

        # top必须被 reshape, 否则在 blob.cpp 中115行会CHECK(data_)报错
        top[0].reshape(self.batch_size, self.target_size, self.target_size, 3)

    def reshape(self, bottom, top):
        pass

    def forward(self, bottom, top):
        """
        forward是必须的
        读取batch数量的图片数目, 以实现批量读取的功能
        """
        for i in xrange(self.batch_size):
            top[0].data[i, ...] = self.batch_loader.next_batch()

    def backward(self, bottom, propagate_down, top):
        """
        backward不是必须的
        """
        pass

class BatchLoader(object):

    def __init__(self, source_dir, target_size):
        self.cur = 0
        self.target_size = target_size
        self.indexlist = glob.glob(source_dir+ '/*.jpg')

    def next_batch(self):
        if self.cur == len(self.indexlist):
            self.cur = 0
            shuffle(self.indexlist)
        item = self.indexlist[self.cur]
        img_tmp = cv2.imread(item)
        img_tmp = cv2.resize(src=img_tmp, dsize=(self.target_size, self.target_size))
        self.cur += 1
        logging.info('load {} images'.format(self.cur))
        return img_tmp

1. 定义prototxt文件

layer{
name: "mylayer"
type: "Python"
top: "images"
python_param{
    module: "MyPythonLayer"
    layer: "myPythonLayer"
    param_str: "'source_dir': '/home/sai/code/face_detection/train/lfw_5590','target_size': 224,'batch_size': 50"
    }
}

• type: 必须是’python’，caffe 编译的时候 config 里面 WITH_PYTHON_LAYER 参数要打开
• module: 自定义 python 层文件的名字
• layer: 自定义 python 层文件中定义的 class 名
• param_str: 字符串，字典的格式，用于设置定义层的参数。要注意param_str的顺序。

运行测试

# -*- coding: utf-8 -*-

import sys
sys.path.insert(0,".../caffe/python")

from __future__ import print_function
import caffe
from caffe import layers as L, params as P, to_proto

def My_layer(source_dir, batch_size, target_size):
    param_str = "'source_dir': source_dir, 'batch_size': batch_size, 'target_size': target_size"
    mylayer = L.Python(module='MyPythonLayer', layer='myPythonLayer', param_str=param_str)
    print(mylayer)
    to_proto(mylayer)

def make(filename):
    with open(filename, 'w') as f:
        print(My_layer(source_dir='/home/sai/code/face_detection/train/lfw_5590', batch_size=50, target_size=100), file=f)

if __name__=='__main__':
    net = caffe.Net('mylayer.prototxt', caffe.TEST)
    net.forward()
    images = net.blobs['images'].data
    print(images.shape)

附1: 安装问题

1. Error: libboost_python.so: undefined reference to `PyClass_Type

   原因分析：libboost_python.so 系统默认链接的是 libboost_python-py2.7.so，而编译 pycaffe 使用的是 python3。

   解决方法:

    cd /usr/lib/x86_64-linux-gnu/
    unlink libboost_python.so
    ln -s libboost_python-py35.so libboost_python.so
   

2. protobuf error: ‘static_assert’ was not declared in this scope

   解决方法：使用 c++11 编译，添加编译参数 -std=c++11

3. cannot find -lboost_python3

   解决方法：将链接的 boost_python3 库，修改为 boost_python-py35