##网络及版本信息docker1 centos7 192.168.75.200docker2 centos7 192.168.75.201物理网络 192.168.75.1/24Docker ver
##网络及版本信息
docker1 centos7 192.168.75.200
docker2 centos7 192.168.75.201
物理网络 192.168.75.1/24
Docker version 1.10.3, build 3999ccb-unsupported ，安装过程略
# calicoctl version
Version: v1.0.0-12-g0d6d228Build date: 2017-01-17T09:01:03+0000Git commit: 0d6d228##1.安装etcd
####下载安装etcd
# ETCD_VER=v3.0.16
# DOWNLOAD_URL=https://github.com/coreos/etcd/releases/download
# curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
# mkdir -p /tmp/test-etcd && tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/test-etcd --strip-components=1
# cd /tmp/test-etcd && cp etcd* /usr/local/bin/
启动etcd
# etcd --listen-client-urls 'http://192.168.75.200:2379' --advertise-client-urls 'http://192.168.75.200:2379'
查看etcd信息
# etcdctl --endpoint 'http://192.168.75.200:2379' member list
8e9e05c52164694d: name=default peerURLs=http://localhost:2380 clientURLs=http://192.168.75.200:2379 isLeader=true##2.下载安装calico
修改网络内核参数
# sysctl -w net.netfilter.nf_conntrack_max=1000000# echo "net.netfilter.nf_conntrack_max=1000000" >> /etc/sysctl.conf
下载calicoctl
# cd /usr/local/bin/ && wget http://www.projectcalico.org/builds/calicoctl
# chmod 755 calicoctl
设置etcd环境变量
# export ETCD_ENDPOINTS=http://192.168.75.200:2379 && echo "export ETCD_ENDPOINTS=http://192.168.75.200:2379" >>/etc/profile
安装运行calico node
# calicoctl node run
Running command to load modules: modprobe -a xt_set ip6_tablesEnabling IPv4 forwardingEnabling IPv6 forwardingIncreasing conntrack limitRemoving old calico-node container (if running).Running the following command to start calico-node:docker run --net=host --privileged --name=calico-node -d --restart=always -e ETCD_AUTHORITY= -e ETCD_SCHEME= -e NODENAME=docker1 -e CALICO_NETWORKING_BACKEND=bird -e NO_DEFAULT_POOLS= -e CALICO_LIBNETWORK_ENABLED=true -e CALICO_LIBNETWORK_IFPREFIX=cali -e ETCD_ENDPOINTS=http://192.168.75.200:2379 -v /run/docker/plugins:/run/docker/plugins -v /var/run/docker.sock:/var/run/docker.sock -v /var/run/calico:/var/run/calico -v /lib/modules:/lib/modules -v /var/log/calico:/var/log/calico calico/node:latestImage may take a short time to download if it is not available locally.Container started, checking progress logs.Waiting for etcd connection...Using auto-detected IPv4 address: 192.168.75.200No IPv6 address configuredUsing global AS numberCalico node name: docker1CALICO_LIBNETWORK_ENABLED is true - start libnetwork serviceCalico node started successfully在docker1查看calico node状态，发现与docker2（192.168.75.201）连接已建立
# calicoctl node status
Calico process is running.IPv4 BGP status+----------------+-------------------+-------+----------+-------------+| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |+----------------+-------------------+-------+----------+-------------+| 192.168.75.201 | node-to-node mesh | up | 01:57:54 | Established |+----------------+-------------------+-------+----------+-------------+IPv6 BGP statusNo IPv6 peers found.##3.配置calico pool
查看默认pool
# calicoctl get pool
CIDR 192.168.0.0/16 fd80:24e2:f998:72d6::/64 删除默认pool，在任意一台node上操作
# calicoctl delete pool 192.168.0.0/16
Successfully deleted 1 'ipPool' resource(s)# calicoctl delete pool fd80:24e2:f998:72d6::/64
Successfully deleted 1 'ipPool' resource(s)创建新的ipPool，在任意一台node上操作
# vi /etc/calico/ippool_10.1.0.0_16.cfg
apiVersion: v1kind: ipPoolmetadata: cidr: 10.1.0.0/16spec: ipip: enabled: true nat-outgoing: true disabled: false# calicoctl create -f /etc/calico/ippool_10.1.0.0_16.cfg
Successfully created 1 'ipPool' resource(s)##4.配置docker，创建docker network
修改集群中每台docker启动参数，重启docker
添加--cluster-store=etcd://192.168.75.200:2379/calico 指定docker集群使用的存储，否则下一步不会成功创建network
# vi /etc/sysconfig/docker
OPTIONS='--selinux-enabled --log-driver=journald --cluster-store=etcd://192.168.75.200:2379/calico'集群中任意一台上docker创建网络
# docker network create --driver=calico --ipam-driver=calico-ipam net1
0501f1b788756d122568e7aed2d7c56fe2de9138f9bd00f6628c4b66c81c7c9b# docker network create --driver=calico --ipam-driver=calico-ipam net2
4b636bf63b23dee13b817c911335823a84ad6d55771a44e89fb81c16f76663ad# docker network ls
NETWORK ID NAME DRIVER54a450c39848 net1 calico 8fdcdecdb0bc net2 calico e0d1a688fef8 none null 0e987140865a host host b5122ac5e20e bridge bridge ##5.测试网络连否连通
docker1启动net1，net2各一个container
[root@docker1 bin]# docker run -itd --net=net1 --name=testnet1 centos /bin/bash579c509e293e25340f10cc188a91136f99ed9021b99f795a9056a683b6b46864[root@docker1 bin]# docker run -itd --net=net2 --name=testnet2 centos /bin/bashc8777a2ff6add64e6abf454828820a6cfee332086a58c769a6cf1e5e0fda8760docker2启动net1，net2各一个container
[root@docker2 bin]# docker run -itd --net=net1 --name=testnet3 centos /bin/bash8bb7be8d86a04631a442a9f43e6be9576a891f704b91042550c5fe632fa11f06[root@docker2 bin]# docker run -itd --net=net2 --name=testnet4 centos /bin/bash422f4466db503b380f646d6eaee14a2f695550669fd4987fadefff438f456a36container ip信息如下
testnet1 10.1.174.193testnet2 10.1.174.194testnet3 10.1.166.129testnet4 10.1.166.130####testnet1上ping其他container
testnet1容器只和docker2上的testnet3容器能通，因为两个container都属于net1网络
[root@579c509e293e /]# ping 10.1.166.129PING 10.1.166.129 (10.1.166.129) 56(84) bytes of data.64 bytes from 10.1.166.129: icmp_seq=1 ttl=62 time=0.400 ms^C--- 10.1.166.129 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 0.400/0.400/0.400/0.000 ms[root@579c509e293e /]# ping 10.1.166.130PING 10.1.166.130 (10.1.166.130) 56(84) bytes of data.^C--- 10.1.166.130 ping statistics ---4 packets transmitted, 0 received, 100% packet loss, time 3000ms[root@579c509e293e /]# ping 10.1.174.194PING 10.1.174.194 (10.1.174.194) 56(84) bytes of data.^C--- 10.1.174.194 ping statistics ---3 packets transmitted, 0 received, 100% packet loss, time 2000ms遇到的问题：
1.docker异常后无法restart testnet3，4容器
docker: Error response from daemon: service endpoint with name testnet3 already exists.解决方案:
etcd中endpoint信息未删除，手动删除吧，查找方法如下
54a450.....是network id，可通过docker network ls查找到
遍历下/calico/docker/network/v1.0/endpoint/54a450c3984853b3942738163cfaaa7dd247686ccc10b8f395dfb807df11e2bb/的所有数据就能找到对应的数据手工删除
# etcdctl --endpoint 'http://192.168.75.200:2379' get /calico/docker/network/v1.0/endpoint/54a450c3984853b3942738163cfaaa7dd247686ccc10b8f395dfb807df11e2bb/5d9cad95e7193e47177eb6d8bdfa25ebc878d8565c48227861^Cf6700136a10c
{"anonymous":false,"disableResolution":false,"ep_iface":{"addr":"10.1.174.198/32","dstPrefix":"cali","mac":"ee:ee:ee:ee:ee:ee","routes":["169.254.1.1/32"],"srcName":"temp5d9cad95e71","v4PoolID":"CalicoPoolIPv4","v6PoolID":""},"exposed_ports":[],"generic":{"com.docker.network.endpoint.exposedports":[],"com.docker.network.portmap":[]},"id":"5d9cad95e7193e47177eb6d8bdfa25ebc878d8565c48227861f6f6700136a10c","locator":"","myAliases":null,"name":"testnet1","sandbox":"bc9abf7c29a9532500aeb9618b22254eab9e73aecc9d4b6c3bf488b6d173791e"}2.node访问其他node上的container不通
默认net1和net2的profile是允许tag相同的访问endpoint，但是calico node默认无法访问，需要修改profile
# calicoctl get profile net1 -o yaml > /etc/calico/profile_net1.yaml
# vi /etc/calico/profile_net1.yaml
- apiVersion: v1 kind: profile metadata: name: net1 tags: - net1 spec: egress: - action: allow destination: {} source: {} ingress: - action: allow destination: {} source: tag: net1#下面是新加的rule - action: allow destination: {} source: net: 192.168.75.0/24 - action: allow destination: {} source: net: 10.1.174.192/32 - action: allow destination: {} source: net: 10.1.166.128/32# calicoctl create -f /etc/calico/profile_net1.yaml
Successfully created 1 'policy' resource(s)10.1.174.192/32和10.1.166.128/32是docker1和docker2的tunl0的ip，手工配置这个还是比较繁琐，应该写成脚本做这个工作
再在集群中任意一台node上ping另外一台node上随便一台net1下的container都能ping通了
