1. 通常批量处理的时候会遇到ip down的问题,会等待非常久,可以加个超时
ssh -o ConnectTimeout=3

2. 如果要在远端机器使用变量怎么办? 因为变量会在本地机器被默认解析, 比如awk的$1, 所以可以加个反斜杠\
ssh -o ConnectTimeout=3 $IP " cat filelist |awk '{print \$1}' "

3. 如果要把远端运行结果赋值给某个变量怎么办?,因为执行的命令也会在本地机器被默认解析,所以也需要加个反斜杠\
ssh $IP “df_data= \`df\`; echo \$df_data ”

4. 另外,如果是逐个IP批处理,很容易遇到执行一个就退出的情况,需要加个n
ssh -n $IP

rp_filter, reverse-path filtering,反向过滤技术,系统在接收到一个IP包后,检查该IP是不是合乎要求,不合要求的IP包会被系统丢弃。该技术就称为rp filter。怎么样的包才算不合要求呢?例如,用户在A网口上收到一个IP包,检查其IP为B。然后考查:对于B这个IP,在发送时应该用哪个网口,“如果在不应该接收到该包的网口上接收到该IP包,则认为该IP包是hacker行为”。

例如:

A: 192.168.8.100

B: (IGMP Query) 10.0.0.1 来自路由器

查找路由表

网卡1为默认路由: 172.17.5.100 172.17.5.1

网卡2 192.168.8.100 192.168.8.1

系统根据路由表,认为10.0.0.1这个IP应该在第一个网卡172.17.5.100上收到,现实的情况是在第二张网卡192.168.8.100上收到了。认为这是不合理的,丢弃该包。致命的问题的,该包是来自路由器的IGMP Query包。

The rp_filter can reject incoming packets if their source address doesn’t match the network interface that they’re arriving on, which helps to prevent IP spoofing. Turning this on, however, has its consequences: If your host has several IP addresses on different interfaces, or if your single interface has multiple IP addresses on it, you’ll find that your kernel may end up rejecting valid traffic. It’s also important to note that even if you do not enable the rp_filter, protection against broadcast spoofing is always on. Also, the protection it provides is only against spoofed internal addresses; external addresses can still be spoofed.. By default, it is disabled.

rp_filter参数在升级到RHEL6之后的版本后出现了比较大的改变,我们看下内核参数解析

在RHEL5的时候这个内核参数的含义:


/usr/share/doc/kernel-doc-2.6.18/Documentation/networking/ip-sysctl.txt

rp_filter - BOOLEAN
        1 - do source validation by reversed path, as specified in RFC1812
            Recommended option for single homed hosts and stub network
            routers. Could cause troubles for complicated (not loop free)
            networks running a slow unreliable protocol (sort of RIP),
            or using static routes.

        0 - No source validation.

        conf/all/rp_filter must also be set to TRUE to do source validation
        on the interface

        Default value is 0. Note that some distributions enable it
        in startup scripts.

在RHEL6 RHEL7之后这个内核参数的含义:


/usr/share/doc/kernel-doc-2.6.32/Documentation/networking/ip-sysctl.txt

rp_filter - INTEGER
        0 - No source validation.
        1 - Strict mode as defined in RFC3704 Strict Reverse Path 
            Each incoming packet is tested against the FIB and if the interface
            is not the best reverse path the packet check will fail.
            By default failed packets are discarded.
        2 - Loose mode as defined in RFC3704 Loose Reverse Path 
            Each incoming packet's source address is also tested against the FIB
            and if the source address is not reachable via any interface
            the packet check will fail.

        Current recommended practice in RFC3704 is to enable strict mode 
        to prevent IP spoofing from DDos attacks. If using asymmetric routing
        or other complicated routing, then loose mode is recommended.

        The max value from conf/{all,interface}/rp_filter is used 
        when doing source validation on the {interface}.

        Default value is 0. Note that some distributions enable it
        in startup scripts.

所以如果需要和RHEL5的行为保持一致,应该设置


net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2

否则在多IP/多网卡/多网段的时候,会出现非预期的丢弃数据包的问题
具体可以参考:
redhat 官方文档: https://access.redhat.com/solutions/53031
案例分析: http://www.cnblogs.com/huazi/archive/2013/02/25/2932021.html

lvs 需要配置下内核参数,否则会发生抢VIP断流之类的异常情况,本文档主要介绍DR模式下
首先是LVS 前端,需要设置两个参数arp_ignore和arp_announce


       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore

       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce

       echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore

       echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce

而后端的节点,也需要设置对应的参数


       echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

       echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

       echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

       echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

这里解释下这两个参数的含义:

arp_ignore - INTEGER
	Define different modes for sending replies in response to
	received ARP requests that resolve local target IP addresses:
	0 - (default): reply for any local target IP address, configured
	on any interface
	1 - reply only if the target IP address is local address
	configured on the incoming interface
	2 - reply only if the target IP address is local address
	configured on the incoming interface and both with the
	sender's IP address are part from same subnet on this interface
	3 - do not reply for local addresses configured with scope host,
	only resolutions for global and link addresses are replied
	4-7 - reserved
	8 - do not reply for all local addresses

	The max value from conf/{all,interface}/arp_ignore is used
	when ARP request is received on the {interface}

简单翻译下:
arp_ignore:定义对目标地址为本地IP的ARP询问不同的应答模式0

0 – (默认值): 回应任何网络接口上对任何本地IP地址的arp查询请求

1 – 只回答目标IP地址是来访网络接口本地地址的ARP查询请求

2 -只回答目标IP地址是来访网络接口本地地址的ARP查询请求,且来访IP必须在该网络接口的子网段内

3 – 不回应该网络界面的arp请求,而只对设置的唯一和连接地址做出回应

4-7 – 保留未使用

8 -不回应所有(本地地址)的arp查询


arp_announce - INTEGER
	Define different restriction levels for announcing the local
	source IP address from IP packets in ARP requests sent on
	interface:
	0 - (default) Use any local address, configured on any interface
	1 - Try to avoid local addresses that are not in the target's
	subnet for this interface. This mode is useful when target
	hosts reachable via this interface require the source IP
	address in ARP requests to be part of their logical network
	configured on the receiving interface. When we generate the
	request we will check all our subnets that include the
	target IP and will preserve the source address if it is from
	such subnet. If there is no such subnet we select source
	address according to the rules for level 2.
	2 - Always use the best local address for this target.
	In this mode we ignore the source address in the IP packet
	and try to select local address that we prefer for talks with
	the target host. Such local address is selected by looking
	for primary IP addresses on all our subnets on the outgoing
	interface that include the target IP address. If no suitable
	local address is found we select the first local address
	we have on the outgoing interface or on all other interfaces,
	with the hope we will receive reply for our request and
	even sometimes no matter the source IP address we announce.

	The max value from conf/{all,interface}/arp_announce is used.

	Increasing the restriction level gives more chance for
	receiving answer from the resolved target while decreasing
	the level announces more valid sender's information.

arp_announce:对网络接口上,本地IP地址的发出的,ARP回应,作出相应级别的限制: 确定不同程度的限制,宣布对来自本地源IP地址发出Arp请求的接口

0 – (默认) 在任意网络接口(eth0,eth1,lo)上的任何本地地址

1 -尽量避免不在该网络接口子网段的本地地址做出arp回应. 当发起ARP请求的源IP地址是被设置应该经由路由达到此网络接口的时候很有用.此时会检查来访IP是否为所有接口上的子网段内ip之一.如果改来访IP不属于各个网络接口上的子网段内,那么将采用级别2的方式来进行处理.

2 – 对查询目标使用最适当的本地地址.在此模式下将忽略这个IP数据包的源地址并尝试选择与能与该地址通信的本地地址.首要是选择所有的网络接口的子网中外出访问子网中包含该目标IP地址的本地地址. 如果没有合适的地址被发现,将选择当前的发送网络接口或其他的有可能接受到该ARP回应的网络接口来进行发送.

这里有个附加的例子解释

Assume that a linux box X has three interfaces - eth0, eth1 and eth2. Each interface has an IP address IP0,

IP1 and IP2. When a local application tries to send an IP packet with IP0 through the eth2. Unfortunately,

the target node’s mac address is not resolved. Thelinux box X will send the ARP request to know

the mac address of the target(or the gateway). In this case what is the IP source address of the

“ARP request message”? The IP0- the IP source address of the transmitting IP or IP2 - the outgoing

interface? Until now(actually just 3 hours before) ARP request uses the IP address assigned to

the outgoing interface(IP2 in the above example) However the linux’s behavior is a little bit

different. Actually the selection of source address in ARP request is totally configurable

bythe proc variable “arp_announce”

If we want to use the IP2 not the IP0 in the ARP request, we should change the value to 1 or 2.

The default value is 0 - allow IP0 is used for ARP request.

其实就是路由器的问题,因为路由器一般是动态学习ARP包的(一般动态配置DHCP的话),当内网的机器要发送一个到外部的ip包,那么它就会请求 路由器的Mac地址,发送一个arp请求,这个arp请求里面包括了自己的ip地址和Mac地址,而linux默认是使用ip的源ip地址作为arp里面 的源ip地址,而不是使用发送设备上面的 ,这样在lvs这样的架构下,所有发送包都是同一个VIP地址,那么arp请求就会包括VIP地址和设备 Mac,而路由器收到这个arp请求就会更新自己的arp缓存,这样就会造成ip欺骗了,VIP被抢夺,所以就会有问题。

arp缓存为什么会更新了,什么时候会更新呢,为了减少arp请求的次数,当主机接收到询问自己的arp请求的时候,就会把源ip和源Mac放入自 己的arp表里面,方便接下来的通讯。如果收到不是询问自己的包(arp是广播的,所有人都收到),就会丢掉,这样不会造成arp表里面无用数据太多导致 有用的记录被删除。

在设置参数的时候将arp_ignore 设置为1,意味着当别人的arp请求过来的时候,如果接收的设备上面没有这个ip,就不做出响应,默认是0,只要这台机器上面任何一个设备上面有这个ip,就响应arp请求,并发送mac地址

参考文档: http://www.cnblogs.com/lgfeng/archive/2012/10/16/2726308.html

1. syslog-ng 是支持模板的,可以自由定义格式,比如
destination d_access {
file(“/opt/itc/syslog-ng/logs/$PROGRAM/$YEAR$MONTH$DAY/access_log”
create_dirs(yes)
dir_owner(“nobody”)
dir_group(“nobody”)
dir_perm(0755)
owner(“nobody”)
group(“nobody”)
perm(0644)
template(“${HOUR}:${MIN}:${SEC} ${HOST} ${PROGRAM} ${MSG}\n”)
);
};

2. 这个例子又刚好会触发一个小窍门,一般加了template的会发现数据丢了第一项,这是因为:
syslog-ng 会把第一列 当做是 $PROGRAM ,之后的那些内容才是 MSG

3. 那么,我想加PROGRAM 让syslog-ng 识别,在收集日志的机器分散出来怎么办? 在source里边定义就好
source s_app {
file(“/opt/itc/nginx/logs/app_access.log”
program_override(“app.4os.org”)
flags(no-parse)
);
};

这个例子里边,就会在原始日志里边自动添加一列 app.4os.org 作为PROGRAM到日志里边去

4. 总结下,实际上,没有template的默认日志格式是怎样的?
${ISODATE} ${HOST} ${PROGRAM} ${MSG}\n

其中日期函数跟定义有关,默认为iso

测试bind-9.8.1-P1 的edns的时候发现,google DNS解析的结果一直在跳,有时候在电信区,有时候在us区

于是给它打了第一个patch,让query log 支持edns的client subnet显示,便于排查

筛选下日志,发现很多edns请求不在我们的ecs ACL范围内,最终落到了google DNS IP 所在的us区域,推测应该是这个造成了google DNS 的错误缓存

于是,把ecs的default区域加入拦截

view "default_ecs" {
match-clients { ecs 0.0.0.0/0; ecs ::/0;};
... ...
};

这样一来,的确是响应了edns请求,结果却是更多的出现了default 区的结果(热数据),或者在client所属区域和default区结果之间跳动(冷数据)

听包,发现google DNS发送过来的请求Scope Netmask 都是0,如果ecs ACL拦截不成功到了default_ecs区域,最终被ecs 0.0.0.0/0; ecs ::/0; 拦截成功,导致返回的Scope Netmask变成了0

Scope Netmask 变为0意味着什么? 意味着这个结果集有效并且范围最大,从而污染所有的subnet client结果集

目前来看,只能给它打个patch

让client subnet 到了default区域的结果集Scope Netmask 为起address netmask长度,控制结果集的有效范围

测试下,把某个网段故意从ecs ACL挪走,让default ecs拦截看看响应是否是我们期待的

正常网段的请求,返回我们ecs ACL的Netmask

上线测试,目前服务正常

 

附上edns 文档,非常重要: 
https://tools.ietf.org/html/draft-vandergaast-edns-client-subnet-01
http://noops.me/?p=653

nginx proxy模式下502 bad gateway 问题

并发测试的时候发现nginx 502 bad gateway 了,看了下日志发现很多upstream Cannot assign requested address的记录
connect() to 192.168.89.170:80 failed (99: Cannot assign requested address) while connecting to upstream

正常判断应该是端口不够用了
不过,我确实开启了: net.ipv4.tcp_tw_recycle = 1 和 net.ipv4.tcp_tw_reuse = 1两个参数
理论上应该可以把timewait 端口重用,查了下这个参数跟tcp_timestamps有关(http://blog.sina.com.cn/s/blog_781b0c850100znjd.html)

if (tmp_opt.saw_tstamp &&
tcp_death_row.sysctl_tw_recycle &&
(dst = inet_csk_route_req(sk, req)) != NULL &&
(peer = rt_get_peer((struct rtable *)dst)) != NULL &&
peer->v4daddr == saddr) {
if (get_seconds() < peer->tcp_ts_stamp + TCP_PAWS_MSL &&
(s32)(peer->tcp_ts – req->ts_recent) >
TCP_PAWS_WINDOW) {
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_PAWSPASSIVEREJECTED);
goto drop_and_release;
}
}

tmp_opt.saw_tstamp:该socket支持tcp_timestamp
sysctl_tw_recycle:本机系统开启tcp_tw_recycle选项
TCP_PAWS_MSL:60s,该条件判断表示该源ip的上次tcp通讯发生在60s内
TCP_PAWS_WINDOW:1,该条件判断表示该源ip的上次tcp通讯的timestamp 大于 本次tcp

因此: 应该在proxy端和后端都开启net.ipv4.tcp_timestamps=1

网卡LACP聚合配置

本文主要介绍多网卡lacp 聚合模式配置,即mode=4模式下RHEL6 bonding 和 RHEL7 team的配置

此模式需要在交换机做配置LACP聚合,具体参考交换机设置
配置范例以双网卡eth0 eth1为例,多网卡类推

RHEL6 配置

1.新增配置文件/etc/modprobe.d/bonding.conf,内容如下

alias bond0 bonding
options bond0 miimon=100 mode=4 lacp_rate=1 xmit_hash_policy=layer3+4

2.更改网卡配置

#/etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

#/etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

#/etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.1.233
NETMASK=255.255.255.0
NETWORK=192.168.1.0
GATEWAY=192.168.1.254

3.重启网络

service network restart

4.注意点: 参见最后附录

 

RHEL7 配置

1. 修改网卡配置

# /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=”eth0″
ONBOOT=yes
UUID=”原来的UUID”
DEVICETYPE=TeamPort
TEAM_MASTER=team1

#/etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
DEVICETYPE=TeamPort
UUID=”原来的UUID”
ONBOOT=yes
TEAM_MASTER=team1

新增配置: /etc/sysconfig/network-scripts/ifcfg-team1
DEVICE=team1
DEVICETYPE=Team
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.168.1.23
PREFIX=24
DEFROUTE=no
TEAM_CONFIG='{“runner”: {“name”:”lacp”, “active”:true, “fast_rate”:true, “tx_hash”:[“ipv4”], “ports”:{“eth0”:{}, “eth1”:{}}}}’
MTU=1476

2. 重新启动机器,是的,重新启动机器

问题研究

 
1. RHEL6网卡in能分摊到不同网卡,但是out只走一个网卡
检查/proc/net/bonding/bond0 文件,看看 Transmit Hash Policy 是否正确,一般layer2 在内网测试可能会有问题
需要配置xmit_hash_policy=layer3+4 或者 xmit_hash_policy=layer2+3

2. RHEL7 systetemctl restart network 网卡不通
恩,是的,重启机器吧

安排一位同学做Docker测试的时候,用的是台380G5的老机器,结果发现找不到磁盘

那位同学搜了下,是驱动被移除了,HP官方不再支持在新OS下的测试,需要显式打开支持

在kernel 引导时加入 hpsa.hpsa_allow_any=1 就可以了

相关文档:
http://serverfault.com/questions/611182/centos-7-x64-and-hp-proliant-dl360-g5-scsi-controller-compatibility
https://www.kernel.org/doc/Documentation/scsi/hpsa.txt

nginx with static libcurl

场景是这样子的: 这边有个nginx 模块 include curl/curl.h,而我的编译参数–with-openssl使用了最新的openssl 1.0.1g,编译出来的nginx直接segfault

去除这个模块或者去掉–with-openssl都能正常使用,推测是系统的libcurl(https)包含了libssl的依赖,与内嵌的openssl产生冲突

于是解决办法就是把libcurl也编译到nginx里边,绕开冲突和依赖

1. 静态编译libssl
cd openssl-1.0.1g
./config –prefix=/usr/src/redhat/BUILD/nginx-1.4.7/openssl-1.0.1g/.openssl no-shared no-threads
make
make install
make install LIBDIR=lib

2. 静态编译libcurl
cd curl-7.36.0
./configure –prefix=/usr/src/redhat/BUILD/nginx-1.4.7/curl-7.36.0/.curl –with-ssl=/usr/src/redhat/BUILD/nginx-1.4.7/openssl-1.0.1g/.openssl/lib/ –disable-ldap –disable-ldaps –without-libidn –enable-static=yes –enable-shared=no

#去除对librt.so的依赖,不介意可以不修改
sed -i /HAVE_CLOCK_GETTIME_MONOTONIC/d lib/curl_config.h

make
make install

3. 修改nginx的Makefile
#替换libcurl.so(lcurl)为静态编译的libcurl.a
sed -i ‘s#-lcurl#curl-7.36.0/.curl/lib/libcurl.a -Lopenssl-1.0.1g/.openssl/lib -lcrypto -lz#g’ objs/Makefile
make
make install

做完这步,就生成了包含libcurl和libssl的nginx了

一次错误mv /* /path/to 操作的恢复

一次错误mv /* /path/to 操作的恢复

描述:执行mv命令的时候没有注意路径,结果把根目录下的大部分目录都挪到了一个新路径中,然后立即
出错命令不能继续,因为 /lib已经被挪走了,/lib下保存有最基本的系统运行库,现代的linux系统大多
数命令已经动态连接了,当/lib路径改变的时候,那你能用来创建目录,或用来copy文件的任何命令,都
不好用了,都会报下面错误:
/lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
这表示已经找不到ld-linux.so.2这个文件了

由于文件都已经被挪走了,我们平时依赖的工具都不好用了。即使你敲入绝对路径,也会提示:
/lib/ld-linux.so.2: bad ELF interpreter: No such file or directory这个错误
尝试了下面的命令
/www/users/abcdefg.net/{ls,ln,mkdir,mktemp,ftp,rsync,mv,cp,rm,ldconfig,scp,sftp,perl,ash,zsh,csh} 报的都是上面的错误,
常规的创建文件,创建目录和目录文件转移操作,及远程或本地复制文件操作都无法进行了。
再尝试找下好用的命令:
cd export declare echo > 这些内置命令都是好用的。

我们看下现场,由于ls已经不好用了。那只好用bash的自动补齐功能(TAB键)来看看有哪些目录和文件
首先检查哪些目录被挪走了,执行下面命令:
/www/users/abcdefg.net [按TAB]
输出如下:
backup bin boot dev ecshop etc home lib lost+found media misc mnt opt proc www_logs ucenter
可以看出 bin和lib都已经被挪过来了,那我们既不能直接执行bin下的程序,也不能执行任何依赖/lib下的程序,而且etc也被mv到这里来
所以我们连修改ld.so.conf并生成新的ld.so.cache的机会也没有,因为我们无法执行创建/etc目录的命令。

开始恢复吧
过程不复杂,我们还有ld-linux.so.2 可用,虽然它已经变了位置。
ld-linux.so.2是linux系统的动态连接器,我们可以用他来执行命令,我们使用它的–library-path参数来重新指定LD_LIBRARY_PATH的位置

再看看根目录还有啥
/www/users/abcdefg.net/lib/ld-linux.so.2 –library-path /www/users/abcdefg.net/lib /www/users/abcdefg.net/bin/ls / -F
输出正常,

恢复/lib
/www/users/abcdefg.net/lib/ld-linux.so.2 –library-path /www/users/abcdefg.net/lib /www/users/abcdefg.net/bin/cp -rfp /www/users/abcdefg.net/lib /lib

恢复/bin
/www/users/abcdefg.net/lib/ld-linux.so.2 –library-path /www/users/abcdefg.net/lib /www/users/abcdefg.net/bin/cp -rfp /www/users/abcdefg.net/bin /bin

恢复其他目录
cd /www/users/abcdefg.net
for i in bin boot etc lib media misc mnt opt ; do rsync -av –progress ./$i/ /$i/; done

然后我们再还原dev目录
cd dev
cp -dRp * /dev

重新检查ssh登陆,此时已经能登陆系统了,仔细检查/boot /etc ,都已经正确恢复,重起系统,所有恢复完毕