linux系统tcpdump丢包问题解决方法

在一台suse linux上使用tcpdump命令抓包,出现“packets dropped by kernel”,一般造成这种丢包的原因是libcap抓到包后,tcpdump上层没有及时取出,导致libcap缓冲区溢出,从而覆盖了未处理包,显示为dropped by kernel,这里的kernel并不是说是被linux内核抛弃的,而是被tcpdump的内核,即libcap抛弃掉。
解决方法:

根据以上分析,可以通过改善tcpdump上层的处理效率来减少丢包率,下面的几步根据需要选用,每一步都能减少一定的丢包率。

1.最小化抓取过滤范围,即通过指定网卡,端口,包流向,包大小减少包数量

2. 添加-n参数,禁止反向域名解析
tcpdump -i eth0 dst port 1234 and udp -s 2048 -n -X -tt >a.pack
大多数情况这样就可以解决
可以通过改善tcpdump上层的处理效率来减少丢包率

3. 将数据包输出到cap文件
tcpdump -i eth0 dst port 1234 and udp -s 2048 -n -X -tt -w a.cap

4. 用sysctl修改SO_REVBUF参数,增加libcap缓冲区长度

华为AR1200-S/AR1200设备web页面配置步骤

配置步骤:

system-view

[HUAWEI]http server enable(开启http服务)

[HUAWEI]interface vlanif 1

[HUAWEI-Vlanif1]ip add 192.168.1.1 255.255.255.0 (配置IP地址)

[HUAWEI-Vlanif1]quit

[HUAWEI]aaa

[HUAWEI-aaa]local-user huawei password cipher huawei@123(配置用户名huawei和密码huawei@123)

[HUAWEI-aaa]local-user huawei privilege level 15(配置用户名权限)

[HUAWEI-aaa]local-user huawei service-type http(配置用户名服务类型)

[HUAWEI-aaa]quit

[HUAWEI]quit

save

y

rp_filter及Linux下多网卡接收多播的问题

有一台双网卡的机器,上面装有Fedora8,运行一个程序。该程序分别在两个网口上都接收多播数据,程序运行是正常的。但是,后来升级系统到Fedora13,发现就出问题了:在运行几秒钟后,第2个网口上就接收不到多播数据了。

能不能收到多播,取决于交换机是不是往这个网口上转发多播数据。程序在起动的时候,会发一个IGMP的AddMembership的消息,交换机将把这个网口加入多播组。当在其他网口上收到该地址的多播包后,会转至这个网口。其后,为了确认该接收者一直在线,交换机会发送一个IGMPQuery消息,接收者反馈一个IGMP Report消息,以确认自己的存在。如果交换机没有收到IGMPReport,则认为该接收者已经断线,就不再往该网口上转发多播包了。

用抓包工具定位了一下,发现程序在启动时确实发了AddMembership消息,这是正常的。在接收下来的5秒时间内,程序能够收到多播数据。接着,交换机发来了一条IGMPQuery,问题来了,这个Fedora13系统却没有反馈Report。这是很奇怪的。按理说,IGMP属于系统自动完成的协议,无需用户干预;那么按照预期,Linux会自动反馈IGMPReport的。事实上,Feodra8和WinXP系统就是这么做的,都很正常。为什么到了Fedora13反而不正常了呢?

在调查“为什么不反馈IGMPReport”的事情上,花了一周时间都没有进展,后来发现其实不至Fedora13,其他的主流linux如Ubuntu10,SUSE14也存在同样的问题。

查了众多论坛都没有一点提示信息。后来,终于在一个英文网站上扫到了一个信息:rp_filter。后来证明,这个关键词是解决问题的关键。reverse-pathfiltering,反向过滤技术,系统在接收到一个IP包后,检查该IP是不是合乎要求,不合要求的IP包会被系统丢弃。该技术就称为rpfilter。怎么样的包才算不合要求呢?例如,用户在A网口上收到一个IP包,检查其IP为B。然后考查:对于B这个IP,在发送时应该用哪个网口,“如果在不应该接收到该包的网口上接收到该IP包,则认为该IP包是hacker行为”。

例如:

A: 192.168.8.100

B: (IGMP Query) 10.0.0.1 来自路由器

查找路由表

网卡1为默认路由: 172.17.5.100  172.17.5.1

网卡2          192.168.8.100  192.168.8.1

系统根据路由表,认为10.0.0.1这个IP应该在第一个网卡172.17.5.100上收到,现实的情况是在第二张网卡192.168.8.100上收到了。认为这是不合理的,丢弃该包。致命的问题的,该包是来自路由器的IGMPQuery包。

The rp_filter can reject incoming packets if their sourceaddress doesn’t match the network interface that they’re arrivingon, which helps to prevent IP spoofing. Turning this on, however,has its consequences: If your host has several IP addresses ondifferent interfaces, or if your single interface has multiple IPaddresses on it, you’ll find that your kernel may end up rejectingvalid traffic. It’s also important to note that even if you do notenable the rp_filter, protection against broadcast spoofing isalways on. Also, the protection it provides is only against spoofedinternal addresses; external addresses can still be spoofed.. Bydefault, it is disabled.

解决方法:

系统配置文件
1. /etc/sysctl.conf
把 net.ipv4.conf.all.rp_filter和net.ipv4.conf.default.rp_filter设为0即可
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
系统启动后,会自动加载这个配置文件,内核会使用这个变量

2. 命令行
显示一个内核变量 sysctl net.ipv4.conf.all.rp_filter
设置一个内核变量 sysctl -w net.ipv4.conf.all.rp_filter=0
设置完后,会更新内核(实时的内存)中的变量的值,但不会修改sysctl.conf的值

3. 使用/proc文件系统
查看 cat /proc/sys/net/ipv4/conf/all/rp_filter
设置 echo “0”>/proc/sys/net/ipv4/conf/all/rp_filter

sysctl.conf文件参数rp_filter

系统:Centos6

影响:

路径:/etc/sysctl.conf

rp_filter – INTEGER 0 – No source validation.
1 – Strict mode as defined in RFC3704 Strict Reverse Path Each incoming packet is tested against the FIB and if theinterface is not the best reverse path the packet check will fail.By default failed packets are discarded.
2 – Loose mode as defined in RFC3704 Loose Reverse Path Each incoming packet’s source address is also tested against theFIB and if the source address is not reachable via any interface the packet check will fail.Current recommended practice in RFC3704 is to enable strictmode to prevent IP spoofing from DDos attacks. If using asymmetricrouting or other complicated routing, then loose mode is recommended.

The max value from conf/{all,interface}/rp_filter is used when doing source validation on the {interface}.

Default value is 0. Note that some distributions enable it in startup scripts.
—–

Red Hat are (correctly) setting rp_filter to 1, strictmode.  In this case a packet coming in eth0 willhave its source address routed out on the same interface that itcame in on (because that’s the default route). However, a packet coming in on eth1 will have it source addressrouted out on a different interface to the one it came in on and itwill be discarded.  Silently.

This is basically asymmetric routing and is quite possibly not whatyou want anyway (it messes up TCP flow control) so there are twoways to fix this: stick with asymmetric routing and permit it orfix the asymmetric routing.

The first one is easiest: in /etc/sysctl.conf change rp_filter=1 torp_filter=2).  You’ll need to load that andrestart the network.  It’s probably easiest toreboot 🙂 to be sure.  I suspect that it was notrestarting enough things that prevented this change from workingbefore.

The second one may be simple as simple as adding those routes thatshould go out on eth1 to the routing table or running some routingdaemon.  It depends on your network topology,basically.  This would be the preferred solutionif it’s practicable.

reverse-pathfiltering,反向过滤技术,系统在接收到一个IP包后,检查该IP是不是合乎要求,不合要求的IP包会被系统丢弃。该技术就称为rpfilter。怎么样的包才算不合要求呢?例如,用户在A网口上收到一个IP包,检查其IP为B。然后考查:对于B这个IP,在发送时应该用哪个网口,“如果在不应该接收到该包的网口上接收到该IP包,则认为该IP包是hacker行为”。

解决方法:

系统配置文件
1. /etc/sysctl.conf
把 net.ipv4.conf.all.rp_filter和net.ipv4.conf.default.rp_filter设为0即可
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0

net.ipv4.conf.eth0.rp_filter = 0
net.ipv4.conf.eth1.rp_filter = 0

net.ipv4.conf.lo.rp_filter = 0

系统启动后,会自动加载这个配置文件,内核会使用这个变量

2. 命令行
显示一个内核变量 sysctl net.ipv4.conf.all.rp_filter
设置一个内核变量 sysctl -w net.ipv4.conf.all.rp_filter=0
设置完后,会更新内核(实时的内存)中的变量的值,但不会修改sysctl.conf的值

3. 使用/proc文件系统
查看 cat /proc/sys/net/ipv4/conf/all/rp_filter
设置 echo “0”>/proc/sys/net/ipv4/conf/all/rp_filter

sysctl常用命令

sysctl -a 查看所有参数

sysctl -p 加载配置文件

sysctl对Linux内核/网络的设置说明

通过/etc/sysctl.conf控制和配置Linux内核及网络设置。

#忽略icmp ping广播包,应开启,避免放大攻击
net.ipv4.icmp_echo_ignore_broadcasts = 1

# 开启恶意icmp错误消息保护
net.ipv4.icmp_ignore_bogus_error_responses = 1

# 开启SYN洪水攻击保护,表示开启SYN Cookies。当出现SYN等待队列溢出时,启用cookies来处理,可防范少量SYN攻击,默认为0,表示关闭
net.ipv4.tcp_syncookies = 1

# 开启并记录欺骗,源路由和重定向包
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1

# 处理无源路由的包
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# reverse-pathfiltering 反向路径过滤,系统收到一个ip包后,会反查该ip包的ip是否与它们到达的网络接口匹配,若不匹配则丢弃。是防ip包欺骗策略。

# The rp_filter can reject incoming packets if their sourceaddress doesn’t match the network interface that they’re arrivingon, which helps to prevent IP spoofing. Turning this on, however,has its consequences: If your host has several IP addresses ondifferent interfaces, or if your single interface has multiple IPaddresses on it, you’ll find that your kernel may end up rejectingvalid traffic. It’s also important to note that even if you do notenable the rp_filter, protection against broadcast spoofing isalways on. Also, the protection it provides is only against spoofedinternal addresses; external addresses can still be spoofed.. Bydefault, it is disabled.
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

#关闭重定向。如果主机所在的网络有多个路由器,你将其中一个设为缺省网关,但该网关在收到你的ip包时,发现该ip包必须经过另外一个路由器,于是该网关就給你的主机发一个“重定向”的icmp包,告诉主机把包转发到另外一个路由器。1表示主机接受这样的重定向包,0表示忽略;linux默认是1,可以设位0以消除隐患。
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0

#禁止数据包转发,不做路由器功能。所谓转发即当主机拥有多网卡时,其中一块收到数据包,根据数据包的目的ip地址将包发往本机另一网卡,该网卡根据路由表继续发送数据包。这通常就是路由器所要实现的功能。

#对比网关:内网主机向公网发送数据包时,由于目的主机跟源主机不在同一网段,所以数据包暂时发往内网默认网关处理,而本网段的主机对此数据包不做任何回应。由于源主机ip是私有的,禁止在公网使用,所以必须将数据包的源发送地址修改成公网上的可用ip,这就是网关收到数据包之后首先要做的工作–ip转换。然后网关再把数据包发往目的主机。目的主机收到数据包之后,只认为这是网关发送的请求,并不知道内网主机的存在,也没必要知道,目的主机处理完请求,把回应信息发还给网关。网关收到后,将目的主机发还的数据包的目的ip地址修改为发出请求的内网主机的ip地址,并将其发给内网主机。这就是网关的第二个工作–数据包的路由转发。内网的主机只要查看数据包的目的ip与发送请求的源主机ip地址相同,就会回应,这就完成了一次请求。 net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# 开启execshield,execshield 主要用于随机化堆栈地址,避免被exploit 程序修改恶意地址,而导致执行攻击程序。
kernel.exec-shield = 1
kernel.randomize_va_space = 1

# IPv6设置
net.ipv6.conf.default.router_solicitations = 0
net.ipv6.conf.default.accept_ra_rtr_pref = 0
net.ipv6.conf.default.accept_ra_pinfo = 0
net.ipv6.conf.default.accept_ra_defrtr = 0
net.ipv6.conf.default.autoconf = 0
net.ipv6.conf.default.dad_transmits = 0
net.ipv6.conf.default.max_addresses = 1

# 增加系统文件描述符限制
fs.file-max = 65535

# 允许更多的PIDs (减少滚动翻转问题);
may break some programs 32768
kernel.pid_max = 65536

# 增加系统IP端口限制 n
et.ipv4.ip_local_port_range = 2000 65000

# 增加TCP最大缓冲区大小
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 87380 8388608

# 增加Linux自动调整TCP缓冲区限制

# 最小,默认和最大可使用的字节数

# 最大值不低于4MB,如果你使用非常高的BDP路径可以设置得更高

# Tcp窗口等
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_window_scaling = 1

Suse linux 11配置多网卡后网络不通的解决办法

问题描述

1.RH2288-1、RH2288-2安装的是SUSE11操作系统。
2.服务器采的是双网卡绑定,bond0在VLAN100、bond1在VLAN200。
3.VLAN100的网关为172.16.0.1、VLAN200的网关为192.168.0.1。
RH2288-1的默认网关为192.168.0.1,RH2288-2的默认网关为172.16.0.1。
4.RH2288-2 ping RH2288-1的bond1网卡 192.168.0.187不通。

告警信息
FY-NMS:~ # ping -c 1 192.168.0.187
PING 192.168.0.187 (192.168.0.187) 56(84) bytes of data.
— 192.168.0.187 ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

处理过程
1.在RH2288-1、RH2288-2 ping 各自默认网关是否正常。
结果正常,无丢包。
2.检查RH2288-1、RH2288-1默认网关配置正确。
RH2288-1配置正确,检查结果如下:
FY-HIS:/etc # route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 bond1
RH2288-2配置正确,检查结果如下:
FY-NMS ~ # route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 172.16.0.1 0.0.0.0 UG 0 0 0 bond0
3.在RH2288-1抓取网络数据包,是否能收到RH2288-2请求的ping请求。
FY-HIS:~ # tcpdump -ni bond1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond1, link-type EN10MB (Ethernet), capture size 96 bytes
09:45:38.900680 IP 172.16.0.188 > 192.168.0.188: ICMP echo request, id 24523, seq 54, length 64
09:45:39.908686 IP 172.16.0.188 > 192.168.0.188: ICMP echo request, id 24523, seq 55, length 64
09:45:40.916731 IP 172.16.0.188 > 192.168.0.188: ICMP echo request, id 24523, seq 56, length 64
09:45:41.924674 IP 172.16.0.188 > 192.168.0.188: ICMP echo request, id 24523, seq 57, length 64
4.根据抓包显示,从RH2288-2 ping RH2288-1 192.168.0.188不通,是因为在RH2288-1上只能抓到RH2288-2的request报文,但没有抓到RH2288-1 的reply的报文。

原因
问题产生的根因应该是SUSE 系统路由表策略,优先级低的路由,它不回复对端的arp请求。

解决方案
1.sysctl -a |grep rp_fi,查看返回值是否都为0。
2.将不为0的值改为0。
如net.ipv4.conf.all.rp_filter = 1,则修改sysctl -w net.ipv4.conf.all.rp_filter=0
3.将该修改值写入配置文件:
vi /etc/sysctl.conf
在该文件中将需要修改为0的值进行更改,需要注意格式与文件中其他配置一致。

建议与总结
在处理服务器侧网络不通时,建议多使用抓包的方法,通过分析网络报文快速定位问题。

DELL R720xd服务器故障处理两例

例一:
一台DELL R720xd服务器开不了机,按开机按钮后,主板风扇转,显示器无任何显示,最小化测试发现仍然开不了机,更换主板后,拔掉所有PCI-E RISE卡,最小化可正常开机,但是接上各硬件后,仍然开不了机,逐个测试发现后置硬盘背板故障,更换硬盘背板后,可正常开机。

例二:
一台DELL R720xd服务器开机后,自检完阵列卡网卡后,出现:
Strike the F1 key to continue,F2 to run the system setup program
此时,键盘一旦按F1或F2后即刻宕机,但是其他按键可以按不会宕机,当然也会继续卡在该界面处,并且idrac卡可以进入,但是硬件日志无任何问题记录,且无法更新固件,上传固件后,由于更新固件依赖lifecycle controller,而此时自检仍未进入该阶段,所以固件也无法更新,并且在idrac里的KVM里按F1或者F2一样会宕机,对于该问题,更换主板后解决。

HP DL180 G6故障处理一例

一台HP DL180 G6服务器开不了机,按下开机按钮后,开机按钮未变绿色,主板风扇正常转,拔下前置硬盘背板、光驱、PCI扩展卡等,做最小化测试,仍然开不了机,更换主板后,问题依旧,更换电源分配板后,问题才解决。

当按下开机按钮后,开机按钮仍是黄色,没有变成绿色,很可能是供电方面的问题,比如电源或电源分配板。

Linux LVM分区扩容方法

现有一台Linux服务器,采用LVM分区,现在新增一块10G硬盘,需将10G硬盘扩容给/data分区,操作步骤如下:

1、vgdisplay查看vg,vg name为VolGroup

2、fdisk -l查看新增硬盘为sdb,
pvcreat /dev/sdb
pvdisplay查看pv

3、df -h
查看/data所在的逻辑卷的名称

4、将/dev/sdb增加到vg里,vg name为VolGroup
查看vg,vg容量已经由39.8G变为49.8G

5、lvextend -L +9.8G 将9.8G增加到文件系统

6、df -h后容量没变,还需要
#e2fsck -f /dev/mapper/VolGroup-lv_root
#resize2fs /dev/mapper/VolGroup-lv_root

7、df -h已经可以看到/data已经由20G变为30G

 

vSphere 6.5密码重置(vCenter, SSO and ESXi)

Everyone knows the situation where you can’t log into a system because you have forgotten the password. The following article explains how to reset the password and regain access to VMware vSphere 6.5 core components including vCenter, SSO and ESXi Hosts.

  • Reset vCenter Server Appliance 6.5 root password
  • Reset SSO Administrator Password (vCenter Server Appliance 6.5)
  • Reset ESXi root password with Host Profiles
  • Gain Administrative ESXi access with an Active Directory
  • Reset ESXi root password (Linux Live CD)

Reset vCenter Server Appliance 6.5 root password

The following method provides steps to recover the vCenter Server Appliance (vCSA) root password. The process is slightly different compared to previous versions as the OS has been changed to PhotonOS. The method is officially supported by VMware and documented in KB2147144.

  1. Take a snapshot of the vCSA to be able to rollback in case of any problems during password recovery.
  2. Connect to the ESXi Host that runs the vCSA and open a remote console.
  3. Reboot the vCSA
  4. Press e immediately after the system starts (When the PhotonOS screen shows up)
  5. Append rw init=/bin/bash to the line starting with linux

  6. Press F10 to boot
  7. In the command prompt, enter passwd and enter a new root password twice
  8. Enter umount / to unmount the root filesystem
  9. Reboot the vCSA by running the command reboot -f
  10. Verify that you can log in with the new root password and delete the snapshot created in step 1.

 

Reset SSO Administrator Password (vCenter Server Appliance 6.5)

The following method provides steps to recover the SSO administrator password on a vCenter Server Appliance (vCSA). The method is officially supported by VMware and documented in KB2146224.

  1. Log in to the vCSA using SSH as root
  2. Enter shell to start the bash shell
  3. Identify the SSO Domain Name (Default is vsphere.local)
    # /usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost

  4. Start the vdcadmintool

    # /usr/lib/vmware-vmdir/bin/vdcadmintool
  5. Press 3 (Reset account password)
  6. The tool asks for the Account UPN to reset. Enter Administrator@<DOMAIN> (identified in Step 3)
  7. The tool generates and displays a new password.
  8. Use the password to log in with the vSphere Web Client and change the password.

 

Reset ESXi root password with Host Profiles

According to VMware KB1317898, “reinstalling the ESXi host is the only supported way to reset a password on ESXi”. However, there is a loophole as you can set the root password with Host Profiles under certain conditions. This method has two requirements:

  • The ESXi hosts needs to be managed by a vCenter
  • vSphere Enterprise Plus License is required to use Host Profiles

The vCenter uses a vpxuser to communicate with ESXi hosts, so it does not depend on the root account. As long as the ESXi host is managed by the vCenter, you can change the configuration without knowing the ESXi root password. This method works with all ESXi 5.x and 6.x versions.

  1. Create a Host Profile with the ESXi you want to reset the root password as reference Host
    Web Client > Right-Click the ESXi Host > Host Profiles > Extract Host Profile…
  2. Navigate to the Host Profile and select Actions > Edit Settings…
  3. Navigate to the root User Configuration
    Security and Services > Security Settings > Security > User Configuration > root
  4. Set the Password configuration to Fixed password configuration and enter a new password.
  5. Click Finish to close the profile configuration
  6. Right-Click the Host Profile and select Attach/Detach Hosts and Clusters…
  7. Highlight the ESXi host, Click Attach > and finish the configuration screen

  8. Put the ESXi host into maintenance mode
  9. Right-Click the ESXi host and select Host Profiles > Remediate…
  10. Finish the remediation wizard. The remediation should take less than a minute, no reboot is required.
  11. Use the new root password to login

 

Gain Administrative ESXi access with an Active Directory

When you don’t have the Enterprise Plus license, you can join an Active Directory to regain administrative access to the ESXi host. This method circumvents the limitation that root PW recovery is not supported.

  1. Login to the vCenter with the vSphere Web Client
  2. Navigate to ESXi > Configure > System > Authentication Services
  3. Click Join Domain…
  4. Enter the domain name and user credentials
  5. Click OK
  6. In the ESXi configuration, open System > Advanced System Settings
  7. Enter Config.HostAgent.plugins.hostsvc.esxAdminsGroup in the search field
  8. Change the settings to match the Administrator group that you want to use in the Active Directory. You can either create a new group in your direcotry or enter an existing group

 

Reset ESXi root password (Linux Live CD)

When you need to recover root access and the methods above are not applicable, the last method explains how to reset the root password with a Linux Live CD. Please be aware that this method is not supported by VMware as KB1317898 states: “reinstalling the ESXi host is the only supported way to reset a password on ESXi”. You can use any current Linux Live CD or installer CD that has a recovery mode. In this example I’m using Knoppix.

  1. Shutdown the ESXi host
  2. Boot the system with the Linux Live CD
  3. Make sure that you can read the gpt partition table, for example with parted /dev/sda print
  4. We are looking for the first fat16 partition with a size of 262MB. IT should be number 5.
  5. Mount the partition
    # mount /dev/sda5 /media/sda5
  6. Verify that there is a current state.tgz in the directory.
  7. # ls -l /media/sda5/state.tgz

  8. The state.tgz file contains the local.tgz file which contains the configuration. Extract both to a temporary directory.
    # cd /tmp/
    # cp /media/sda5/state.tgz /tmp/state.tgz
    # tar -xf state.tgz
    # tar -xf local.tgz
  9. Edit the shadow file and remove the root password
    # vi etc/shadow

    Remove the hashed password until the second colon:

    You want a file that looks like this:

  10. Save the file and exit the editor (<ESC> :wq <ENTER>)
  11. Recreate state.tgz with the changed shadow file
    # tar -czf local.tgz etc
    # tar -czf state.tgz local.tgz
  12. Move state.tgz back to ESXi partition and make sure to overwrite the old file
    # mv state.tgz /media/sda5/
  13. Reboot to ESXi. You should be able to access the DCUI or log in as root without a password.