All ports blocked, ping works, firewall and apparmor off

[Tony Baechler - BATS]

Hi all,

I've been struggling with this for days and am getting nowhere. I'm totally
baffled and have tried everything I can think of, including several Google
searches. I am sorry for the lack of information, but I'm not sure what to
post as I've never seen anything like this before.

I'm running a Ubuntu 14.04.1 server. It has ssh, Postfix, Apache2, etc. I
upgraded from kernel 3.13.04-6-generic to 3.13.04-8-generic and rebooted.
Everything was working fine, but I couldn't connect after the reboot. I
tried nmap and it says all ports are closed. My first thought is that it
must be a firewall, so I completely purged ufw and iptables but no luck. I
restored /etc/network/interfaces from a known good backup.

After reading various forum posts, I again installed iptables and copied and
pasted the script from the official Ubuntu wiki to disable the firewall,
just in case old rules were left. That should allow all incoming and
outgoing traffic. Apparently the script works as pinging works fine even
though nmap says all ports are closed. I've tried rebooting several times
just in case. I ran e2fsck and my disks are clean. I am able to boot into
a separate rescue system, so it's definitely not an obvious hardware
problem. I also ran update-grub and update-initramfs just in case.
Finally, I reconfigured grub-pc and installed the bootloader on both
/dev/sda and /dev/sdb. I'm running software RAID and I didn't install it on
/dev/md1, but I don't think that would matter. I have to access the server
remotely and writing to their support was of no help.

What's really strange is that it boots fine with kvm from the rescue system.
I can get to the login prompt and everything seems to be fine. It acts
like a boot problem, but I don't see why ping would work if it isn't
booting. Nothing gets written to syslog, so it acts like it's an init
problem, but I didn't change anything that I know of and I saw no errors
with kvm. Without kvm, it doesn't seem to boot as I mentioned. Just in
case, I removed or purged qemu, ufw, iptables, apparmor, etc. I also tried
upgrading to a completely different kernel, specifically
3.16.0-33-lowlatency. It's an Intel x86-64 processor with 32 GB of RAM. It
did work fine after the initial install. According to mdadm, my RAID arrays
are fine.

Again, I've tried everything that comes to mind but I'm out of ideas. I
don't want to do a fresh install, but I don't know what else to do. I can
post more specific information, but I don't know what would help. I've gone
through everything which seems relevant and mostly things are still at the
defaults. There are no wireless devices or other network interfaces except
eth0 and there shouldn't be a firewall issue. Does anyone here have any
ideas? Please help! Thank you.

[Brian Poissant]

Maybe it's a protocol problem given that point works (ICMP) and everything
else doesn't (TCP/UDP).

Post by Tony Baechler - BATS
Hi all,
I've been struggling with this for days and am getting nowhere. I'm totally
baffled and have tried everything I can think of, including several Google
searches. I am sorry for the lack of information, but I'm not sure what to
post as I've never seen anything like this before.
I'm running a Ubuntu 14.04.1 server. It has ssh, Postfix, Apache2, etc. I
upgraded from kernel 3.13.04-6-generic to 3.13.04-8-generic and rebooted.
Everything was working fine, but I couldn't connect after the reboot. I
tried nmap and it says all ports are closed. My first thought is that it
must be a firewall, so I completely purged ufw and iptables but no luck. I
restored /etc/network/interfaces from a known good backup.
After reading various forum posts, I again installed iptables and copied and
pasted the script from the official Ubuntu wiki to disable the firewall,
just in case old rules were left. That should allow all incoming and
outgoing traffic. Apparently the script works as pinging works fine even
though nmap says all ports are closed. I've tried rebooting several times
just in case. I ran e2fsck and my disks are clean. I am able to boot into
a separate rescue system, so it's definitely not an obvious hardware
problem. I also ran update-grub and update-initramfs just in case.
Finally, I reconfigured grub-pc and installed the bootloader on both
/dev/sda and /dev/sdb. I'm running software RAID and I didn't install it on
/dev/md1, but I don't think that would matter. I have to access the server
remotely and writing to their support was of no help.
What's really strange is that it boots fine with kvm from the rescue system.
I can get to the login prompt and everything seems to be fine. It acts
like a boot problem, but I don't see why ping would work if it isn't
booting. Nothing gets written to syslog, so it acts like it's an init
problem, but I didn't change anything that I know of and I saw no errors
with kvm. Without kvm, it doesn't seem to boot as I mentioned. Just in
case, I removed or purged qemu, ufw, iptables, apparmor, etc. I also tried
upgrading to a completely different kernel, specifically
3.16.0-33-lowlatency. It's an Intel x86-64 processor with 32 GB of RAM.
It
did work fine after the initial install. According to mdadm, my RAID arrays
are fine.
Again, I've tried everything that comes to mind but I'm out of ideas. I
don't want to do a fresh install, but I don't know what else to do. I can
post more specific information, but I don't know what would help. I've gone
through everything which seems relevant and mostly things are still at the
defaults. There are no wireless devices or other network interfaces except
eth0 and there shouldn't be a firewall issue. Does anyone here have any
ideas? Please help! Thank you.
--
ubuntu-users mailing list
https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

[Karl Auer]

Post by Tony Baechler - BATS
Everything was working fine, but I couldn't connect after the reboot.
[...]
I completely purged ufw and iptables but no luck.

If you cannot connect to it, how are you modifying the configuration?
Just to double-check, this is a real system, not a virtual? Do you still
have physical access to the system?

You mentioned calling someone's support - whose?

Post by Tony Baechler - BATS
What's really strange is that it boots fine with kvm from the rescue system.
I can get to the login prompt and everything seems to be fine. It acts
like a boot problem, but I don't see why ping would work if it isn't
booting.

Add a crontab entry that runs every minute as root, collects some
information (the output from runlevel, ps, dmesg, ifconfig, iptables,
mount - whatever you can think of), and writes it into a known location
(but NOT /tmp). Reboot, wait at least ten minutes, then go in with the
rescue system and look at what's been written. If nothing's been written
then yes, you have a boot problem. I suggest you write a very simple
one first and see if it works at all. That way you haven't wasted a lot
of time if it doesn't. If it does work, go wild with version 2 :-)

Perhaps the system is booting into single-user mode for some reason. You
could try adding a job in /etc/init.d/rc1.d that collects info if level
1 is entered.

Also, check the default run level in /etc/init/rc-sysinit. It should be
2.

Check the kernel command line.

Check the BIOS boot order - this is a very long shot.

Also, check /etc/resolv.conf. Make sure the nameservers are correctly
entered and reachable from that system, otherwise all sorts of weird
delays can happen, especially if things like Apache try to check their
own address, or things like ssh try to check yours.

Check the IP address you have configured. Make sure it is legal - not a
broadcast or network address. Check the mask, check the gateway. Easy to
get wrong. It seems unlikely if you can ping the address, but still -
check it.

Check that the IP address of your server is not a duplicate. If some
other system has your server's IP address, your server may not be able
to bring up networking, but the other system with that address may well
respond only to ping, either because it doesn't have services
configured, or is firewalling you.

How are you trying to (for example) connect with ssh? Via the known IP
address or via the name of the system? If via the name, try via the IP
address.

If you connect by name, and the system has DNS entries for IPv4 and
IPv6, and YOUR system has IPv6 enabled, the connection will be attempted
via IPv6. Specifying the IP address rather than the name bypasses that
mechanism. If connection via the address works and connection via the
name doesn't suspect DNS issue at your end, or an IPv6/IPv4 issue such
as misconfigured IPv6.

Try connecting from a completely other machine in a completely other
well-maintained network. Just to make sure it's not a problem at your
end.

How long have you waited for the system to come up? Some networking
issues cause a delay of up to a minute or more.

Maybe try re-installing just the new kernel?

Regards, K.
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Karl Auer (***@biplane.com.au)
http://www.biplane.com.au/kauer
http://twitter.com/kauer389

GPG fingerprint: 3C41 82BE A9E7 99A1 B931 5AE7 7638 0147 2C3C 2AC4
Old fingerprint: EC67 61E2 C2F6 EB55 884B E129 072B 0AF0 72AA 9882

[Tony Baechler - BATS]

I have tried your suggestions as far as I'm able and it would seem to be a
boot problem. I think you're right that it's somehow stuck in single user
mode, but I don't know how to undo it. Obviously networking isn't coming
up. I'll answer your specific questions below.

Post by Karl Auer

Post by Tony Baechler - BATS
Everything was working fine, but I couldn't connect after the reboot.
[...]
I completely purged ufw and iptables but no luck.

If you cannot connect to it, how are you modifying the configuration?
Just to double-check, this is a real system, not a virtual? Do you still
have physical access to the system?

Yes, it's a real dedicated server. I never had physical access. Everything
is through the rescue system. I'm in the US and it's in Germany, so it
would be impossible for me to physically access it. If I could, I could
figure out if there is an error on the screen and that would probably clear
up the mystery.

Post by Karl Auer
You mentioned calling someone's support - whose?

No, I wrote to them. They are Hetzner. <http://www.hetzner.com/> and they
told me that they give me full root access, so I'm on my own.

Post by Karl Auer

Post by Tony Baechler - BATS
What's really strange is that it boots fine with kvm from the rescue system.
I can get to the login prompt and everything seems to be fine. It acts
like a boot problem, but I don't see why ping would work if it isn't
booting.

Add a crontab entry that runs every minute as root, collects some
information (the output from runlevel, ps, dmesg, ifconfig, iptables,
mount - whatever you can think of), and writes it into a known location
(but NOT /tmp). Reboot, wait at least ten minutes, then go in with the
rescue system and look at what's been written. If nothing's been written
then yes, you have a boot problem. I suggest you write a very simple
one first and see if it works at all. That way you haven't wasted a lot
of time if it doesn't. If it does work, go wild with version 2 :-)

I set a cron job to write the output of dmesg to a file every minute and the
file wasn't created. I also added a similar line to /etc/rc.local and that
wasn't created either. It would seem to be a boot problem. Now that I
think about it, I did at one point try to go to single user mode to make a
full backup. I wasn't able to connect, but rebooting via the web-based
robot seemed to fix the problem at the time. Now that I'm again totally
locked out, it would seem that it went back to single user mode, thus my
question of how to get it to boot normally. Again, KVm booted to a normal
login prompt with nothing about single user mode.

Post by Karl Auer
Perhaps the system is booting into single-user mode for some reason. You
could try adding a job in /etc/init.d/rc1.d that collects info if level
1 is entered.

I tried rcS.d but nothing happened. I'll look at rc1.d.

Post by Karl Auer
Also, check the default run level in /etc/init/rc-sysinit. It should be
2.

Yes, it's 2 and I haven't changed anything in that directory.

Post by Karl Auer
Check the kernel command line.

I checked /etc/default/grub several times and /etc/grub.d and
/boot/grub/grub.cfg and they look fine. The job to write /proc/cmdline to a
file doesn't seem to be working.

Post by Karl Auer
Check the BIOS boot order - this is a very long shot.

I can't since I have no physical access to the machine, but I don't think
that's it. Then again, it almost seems like a BIOS issue, but why would
ping work?

Post by Karl Auer
Also, check /etc/resolv.conf. Make sure the nameservers are correctly
entered and reachable from that system, otherwise all sorts of weird
delays can happen, especially if things like Apache try to check their
own address, or things like ssh try to check yours.

Yes, it checks out fine. It's using the Hetzner nameservers.

Post by Karl Auer
Check the IP address you have configured. Make sure it is legal - not a
broadcast or network address. Check the mask, check the gateway. Easy to
get wrong. It seems unlikely if you can ping the address, but still -
check it.

Yes, that's why I restored /etc/network/interfaces from a known good backup.
It looks fine.

Post by Karl Auer
Check that the IP address of your server is not a duplicate. If some
other system has your server's IP address, your server may not be able
to bring up networking, but the other system with that address may well
respond only to ping, either because it doesn't have services
configured, or is firewalling you.

I don't think that's the case. The rescue system on the same IP address
works fine and I doubt if Hetzner would assign duplicate addresses.
Apparently the rescue system is loaded from the Hetzner tftp server when I
activate it from the robot.

Post by Karl Auer
How are you trying to (for example) connect with ssh? Via the known IP
address or via the name of the system? If via the name, try via the IP
address.

I've tried both. That's why I tried nmap from an outside system and it said
all 1,000 ports are closed.

Post by Karl Auer
If you connect by name, and the system has DNS entries for IPv4 and
IPv6, and YOUR system has IPv6 enabled, the connection will be attempted
via IPv6. Specifying the IP address rather than the name bypasses that
mechanism. If connection via the address works and connection via the
name doesn't suspect DNS issue at your end, or an IPv6/IPv4 issue such
as misconfigured IPv6.

Yes, I think IPV6 isn't configured correctly but would that block IPV4
connections?

Post by Karl Auer
Try connecting from a completely other machine in a completely other
well-maintained network. Just to make sure it's not a problem at your
end.

Yes, I tried from a different Hetzner server in a completely different
datacenter and a different IP address block. I can connect to other servers
fine from my local box. I have a UK server for example which connects fine.

Post by Karl Auer
How long have you waited for the system to come up? Some networking
issues cause a delay of up to a minute or more.

Several hours.

Post by Karl Auer
Maybe try re-installing just the new kernel?

What do you mean? I removed the old "generic" kernels completely and
installed "lowlatency" to eliminate a kernel problem. The machine has 32 GB
of RAM. I could switch back to the 3.13.0-48-lowlatency kernel as it's
currently booting 3.16.

Post by Karl Auer
Regards, K.

[Tony Baechler - BATS]

Hi all,

I finally solved the problem. It turnbed out that it was in fact a boot
problem. Apparently, it couldn't mount swap, so it stopped and waited for a
keypress. Obviously there was no way to do that, so it never finished
booting. I'm not sure why I missed it before. It is back up and seems to
work fine. What I don't understand is why swap wouldn't work on a RAID1
array, but the machine has 32 GB of memory, so I guess it's no great loss.
Thank you very much for your help.

[Liam Proven]

Post by Tony Baechler - BATS
What I don't understand is why swap wouldn't work on a RAID1
array, but the machine has 32 GB of memory, so I guess it's no great loss.

You could try using the swapspace tool to make an automatic swapfile
on the main partition.
--
Liam Proven • Profile: http://lproven.livejournal.com/profile
Email: ***@cix.co.uk • GMail/G+/Twitter/Flickr/Facebook: lproven
MSN: ***@hotmail.com • Skype/AIM/Yahoo/LinkedIn: liamproven
Cell/Mobiles: [phone] (UK) • [phone] (ČR)