Problems with connectivity to site in S. Africa - August '06 Network logo

Les Cottrell. Page created: August 4, 2006

Central Computer Access | Computer Networking | Network Group | More case studies
SLAC Welcome
Highlighted Home
Detailed Home
Search
Phonebook

Problem

From looking at the PingER monitoring data we noticed that we were unable to gather data from brunsvigia.tenet.ac.za, the PingER monitoring host in S. Africa.

Testing

We tried pinging brunsvigia:
232cottrell@pinger:~>ping brunsvigia.tenet.ac.za PING 
brunsvigia.tenet.ac.za (196.21.99.222) 56(84) bytes of data.
--- brunsvigia.tenet.ac.za ping statistics ---
1065 packets transmitted, 0 received, 100% packet loss, time 1064087ms
Then we tried a traceroute:
233cottrell@pinger:~>traceroute brunsvigia.tenet.ac.za traceroute to 
brunsvigia.tenet.ac.za (196.21.99.222), 30 hops max, 38 byte packets  
1  rtr-test-test (134.79.243.1)  0.257 ms  0.269 ms 0.243 ms  
2  rtr-core1-p2p-test (134.79.252.5)  0.236 ms  0.304 ms 0.241 ms  
3  rtr-dmz1-ger (134.79.135.15)  0.517 ms  0.340 ms  0.235 ms  
4  i2-gateway.stanford.edu (192.68.191.83)  0.377 ms  0.306 ms 0.375 ms  
5  hpr-svl-hpr--stan-ge.cenic.net (137.164.27.161)  0.937 ms  1.413 ms  0.931 ms  
6  lax-hpr--svl-hpr-10ge.cenic.net (137.164.25.12)  8.864 ms  8.585 ms  8.291 ms  
7  abilene-LA--hpr-lax-gsr1-10ge.cenic.net (137.164.25.3) 8.550 ms 8.367 ms  8.390 ms  
8  snvang-losang.abilene.ucaid.edu (198.32.8.95) 16.466 ms 15.852 ms  15.898 ms  
9  dnvrng-snvang.abilene.ucaid.edu (198.32.8.2)  40.882 ms 40.684 ms  50.722 ms 
10 kscyng-dnvrng.abilene.ucaid.edu (198.32.8.14)  51.273 ms 51.230 ms 51.417 ms 
11 iplsng-kscyng.abilene.ucaid.edu (198.32.8.80)  71.124 ms 60.529 ms  60.709 ms 
12 chinng-iplsng.abilene.ucaid.edu (198.32.8.76)  64.321 ms  64.263 ms  64.322 ms 
13 nycmng-chinng.abilene.ucaid.edu (198.32.8.83)  89.426 ms  84.357 ms 84.314 ms 
14  * * *
15  * * *
We found the same results for www.aims.ac.za. However pings and traceroutes worked to: www.museumsnc.co.za. At this point we surmised that we may have lost connectivity to academic and research sites in S. Africa. We sent email to our contact at TENET, and also to Abilene Network Operations at 4:47om Thursday August 3rd, 2006. We received email from the Abilene NOC stating

Les,

Are you still seeing this problem?  We just did a trace from the Los Angeles Abilene router that made it.  We're also able to access www.museumsnc.co.za.

The TENET connection off of the the New York router is stable.

Thanks,
Stacy W. Bengochea
We made more ping and traceroute measurements that indicated 
brunsvigia was still not pingable but www.aims.ca.za was
pingable.
66cottrell@pinger:~>ping www.aims.ac.za
PING kingklip.aims.ac.za (196.21.78.18) 56(84) bytes of data.
64 bytes from ssh.aims.ac.za (196.21.78.18): icmp_seq=0 ttl=49 time=320 ms
64 bytes from ssh.aims.ac.za (196.21.78.18): icmp_seq=1 ttl=49 time=321 ms
64 bytes from ssh.aims.ac.za (196.21.78.18): icmp_seq=2 ttl=49 time=321 ms
64 bytes from ssh.aims.ac.za (196.21.78.18): icmp_seq=3 ttl=49 time=321 ms
64 bytes from ssh.aims.ac.za (196.21.78.18): icmp_seq=4 ttl=49 time=321 ms

--- kingklip.aims.ac.za ping statistics ---
6 packets transmitted, 5 received, 16% packet loss, time 5055ms rtt min/avg/max/mdev = 320.950/321.152/321.340/0.732 ms, pipe 2 67cottrell@pinger:~>traceroute www.aims.ac.za traceroute to kingklip.aims.ac.za (196.21.78.18), 30 hops max, 38 byte packets
 1  rtr-test-test (134.79.243.1)  0.532 ms  0.401 ms  0.815 ms
 2  rtr-core1-p2p-test (134.79.252.5)  0.482 ms  0.318 ms  0.382 ms
 3  rtr-dmz1-ger (134.79.135.15)  0.466 ms  0.443 ms  0.377 ms
 4  i2-gateway.stanford.edu (192.68.191.83)  0.479 ms  0.442 ms  0.387 ms
 5  hpr-svl-hpr--stan-ge.cenic.net (137.164.27.161)  0.962 ms  0.932 ms  0.865 ms
 6  lax-hpr--svl-hpr-10ge.cenic.net (137.164.25.12)  8.484 ms  8.545 ms  8.727 ms
 7  abilene-LA--hpr-lax-gsr1-10ge.cenic.net (137.164.25.3)  17.350 ms  8.449 ms  8.427 ms
 8  snvang-losang.abilene.ucaid.edu (198.32.8.95)  26.325 ms  15.907 ms  22.584 ms
 9  dnvrng-snvang.abilene.ucaid.edu (198.32.8.2)  40.684 ms  40.763 ms  40.701 ms 10  kscyng-dnvrng.abilene.ucaid.edu (198.32.8.14)  51.307 ms  51.497 ms  63.688 ms
11  iplsng-kscyng.abilene.ucaid.edu (198.32.8.80)  60.550 ms  60.496 ms  70.058 ms
12  chinng-iplsng.abilene.ucaid.edu (198.32.8.76)  64.301 ms  73.244 ms  71.054 ms
13  nycmng-chinng.abilene.ucaid.edu (198.32.8.83)  84.444 ms  84.492 ms  84.545 ms
14  * * *
68cottrell@pinger:~>date
Thu Aug  3 20:51:40 PDT 2006 

Resolution

On Friday 8/4/2006 at 6:07 am we received email from Duncan Martin of TENET stating:

Hi Les,

We experienced two huge DDOS attacks on Wednesday.  
The first started at about 00:00 GMT and lasted about 6 hours; 
the second started at about 13:30 GMT and was terminated 
within about 20 minutes.  In both cases we had to de-announce 
the attacked addresses.

And then this morning between 02:00 and 06:00 we had a 
power failure in the building in which brunsvigia is located.

All should be back to normal now.
Kind regards,
Duncan

Page owner: Les Cottrell