Reporting COMSATS Pakistan not Reachable

Les Cottrell and Gary Buhrmaster, July 8, 2007

Reported Problem

On 7/7/07 we noticed we were unable to ping pinger.comsats.edu.pk. It gave Time to live exceeded:

pinger@pinger $ ping pinger.comsats.edu.pk
PING pinger.comsats.edu.pk (202.83.169.113) 56(84) bytes of data.
From rwp44.pie.net.pk (202.125.159.209) icmp_seq=0 Time to live exceeded
From rwp44.pie.net.pk (202.125.159.209) icmp_seq=1 Time to live exceeded

--- pinger.comsats.edu.pk ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1013ms
, pipe 2

First Look

We observed that the the route to pinger.comsats.edu.pk had a routing loop at the Pakistan Internet Exchange (PIE) router in Rawalpindi (location confirmed by Visualroute).

pinger@pinger $ traceroute pinger.comsats.edu.pk
traceroute to pinger.comsats.edu.pk (202.83.169.113), 30 hops max, 38 byte packets
 1  rtr-iepm-test (134.79.243.1)  0.770 ms  0.402 ms  0.476 ms
 2  rtr-core1-p2p-iepm (134.79.252.5)  0.491 ms  0.450 ms  0.484 ms
 3  rtr-dmz1-ger (134.79.135.15)  0.487 ms  0.453 ms  0.487 ms
 4  slac-rt4.es.net (192.68.191.146)  0.486 ms  0.455 ms  0.485 ms
 5  slacmr1-slacrt4.es.net (134.55.209.93)  1.004 ms  0.434 ms  0.482 ms
 6  snv2mr1-slacmr1.es.net (134.55.217.2)  2.489 ms  120.952 ms  1.900 ms
 7  snv2sdn1-snv2mr1.es.net (134.55.207.37)  0.964 ms  1.333 ms  0.948 ms
 8  snvrt1-snv2sdn1.es.net (134.55.221.37)  0.970 ms  0.979 ms  1.074 ms
 9  188.ATM1-0.BR2.SJC1.ALTER.NET (204.255.174.49)  1.863 ms  2.012 ms  1.428 ms
10  154.ATM3-0.XR1.SJC1.ALTER.NET (152.63.51.174)  1.965 ms  1.981 ms  2.428 ms
11  0.so-0-0-0.XL1.SJC1.ALTER.NET (152.63.55.114)  1.996 ms  2.493 ms  2.444 ms
12  0.so-1-1-0.XT1.NYC8.ALTER.NET (152.63.21.101)  72.992 ms  72.960 ms  72.933 ms
13  0.so-3-0-0.XR1.NYC8.ALTER.NET (152.63.19.30)  72.960 ms  72.966 ms  93.427 ms
14  183.ATM6-0.IG3.NYC8.ALTER.NET (152.63.26.49)  72.933 ms  72.977 ms  73.444 ms
15  pctl-gw.customer.alter.net (208.192.182.34)  296.934 ms  296.430 ms  296.435 ms
16  pos1-2.rwp44gsrc2.pie.net.pk (202.125.159.23)  316.434 ms  316.290 ms  316.012 ms
     MPLS Label=567 CoS=7 TTL=1 S=0
17  rwp44.pie.net.pk (202.125.159.210)  391.892 ms  408.267 ms  458.021 ms
     MPLS Label=591 CoS=7 TTL=1 S=0
18  rwp44.pie.net.pk (202.125.159.209)  319.844 ms  318.206 ms  315.996 ms
19  rwp44.pie.net.pk (202.125.159.210)  315.890 ms  315.771 ms  317.949 ms
     MPLS Label=591 CoS=7 TTL=1 S=0
20  rwp44.pie.net.pk (202.125.159.209)  316.435 ms  316.130 ms  316.097 ms
21  rwp44.pie.net.pk (202.125.159.210)  316.878 ms  320.811 ms  315.959 ms
     MPLS Label=591 CoS=7 TTL=1 S=0
22  rwp44.pie.net.pk (202.125.159.209)  316.946 ms  316.285 ms  315.967 ms
23  rwp44.pie.net.pk (202.125.159.210)  316.428 ms  316.304 ms  332.451 ms
     MPLS Label=591 CoS=7 TTL=1 S=0
24  rwp44.pie.net.pk (202.125.159.209)  316.469 ms  316.291 ms  316.453 ms
25  rwp44.pie.net.pk (202.125.159.210)  316.447 ms  316.328 ms  316.479 ms
     MPLS Label=591 CoS=7 TTL=1 S=0
26  rwp44.pie.net.pk (202.125.159.209)  337.333 ms  316.372 ms  315.948 ms
27  rwp44.pie.net.pk (202.125.159.210)  316.393 ms  316.309 ms  320.434 ms
     MPLS Label=591 CoS=7 TTL=1 S=0
28  rwp44.pie.net.pk (202.125.159.209)  316.443 ms  337.849 ms  316.434 ms
29  rwp44.pie.net.pk (202.125.159.210)  319.456 ms  316.331 ms  316.451 ms
     MPLS Label=591 CoS=7 TTL=1 S=0
30  rwp44.pie.net.pk (202.125.159.209)  316.907 ms  316.829 ms  324.821 ms

Reporting

After observing that the problem had not been fixed 24 hours later, we tried to report the problem to noc@pie.net.pk the advertised mail address for problem reporting. However the mail bounced. We also tried noc@iti.net.pk. We then tried mansoor@pie.net.pk (the responsible person for the ASN record for 17557, the advertised ASN for pie.net.pk). However this email also bounced. We tried to check where else to report problems but www.pie.net.pk and www.iti.net.pk gave Server not found. We tried www.ptcl.com.pk which gave us PTCL Corporate Customer Support in Islamabad as 3cisb@ptcl.com.pk. We sent email to this addrerss but with no answer (it was a Monday 4am Pakistan time).

Further investigation of the MX records showed that the MX server for pie.net.pk was asher.pknic.net.pk using dig:


; <<>> DiG 8.4 <<>> pie.net.pk MX
;; res options: init recurs defnam dnsrch no-nibble2
;; got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5926
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUERY SECTION:
;;      pie.net.pk, type = MX, class = IN

;; AUTHORITY SECTION:
pk.                     3H IN SOA       ns.pknic.net.pk. ashar.pknic.net.pk. (
                                        1137368302      ; serial
                                        4H              ; refresh
                                        2H              ; retry
                                        1w3d            ; expiry
                                        6H )            ; minimum


;; Total query time: 50 msec
;; FROM: pinger to SERVER: 134.79.18.45
;; WHEN: Sun Jul  8 18:23:20 2007
;; MSG SIZE  sent: 28  rcvd: 79

However the MX host ashar.pknic.net.pk was not resolvable. So we could not even send email about the email address being broken let alone the routing problems.

Resolution

As of 20:15 7/8/07 PDT Eli Dart of ESnet reported that the problem appeared fixed:

pinger@pinger $
pinger@pinger $ ping pinger.comsats.edu.pk
PING pinger.comsats.edu.pk (202.83.169.113) 56(84) bytes of data.
64 bytes from 202-83-169-113.reverse.ntc.net.pk (202.83.169.113): icmp_seq=0 ttl=42 time=323 ms
64 bytes from 202-83-169-113.reverse.ntc.net.pk (202.83.169.113): icmp_seq=1 ttl=42 time=323 ms

--- pinger.comsats.edu.pk ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1012ms
rtt min/avg/max/mdev = 323.113/323.116/323.120/0.568 ms, pipe 2
pinger@pinger $ traceroute pinger.comsats.edu.pk
traceroute to pinger.comsats.edu.pk (202.83.169.113), 30 hops max, 38 byte packets
 1  rtr-iepm-test (134.79.243.1)  0.485 ms  0.468 ms  0.368 ms
 2  rtr-core1-p2p-iepm (134.79.252.5)  0.359 ms  0.325 ms  0.369 ms
 3  rtr-dmz1-ger (134.79.135.15)  0.478 ms  0.429 ms  0.500 ms
 4  slac-rt4.es.net (192.68.191.146)  2.112 ms  0.437 ms  0.484 ms
 5  slacmr1-slacrt4.es.net (134.55.209.93)  0.615 ms  0.467 ms  0.476 ms
 6  snv2mr1-slacmr1.es.net (134.55.217.2)  0.865 ms  0.827 ms  0.858 ms
 7  snv2sdn1-snv2mr1.es.net (134.55.207.37)  0.864 ms  0.954 ms  1.107 ms
 8  snvrt1-snv2sdn1.es.net (134.55.221.37)  2.579 ms  1.424 ms  0.975 ms
 9  188.ATM1-0.BR2.SJC1.ALTER.NET (204.255.174.49)  1.492 ms  1.917 ms  1.478 ms
10  154.ATM3-0.XR1.SJC1.ALTER.NET (152.63.51.174)  2.489 ms  1.954 ms  1.991 ms
11  0.so-0-0-0.XL1.SJC1.ALTER.NET (152.63.55.114)  2.479 ms  2.444 ms  2.566 ms
12  0.so-6-0-1.XT1.NYC8.ALTER.NET (152.63.0.162)  73.500 ms  73.479 ms  73.333 ms
13  0.so-3-0-0.XR1.NYC8.ALTER.NET (152.63.19.30)  73.412 ms  73.488 ms  73.934 ms
14  183.ATM6-0.IG3.NYC8.ALTER.NET (152.63.26.49)  73.561 ms  73.921 ms  73.406 ms
15  pctl-gw.customer.alter.net (208.192.182.34)  296.974 ms  296.952 ms  296.863 ms
16  pos1-2.rwp44gsrc2.pie.net.pk (202.125.159.23)  416.967 ms  461.364 ms  537.902 ms
     MPLS Label=567 CoS=7 TTL=1 S=0
17  rwp44.pie.net.pk (202.125.159.210)  316.331 ms  316.336 ms  317.447 ms
     MPLS Label=591 CoS=7 TTL=1 S=0
18  221.120.235.166 (221.120.235.166)  325.879 ms  322.847 ms  321.939 ms
19  202-83-169-113.reverse.ntc.net.pk (202.83.169.113)  322.466 ms  327.761 ms  323.940 ms

Looking at the history plots from PingER it appears that the link is fragile as seen from SLAC with several periods of unreachability in the last few days. PingER Loss & RTT

Looking at the measurements from pinger.comsats.edu.pk to pinger2.niit.edu.pk however it appears the pinger.comsats.edu.pk was up (no 100% loss) and measuring except for one outage, whereas SLAC to COMSATS was down 4 times in teh same period. Also the MX host for pie.net.pk, i.e. ashar.pknic.net.pk still does not resolve.

On the other hand Azher Amin from NIIT reported that that either there was no electricity in Comsats during the downtime (Sunday) or their FO link/equipment was cutoff.

The www.pie.net.pk server was still inaccessible and the MX host for pie.net.pk was still not responding Sunday 7/8/07 10:00pm (PDT), even though pinger.comsats.edu.pk was accessible.