SLAC logo

Effect of change from Internet2 to ESnet for the route from SLAC to Europe Network logo

Les Cottrell. Page created: June 12 2003.

Central Computer Access | Computer Networking | Network Group | ICFA-NTF Monitoring
SLAC Welcome
Highlighted Home
Detailed Home
Search
Phonebook

Introduction

This case shows the effect of changing the route from SLAC to Europe to use ESnet rather than Stanford/CalREN2/Internet2. The routing change was made at about 3:45pm PDT June 11/2003. Prior to this time the default routing to Europe was via Stanford University / CalREN2 / Abilene/Internet2. After the change it was via ESnet.

Problem reports

Most of the throughputs measured by iperf from to European hosts stayed the same (the information in the table below came from our regular IEPM-BW measurements):
Iperf throughputs before and after change
HostInitial carrierInitial Mbits/sAfter Mbits/s
1.cesnet.czI2220-250125*
1.clrc.ac.ukES4040
1.dl.ac.ukES4040
1.in2p3.frI212060
1.mib.infn.itI22525
1.nikhef.nlES350350
1.pd.infn.itI2115115
1.roma1.infn.itI21313
1.switch.chI2100-250100-250
2.cern.chES400-500400-500
* The 125Mbits/s limitation was caused by the TCP windows/streams setting being non optimal for this path (see below).

The hostnames are anonymized for privacy reasons. It can be seen that the throughput to only CESnet and IN2P3 appear to be affected. In some cases (e.g. UK, and NL) this is since the route did not change, in other cases (e.g. IT) maybe the throughput is not large enough to notice a change in bottleneck. The large variation in the SWITCH.CH case are due to large variations (due in turn to cron jobs) in the load on the 1.switch.ch host.

CESnet

After the change, the iperf throughput from SLAC to CESnet in Prague, Czech Republic dropped from about 250 Mbits/s to about 150 Mbits/s, but the ping RTT stayed constant. The route from SLAC to CESNET reduced from 17 to 13 hops (that is good), the last 6 hops were identical in both cases, the changes were for hops 3-11 for the Internet2 route and hops 3-7 for the ESnet route. The route from CESNET to SLAC after the change are pretty symmetric with the route from SLAC to CESNET.

IN2P3

In the case of SLAC to IN2P3 in Lyon France, the iperf throughput dropped from about 125 Mbits/s to about 60 Mbits/s. Also the ping RTT increased from about 150ms to about 160ms. The traceroutes before and after show the number of hops changing from 17 to 15 and the first 2 and last 2 hops being identical. In the former case (Internet2) the peering from Abilene/Internet2 was to GEANT, after the routing change ESnet handed off the packets to opentransit. I would have exepected IN2P3 to have been routed by GEANT and RENATER so perhaps opentransit is a backup route.

Resolving

We sent email to trouble@es.net describing the problem, and also to Anne Marie Lutz of IN2P3 identifying the possible use of a backup route.

IN2P3

Mike O'Connor of ESnet responded via Email6/12/03 at 11:07am PDT: ESnet had been filtering a GEANT route to ccsvsn04.in2p3.fr. The 134.158.0.0/16 route from GEANT is now being accepted in NY. Since the route was corrected the IPERF times have recovered to the previous performance levels for node1.in2p3.fr. The previous path traversed an opentransit peering in Chicago with the shaping set to: vbr peak 135m sustained 67m burst 100 The proper path via GEANT peers at OC48 in NYC.

The changes are nicely illustrated by the ABwE available bandwidth tool results which are updated every minute.

CESnet

The iperf configuration was 8 parallel streams with 1024KByte window. It appears that though this worked well for the Internet2 path, it is much less than optimal for the ESnet path. This is also confirmed by the ABwE results which indicated that available bandwidth increased when we moved the path from Abilene to ESnet (note the step up on the blue line from about 200 Mbits/s to 280Mbits/s), at around 4pm. Noting that the single stream iperf (measured by IEPM-BW with a 1024KByte window) was about 7Mbits/s and we want to achieve about 280MBits/s, we tried 48 streams with a 256KByte window and achieved about 300Mbits/s. We therefore used tcpload.pl to scan through various window sizes and streams to find the optimal configuration and used 30 streams and a 512 kByte window.

Resolution

IN2P3

When we switched over from Internet2 to ESnet, the ESnet route to IN2P3 was less than optimal and had rate limiting on it. This was fixed by ESnet peering directly with GEANT for the IN2P3 path.

CESnet

The ESnet route to CESnet was very good, however the measurement tool (iperf) needed re-configuring (change the window and streams) to work well on the new path. ABwE was very valuable in identifying this since it showed immediately that the new path via ESnet had similar available bandwdith to the Internet2 path.
Page owner: Les Cottrell