To provide information to physicists on International
connectivity for a talk scheduled at
SLAC for April 5th, 2001,
and to help understand the needs for improvements to meet
the needs of
BaBar
regional computing centers at
IN2P3 Lyon
France, INFN Rome Italy and
RAL
Oxford UK, and other HENP experiments, I put together this
page to provide a repository of information concerning major
International links (mainly as seen from SLAC).
Methodology
Where possible I contacted and gathered information from networkers at
other sites and from
ESnet to understand the network configurations.
In addition I used the SLAC reverse
traceroute server to find the routes from SLAC to the remote site
and to identify the Autonomous Systems (AS) along the path.
I also used
pipechar a tool for reporting dynamic network characteristics
in particular the bottleneck bandwidth to get estimates of
bandwidths. Finally I used
iperf,
as described in
Bulk throughput measurements to measure the maximum TCP
data rate achievable at the time.
I used a Sun Solaris 5.7 host at SLAC with a Gbit
Ethernet connection to the network and thence to the SLAC site border
to make the pipechar and iperf
measurements.
Connectivity
The SLAC LAN provides connectivity to the outside world at up to Gbit speeds
via its so-called DMZ.
SLAC connects to all the above sites via Esnet. The ESnet
link to SLAC is at OC3 (155Mbps) to the Sunnyvale GigaPoP.
The GigaPoP connects to the QWest ATM cloud by an OC12 (622Mbps) link.
SLAC also has a 622Mbps link to
Internet 2 via a Gigabit Ethernet link from SLAC to
Stanford University.
The
RENATER (France),
DFN (Germany),
GARR
(Italy), and
UKERNA (UK)
academic and research networks have also
signed the
MoU
with
UCAID
for the development, together with other
worldwide research networks, of
Internet2, so this might be an
alternative way to reach some of the sites from SLAC.
The main exchange points between ESnet and other ISPs are in New York and
Chicago (STARTAP),
with the trans-Pacific networks connecting via the Sunnyvale
hub. The New York exchange points are in two locations as seen in the figure
below prepared in September 2000 by Jim Leighton of ESnet.
CERN, Geneva, Switzerland
The traceroute indicates that
path from SLAC goes via ESnet to Chicago and thence to
CERN.
The pipechar indicates that the bottleneck
is in ESnet and is about 65Mbps. Looking at
Netperf throughput on transatlantic-line CERN (sunstats) & CERN's PoP in Chicago
it appears that throughput achievable from CERN to the US
increased by about a factor of 2 in mid-January 2001.
The plans indicate that CERN is looking to
double the bandwidth in the short term future.
IN2P3, Lyon France
The traceroute from
SLAC to
IN2P3 indicates the route goes
over ESnet to Chicago and then crosses the Atlantic on the French PHYnet.
The traceroute from IN2P3 to SLAC indicates
that the routes are symmetric.
The pipechar indicates the bandwdith across the
Atlantic and between ESnet and PHYnet is about 43Mbps.
This is in rough agreement with the plans
which also call for upgrades.
The iperf measured maximum TCP throughput was 29Mbps.
More information on the
French academic and research networks can be found in
NRD's and
Renater's map.
RAL, Oxford, UK
The traceroute from SLAC to RAL indicates
that the route goes from SLAC to ESnet at Sunnyvale, to New York where it
joins
JANET and crosses the Atlantic to RAL.
The pipechar
indicates that there are bottlenecks starting in JANET around 45Mbps.
A pipechar from Glasgow to SLAC
also confirms botlenecks in JANet around 45Mbps.
A bulk throughput iperf measurement
made on March 29 2001, around 2:00pm PST
indicates that the maximum TCP throughput available at the time was about
40Mbps.
The current trans Atalantic link to the UK was
upgraded to 6 * OC3s on March 23, 2001.
The peering to ESnet was
upgraded to 85Mbps early February 2001. The
JANet Backbone map shows the
topology and link speeds.
INFN, Rome, Italy
The traceroute from SLAC to INFN
shows the route goes from SLAC via the ESnet Sunnyvale hub to New York
then via
DANTE to the European
TEN-155 network and
thence to the Italian
GARR academic research network and to Rome.
The
pipechar
indicates that the bottleneck starts on the ESnet and DANTE/TEN-155
and is 45Mbps.
The iperf measurements indicate a maximum TCP bandwidth attainable of 26Mbps.
The TEN-155 link to
the GARR backbone in Milan is at 155Mbps, the GARR backbone is
fully meshed at 155Mbps, and the link from GARR to
INFN Rome is also at 155Mbps.
The GARR-B
Network Map and Statistics gives more details on the links.
DESY, Hamburg, Germany
The traceroute
shows the route from SLAC to
DESY
goes via the ESnet Sunnyvale hub to
New York, across the Atlantic onto DANTE and onto the TEN-155 European backbone
to the German academic research network (WIN) and thence to DESY.
The pipechar measurement indicates that
the links between ESnet and DANTE and from DANTE through
WIN to DESY are at ~45Mbps (T3). From other sources we know the link
between ESnet and DANTE is actually 50Mbps, and the Transatlantic link is
90Mbps.
KEK, Tokyo, Japan
The traceroute from SLAC to
KEK
indicates the route goes from SLAC to the ESnet Sunnyvale Hub and
thence to Japan and KEK. The
pipechar indicates the bandwidth from
ESnet to KEK is limited to 10Mbps.
IHEP, Beijing, China
Information on the link from SLAC to
IHEP can be found at:
Comparison of ihep.ac.cn with
cas.ac.cn.
The traceroute from SLAC to IHEP indicates
the route goes via KEK (see above) and then via HEPnet-J to IHEP. This latter
link is a 128kbps fiber link.
The pipechar measurement shows the
10Mbps link from Esnet to KEK, and then a lower speed link via HEPnet-J
to IHEP.
BINP, Novosibirsk, Russia
Information on the Novosibirsk
link can be found at
Network connectivity between SLAC and BINP, Novosibirsk.
The traceroute shows the route
goes from SLAC to the ESnet Sunnyvale hub to KEK and thence visa HEPnet-J
to Novosibirsk.
The pipechar indicates that the bottlneck is
is 1.5Mbps from HEPnet-J to BINP. This does not agree with our
understanding that this link is a fiber 128kbps link.
Summary
The table below shows a summary of the information gathered above, plus
some historical meassurements made at earlier times. The column
labelled AS gives the Autonomous Systems (almost the same as the Internet
Service Providers) traversed by the route between SLAC and the destination site
(in column 1), apart from KEK it does not provide the AS at the end sites. The
Peering @ gives the location where ESnet peers with the next AS providing
connectivity to the destination site. The Bandwidth Summer 2000 was the
bandwidth between SLAC and the destination site. The Bandwidth March 2001 is the bottleneck bandwidth
between the SLAC site and the destination site. If there is an asterisk (*) then
the bandwidth is as measured by
iperf if a tilde (~) then it is measured by
pipechar, otherwise it is by knowing the stated link speeds.
The bandwidth between two nodes
one at each site may be lower than the bandwidth between the borders o
the two sites, so in
general we would expect the pipechar (or prior to March 2001 the
pchar) measured
bandwidth from host to host
to be less than the bandwidth from site boundary to site boundary.
We minimized this effect at the SLAC end by using a measurement host with high
speed (Gigabit Ethernet) connectivity to the SLAC core network and thence
the border.
Also the maximum iperf TCP bandwidth that can be achieved through a link
is typically 80-95% of the unloaded link speed (see
Bulk Throughput measurements), so the iperf bandwidths would be
expected to be the lowest.
| Site |
City |
Country |
AS |
Peering @ |
Bandwidth Feb 2000 |
Bandwidth Sep 2000 |
Bandwidth Mar 2001 |
Bandwidth Jul 2001 |
| CERN |
Geneva |
Switzerland |
ESnet, CERN |
Chicago |
12Mbps* |
22Mbps~, 27Mbps*, 34Mbps |
65Mbps~, 63Mbps*, 155Mps |
80Mbps~,
85Mbps*, 155Mbps |
| IN2P3 |
Lyon |
France |
ESnet, PHYnet |
Chicago |
6Mbps~, 4.5Mbps* |
21Mbps* |
45Mbps~, 29Mbps* |
80Mbps~,
90Mbps*,
155Mbps |
| RAL |
Oxford |
UK |
ESnet, JANet |
New York |
|
8.9Mbps~, 6.5Mbps*, 10Mbps |
45Mbps~, 40Mbps* |
35Mbps~,
45Mbps*, 50Mbps |
| INFN |
Rome |
Italy |
ESnet, TEN-155, GARR |
New York |
1.5Mbps~, 1.5Mbps* |
7Mbps* |
45Mbps~, 26Mbps* |
45Mbps~,
27.5Mbps* |
| DESY |
Hamburg |
Germany |
ESnet, TEN-155, WIN |
New York |
|
|
45Mbps~ |
| KEK |
Tokyo |
Japan |
ESnet, KEK |
Sunnyvale |
|
|
10Mbps |
| IHEP |
Beijing |
China |
ESnet, KEK, HEPnet-J |
Sunnyvale |
|
|
128kbps |
| BINP |
Novosibirsk |
Russia |
ESnet, KEK, HEPnet-J |
Sunnyvale |
|
|
128kbps |
It can be seen that there is considerable improvement (factors of 75% to
1600%) in the last year in the maximum TCP throughput achievable from SLAC to
the four major international sites of interest to Babar. This is illustrated
in the figure below.
Page owner: Les Cottrell
