3. Backbone network
• Backbone network ("CUDN" — Cambridge University Data
Network) runs like an ISP to 175 institutions (departments and
colleges)
• No border firewall — it's all "outside"
• Only simple access controls and DoS protection, typically for
the network itself
• Firewalling left to institutions: appropriate for them
• Easy to deliver unfiltered, maximum bandwidth connections
• Prices: 1G ~£2,500/year, 10G ~£4,500/year.
6. VLAN options
• Layer 2 circuit [point-to-point VLAN] using EoMPLS
(undesirable)
• Layer 3 routed subnet with optional access lists
• Layer 3 routed subnet as part of MPLS VPN
• Creates a private network between a group of
institutions for a particular service or project
• Need a gateway router/firewall/server to get out of
the network to backbone/Janet
8. Dark fibre network
• University and Colleges own a private dark fibre network covering relevant
parts of the city (the "GBN" — Granta Backbone Network)
• Redundant paths between most areas
• Circuits are singlemode and spliced end-to-end
• Anyone can rent circuits and build/extend their own network, (departments/
colleges, Janet, Anglia Ruskin University)
• (The CUDN rents circuits just like anyone else.)
• Ideal for delivering Janet Lightpath and Aurora services
• Bypasses security concerns when passing sensitive data?
• Price for A/Y 2016-17 is £13.22/100m/year
9. GBN
Data
Centre
Old Addenbrooke’s
University Press
Hills Road 6th Form
Strangeways
Clinical School
Chemistry
Architecture
Long Road 6th Form
Engineering
Fitzwilliam Museum
Peterhouse
Downing Site
New Museums Site
Trinity
St John’s
West
Lucy
Cavendish
Wolfson
CourtAstronomy
Mill Lane North
Cavendish
Sidgwick Site
Great St Mary’s
Gonville & Caius
King’s
St Catharine’s
Thompson’s Lane St John’s Road Park Parade
Queens’
East
Queens’
West
Darwin
Harvey Court
Caius Pavilion
Grasshopper
Lodge
Newnham
Owlstone Croft
South Green Lodge
Wolfson
Selwyn
Gardens
Leckhampton
St Chad’s Selwyn
Wilberforce Road
Sports Ground
Cripps
Court
Emmanuel
Christ’s
JesusSidney SussexADC
St Edmund’s Kettle’s Yard
Murray
EdwardsFitzwilliam
Churchill
Girton
University Farm
St John’s
East
Trinity Hall
Clare
Old
Schools
Botanic Garden
Chaucer Road
Latham Road
Downing Parker’s Piece
Fenners
St. Paul’s Road
Kelsey
Kerridge
Magdalene West
Magdalene East
Chesterton Lane
The Colony Shire Hall
Magrath Avenue
Corpus
Christi
Free School Lane
Cambridge
Assessment
Harvey Road
Open
University
Workers’
Educational
Association
Glisson Road
Gresham Road
Russell Street
Pembroke
Panton Street
CRUK
MRC LMB
Barton Road East
Grange Road
South
Newnham
Village
Newnham
House
Newnham
Terrace
Gwen
Raverat
Malting House
Hughes Hall
Mill Road West
Robinson
Library Memorial Court
KGH
Clare Hall
MathsIMS
Huntingdon Road
Madingley Road
East
Fossedene
Halifax Road
Westminster
Zoo Sub Dept
Madingley Hall
Whittle
UIS
Computer Lab
Earth Sciences
Soulsby
Veterinary Medicine
Gravel Hill Farm
Laundry Farm
Hauser Forum
Materials
Science
& Metallurgy
Sports
Centre
Institute for
Manufacturing
Residences
Grange Road
Forvie Site
Storey’s Way
Wychfield
Saxon Street
Anglia Ruskin
University
Midsummer
Common
Newmarket
Road
Maids
Causeway
Histon Road
Canterbury Street
Jesus Green
River Cam
River Cam
Barton Road West
Mill Lane South
Boat Houses
Darwin Line
Book Line & Thinker
Turing Loop
Cats & Queens’
Storey’s Way to NMS
Zoo Line
Bumps & Bruises
Stars & Bytes
Newton Line
Interchange Stations
Under Consideration
Leased Fibre Line (Redstone)
Leased Fibre Line (Virgin)
Site Fibre Line (Clinical School)
The Janet Network
Faculty of Education
Homerton
Burrell’s Field
Needham
Research
Institute
Bene’t Street
Mornington Crescent
1 2 3 4 5 6 7 8 9
1 2 3 4 5 6 7 8 9
E
D
C
B
A
F
E
D
C
B
A
F
Transport for Cambridge
Chemical
Engineering &
Biotechnology
open summer ‘16
open summer ‘16
~6km
direct;~10km
by
fibre
11. Dark fibre challenges
• The GBN is easy — buy your [cheap] switches
and transceivers; rent a circuit and feed your
VLAN across it
• Ideal for regular, point-to-point high bandwidth
data transfers (e.g. MRI brain scanner to HPC
cluster)
• Difficulty is scaling to become a multipoint
service: you end up building your own network
12. Active network challenges
• CUDN allows transfers across an existing connection, however...
• Institutions expect 10G links to run at 10G
• ... and without disruption to / because of their regular traffic
• Traffic spikes (to/from the institution or across the backbone) can
interrupt high speed flows and take time to recover (TCP
sawtooth problem)
• “I’m only getting 3Gbit/s” — could be the backbone, but could
also be the disk, transfer protocol, firewalls, local institutional
network, remote institutional network, remote server — testing
requires clean, directly-connection host with iperf
13. QoS?
• QoS may be necessary to smooth out flows and
avoid disruption — bandwidth might not be
sufficient if you're operating at near line rate
• We already do QoS for phones and (soon)
CCTV
• Remember — you can’t create bandwidth, just
decide how to use it (and is a political
problem)
14. Security developments
• University looking to harden the network and attached hosts
from cyber attacks
• New border IDS/IPS solution (NOT a firewall) — has to operate
at 20Gbit/s... at the moment
• Upgrading will be more expensive than just transceivers and
patch cords
• Push to introduce more institutional firewalls
• VLANs allow bypass but not if the clients can be separated
• Need to improve control plane security
15. Troubleshooting problems
• Output queue drops (link bandwidth exceeded)
• Input queue drops (exceeded internal switch capacity)
• Ask your vendor — they should have packet walks, block diagrams
of buses, bandwidth, port groups, buffer sizes and oversubscription
ratios
• We had to swap ports around on our core routers to better distribute
traffic across buses and solve problems (delivered speed increase
from 3-4Gbit/s to 8-9Gbit/s)
• Don't always go for the largest number of ports per slot, especially
towards the centre of the network
• Beware port aggregation and traffic hashing: 4x 10G LACP ≠ 40G
16. Things to remember
• Links are never 50% loaded — busy 50% of the
time, measured over a defined period: it’s either
busy or not busy; buffers cope with spikes
• Beware port aggregation and traffic hashing: 4x
10G LACP ≠ 40G
• We have 3x [2x 10GE] links to Janet: 60Gbit/s?
• Speeding up the network speeds up DoS attacks
17. Transferring data
• We leave this up to the scientists...
• But SSH implementations can have inherent
limitations, in particular OpenSSH
• Special version or hacks (our HPC have their
"SSH download accelerator”)
• We [Networks] don’t particularly care but we don’t
want them to bust the network for everyone else
