A presentation at Networkshop46, by Sam Wilson

1. Reprocuring a
large campus
network, part one
Sam Wilson,
University of Edinburgh

2. Sam Wilson
Communications Infrastructure Section
IT Infrastructure Division
Information Services
The University of Edinburgh
Sam.Wilson@ed.ac.uk

3. Agenda
>University
>History and Current Situation
>Procurement
>Lessons

4. > The University of Edinburgh has a large computer network
> The network has grown organically over the years
– We’ve tried to be economical with equipment
– “Technical debt” - network needs to be refreshed
> We’re part way through the reprocurement
– This may be the most frustrating talk you
hear today
Summary

5. A slide from our Bidders Conference

6. Oops! That was 2017.
In 2018 we're 23rd=.
A slide from our Bidders Conference
Perhaps we should
buy a new network or
something.

7. >…apparently this kind of marketing makes network
providers give you things cheap. Who knew?
I’m a techy, but…

8. > It’s not all about technology
> “Digital Transformation” – everything is digital
– Data networking is fundamental
> Data Driven Innovation
– High capacity
– Rapid change
> Student Experience
– High quality WiFi
– Stability and performance
Business drivers

9. History – How did we get here?

10. > Plan installation of LAN across campus
> 10 Mbps Ethernet, 100 Mbps FDDI, 2 Mbps
serial
– Thick and thin ethernet
– Extensive fibre installation
> 1000 hosts, growing to 3000 hosts over
~5 years
> One of first two UK Universities to buy
Cisco routers
> IP(v4), IPX, AppleTalk, DECNet Phase IV,
bridging (for Pink Book)
Indulge me – 1988-1990

11. > 2014: New director of IT Infrastructure
– First task: infrastructure review
– Some kit up to 10 (20?) years old
– No budget for renewal
> Network has grown to >100K outlets
– Integral to University business, operations
and strategy
– Phones, WiFi, BMS, CCTV, Things…
> In the interim network budget -> Estates
– Building refurb cycle (10-20 yr) != network
replacement cycle (3-5 yr)
Fast forward to 2014
Oops!

12. Let’s buy a new network!

13. We’re a University; we have committees and projects and such
Well maybe not quite yet...
2014: internal infrastructure review
2016: PTS Consultants – consider network strategy
Nov 2016: PTS report – “buy a new network”
“we need a strategy” – set up a project
May 2017: Budget approved

14. > 5 main campuses
– University estate approx. 4 x 10 miles (or 10 x 10)
– 220+(ish) buildings
– Mainly owned or leased fibre
> Over 100,000 outlets
– 30K+ active ports
– 10M, 100M, 1G, 10G, 40G
– 2700 switches, 2700 WAPs
– 600+ racks
– ~13 core and distribution routers
> Constant churn
– 10 year building programme
Scale

15. > Wired and wireless networks
> Data centre firewalls
> Wireless firewall
> Other security features
> Management and monitoring
> Automation/Orchestration
> Future expansion
> Professional services
> 5+2+2+2+2 year contract
Out of scope
Scope
In scope
> Data centre networking
> Upgrade or replacement of
cabling unless essential for
solution
> Wireless p2p links
> Student residences
> (Perimeter firewall)

16. > Budget: [REDACTED]
> Project Board: senior representatives across UoE
> Project Team: ~7 core members
– Director ITI
– 2 x Communications Infrastructure Section
– Procurement Officer
– Project Manager
– 2 x PTS Consultants
> SMEs: Security, Academic computing, Lawyer (~10)
Now can we start the procurement, Daddy?

17. > Multiple rounds of dialogue – not just an ITT
> “Down select” at each round to reduce field
– Kind of knockout stages
> Drive down price
> Co-design?
– collaborate on what’s possible/available
– range of potential solutions
– beware of steering bidder and then rejecting
Procurement by Competitive Dialogue

18. > There are things I can’t tell you:
– because we’re only part way through the
process and we don’t know yet;
– because they’re confidential –
commercially sensitive or NDA;
– because we’ve decided not to tell anyone
– like the budget
Frustrations

19. > Premarket engagement
> OJEU Contract Notice
> ESPD
– Like PQQ – select top 10
> ITPD
– 10 provide outline solution – select 5
> ITCD
– Lengthy dialogue
– 5 provide developed solution – select 3
> ITCD 2
– Select 2
> ITT
– Select 1
Programme
YOU ARE
HERE!

20. > June-July 2017: Premarket engagement
– 6 manufacturers selected by consultants
– Alcatel Lucent, Cisco, Dell EMC, Extreme, HPE, Huawei
– ½ day presentations
– Own research
> What's on the market?
– Pure L2 or pure L3 backbone? Y
– VLANs or no VLANs? Y
– Overlays, tunnels, data/control plane...? Y
– NAC for security or configuration? Y
– Automation and orchestration? Y
– SDN (some or all of the above)? Y
– Firewalls and security features? Y
– You name it…
Pre-market engagement
?!

21. > September 2017:
– Backward looking scored Qs
– Includes draft ITPD document so they know what's coming
> About 20 responses
> Mid October: Choose top 10 to receive ITPD
– Formal scoring
> No, I can’t tell you who any of them were
Contract Notice and ESPD

22. > 3 Nov: ITPD Published
> 7 Nov: Bidders conference
> ITPD documents
– 91 page project brief - simplified network model
– MRs, scored Qs (4 pages), pricing spreadsheet
> Responses required
– MRs
– Outline solution (10 pages)
– Scored Qs & As (4+10 pages)
– Pricing spreadsheet
ITPD

23. > Responses received early December
– 10 x ((10 + 14 pages) + MRs + pricing spreadsheet)
> ½ day meetings with bidders to present their outline solutions
and answer questions
– Includes manufacturers
– NOT scored
> Answers scored
> Sorry letters and ITCD invitations issued 22 Dec
– We spoiled everyone’s Christmas
> You think that sounds like a lot of work?
ITPD Responses

24. > Mid-Jan to late Feb 2018
> 2 + 1 day with each of 5 bidders
– Discuss outline solution
– Questions arising from solution
– (Some) ways that the University is different from the model
– Bidders may change original solution and costings
> Generate ITCD document with scored Qs
– Expanded from ITPD brief and Qs
– But ask original Qs again - make sure the answers
haven't changed
> Responses required
– 10 page developed solution
– 30 page scored questions (8+22)
– Pricing spreadsheet
> Issued 21 Feb 2018
> Responses 14 March 2018
ITCD

25. > ITCD 2 (3->2) – March-June 2018
> ITT (2->1) – July-September 2018
> Selection and award – September-November 2018
> Implementation – November 2018 for 18 months
> Contract – 5+2+2+2+2 years
And next...?

26. > Budget: [REDACTED]
> 20 initial responses: [REDACTED]
> 10 ITPD bidders: [REDACTED]
> 5 ITCD bidders: [REDACTED]
> 3 ITCD bidders: [REDACTED]
> Solutions we’ve been offered? [REDACTED]
> Leaning toward any particular outcome? Not redacted - because
we have to judge all bids on their merits.
The frustrating bit summarised

27. Lessons?

28. > Competitive dialogue is a **lot** of work
– ITCD had 7 staff almost full time for 6 weeks, not
including scoring
– Also for the bidders
> We might have too many stages
– Each stage takes a lot of effort (info pack, dialogue,
evaluation, down selection)
– 10 - 4 - 2 - 1 rather than 10 - 5 - 3 - 2 - 1?
> The process can change your views
– I can’t tell you how. Sorry
Is there anything we'd do differently next time?

29. End of part 1
(If you want to see part 2 you’ll have to invite us back
next year)

30. Spare Slides

31. > ESPD: European Single Procurement Document
– Like PQQ – backward looking “have you done this before?”
> ITPD: Invitation To Participate in Dialogue
– Outline solution based on model design
> ITCD: Invitation To Continue in Dialogue
– After down selection
> ITT: Invitation To Tender
Alphabet soup

32. Another Bidders Conference slide. I’ve included it because…