What Does it Take to Deliver a Solution to Process Over $2B in Loans from Inception to Production?
•Download as PPTX, PDF•
0 likes•228 views
SpringOne 2020 What Does it Take to Deliver a Solution to Process Over $2B in Loans from Inception to Production? Srinivas Machani, Chief Architect at Fiserv Clint Myers, Vice President, Architecture / CTO - AP at Fiserv
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to What Does it Take to Deliver a Solution to Process Over $2B in Loans from Inception to Production?
Similar to What Does it Take to Deliver a Solution to Process Over $2B in Loans from Inception to Production? (20)
More from VMware Tanzu
More from VMware Tanzu (20)
Recently uploaded
Recently uploaded (20)
What Does it Take to Deliver a Solution to Process Over $2B in Loans from Inception to Production?
- 1. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 20201 To move money and information in a way that moves the world Our Aspiration Clint Myers VP, CTO Account Processing Srinivas Machani Director, Architecture Account Processing
- 2. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 20202 Our Purpose: To deliver superior value for our clients through leading technology, targeted innovation and excellence in everything we do U.S. Bank Account Processing Market Share 36% 4,000+ Financial Institutions
- 3. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 7,655 Approved Status Applications Totaling: $878M 3 18,000+ U.S. small businesses awarded loans ... …totaling nearly: $1.4B Tens of thousands of SBA PPP loan applications processed
- 4. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 20204 Applicant Portal Application Review Workflow Financial Institution Portal SBA Integration Fiserv SBA
- 5. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 Deploy Design Research Develop “Vision without action is a daydream, but action without vision is a nightmare.” ― Kaihan Krippendorff Agility Time to Market
- 6. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 Shift from a reactive to a pro-active strategy. Self-Healing Reliability
- 7. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 Additional workers can be added or removed depending on the load Auto-Scaling Scalability
- 8. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 Run more applications on less hardware Application Density Cost
- 9. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 20209 104,794new lines of code Orchestrating countless cloud containers via Tanzu Application Service, an enterprise Kubernetes platform for auto-scaling and self-healing Predictive load models defined minimum containers per microservice 436 28 days Code releases over 8 Developers 3 UX designers 2 QA 2 Architects
- 10. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 202010 Loans totaling nearly $1.4 Billion into the hands of small businesses “Fiserv helped us get over 2,000 PPP loans approved through the SBA in two days. This is a HUGE win for our company and our customers. With genuine tears in my eyes, and on behalf of our customers, our employees and our company…THANK YOU!” - Jeff Starke, CTO, Bank OZK
- 11. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 Building Blocks CI/CD Jenkins, Artifactory, Twistlock API and DB Design Spring Boot Features and DB schema Infrastructure build out TAS/TKG, Splunk and APM 11
- 12. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 202012 Infrastructure and Data Flow
- 13. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 Spring allowed us to develop simpler API Spring Data JPA allowed us to develop API much more quickly Spring multi-threading has enabled us to write scalable code Few annotations/features allowed us to build first class API @Async allowed us to process data asynchronously for 100s SBA loan applications simultaneously spring-boot-starter-web; spring-boot-starter-data-jpa; spring-boot-starter-web-services Building API
- 14. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 202014 Account Processing Pull Docker containers Change Request 1 5 Service Point 2 3 BU Manager Operations Manager Auditing Product Deployment Manager 4 Development Manager Artifactory CI / CD Pipeline TAS
- 15. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 Dynatrace APM – SBA API 15
- 16. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 Dynatrace APM - Infrastructure 16
- 17. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 Splunk - Logs 17
- 18. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 • Database Design • Naming conventions • Code simplification Learnings 18 • Spring Data JPA • Unintentional data mutation • Transaction boundaries
- 19. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 Beyond SBA 19 Monolith
- 20. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 Split Source Code (Domain Driven Design) 20 Account Management Payments Safe Deposit Box Servicing Stop & Hold Cards Party Party Account Rel Party Account Ser Document Man eAgreement Lending Collateral ePreferences
- 21. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 Restructure Organization (Around Microservices) 21 Team A Team B Team C Team D Account Management Payments Stop & Hold Party Cards Document Man Safe deposit box Party Account Rel Lending E-Agreement Servicing Party Account Ser Collateral E-Preferences Superior focus on producing automated testing 71 Changes / Year 71 Changes / Year 67 Changes / Year 70 Changes / Year Teams balanced around the number of predictive changes in each microservice
- 22. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 Containerize 22 Account Management Servicing Safe Deposit Box Payments Stop & Hold Party Account Ser Party Party Account Rel Collateral Cards Lending ePreferences Document Man eAgreement Utilize predictive load models to define minimum containers per microservice Orchestrate countless containers via TKG for auto-scaling and self-healing
- 23. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 Follow Fiserv 23 To deliver superior value for our clients through leading technology, targeted innovation and excellence in everything we do Our Purpose Twitter: Follow @Fiserv | LinkedIn: Connect Fiserv | Online: Fiserv.com
- 24. © 2020 Fiserv, Inc. or its affiliates. | FISERV CONFIDENTIAL FORTUNE World's Most Admired Companies® 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 Thank You! 24
Editor's Notes
- In just a couple of weeks, X new lines of code were written, and we issued around 24 releases in two weeks. Agility is the key: when legislation passed, we had a minimum viable product in 3-5 days. Running on a cloud platform meant we didn’t have to provision new servers. We were prioritizing enhancements, iterating, developing as much as we could in a given day and then pushing out code into Pivotal for nightly releases. Clients were able to realize the benefits in virtual real time. The platform allowed us to autoscale: In an unknown world of PPP, didn’t know how much we could sell. We were walking into the dark. We had a high comfort level using the pivotal platform. If we were a victim of our own success. If we sold 50 percent more than we thought, we wouldn’t have to worry about outages. Self-healing: We needed it to be up all the time, and a cloud environment enabled that. In one hour, we could process several million dollars in loans. Detect some part of code not acting correctly and automatically restart without human intervention.
- In just a couple of weeks, X new lines of code were written, and we issued around 24 releases in two weeks. Agility is the key: when legislation passed, we had a minimum viable product in 3-5 days. Running on a cloud platform meant we didn’t have to provision new servers. We were prioritizing enhancements, iterating, developing as much as we could in a given day and then pushing out code into Pivotal for nightly releases. Clients were able to realize the benefits in virtual real time. The platform allowed us to autoscale: In an unknown world of PPP, didn’t know how much we could sell. We were walking into the dark. We had a high comfort level using the pivotal platform. If we were a victim of our own success. If we sold 50 percent more than we thought, we wouldn’t have to worry about outages. Self-healing: We needed it to be up all the time, and a cloud environment enabled that. In one hour, we could process several million dollars in loans. Detect some part of code not acting correctly and automatically restart without human intervention.
- In just a couple of weeks, X new lines of code were written, and we issued around 24 releases in two weeks. Agility is the key: when legislation passed, we had a minimum viable product in 3-5 days. Running on a cloud platform meant we didn’t have to provision new servers. We were prioritizing enhancements, iterating, developing as much as we could in a given day and then pushing out code into Pivotal for nightly releases. Clients were able to realize the benefits in virtual real time. The platform allowed us to autoscale: In an unknown world of PPP, didn’t know how much we could sell. We were walking into the dark. We had a high comfort level using the pivotal platform. If we were a victim of our own success. If we sold 50 percent more than we thought, we wouldn’t have to worry about outages. Self-healing: We needed it to be up all the time, and a cloud environment enabled that. In one hour, we could process several million dollars in loans. Detect some part of code not acting correctly and automatically restart without human intervention.
- In just a couple of weeks, X new lines of code were written, and we issued around 24 releases in two weeks. Agility is the key: when legislation passed, we had a minimum viable product in 3-5 days. Running on a cloud platform meant we didn’t have to provision new servers. We were prioritizing enhancements, iterating, developing as much as we could in a given day and then pushing out code into Pivotal for nightly releases. Clients were able to realize the benefits in virtual real time. The platform allowed us to autoscale: In an unknown world of PPP, didn’t know how much we could sell. We were walking into the dark. We had a high comfort level using the pivotal platform. If we were a victim of our own success. If we sold 50 percent more than we thought, we wouldn’t have to worry about outages. Self-healing: We needed it to be up all the time, and a cloud environment enabled that. In one hour, we could process several million dollars in loans. Detect some part of code not acting correctly and automatically restart without human intervention.
- In just a couple of weeks, X new lines of code were written, and we issued around 24 releases in two weeks. Agility is the key: when legislation passed, we had a minimum viable product in 3-5 days. Running on a cloud platform meant we didn’t have to provision new servers. We were prioritizing enhancements, iterating, developing as much as we could in a given day and then pushing out code into Pivotal for nightly releases. Clients were able to realize the benefits in virtual real time. The platform allowed us to autoscale: In an unknown world of PPP, didn’t know how much we could sell. We were walking into the dark. We had a high comfort level using the pivotal platform. If we were a victim of our own success. If we sold 50 percent more than we thought, we wouldn’t have to worry about outages. Self-healing: We needed it to be up all the time, and a cloud environment enabled that. In one hour, we could process several million dollars in loans. Detect some part of code not acting correctly and automatically restart without human intervention.
- Building with agility”: Show agile development graphic. In just a couple of weeks, X new lines of code were written, and we issued around 24 releases in two weeks. As we were building, the SBA and U.S. treasury were still working out the details. And as we’re working to support these loans, the Small Business Association and the U.S. Treasury were still working out the details on how the loans would work. A new announcement would come out that changed the loan terms or change the required fields on the applications. Cloud capabilities supported this agility.
- How do you build an enterprise application in a few days? Hi! I’m Srinivas Machani with Fiserv and I’ll be talking about our journey through the valleys, peaks and springs of SBA loans application that Fiserv developed within 4 weeks to help the US economy. I’ve been with Fiserv for 9 years and have 24+ years of experience in technologies ranging from front end (UI) to middleware to backend systems. We started our development on March 31st SBA First round funding lasted 14 days – Apr 14th and second round started on Apr 27th Like all good platforms, we started with a solid foundation, build out a PCF platform, that is operationally ready Then we had a team of developers develop the APIs with a DB design that supported the SBA PPP Loan functionality Last but not least we created the the CI/CD pipelines that would enable us to move from Dev to Ops as quickly as possible
- Lenders and Borrowers Review and approve the applications. Walk through the diagram left to right Disparate protocols and technologies. MFA with OTP generation. React JS front end, that loads on the client machine, redirects the API requests to Springboot, exposed on the internet. SBA Gov exposed SOAP. Based web services, needed to get certified. Credentials were given on the day of Go Live (March ?) Had to tweak the Apigee bundle to successfully send to SBA **First round of funding we used RPA to send applications, then we developed the Gatling Gun to rapid fire all the 25K or so applications
- Actual Development: Agility is the key Agility is the key: when legislation passed, we had a minimum viable product in 3-5 days. Running on a cloud platform meant we didn’t have to provision new servers. We were prioritizing enhancements, iterating, developing as much as we could in a given day and then pushing out code into Pivotal for nightly releases. Clients were able to realize the benefits in virtual real time. The platform allowed us to autoscale: In an unknown world of PPP, didn’t know how much we could sell. We were walking into the dark. We had a high comfort level using the Tanzu platform. If we were a victim of our own success. If we sold 50 percent more than we thought, we wouldn’t have to worry about outages. Self-healing: We needed it to be up all the time, and a cloud environment enabled that. In one hour, we could process several million dollars in loans. Detect some part of code not acting correctly and automatically restart without human intervention. DB Design, Spring Data JPA Secure forms XML, login and OTP for Github repository, Spring JPA, XML staging service Gatling gun Spring Libraries utilized: Spring fox - library for generation swagger hub documentation Spring Dev tools - changes in code will compile and deploy to embedded Tomcat server Spring Boot - auto configuration for properties and embedded Tomcat; Java Build pack in PCF (Open JDK) Javax.mail Jackson for JSON PDF part FOP - Apache library (combining HTML with XML to generate the PDF) Spring Scheduler @Async for parallel Lombok - generates the getters and setters but didn’t work as well used Eclipse IDE Session handling using DB; Session timeout - adaptive timeout ; session gets extended (15 minute idle logout) Spring boot maven plugin for generating the jar
- In just a couple of weeks, 5K new lines of code were written, and we issued around 24 releases in two weeks. Sonarqube scans, TwistLock image scans Jenkins, Github with webhooks and security scans, stored the images in Artifactory where the Twistlock scans kicked off We had to take few shortcuts in the defined process, when you are moving at breakneck speed and there isn’t a version of app that you are going to break anyway, we could afford to move faster We externalized all the parameters that could change Numerous changes: Changes to Spring Environment variables: File upload limits, timeouts, DB user passwords, SBA URLs; SPRING_JPA_HIBERNATE_DIALECT; Vendor Keys, SPRING_DATASOURCE_DRIVERCLASSNAME spring.datasource.hikari.connection-timeout: 60000 spring.datasource.hikari.minimum-idle: 200 spring.datasource.hikari.maximum-pool-size: 500 spring.datasource.hikari.idle-timeout: 10000 spring.datasource.hikari.max-lifetime: 10000 spring.servlet.multipart.max-file-size: 15MB spring.servlet.multipart.max-request-size: 15MB
- All good applications in production must have APM and logging (Splunk) Tanzu has built in plugins for Splunk, all standard out was directed to Splunk We started monitoring the traffic via Dynatrace. We had Dynatrace working and hooked up to the TAS platform with some agents installed and then built out the dashboards required to monitor our APIs Explain the various graphs in the chart…the DB locks were due to some transactional queries and Gatling Gun running without @Async for parallel
- We found some interesting information using Dynatrace…the pods were crashing sporadically… We did not implement the pagination (used Criteria building API), first one should retrieve only 10 records and the count—>this one got all the records, so we saw SQL spewing 1000s of lines in Splunk Turned auto scaling on
- How big were the pods max memory 6GB Max, important logs
- Database Design: Naming conventions for database and java naming for entity classes Reduction in mapping of column names with property name of the java class Follow JPA naming strategies in setting up DB tables Spring Data JPA : Any changes in entity object were reflected in the database (Unintentional data mutation) Detached the entities from the context while manipulating and saved it back Refrain from making changes directly to the entity object but use value objects Define transaction boundaries for each method (especially for committing back to DB) This platform will live for another year but that is the beauty of the platform, it scales and auto heals so we don’t need to worry about it! We did not implement the pagination (Criteria building API), first one should retrieve only 10 records and the count—>this one got all the records Good things: PEN Test changes to the application, parametrize all the values that can change APM, analysis, dash boards, agile teams working around the clock
- How to microsize Existing SOAP services that were in a monolith? DDD Eric Evans; Considered various angles, frequency or transactions, changes, business functionality
- Team restructuring around domains and agile methodologies
- Final picture and talk about routing using Apigee or considering Spring Cloud Services and Gateway and cert-manager that manages the certificates at a cluster level learnt something new in this conference yesterday from Chris TKGI, HPA
- Every day, our 40,000 associates worldwide are here with one thing in mind – focused on how we deliver superior value for you – through leading technology, targeted innovation and excellence in everything we do. Did I mention Fiserv was named Fortune Magazine’s World’s most admired companies 6 years in a row and we a Fortune 200 company, 45000 employees, 35 years old