Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to Random thoughts on sql server performance
Similar to Random thoughts on sql server performance (20)
Recently uploaded
Recently uploaded (20)
Random thoughts on sql server performance
- 1. Random Thoughts on... Performance and Troubleshooting SQL Server Nigel Foulkes-Nock 1
- 2. About me Nigel Foulkes-Nock • SQL Server DBA with TargetGroup, Newport • Previous Barclays Partner Finance Hewlett Packard • Worked with SQL Server versions from 6.5 through 2016 2
- 3. Session Purpose • Discuss real-life Performance issues • Provide ideas and suggestions • Get you thinking 3
- 4. Disclaimer • Test in a non-Production environment • Follow your Change process • Speak with your DBA 4
- 5. My Experiences • When you can’t change the Query • When you CAN change the Query 5
- 6. The Server is running slowly... • What do I know? • It’s slow • 2 CPU / 16Gb Memory • Minimal Workload 6
- 7. The Server is running slowly... • RDP connection sluggish • Task Manager | Performance • Eeek! 7
- 8. The Server is running slowly... Max Memory Setting 8
- 9. The Server can breathe again • Max Memory changed to 12Gb • 4Gb free for O/S • Everyone is happy 9
- 10. CPUs – More is good, right? • Queries running longer • 8 CPU, 64Gb Memory 10
- 11. CPUs – More is good, right? • Check the Defaults 11
- 12. CPUs – More is good, right? • Check your queries with sp_blitzcache or similar tool • Set “Cost Threshold for Parallelism” and consider “Max Degree of Parallelism” 12
- 13. Making the right decision • Query Plans • Estimated Query Plan • Actual Query Plan • One more... 13
- 14. Making the right decision • Query Plans 14
- 15. Statistics • What’s in the columns • How unique is the Data? • Consider Phone Number or Gender • More unique = better 15
- 16. Indexes • Not for free! • High Profile option • Check your Data Types • Index Tuning Advisor - consider your actions 16
- 17. Maintenance • Statistics need updating • Indexes need de-fragmenting • Ola! 17
- 18. What are you doing now? • Live Query Statistics • SQL Server 2016 and 2014 • DMV – sys.dm_exec_query_profiles 18
- 19. What are you doing now? • Live Query Statistics • SQL Server 2016 and 2014 • DMV – sys.dm_exec_query_profiles 19
- 20. What are you doing now? • Live Query Statistics • SQL Server 2016 and 2014 • DMV – sys.dm_exec_query_profiles 20
- 21. Hindsight • Involve your Developers • Simplify Queries • Temporary Tables (not Table variables) 21
- 22. Change the Queries • Avoiding “Data Bloat” • Choose Datatypes wisely • Fix early (just say no!) 22
- 23. Go faster by doing less • Performance was poor • Virtual Server – hot-upgradable • Updates, removing invalid Characters • 3Tb of Data Updates but 200Gb of Data 23
- 24. Go faster by doing less • SQL Table – 200Gb • 3 Characters to update, 5 Fields • 200Gb * 3 Characters * 5 Fields = 3000Gb 24
- 25. Go faster by doing less • Only use required Columns • Filter rows, early! 25
- 26. Conclusion • It all happened – really! • Resources to follow • Nigel Foulkes-Nock NigelFN@Outlook.com 26
- 27. Resources • Where you can’t change the Query • Memory Issues • https://www.brentozar.com/blitz/max-memory/ • CPU Simple http://blog.idera.com/sql-server/is-your-sql-server-slow-maybe-its- trying-too-hard/ Advanced • http://www.littlekendra.com/tag/cost-threshold-for-parallelism/ 27
- 28. Resources • SQL Server Query Plans • https://www.simple-talk.com/sql/performance/execution-plan-basics/ • https://www.sqlsentry.com/plan-explorer • Statistics and Indexing • http://blog.sqlauthority.com/2010/01/25/sql-server-find-statistics-update-date- update-statistics/ • https://www.simple-talk.com/sql/performance/sql-server-statistics-basics/ • https://www.pythian.com/blog/sql-server-statistics-maintenance-and-best- practices/ • https://www.mssqltips.com/sqlservertip/1206/understanding-sql-server-indexing/ • https://www.brentozar.com/sql/index-all-about-sql-server-indexes/ • Database Maintenance • https://ola.hallengren.com/ 28
- 29. Resources • Query Optimization Tips • http://blog.sqlauthority.com/2013/08/28/sql-server- tips-for-sql-query-optimization-by-analyzing-query- plan/ • Temporary Tables - gets advanced • http://sqlblog.com/blogs/paul_white/archive/2012/08 /15/temporary-tables-in-stored-procedures.aspx • Temporary Tables - against the use of… (for balance) • https://sqlperformance.com/2016/04/t-sql- queries/knee-jerk-temporary-tables 29
- 30. Resources • Are you using the correct Data Types? • https://www.connectionstrings.com/sql-server-data-types- reference/ • Where to go next - Advanced Links • https://www.brentozar.com/responder/ • https://www.simple-talk.com/sql/performance/a- performance-troubleshooting-methodology-for-sql-server/ • http://www.sqlskills.com/blogs/paul/wait-statistics-or- please-tell-me-where-it-hurts/ 30
- 31. Resources • Teach Yourself SQL Server Performance Tuning • http://www.littlekendra.com/2016/08/11/teach- yourself-sql-server-performance-tuning-dear-sql-dba- episode-12/ • http://www.red-gate.com/products/dba/sql- monitor/entrypage/hard-earned-lessons-3 • https://www.brentozar.com/sql/sql-server- performance-tuning/ 31
- 32. Fin 32
Editor's Notes
- Good afternoon. I'd like to introduce myself. I am Nigel Foulkes-Nock, and I am a DBA with TargetGroup, a software and Financial service provider based in South Wales. TargetGroup are on the SQL Relay “Awesome Employers” list, and more importantly came in the top 5 at the Cardiff Half Marathon Corporate Challenge last Sunday.
- This talk is all about improving the performance of SQL Server. It's billed as a fairly basic talk - there’s plenty of ground to cover in 30 minutes. I won't be going into any area in any great depth, no demos but plenty of food for thought. Let’s get a bit of feedback to start off with: How many people have experienced performance problems… with SQL Server? OK, that’s pretty much everyone. How many people managed to fix their problems? Got someone else to fix them? Finally, how many people have caused performance problems in the past? OK, that gives us all an idea of other people’s experiences, highlights our frustration and gives me an idea (for the last question) of how truthful you are. Over the next 30 minutes, I'll be sharing some of my experiences with you. I'll be talking about issues that I have found, how I investigated further and the eventual outcome The session is aimed at making you think - you may think “I know that already”, which is great. You may think, "ah, I can use some of those techniques", or even "oh heck, I've created that issue" (in which case you can promptly start looking at how you can resolve it when you get back to the office). Exact details of the issues have been masked to protect the guilty. I hold no malice for people bringing me "learning opportunities", though it's never good for someone to bring the same issue again and again.
- Disclaimer Any suggestions or thoughts brought on from this talk should be performed in a safe environment (yeah, right) and any production changes should follow your existing change control process. I think I've covered everything in my disclaimer there. Don't go beyond your remit - if you're a Developer and you have DBAs at your organisation, don't go making Server level changes or installing stuff - go speak to your DBAs. If you've already got a DBA but you notice some things a bit screwy in your environment, go and talk to them. There's nothing we like more than someone with ideas to help us - just don’t go in and change things. Even if you have the access, it’s still not OK. If you have no DBA, lobby your management to get one. Better still, why not consider becoming a DBA? Happy to talk about this, and all sorts of other things after the session - just come and talk!
- Now, I’ve split my experiences into two parts: What to do when you can’t change the Query In this section, I’ll talk about changes that you can make without changing the code. That doesn’t mean that they are not Changes (Change with a capital “C” here), they just don’t require “code changes”. These changes can be really useful in many situations - you might be dealing with a Commercial Application, a piece of Legacy Software where all of the Developers have moved on, or maybe there’s just too much involved in testing any changes made to the logic of the existing code. We’re just interested in the Environment. What to do when you CAN change the Query I nearly wrote “Performance Tuning when you CAN fix the Queries?”, but I didn’t. Partly because the great Brent Ozar does a training course under this title, but more so because when it comes to queries, there’s often someone far more qualified to change code and fix Performance Issues. How about the people who caused the issues in the first place? Developers, Report Writers, Users can be quite receptive to polite suggestions towards how they might be able to make their queries play nicer with the Server and other queries. After all, they know the data, they know the results that they’re looking for and they have the expertise in testing the Changes that they’ve made. Often, they just need a bit of polite encouragement.
- The call came in - “Nigel (never Nige), will you look at this server - it’s slow”. There wasn't a huge amount of data on there. The Server specification was reasonable for what was involved - 2 CPU / 16Gb Memory, not a lot of data, I logged on and it felt a bit sluggish. I checked the "Performance" Tab in Task Manager - the Server had 5Mb of Memory left.
- Upon further investigation, the memory settings had been left at the default - essentially allowing SQL Server to “take as much as it wants”, so it did just that - it took everything. With no Memory available to the Operating System and other Services, Windows Server just started using the Paging file instead of memory and that’s why everything became really slow... Memory is quick, disk is slow.
- I changed the Maximum Memory setting within SQL Server to 12Gb which gave the Operating System 4Gb to "breathe", re-started SQL Server and all was happy once more.
- CPU - More is Less sometimes Now, I'm going to continue on the theme of the last improvement to talk about another situation where SQL Server can have too much as well as too little. Falling hardware costs and ever increasing Database Sizes mean that plenty of SQL Servers are being specified with multiple processors. Trouble is, they're not always configured to work with them properly. Another time, I started to look into a situation whereby Queries were taking longer and longer to run each day. I started by checking the basic environment - after all, 8 CPU, 64Gb of memory, for the workload it should have been sufficient. No massive increase in data to be processed so no real reason for the slowdown. I checked the basic SQL Server setup details and there was a clue on the Options tab of SQL Server Properties. The value for "Cost Threshold for Parallelism" was set to the default value of 5. Also, the "Max Degree of Parallelism" was set to the default of 0.
- Most Servers these days will have more than one processor, this one certainly did. Running a query using multiple processors is always going to run faster than one a single processor, right? It Depends. Before SQL Server runs a query, it works out the most efficient way that it can run. It creates an Estimated Execution Plan, together with calculating an arbitrary cost for running this Query. The "Cost Threshold for Parallelism" value is used to determine whether SQL Server should use multiple Processors to run this query. If it's set too low, then you can have a situation whereby too many queries are being run across multiple processors. The default value for “Cost Threshold for Parallelism” is 5. Not five seconds, just 5 - the measurement used is an arbitrary value that fokelore says is based on one of the original SQL Server developers PC. That said, 5 is a low value. Current best practice is to set this to 50 in order to let a lot of the smaller queries just run happily along on single processors. You see, when you run a query over multiple processors then you often get unexpected delays (Waits) - SQL Server will split up the work and then gather together the results at the end.. I can relate this back to the race that I spoke about on the intro slide. I was part of a Corporate team, but the race wasn’t over until all of the team were over the line. No matter how quick our best runner finished, they had to wait for me until we were considered finished. In my case, I changed the configured values and suddenly SQL Server started running most of the smaller queries more quickly.
- You can check the “Cost” of executed Queries by using a tool like sp_blitzcache from Brent Ozar unlimited, available as part of the “First Responder” kit a free download (link on the Resources slide).
- I mentioned Query Execution Plans earlier - SQL Server uses them to determine the most efficient way of processing a Query. They don’t always work as you’d expect them to. You can't dictate exactly how SQL Server is going to run a query. Well, you can, but that’s like trying to dictate your children's choice of friends, it's never a good idea to force the situation. Far better to help guide them towards the right decision. Reading and interpreting Query Plans is a huge subject. It’s a really interesting one and there’s plenty of Resources available to investigate further. I’ll provide a few links in the notes.
- You can view the “Estimated Query Plan” before running a Query and look at the “Actual Query Plan” afterwards by using the settings in SQL Server Management Studio (SSMS). You can view the plan in SSMS, or better still use the excellent “SQL Plan Explorer”from SQLSentry which is a completely free tool that makes SQL Query Plans far easier to read (especially the big ones).. We had an issue whereby a Query was running slow - too slow. I brought up the Estimated Query Plan and what did I find? A few Tables were being Joined, Indexes existed on the Columns being used for the Joins but it was clear that these Indexes weren’t being used, and the Query was running slowly. Why was this? Lets talk about a few things
- Statistics SQL Server keeps Statistics on columns to keep a track of the distribution of values in those columns. As an example, if you have a column that contains Phone Numbers then the Statistics will show SQL Server that most of the values (perhaps all) are unique, so an Index is likely to be useful. On the other hand, a column that contains Gender is likely to have very few values so it would be less useful. Statistics change over time and it's really easy for Statistics to get out of date, causing SQL Server to not use an Index and do a Table Scan instead - i.e. read all of the data in the table. I checked how fresh the Statistics were - they were old and stale, so I updated them (really easy to do). Re-running the Query, SQL Server had a better idea of what was in the data and chose a better way of returning that data in the Query.
- Indexes Indexes can make it quicker to run a query, either by improving the speed of joins between Tables, or also when they include all of the columns needed from that table for a query - in that case, SQL only needs to read the Index instead of the entire table. One difference between Indexes and Statistics is that in my experience, Managers know the word Index. “Add an Index, do it NOW” is a common expression when performance takes a nose-dive. I can happily talk for hours about Indexes, but don’t worry, not right now. Just think of Indexes at mini-tables. They contain the Columns that you’re indexing on, together with a link back to the main Index that is used to hold the complete set of data. The point that you should get from the above statement is that they take up disk space and they need maintenance. Each time that a Table is updated, if the update includes columns that are part of the Index, the Index will need updating too. You can quickly arrive at a situation whereby the maintenance of your Indexes has more of an effect than the problem that they are solving. Also, when Joining Tables, make sure that the columns that you join on are of the same DataType. For example, if you’re using INT to store Customer Number in one Table and Varchar(25) to store it in another Table then joining those Tables won’t using any Indexes on those columns. You can see that this is happening if you see a “CONVERT_IMPLICIT” predicate in your Query Plan Index Tuning Advisor can be a bit over-eager in making suggestions - you need to approach with caution and learn about your Data before making changes. Personally, I prefer to use sp_BlitzIndex from Brent Ozar Unlimited but even then you need to exercise caution and that tool won’t make suggestions on certain types of Indexes. Indexes need maintenance too - they get Fragmented over time - they will need re-building and re-organising. It's worth checking and de-fragmenting your Indexes
- A great tool for Maintenance is provided free by Ola Hallengren. This is highly configurable so that you can set it to only work on Tables that are fragmented beyond a certain level or above a certain size. Details on the Resources page once more. In my experience, updating Statistics can have more of an effect than de-fragmenting Indexes. Also, de-fragmentation can cause an “IO storm” of updates - especially if you’re doing Log Backups / High Availability, so it’s probably best done less frequently.
- Live Query Statistics SQL Server Management Studio 2016 introduced a new feature called “Live Query Statistics”. This allows you to view a Query Plan as a the Query runs. is running in an animation. There’s a really nice graphical representation of the Query Plan with the arrows showing the number of rows processed - estimated rows and actual rows. It really helps with the learning of exactly where the bottlenecks are in running your query.
- Not only that, it can be run against SQL Server 2014 as well as SQL Server 2016.
- But wait, there’s more - this information is available not just in Graphical format - it’s exposed using a Dynamic Management View - sys.dm_exec_query_profiles. This DMV provides a huge amount of data about a running Query - provided that you ask it. To use sys.dm_exec_query_profiles, you need to set “Include Actual Execution Plan” in SSMS before running your query, or run the Query with the command “SET STATISTICS PROFILE ON”. When would you want to use this, certainly not every time you run a query. We had a situation where a Query wouldn’t complete and we were stuck. Started Troubleshooting by viewing the Estimated Execution Plan. That gave a few clues - I reviewed the Indexes, the Statistics and added an Index or two with the aim of making the Query Plan more efficient. The Estimated Query Plan returned a far lower value, but still the Query wouldn’t complete. Further optimisation was necessary, but I needed to view the “Actual Query Plan” - but the Query wouldn’t complete. Catch-22 situation. Using Live Query Statistics, either in it’s graphical form or by using sys.dm_exec_query_profiles then I can get a better idea of where the query is stuck. It’s also a really nice way to see where SQL Server is choosing a bad plan based on bad estimates. I’ve got a little query that tells me parts of the running query have completed where the Estimated Rowcount differs from the Actual Rowcount. Potential reasons behind this include out of date Statistics, User Defined Functions, Table Variables, Distributed queries, Implicit Conversions and plain old Complex queries.
- Hindsight This last section was written with the benefit of hindsight. Fixing this specific issue didn’t involve the use of Live Query Statistics. I had a chat with the Developer, nice chap. He took the problematic code and examined it. Now, many overly complex queries with huge sub-queries can be re-written. Data retrieved by the sub-queries can be extracted into Temporary Tables at the beginning of the query. These Tables can be Indexed just like any other Tables and breaking down a Query like this can provide not only higher performance for the Query, it can also make it easier to read and troubleshoot. Just make sure that you’re not copying great swathes of data and creating yet another problem. Sure enough, there were many sub-queries, the query was re-written as per the guidance above.. Performance increased significantly following the re-write, it completed in a tenth of the original time and everyone was happy. It just goes to show that sometimes things can just get too complicated for SQL Server to process in one go. A word about Table Variables I mentioned “Temporary Tables” note that I’m not talking about “Table Variables”. If you see any of these then that’s certainly a prompt to re-write the Query. Table Variables look like a great idea until you realise that SQL Server can’t deal with them properly when estimating Query Plans. Oh, now I’ve strayed into the second part of the talk
- Move quicker by avoiding “Data Bloat” I was doing the Daily Checks one morning, and noticed some growth on one of our Databases. I took a quick look at the "Top Tables by Disk Space" Report (one of the Microsoft supplied ones that you can access from by right-clicking the Database, then selecting Reports / Top Tables by Disk Space. Upon viewing the Report, I was surprised to see an unfamiliar table at the top of the tree. This table wasn't part of what I'd regarded as our main workload, I couldn’t understand why it was there - and based on dividing the Table size by number of rows, each row seemed inordinately large - something just didn't look right. I examined the structure of the Table and it appeared that the Developer hadn't taken much care in their choice of DataTypes. Choosing the wrong DataType can make a massive amount of difference to the ongoing size of a Table. Simple things like using INT or BIGINT for Columns where a smaller DataType would suffice. Examples like "Month of Year" - BIGINT (that's 8 bytes) where TINYINT would have done (just 1 byte), using numeric fields with massive scope for values / precision, using fixed widthCharacter fields that are larger than the expected values “just in case”. Use of UNICODE Data Types where there’s no possibility of non-standard ASCII values being held. By using the correct DataTypes, it was possible to reduce the size of the Table significantly. Most of the Indexed columns in the Table were affected too, so the Indexes were significantly larger than they should have been. It wasn't a simple job to rectify this situation - we were running out of space because of... the overly sized Table and if you simply change the Column definition then guess what? The Table gets bigger. This is due to the way that SQL Server processes Table definition changes. The only route is to copy the Data into another Table (or out to a file and back). Even then, you are likely to need to do it in Batches to avoid filling your Log file. It’s not a pretty situation but it get worse as you go along, so best to catch it early. Best to catch it in a review of the code before it gets to Production but hey, we live in the real world. Another "special feature" about this Table was a series of about 150 numeric columns, each of which may be populated, or more likely would be set to "null". Either value would take the same amount of space regardless. The use of "Sparse" columns can come in handy in a situation like this - provided that the vast majority of the entries in the "Sparse" column are likely to be "null" then by defining the column as "Sparse", the disk space that the Table uses would be significantly reduced.
- Increase performance by doing less I had a call to say that SQL Server performance was poor. This was on a Virtual Server, so we could add CPU or memory to make it go quicker. Guess what they wanted us to do? I fired up SQL Profiler and set a trace to show me what was going on. I set a filter to only record queries that were taking more than a few seconds. One little point to note here is about SQL Profiler - make sure that you only use it when absolutely necessary and always switch it off straight afterwards otherwise you'll create yet another Performance issue, all by yourself. After the main queries had run, I examined the Profiler output only to find that it involved data updates. Multiple updates. Very similar updates. Nothing wrong with the work that needed to be done, let’s just say that it could have been done more efficiently. The updates were to remove Control Characters from a Table prior to further processing.. Each update to remove an invalid character was run as a separate statement. There were three different characters being removed across five different fields, all done as a separate SQL Update Statements. Now, this SQL Table was 200Gb in size. Let us do a little calculation... 200Gb (Table Size) x 3 (Characters) x 5 (Fields) = 3,000 Gb That's 3 Terabytes of data moving around SQL Server just to do this data update. The Server had 16Gb of memory allocated, OK, that’s not a lot. 12Gb of this was allocated to SQL Server. Now, SQL Server just loves memory. SQL Server is one big cache. As you query a Database or update Data, SQL Server will deliver the information and perform the update in memory. If the data that you need is in memory right now, it'll let you have it. If it's not in memory then SQL Server will load it from disk, then hand it to you (or update it), from memory. This will happen until SQL Server fills the memory that it has to work with, then it writes any changed data back out to disk before dropping that data from memory to make space to load something else into memory. A constant shuffle. Remember - Memory is quick, disk is slow. Reading through 3Tb of data to do the updates when 200Gb would do will make a significant difference to the speed of your processing. The Query was updated to perform all of the updates in one go and performance increased significantly.
- Don’t use so much Data Only use the Columns that you require Look out for “SELECT * FROM” within your code. Chances are, if you’re using that, no Indexes are being used and you’re shifting far more data than you need (or want) to. Not just that, if you’re performing a Pivot or any form of GROUP BY operation, you only want to perform the action based on the relevant columns. Filter rows, filter early Another common issue that I’ve encountered is a series of SQL Statements that carefully copy another set of the Data, update, embellish that data and finally use a DELETE FROM or add a WHERE clause to only return part of the Data. Often, that filtering can be done earlier in the process. Along with only using the Columns that you need, this can make a tremendous difference to the performance of a query.
- Conclusion At the beginning of this session, I promised you would hear about some of my real-world examples. Everything in this talk truly happened - even if I can’t give all of the grizzly details. The following slides will be available after the event. They cover background information about the Topics that I discussed. They will help guide you in the right direction. If you have any questions, please get in touch, either following the talk or feel free to drop me an e-mail. Thank you for your time. Nigel Foulkes-Nock NigelFN@Outlook.com