My Presentation from the ILUG 2010 Belfast conference. Here the Abstract: "Come to this session to learn on real examples how to read and understand those NSD files that can give you so mush information for troubleshooting and debugging your Domino Servers, Notes Clients and even your Applications. You will learn how to find what files and documents has been open, what agents has been running and how much memory was available, as the last crash occurred. You will also see all kinds of tips and tricks around the system diagnostics, that will allow you to troubleshoot problems faster and more effective. "

1. NSD ‘Crash Course’
Gregory Engels, Kompurity
ILUG 2010

2. Some Rights Reserved
Copyright 2010 by Gregory Engels,
Kompurity. Except where otherwise
noted, this work is licensed under the
Creative Commons Attribution Share
Alike 3.0 License

3. About me
Gregory Engels
Born in Moscow, live in Frankfurt, Germany
since 1989
Been around Lotus Notes since 1998
Advanced Certified Domino Developer and
Admin and Security R5, R6, R7, R8
Co-Author "Migrating
Microsoft Exchange 2000/2003 to IBM Lotus
Notes 7"
Founder Kompurity http://www.kompurity.de
Specialize in Hosting, Infrastructure
management
Blog: http://inotes.de
Twitter: dichter

4. Agenda
• NSD
• ADC
• Upgrading NSD
• LND

5. • Notes Systems Diagnostic
• binary on W32 and iSeries
• shell script nsd.sh on Unix
What is NSD

6. Running the NSD
• Manually
• Automatically as part of Fault
Recovery

7. • Enabled by default
• Enable it in the
Server document
• Go enable it now.
Fault Recovery

8. Help is on the way
• nsd -help

9. Automated Data
Collection (ADC)

10. Enable in the Config Doc

11. Enable in Desktop Policy

12. Fault Reports DB

13. Put this in the nsd.ini
• canceldisable=1
• (BOFH tip of the day)

14. Real Life
• ADC Enabled 9 Month ago
• User Population c.a. 6000 Users
• 1700 Users have created at least 1
NSD
• c.a. 13000 Client NSD
• 13 Users (0.05%) created 1700 NSD
(14%)

15. How often does Notes
Crashes?
0
10
20
30
40
50
60
70
80
90
100
1 5 9 14 21 30 52 75 163 1258
Number of Users that had N Crashes
Occurency

16. Number of NSDs per
User
0
375
750
1125
1500
Occurences

17. NSD’s per Day
0
18
36
54
72
90
22.08.2009 25.08.2009 28.08.2009 31.08.2009 03.09.2009 06.09.2009 09.09.2009
12.09.2009 15.09.2009 18.09.2009 21.09.2009
24.09.2009

19. Upgrading NSD

20. • See TN #4013182 for ftp links
• See TN #1233676 for list of SPR’s
fixed
• Version of NSD is
<NotesVersion><Fixpack>.<YNNN>
• NNN = Days since year begin
• example: NSD 8.0.10.8057 (Release
8.0.1) = Feb 27 2008 (57 Days since
Jan 01)

21. Reading the NSD log

22. Finding the log
• Logs are stored in the
IBM_TECHNICAL_SUPPORT
subdirectory
• in the Lotus Notes Domino Fault
Reports Database lndfr.nsf
• ND6 - nsd_all_<Platform>_<Host>_MM_DD@HH_MM.log
• ND7 -
nsd_<Platform>_<ServerName>_YYYY_MM_DD@HH_MM_SS.log

23. Contents of the NSD
• Rather verbose
• Process Info (Call Stack)
• Memcheck (Domino Memory
Objects)
• System Information
• Environment Information

24. • Process Information (Call Stacks) –
Process Information is composed of
the list of all running processes
system wide, followed by a list of all
Domino specific processes.
• the code path involved in a particular
problem.

25. • Memcheck (Domino Memory Objects)
– The Memcheck section dumps
information about Domino-specific
structures.
• memory pools (both private and
shared)
• list of open resources such as open
database, open view, open documents,
connected users, and open files.

26. • System Information – This section
provides information regarding
version of OS,
• kernel configurations, patch
information, disk information,
network connections, and memory
usage.

27. • Environment Information:
• user environment,
• notes.ini,
• file listing

28. Finding the fault
• Look for the words FATAL, PANIC, ETC
### FATAL THREAD 67/135 [ nSERVER:0908: 2692]
### FP=0x0ae5ec70, PC=0x6018eb23, SP=0x0ae5e2f8
### stkbase=0ae60000, total stksize=262144, used stksize=2424
### EAX=0x010d088c, EBX=0x00000000, ECX=0x00ba0000, EDX=0x00ba0000
### ESI=0x0ae5e904, EDI=0x00001d34, CS=0x0000001b, SS=0x00000023
### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000
Flags=0x00010202 Exception code: c0000005 (ACCESS_VIOLATION)
Process ID: Thread ID

29. Reading the Callstack
############################################################
### FATAL THREAD 1/4 [ncompact: 3044: 2620]
### FP=0x0012e644, PC=0x60193297, SP=0x0012dacc, stksize=2936
### EAX=0x00e3088c, EBX=0x00000000, ECX=0x00900000, EDX=0x00900000
### ESI=0x0012e1d8, EDI=0x00d495d8, CS=0x0000001b, SS=0x00000023
### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010202
Exception code: c0000005 (ACCESS_VIOLATION)
############################################################
@[ 1] 0x60193297 nnotes._Panic@4+631 (60a30013)
@[ 2] 0x600027a6 nnotes._LockHandle@12+294 (3963e26,12e674,12e680)
@[ 3] 0x600025e7 nnotes._OSLockObject@4+23 (60c7df40)
@[ 4] 0x606c09cc nnotes.CArchSetPolicy::GetPrivateArchiveSettings+3404 (1)
@[ 5] 0x606bf1a8 nnotes.CArchSetPolicy::GetArchiveSettings+1032 (0)
@[ 6] 0x606be9f9 nnotes.CArchSetPolicy::Init+553 (33a0a99)
@[ 7] 0x606c7554 nnotes._PolicyCheckServerArchiveEnabled@4+52 (a9)
@[ 8] 0x606e2fc1 nnotes._CheckServerArchiveEnabled+113 (10900a9)
@[ 9] 0x00402111 nCompact._CompactFile@16+529 (0,a9,81000000,0)
@[10] 0x00401b61 nCompact._AddInMain@12+2913 (3,3,398644)
@[11] 0x004028cf nCompact._NotesMain@8+47 (3,400000)
@[12] 0x004029d6 nCompact._notes_main+214 (0,0)
@[13] 0x004028f6 nCompact._main+22 (3,3b2da0)
@[14] 0x00402cc1 nCompact._mainCRTStartup+197 (0,0)
@[15] 0x7c816ff7 KERNEL32.RegisterWaitForInputIdle+105 (402bfc,0)
Pid : Tid

30. Finding the Resources
• Search for “Open Databases”
• Search for “Open Documents”

31. DBH NOTEID HANDLE CLASS FLAGS IsProf #Pools #Items Size Database
531 7330 0x24ff 0x0001 0x0200 Yes 1 4 2984 d:notedata
drmailjsmith.nsf
Open By: CN=John Smith/O=ACME/C=US
Flags2 = 0x0404
Flags3 = 0x0000
OrigHDB = 531
First Item = [ 9471: 836]
Last Item = [ 9471: 1228]
Non-pool size : 0

32. Note Class Description
• 0x0001 Data Note (document)
• 0x0004 Form Note
• 0x0008 View note
• 0x0040 ACL Note
• 0x0200 Agent Note
• 0x0800 Replication Formula Note

33. Search for VTread info
• Search on KEYWORDs “Resource
Usage” or “Resource Usage Summary”
• Search on KEYWORD “Process” until
you located the necessary process ID
• Search on the PHYSICAL THREAD ID
of the thread you are interested in
(cosection after you have looked at
call stacks)

34. Troubleshooting NSD

35. • Try NSD -info
• If this runs to completion, perhaps
there is a problem with the pid.nbf
file
• Try upgrading NSD
• Check if symbols are installed
• lotusnotes.sym

36. LND Tool

37. • See TN # 4019151 for download link
• ftp://ftp.software.ibm.com/software/
lotus/tools/LND_2.8_Setup.zip

40. Auto KB Seatch

41. See the Process
Resources

42. Works on Right-Click
in Explorer

43. Questions

44. Thank You!
• Please fill out the evaluation (so I can
get back next year)
• Contact: Gregory.Engels ※
kompurity.de
Blog http://inotes.de
twitter: dichter

45. Backup

46. NSD parameters
• nsd –stack (collects only call stacks, speeds execution)
• nsd –info (collects only system info)
• nsd –noinfo (collects all but info)
• nsd –memcheck (collects only memcheck info)
• nsd –nomemcheck (collects all but memcheck)
• nsd –perf (collects process memory usage)
• nsd –noperf (collects all but performance data)
• nsd –handles (collects OS level handle info)
• nsd –nohandles (collects all but handle info)
• nsd –kill (kills all notes process and associated memory)
• nsd –monitor (attaches and waits for exceptions)
• nsd –p (runs against a specific process – call stacks only)