Things WILL get VERY technical when two experts face-off in a unique session that explores polar perceptions regarding various types of logs, verbosity levels, data extraction, responses for alerts, and more. Be it Domino, Sametime, or Traveler operating on-prem. or in Hybrid and Cloud environments, it is vital to have an understanding of log data structure, what is (or isn't) logged and why, and how to search logs effectively. But aren't there ways to find your information without having to pipe everything into the log? Where does one's best practice end and another's begin? From this collision of opposing viewpoints and real-world stories, you'll take away knowledge and tools ready to deploy to various scenarios, products, and log types.
3. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Speaker
• Ben Menesi
§ Head of Product at Ytria
§ IBM Notes Administration & Development
§ Certified Domino Admin, Dev and Adv.
Security Professional v7+
§ Speaker at ICS / ESS tech conferences
around the globe
§ IBM Champion ‘14 – ‘16
4. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Speaker
• Chris Miller
§ Director, Collaboration at Connectria
§ @IdoNotes on everything
§ IBM Champion ’11 – ‘16
5. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Questions to ask ourselves
Does Domino always
display what is logged?
Does Domino always
log what it displays?
6. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Agenda
Mail Logging
Usage Activity
Replication Logs
Web Server Logs
DDM vs. Others
Logging Misc.
1.
2.
3.
4.
5.
6.
7. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Mail Logging
• Daily logging is different than troubleshooting logging
§ The goal is reduce logged data and utilize built in tools
• MTC can be your friend across a domain for daily logging
issues
8. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Mail Logging
• Other alternatives to MTC:
§ Move4Ideas Audit4Mail: Server task, intercepts all mails via
HTTP, Router, SMTP or AMGR
• Can show custom items, keeps light headers for longer time and
accessible by helpdesk, without the Domino Administrator client
§ GSX Mail Monitor
9. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Mail Tools
• Mail File Digest - summary report for mailfiles and mail-in
databases from OpenNTF
§ http://idonot.es/mailfiledigesttool
• Server add-on, requires downtime to get done, plus only supports
8.0.1, 8.5.2 and 9
• Implement Converter_Log_Level=10
§ Removes tons of unnecessary log entries on mail conversion
• Log_MailRouting=0
§ Yes this is an actual value when used with one we discuss later
10. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Bonus Tip: One remedy for all mail rule issues
• Mail rules are frequent troublemakers for Administrators
§ Let’s understand their architecture a bit better
§ Tip: If you ever struggle with Mail Rules, just create a new one.
Mail Rule 1 CalendarProfile
$Filterformula
$Actions
$Conditions
Ordernum
$Filterformula_1
§ More info:
http://techlab.ytria.com/5243/ibm-notes-
secrets/mail-rules-architecture-sort-
scanez
11. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
User Activity
• Who’s accessing what?
§ Increasingly important question whether you’re:
• Upgrading / consolidating your environment
• Considering going to the cloud
• In the process of migrating
§ What data to use and HOW to use it
• Frequently used databases and apps that aren’t used at all?
12. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
User Activity
• Possible Data Sources
Activity
Trends
Database
User
Activity
Log.nsf
User
Activity
3rd Party
Solutions
§ Valuable high level
reports
§ Not very customizable
§ Lack of customization
leads to inaccuracies
§ Very detailed logs
§ Really crucial info
buried inside one text
field
§ Unless through 3rd
party tools: NONO
§ Great amount of detail
§ Only available per
database
§ To be discussed later
§ Amazing products out
there
§ Some installed on
server side
§ Some client side
§ To be discussed later
13. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
User Activity
• Activity Trends – How does it work?
Log.nsf Catalog.nsf Names.nsf Statistics
Activity Trend Collector
Activity.nsf
14. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
User Activity
• Activity Trends – Why is it good?
§ Incorporates valuable data and provides high level reports on:
• Databases usage (Active / Inactive, space, growth,
• Users (Active users, connection times, databases used, R/W counts,
Data sent and received from server)
• Server tasks (High level reports on HTTP, IMAP, LDAP, POP3,
Replication etc…) According to configuration
• Server Disk Space (Usage, growth, etc...)
• Server Statistics
§ Tip: Use the Domino Administrator client to visualize data:
15. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
User Activity
• Activity Trends – Why is it good?
§ Interactive Analysis in the Domino Administrator client
16. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
User Activity
• Activity Trends – Cons
§ Summary data reports: you can’t really dig deeper
§ Reports aren’t customizable
– E.g. Database marked ‘Active’ – yet only a Server and an Administrator
has used it within the past 3 months
§ Does not properly calculate Space Used Percentages
17. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
User Activity
• Log.nsf
§ There is a TON of useful, raw data captured in User Activity
Documents
§ Yet, using log.nsf for User Activity Analysis is the common Enemy
§ What is actually recorded in a User Activity Event?
Date & Time Reads Writes User
Bytes Read Bytes Written DB Open Time
18. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
User Activity
• Log.nsf – What is recorded?
Date & Time Reads Writes User
Bytes Read Bytes Written DB Open Time
19. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
User Activity
• Recorded User Activity
§ User Activity can be a *VERY* rich source of data (incl. HTTP!)
§ Tip: No_Force_Activity_Logging=0 (Default value) allows on ALL
dbs
§ Tip: Set confidential to prevent users with lower than Designer
access
20. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
User Activity
• Prior to ODS 48
§ Entry size allocated 44 bytes, object size 61600 bytes (max 1400)
• ODS 48+ (undocumented improvement)
§ Entry size 92 bytes. Object size 128800 bytes (same max 1400)
§ Curious: only two additional counters added, yet ODS48+ entry
size is over 2x the size of ODS48 – why?
Date & Time Reads Writes User
Date & Time Reads Writes Updates Deletes User
21. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
User Activity
• Does Notes only capture what it displays?
§ Nope
§ Read more: http://techlab.ytria.com/5448/lotus-notes-
articles/interpret-user-activity-traces-portrait-global-usage/
Reads Adds Updates Deletes
Data
Non
Data
Σ
Data
Non
Data
Σ
Data
Non
Data
Σ
Data
Non
Data
Σ
22. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
User Activity
• Cons
§ Only available per database
• You have free tools available to facilitate working with this data
across multiple databases: http://www.agecom.com.au/useractivity
– Or implement via W32_NSFDbGetUserActivity
– Or get Ytria’s databaseEZ
§ Makes no difference between users and servers
• Filtering this info is easy in tools like databaseEZ
§ Only maintains 1400 entries
• Funny you should say that…
23. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
User Activity
• Deriving usage frequency from User Activity
§ Database A is more frequently used than Database B
Database A
Entry Date
Entry Date#1400
#1
75 days
Database B
Entry Date
Entry Date
120 days
#1400
#1
24. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
User Activity
• 3rd Party products for usage analysis
§ Domino Doublecheck
• http://goo.gl/maL0fo
§ Panagenda iDNA
• https://www.panagenda.com/products/idna
§ Ytria databaseEZ & consoleEZ
• http://ytria.com/databaseez
• http://ytria.com/consoleEZ
25. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Replication Logging
• Replication is a subject that raises various questions such as
§ How do I make sure replication runs smoothly?
§ What do I do when problems occur (and problems do occur)
§ The log file and log_replication is pretty useless
26. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Replication Logging
• Various Replication diagnostic tools are available such as the
Log_Replication notes.ini parameter.
§ What if we want to know what exactly happens during replication?
• Set Log_Replication=3 to get information about each replicated note:
27. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Replication Logging
• Can we go even higher in Log_Replication?
§ Why not go to 4
• This will show detailed logging information about each replicated
field
§ Or how about 5
• Adds some additional logging summary
• http://www-01.ibm.com/support/docview.wss?uid=swg21214739
28. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Replication Logging
• What about Cluster Replication?
§ Cluster Replication isn’t always logged (Check
initiatedby item on replication logs)
• Use RTR_Logging (manage information written into log.nsf)
– 1 – Default
– 2 – All context structure changes
– 4 – Log Attempted & Performed replications
– 8 – Log iterations through main polling loop
– 16 – Verbose debug logging
– 32 – Log all lock operations
29. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Replication Logging
• Replication Logs in your Log.nsf
§ There’s more data in there than you might see
• Categorize your Replication Logs view by “Initiatedby” to differentiate
between replication events carried out by the Replicator and Cluster
Replicator.
• Use a formula @IsAvailable(“Eventlist”) to display exclamation mark
next to events, to instantly see replication events that need attention
• Use @Elements(Pathname) to get the number of databases (and
directions) replicated per session.
– Makes it easy to map replication events to connection documents!
30. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Replication Logging
• Improved Replication Logs view
§ Let’s take a look
31. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Web Server Logging
• Filtering and excluding is a key to reducing log clutter
§ Server document -> Internet Protocols -> HTTP -> Exclude
§ By listing file extensions (*.gif) n URL request they are removed
§ Use full MIME type listings to remove image logging
• Content-Type: image/gif
32. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Web Server Logging
• Consolidating HTTP & Miscellaneous Logs
§ Use the HTTPLogUnauthorized=1 Notes.ini Parameter
• Logs 401 HTTP Errors in console AND Misc. log events
• HTTP Users attempting to access resources they aren’t entitled to
access + user authentication requests.
33. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
DDM vs. Other methods
• DDM is about building proper triggers for logged and non-
logged data
§ DDM takes effort to configure properly
§ Domino exposes all errors but does not technically log them
34. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
DDM vs. Other methods
• Customer Example
§ DDM constantly communicates disk space low
§ Customer used to running low on storage
§ 90% of databases with <25%
§ Discovered Program doc.
turned off in 2009
35. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
DDM vs. Other methods
• How to extract more from Log.nsf?
§ There is MUCH more data in your logs than you know
• Each event comes with a status code, severity, server add-in
§ Extract more data from your log documents using formulas such
as:
• # of events, unique events
• Highest severity event encountered
• # of various severity events
• Total # of messages transferred per each Mail Routing document
36. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
DDM vs. Other methods
• How to extract more from Log.nsf?
§ More Info: More info:
http://www.slideshare.net/BenedekMenesi/ibm-connected-2015
37. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
DDM and Stat Tools
• Andy Pedisich statrep.nsf customization
§ http://www.andypedisich.com/blogs/andysblog.nsf/dx/resources.h
tm
38. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Agent Logging
• Debugging Agents through Print statements is BAD
§ How can we log and control our agents better?
• OpenLog on OpenNTF
§ https://www.openntf.org/main.nsf/project.xsp?r=project/OpenLog
• Thomas Lindberg Agent Controller (OpenNTF)
§ https://www.openntf.org/main.nsf/project.xsp?r=project/AgentCon
troller
• AgentEZ
§ http://www.ytria.com/agentEZ
39. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Logging Misc.
• Clean up your logs even more
• Log_disable_session_info=1
§ This will remove when sessions are closed
§ It is on by default on all servers
• Log_sessions=0
§ Do we really need to know when a user has a session?
§ This works in conjunction with Log_MailRouting=0 or alone
40. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Logging Misc.
• Log filtering
§ This is built into all your Domino servers in events4
§ See Chris' article http://idonot.es/filterlognsf
41. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Logging Misc.
• Log File Retention
§ Default Deletion time specified via “Log=“ Notes.ini Parameter
• Don’t use this!
§ Use “Remove Documents not modified in this file”
§ Use Archive Profile(s) to split into multiple databases
§ Use console logs to retain old log data
• Log Mirroring, Mirror retention
§ Conclusion
42. 2016ConnectThe Premier Social Business and Digital Experience Conference
MakeEveryMomentCount
Thank you
• Thank you for attending!
§ We’d love to hear from you
ca.linkedin.com/in/benedekmenesi
Twitter.com/BenMenesiTwitter.com/Idonotes
Ben Menesi
Linkedin.com/in/idonotes
Chris Miller