In discussions with some colleagues from my previous employer, it was very enlightening to hear their perspectives on SharePoint. From their point of view, SharePoint wasn’t an application unless it had a customization package sitting on top of it and unless it had a customization there was no reason for testing. Really? A sign of inexperience for sure. Why? Any complex product that requires the amount of configuration and infrastructure that SharePoint does must be tested and tested ongoing especially after applying service packs, adding customizations and or third party add-ons. Why? Ask yourself these questions:
- How do I know SharePoint is configured properly?
- The servers, network and storage are optimized?
- The required features are activated?
- Customizations are optimized?
- The configuration is delivering the performance it was designed to provide?
- A new change or update was done successfully? It’s impact to performance? Functionality?
- A new change or update didn’t impact performance in a negative way?
- Proactive plan for migrations and consolidation?
- That capacity plans can be acted upon in a smart way?
What’s the risk / monetary impact of not having proper QA? My employer at the time incurred an additional $700k in additional cost yearly (figure assembled by procurement and product management offices.) due to not utilization proper Change Management, Development and Quality Assurance practices. This figure doesn’t include the lost productivity that occurred over a 2-3 month period each year, negative perception of the poor service, regions leveraging poor service to justify their own SharePoint farms (service duplication cost overhead, fractured KM model and compliance mgmt. control plan) and which resulted in high stress for the service owner.
Needless to say this companies SharePoint infrastructure was not very stable, was riddled with problems and for the SharePoint Admins, it was a nightmare to support and troubleshoot. They couldn’t answer any of the aforementioned questions. When escalated to their executive management they did nothing (Execs had been with the company since the beginnings and didn’t have the experience and open mind to understand this perspective). Why? The executive didn’t want to risk looking foolish regarding the infrastructure they (he) helped design, deploy and manage. Managements directive was to hide the problems from the business and blame the SharePoint staff. Have the same problem? Search on my Governance articles, this is a clear case for Governance. So where do you start? Document the following:
- Each service you will activate
- Each of the features activated
- Any default settings you plan to change
- The expected performance of page loads, provisioning, search results and various jobs
- Any customizations you will deploy
- Any third-party applications or add-ons you will deploy
- Performance under load
What this gives you is a list of test cases to build on for your test plan. For example, you configure Profile Services, how do you make sure it’s configured properly? You run an import and check the user profile database. Another example would be how do you make sure farm performs properly? You configure load testing software and simulate 250, 500, 1000 and 2000 users on the farm carrying out transactions (e.g. page browsing and or document uploads). The following are sample templates based on IP I have developed over the years working with a variety of clients:
- Test Guidelines example
- Test Plan example
- Test Cases example
- Data Set Build Scripts (Contact me firstname.lastname@example.org)
- Load Simulation Scripts (Contact me email@example.com)
- SharePoint Configuration Document (Contact me firstname.lastname@example.org)
- SQL Server Configuration Document (Contact me email@example.com)
To help get you started here are some tests you should carry out:
- Profile import – run import and check profile database
- Search crawl – upload some documents, run crawl and search on documents
- Business Services Catalogue – provision a catalogue entry and consume it with a list.
- Meta Data Services – provision and import some properties.
- Page load times – configure load testing software and simulate 250, 500, 1000 and 2000.
- Search result times – configure load testing software and simulate 250, 500, 1000 and 2000.
- Form services – create a form in InfoPath, publish it, view it, enter data and save it.
- Workflow – create a workflow for a document review and approval.
- Workflow – create a workflow with a pause 1, 5, 10 minute.
- Add a content database – add a content database to make sure SQL and storage are configured properly.
- Add a content source – add a content source to search and run crawl.
- Provision a site collection – provision a site collection to make sure your process works.
- Provision a site collection – provision a site to make sure your process works.
- Provision a subsite – provision a site to make sure your process works.
- List and Libraries – create them and delete them.
- Documents – upload, view and download.
- Pictures – upload, view and download.
- List items – create, view and delete and item.
- Load test – load each test at 1,10, 100, 25, 500, 1000, 2000 users.
- Check SharePoint Farm health analyzer.
- Check event logs on all servers.
- Check storage trends and logs.
- Check networking device(s) trends and logs.
As far a amount of data to simulate I suggest 1 TB for small companies (Up to 1000) employees), 4 TB for medium sized companies (1000 – 50,000 employees) and 10 TB for large employees (over 50,000 employees). There are scripts available that can do this for you or you can create scripts that does this as I did. We creates a 1000 site collections and uploaded enough data etc. to simulate production. Note if your really detail oriented throw in some large lists and libraries as this will impact servers and cause spikes.
This is a short list but helps make sure your core services are working correctly. These sorts of tests (with more detail) are especially important when you host composite application on SharePoint. Also, refer to your installation guide for more ideas regarding what to test, look for services and features your configuring and other settings. Finally, with all the moving parts it’s important to be able to test components to make sure nothing breaks after major code changes or applying service packs. Some final advice, the Quality Assurance team must be run by an Executive with empowerment and no ties to any infrastructure or application legacy decisions. This person and the team must be open minded and approach QA with the companies best interests in mind. Want to read more? Have a look at SharePoint and test quality assurance, Capacity and baselining Part 1, Capacity and baselining part 2 and SharePoint Performance Baselining Part 4. Also, explore other examples such as Sample Test report.
In the end what did we find?
- Poorly written code – full of errors and coding flaws
- The development had been done with out any oversight
- Infrastructure had over provisioned the SAN and it ran out of IOPs and could not handle peak loads
- Infrastructure had over provisioned the network switches connecting the SharePoint environment and could not handle peak loads
- The change management process didn’t include necessary quality checks
Have feedback, lessons to share or ideas? Contact me.