Thoughts on Robert Bogue’s Post: How Best Are Your Best Practices

Robert Bogue raises some challenging questions and makes some powerful statements in his post, How Best are Your Best Practices, regarding his experiences and struggles with reaching consensus on what a best practice is and applying it. I was in the middle of posting a comment to his post when I realized my thoughts on the post were quite a bit more than a couple of lines in a comment box. And without further adeu, here are my thoughts…

Doing what we know we should do

In his post, Robert asks, “when are we going to do what we say?” This question was in context of his findings that there are very few professionals out there that actually use the best practices they know. From a consulting professional perspective, I feel that failing to do what we know we should do is ethically wrong. Clients and customers pay for our expertise and deserve to get it. While I agree that the SharePoint space is so large that even experienced professionals are learning something new about SharePoint on a daily basis, we need to ensure that make use of the best practices that we do know in every project that we undergo regardless of its size and complexity. This is our duty as professionals.

On defining best practices

I agree with Robert when he states that the definition of best practices has been a problem space in software development that has yet to be resolved. There are many reasons for this. But let me list a couple:

  • Academically, software engineering methodologies regarding the definition of development practices are still in their infancy. While a lot of progress has been made in this area in the last few decades, software development is not a hard science. Some of us consider it more of an art than a science.
  • While progress has been made academically in this problem space, there is a large disconnect between academic approaches to the problem and practical approaches used in the industry. I think some of this has to do with the cost of trying different frameworks that generate process and procedures that still end in leading a good number IT and software projects to failure.

So what can we do to help reach consensus on best practices? Honestly, I don’t have a good answer to this question right now. What I can tell you is that what ever practice we do try to use, whether a best practice or not, we should make the effort to take review how the practice helped or hindered us, learn from the experience, and share it so that we may continue to define new best practices and debunk those practices that are not so best anymore (assuming they really were a best practice at one time).

SharePoint Reading List

A colleague recently asked me for tips on how to prepare for the 70-630 exam (TS: Microsoft Office SharePoint Server 2007, Configuration) and asked me to look at some sites with sample questions. Frankly sample questions are great to prepare you for an exam, but they only really help if you have already studied necessary material first. I highly recommend and enjoy getting solid hands on instructor lead classroom based training (for example check out the courses offered by the Ted Pattison Group that are actually taught by SharePoint MVPs). I also highly recommend to read, read, and read (yes that includes reading authoritative material on the Internet, however, there is something about reading an old fashioned paper and ink book that I enjoy so much more). With that said, I want to share my existing SharePoint library as well as books on my shopping list.

  1. Inside Microsoft Windows SharePoint Services 3.0 by Ted Pattison and Daniel Larson – This is a must read for all SharePoint developers. If you are working with WSS 3.0 and/or MOSS 2007, then you have to read this book. Personally, I have read it cover to cover three times and refer to it on a regular basis. The book is organized very well and cover the fundamental topics for developing anything SharePoint. It also includes download-able examples that are easy use and help to expose you to best development practices and the object model rapidly without overwhelming you. Highly recommended for developers.
  2. Inside Microsoft Office SharePoint Server 2007 by Patrick Tisseghem – Another solid read. This book covers development with the added value features of MOSS 2007 that were not covered in the Pattison/Larson book (look at #1 above). I don’t use this book as much as I use the Pattison/Larson book since I find that most of my projects really only require use of the WSS 3.0 framework, but when I do need to develop against the BDC and Search (or other MOSS 2007 features) this is my reference of choice. Recommended for developers.
  3. Microsoft Office SharePoint Server 2007 Administrator’s Companion  by Bill English – As a developer, this 1200 page monster-sized book was not my typical read. It’s target audience is the system administrators, network infrastructure, and other non-developer type IT professionals that need to manage and administrate MOSS 2007. However, this book provided me with a solid foundation for preparing for the 70-630 exam. Not to mention, I believe that developers really need to understand how the platforms we develop operate to get clear insight as to how power users use the solutions and software we build (this is another topic that I need to blog about some day, great progress has been done in software engineering to consider end user interface design but the smaller audience of power users and administrators, who are also end users, get left out). Highly recommended for administrators. Recommended for developers.

That’s it. Three books. You take those three books plus the documentation and white papers on MSDN and TechNet and the great blogs (not the junk blogs which are way too abundant) that are on the Internet and you have enough resources to prepare for the 70-630 exam.

There have been some recent announcements of upcoming titles that I want to pick up and add to my library. Here is a quick list:

I know that most of these books are from Microsoft Press and that is not because I am partial to them. There are some other great books out there from Wrox other technical references. However, the books listed up there have been authored by (or with) SharePoint MVPs whose work I professionally admire and respect.

If you need more SharePoint books for your library, then check out Andrew Connell’s “Best Damn SharePoint Books” List.

SharePoint Ramp Up Webcasts for ASP.NET Developers

I wish I would have seen this schedule earlier since the first webcast was today. Nevertheless, I will be trying to catch the others live (or on demand if I miss them). Even as an experienced WSS3/MOSS2007 developer, getting the chance to see a couple of MVPs (Andrew Connell and Rob Bogue) introduce the basics is great review and a chance to check your skills.

Date (all times EDT) Topic & Registration URL Presenter
Tues, May 20 : 12-1p Web Parts Rob Bogue
Wed, May 21 : 12-1p Data Lists Rob Bogue
Tues, May 27 : 12-1p Silverlight Andrew Connell
Wed, May 28 : 12-1p Event Handlers Andrew Connell
Tues, June 3 : 12-1p Site Branding Andrew Connell
Wed, June 4 : 12-1p Workflow Rob Bogue
Tues, June 10 : 12-1p Web Services Andrew Connell
Wed, June 11 : 12-1p Page Navigation Andrew Connell
Tues, June 17 : 12-1p User Management Rob Bogue
Wed, June 18 : 12-1p Content Types Rob Bogue

If you have any interest in SharePoint development, then you don’t want to miss these webcasts.

Creating Solution Packages for WSS 3.0 and MOSS 2007 Deployments

A couple of weeks ago I posted an apology for not following through and posting anything about SharePoint solutions and features like I had promised almost a year ago. This post is a start on coming through with that promise. The only thing that is different from what I had promised in that post is that I will not focus on using Visual Studio 2005 or any other IDE. Rather, my approach will be to help you learn the essentials so that you don’t have to rely on a specific version of an IDE to accomplish the task.

Today in this post, I am going to explain why solution packages are important, what makes up a solution package, how the solution deployment works, and how to create them to effectively deploy solutions and features to a WSS 3.0 or MOSS 2007 farm.

Importance of Solution Packages

SharePoint deployments vary in size from a stand-alone installation on a single server to large multi-server farms. A solution or feature may require configuration changes to the web application, placing files on the file system, and/or deploying assemblies. As such, manual deployments can be administratively difficult since the possibility of deployment error increases with the number of servers in the farm as features and solutions are moved from a development environment to a test environment and finally to a production environment. In order to reduce deployment complexity and possibility of error, SharePoint uses solution packages a deployment mechanism to ensure consistent deployment of solutions and features to all servers in the farm. This allows the testing of a deployment and enables system administrators to create scripted installations (a typical requirement in most enterprise-class IT organizations).

Anatomy of a Solution Package

A solution package is a compressed CAB file with a WSP (.wsp) extension that contains the contents to be deployed including its required additional meta data.

The contents inside the WSP file may include, but is not limited to, assemblies, as?x files, feature manifests, schema XML files, site definitions, WEBPART files, DWP files, InfoPath forms, resource files, images, javascript files, other media files, html files, XML files and the solution manifest. In other words, it can contain just about any type of file you can think of. I may go into details of some of the SharePoint specific content files in some future post.

Every solution package requires additional meta-data to provide the WSS run time with instructions on what to do with the content of the WSP file. The meta-data is stored in a manifest file in XML that must be located in the root of the WSP file; the file is appropriately named manifest.xml. Perhaps I will dive into the details of the Solution schema that defines the XML structure of the solution manifest in another post in the future.

How Solution Packages Work

Basically, they just do.

But in all seriousness, the deployment of a solution package happens in two stages. In the first stage, or the installation stage, WSS takes the WSP file and copies it to the configuration database. In the second stage, or the deployment stage, WSS takes the WSP file from the configuration database, extracts the content(s), and places the content(s) as specified in the package meta-data. Simple huh? Obviously, a system administrator needs to tell WSS to install and deploy a package. This is easily accomplished by executing three simple STSADM utility commands.

stsadm -o addsolution -filename my.wsp
stsadm -o execadmsvcjobs
stsadm -o deploysolution -name my.wsp

The addsolution and deploysolution operations both take additional operational parameters that may be needed depending on the needs of actual solution to be deployed. It should be noted that the  STSADM utility commands only need to be executed on the server that is hosting the Central Administration application. Let’s look at what each command actually does:

  1. The first line (stsadm -o addsolution -filename my.wsp)creates a timer job that asynchronously performs the installation stage of the solution deployment. When the job executes, WSS takes the specified WSP file and copies it to the configuration database.
  2. The second line (stsadm -o execadmsvcjobs) forces WSS to execute any timer jobs that are pending. This is necessary because the actual deployment stage cannot commence until the installation stage has been completed successfully.
  3. The last line (stsadm -o deploysolution -name my.wsp) creates another timer job that asynchronously performs the actual deployment stage of the solution deployment. When the timer job executes, WSS gets the WSP file from the configuration database and deploys the contents of the package based on the instructions defined in the solution package meta-data. It should be noted that the WSS service on each server in the farm performs this task at the same time as triggered by the timer job ensuring a consistent and timely deployment on all servers in the farm.

Another great thing about the solution deployment mechanism is that it also simplifies the removal of a solution from a farm. Solution removal also happens in two stages. In the first stage, or retraction stage, WSS uses the solution package meta-data to remove the contents from each server in the farms. In the second and final stage, or deletion stage, WSS deletes the copy of WSP file from the configuration database. A system administrator can accomplish this easily be executing three STSADM utility commands on the server hosting the Central Administration application.

stsadm -o retractsolution -name my.wsp
stsadm -o execadmsvcjobs
stsadm -o deletesolution -name my.wsp

The retractsolution and deletesolution operations both take additional operational parameters that may be needed depending on the needs of actual solution to be removed. Like we did with the deployment commands, let’s now review what each command actually does:

  1. The first line (stsadm -o retractsolution -name my.wsp) creates a timer job that asynchronously tells the WSS service on each server in the farm to go get the WSP file from the configuration database, take a look at the solution package meta-data, and retract any features, files, and/or assemblies that were previously deployed. This command initiates the retraction stage of the removal process.
  2. The second line (stsadm -o execadmsvcjobs) forces WSS to execute any timer jobs that are pending. This is necessary because the actual deletion stage cannot commence until the retraction stage has been completed successfully.
  3. The last line (stsadm -o deletesolution -name my.wsp) creates another timer job that asynchronously performs the deletion stage of the solution removal. This job tells the WSS run-time to delete the copy of WSP file from the configuration database.

And that is how solution packages work.

Creating a Solution Package

While SharePoint elegantly utilizes Microsoft’s most advanced bleeding edge technology (such as SQL Server, the .NET 3.5 framework, etc.), the actual process of creating a WSP file is ironically a bit archaic. If you are using Visual Studio 2005 for development, then the Visual Studio 2005 extensions for WSS 3.0 can jump start the process of creating the WSP file (but it won’t save you as much time as I expected it to when I first used the extensions). The process of creating a WSP file is the same as creating a  CAB file using the MAKECAB operating system utility. The MAKECAB utility creates the WSP file by reading and processing the cabinet definition contained in a DDF file, also known as the Data Description File. Creating a DDF file is not difficult and can be done with any plain text editor, but the maintenance of the DDF file is manual (this is why I called the process a bit archaic) and become quite tedious with large solutions.In any case, let’s build a sample solution package.

The first step is to create the data description file (DDF) that describes the files for the CAB archive. You can use any text editor to create the file (download the sample DDF file).

; ** my.wsp (Alonso Robles) **
.OPTION EXPLICIT                   ; Generate Errors
.Set CabinetNameTemplate=my.wsp
.Set DiskDirectoryTemplate=CDROM   ; All cabinets go in a single directory
.Set CompressionType=MSZIP         ; All files are compressed in cabinet files
.Set UniqueFiles="ON"
.Set Cabitet=on
.Set DiskDirectory1=Package
; All solution packages require additional metadata
manifest.xml   manifest.xml
; Deployment files, Schemas, and Assemblies
my.html        LAYOUTS/mysample/my.html

There are lot of things going on in the DDF file, but here is what I think is important to understand.

  • Comments in the file begin with a semi-colon
  • .OPTION lines tell the MAKECAB utility what options to use when building the CAB archive
  • .Set lines tell the MAKECAB utility what values to apply to certain parameters that are used when building the CAB archive
    • Make sure the CabinetNameTemplate parameter is set to what the name of the WSP file will be (in the example above the MAKECAB utility will generate a CAB archive named my.wsp)
  • Each content file to be added to the CAB archive is listed on a line in a space separated source-destination format
    • For example, this CAB archive will contain an HTML file (my.html); the last line in the example above tells the MAKECAB utility to grab the my.html file from the file system (the source location) and place copy it to LAYOUTS/mysample/my.html in the CAB archive (the destination location)
    • Double quotes can be used to define sources that contain spaces in the path or file name
    • Make sure to include the solution meta-data XML file (manifest.xml) at the root of the CAB archive

The next step is to create the solution manifest file (download the sample solution manifest file).

<Solution SolutionId="01234567-89AB-CDEF-0123-456789ABCDEF"
	 xmlns="http://schemas.microsoft.com/sharepoint/">
  <TemplateFiles>
    <TemplateFile Location="LAYOUTS/mysample/my.html" />
  </TemplateFiles>
</Solution> 

The example is a very simple solution manifest, but there are a couple of things to take away from it.

  • Every solution has a GUID (globally unique identifier) that identifies uniquely
  • As a general rule, every deployment file, schema, and/or assembly contained in the CAB archive must also be listed in the solution manifest or another manifest listed in the manifest.xmlfile; otherwise it will be ignored by the WSS runtime
  • The locations of the deployment file, schema, and/or assembly in the manifest refer to the location in the CAB archive

The last step is to use the MAKECAB utility to generate the solution package – the actual WSP file (you might want to download the sample HTML file if you want to try to create the sample solution package yourself). This is done by using a command window and executing a simple command:

makecab /f my.ddf

And that’s how a solution package (WSP file) is created.

I hope you enjoyed this post. Now I am not making any promises that I might have to apologize for, but I hope to post about other topics such as the solution and/or feature manifests or other fun SharePoint topics in the future.

References & Additional Linkage

Avoiding SharePoint Project Failure

Joel Oleson wrote an interesting post sharing his insights on SharePoint project failures inspired by a five part post written by Paul Culmsee in cleverworkarounds.com. Joel’s insights point out that SharePoint projects are destined for failure when:

  1. Developers are assumed to be the administrators,
  2. SharePoint was selected as a solution before the problem was defined,
  3. The deployment is not planned,
  4. Budget is not allocated, and/or
  5. Those who will be managing it have not played with it.

After reading Joel’s post, I took the time to read the Why SharePoint Projects Fail (Part 1, 2, 3, 4, and 5) posts by Paul Culmsee. The posts are a great read for project managers, technical leads, and sales folks who are working on (or will be working on) a SharePoint project. Paul focuses his discussion on the inherent problems found in project planning, failure to identify the problem to be remedied, mixed with the expectations of SharePoint being a solution to all problems, and underestimating the complexity of SharePoint as a product.

In my opinion, all of the points raised by Joel and Paul are valid and I have a few insights to add to the list.

Academically speaking, the success of a software project is a result of good risk management, mitigation, and contingency planning. Having a clearly defined and well accepted problem statement is a key element that allows us to identify the risks involved in order to manage them. Paul’s history lesson on the wicked problem is an important lesson to keep in mind when starting any software project.

All academics aside, the SharePoint product is a powerful, robust, complex, and versatile platform that can be used as architect a soution to solve many different business problems. However, those same power, robustness, complex, and versatile features make SharePoint a double edge sword. Just as it can be used to solve many problems, it can create many problems. In terms of project management, those problems are potential risks that may result in project failure.

My advice to help avoid SharePoint Project failure is simple and follows below.

Never identify the solution before the problem. Understanding the problem is essential to project success. Clear identification of the business problem will help stakeholders decide whether or not SharePoint is the right tool (or one of the tools needed) to do the job.

If SharePoint is being considered for the solution, then get an experts opinion. Chances are that most stakeholders and project managers have only heard about the SharePoint buzz. Let alone used SharePoint at all. It’s very possible that the technical expert on the project team has only limited knowledge about SharePoint. As such, it is essential that an expert is consulted to help the stakeholders decide whether SharePoint is the right tool for the job. As I pointed out earlier, SharePoint is a great platform for creating many solutions to business problems, but it is not the holy grail. Consulting an expert will also allow the project team to have better resource and budget estimates needed to complete the project at hand and help identify potential risks that will need to be mitigated.

If SharePoint is selected as the platform, ensure the project team has adequate training and don’t be afraid to ask an expert for help. Once SharePoint has been selected, make the investment and get the team properly trained. Be sure to consider having a SharePoint expert available to help with the project. The upfront training investment and having a SharePoint expert on the team will help with risk identification and mitigation throughout the project. In addition, familiarity with SharePoint features will result in cost savings by keeping the costs of custom development down.

On SharePoint Solutions and Features

So back in July of last year (2007), I promised to post some tutorials and walk-throughs about creating SharePoint solution packages (also known as WSP files) using some extensions for Visual Studio 2005. Almost a year later, I haven’t done it. This is because I everytime I started working on a tutorial I found that there are so many options and decisions that need to be made to create a proper package. I still plan on posting about it soon, however, I think I will start with some simpler tutorials to explain how to organize and scope the features that are contained in the solution. In any case, I just wanted to let everyone know that I haven’t forgotten about my promise and why it has taken so long to get around to it.

Microsoft Certified Technology Specialist: Microsoft Office SharePoint Server 2007 – Configuration

I forgot to post about this, but last February I took and passed the MOSS 2007 Configuration exam and became a Microsoft Certified Technology Specialist.

MCTS MOSS 2007 Configuration