Survey Results: Consultant Selection Practices of Public Agencies

I recently conducted an online survey of public agencies across the nation on some of their consultant selection practices.  The survey was open from August 18, 2011 and August 26, 2011.  During that time, there were a total of 61 responses from 47 public agencies in nine states plus the District of Columbia and Canada.

Summary of Results:  While practices varied by agency as reflected in the answers, most agencies have evaluation committee members read and rate proposals prior to the evaluation committee actually meeting.  At the evaluation committee meeting, the members discuss their scores and reasoning, and based on those consensus developing conversations, individuals may modify their scores.  These modified scores are then rolled up into a committee total reflecting the basis of selection.  Most agencies keep the scores and comments of the evaluators and committee as a matter of public record, only disclosing the information after an award has been made.  The names of the evaluators are typically included on the score sheets and then subject to disclosure upon request.

Survey Results:  The following are the results from the survey's nine questions:

1.  Evaluation Committees: Do your agency's evaluation committees meet in person as a group?

• 89% - Yes
• 3%  -  No
• 8%  -  Sometimes

2.  When Are Proposals Scored:  When do members of of the evaluation committee assign their scores to proposals?

• 57% - Before the committee meeting
• 36% - At the committee meeting for the first time
• 7%  -  Other

3.  Purpose of Evaluation Committee Meeting:  What is the purpose of the evaluation committee meeting together as a group?

• 68% - Develop consensus scores for each criterion based on average of individual committee member scores that were rated before the committee meeting, and possibly modified during the meeting.
• 10% - Develop one committee score for each firm, but not developed from average of scores from individual committee members.
• 5%  -  Develop overall group consensus on top rated firm(s) without using or documenting any scores, but based on what firms seem to be best for the agency.
• 2%  -  The evaluation committee does not ever meet as a group, but relies on individual scores to determine highest ranked firm.
• 2%  -  Develop overall group consensus (no score) for selection where individual committee members have not rated proposals prior to the meeting.
• 10% - Other

4.  Basis of Consensus Scores:  If the evaluation committee develops consensus scores, what is it based on?

• 64% - Each evaluation committee member developing or modifying their score based on the conversation of the committee and rolling those numbers into a consensus score.
• 19% - We don't develop consensus scores.
• 14% - One group consensus score for each firm.
• 5%  -  Other

5.  Scores as a Matter of Public Record:  What does your agency do with scores and written comments from evaluation committee members (including those developed before and during the evaluation committee meetings)?

• 73% - We keep the scores and comments as part of the public record.
• 6%  -  We keep the scores and comments as part of the public records, but mark them as draft documents.
• 5%  -  Individual committee members don't write down scores and comments since it's all developed in the actual committee meeting.
• 3%  -  We shred the scores and comments.
• 11% - Other

6.  Disclosing Individual Scores:  When does your agency release written comments and scores of individual committee members to the public?

• 29% - Only after award is made.
• 26% - Only after the contract is executed.
• 21% - Anytime, upon request.
• 10% - We never release these documents to the public.
• 7%   - We don't have individual scores from committee members.
• 2%   -  Before the contract is executed.
• 0%   -  Before an award is made.
• 0%   -  We shred the scores and comments
• 5%   -  Other

7.  Disclosing Committee Scores:  When does your agency release the selection record from the full evaluation committee?

• 39% - Only after award is made.
• 26% - Only after the contract is executed.
• 15% - Anytime, upon request.
• 10% - Before an award is made.
• 3%   - Before the contract is executed
• 0%   - We never release these documents to the public.
• 0%   - We shred these documents and thus never release them to the public.
• 7%   - Other

8.  Inappropriate Written Comments:  How does your agency handle comments on an individual evaluator's score sheet that includes inappropriate comments about a firm?

• 48% - We don't have a problem with evaluators writing down inappropriate comments.
• 22% - We let the record speak for itself and don't take any action.
• 15% - Our score sheets don't include a place for comments.
• 12% - We ask the individual to change the comments to be more appropriate and professional.
• 3%  -  We shred all score sheets of individual evaluators so this isn't a problem for us.

9.  Disclosing Evaluator Names:  How does your agency handle the disclosure of the names of evaluation committee members?

• 40% - Disclose these names on the score sheets.
• 23% - Disclose these names anytime upon request.
• 23% - Never disclose these names.  We use letters or numbers on the score sheets to designate the names of the evaluators.
• 0%  -  Disclose these names in the RFP or RFQ.
• 13% - Other

Note:  Percentages don't always add up to 100% due to rounding.  Not all respondents answered all questions.

Mike Purdy's Public Contracting Blog 

© 2011 by Michael E. Purdy Associates, LLC 

http://PublicContracting.blogspot.com