Page 1
Evaluation Tool Summary:
Client/Customer Satisfaction Survey
by Ken Smith and John Tull
1. Tool Description
A self-administered card or form is given to the person receiving assistance, to fill out at the conclusion of
service by the project. Each person who receives more than quick directions or a brochure is asked to
complete a short questionnaire indicating assessment of the services received and suggestions for
improvement.
Toolkit:
Models and Elements of This Tool
a.
Type of Legal
Aid Project
b.
Benchmark Model/
Champion
c.
Instruments
d. Tool Elements
(1)
Sample
Output
(2)
Method
Description
(3)
Evaluation
Report
Pro per
assistance
project
Andrea Agloro, Exec. Director
Self Help Assistance Center
(SHAC), Sonoma County Legal Aid
Santa Rosa, CA
Exhibit 1* Exhibit 3 Tab 13*** Tab 13***
Pro per
assistance
project
James Mensing
Center for Family, Children & the
Courts
Administrative Office of the Courts
San Francisco, CA
Exhibit 2**
Web- or Kiosk-
based self-help
project
Bob Cohen, Exec. Director
I-CAN! Project
Legal Aid Society of Orange County
Santa Ana, CA
Exhibit 4
General Civil
Legal Aid
Program
Bob Oakley, Exec. Director
Northwestern (PA) Legal Services
Erie, PA
Exhibit 5
* This is a brief exit survey, with four questions, that was used effectively for management purposes.
**
Model instrument containing a more extensive list of questions that can be adapted to most types of legal
services projects.
***
For copies of the SHAC evaluation report and method description, see the hard-copy Toolkit at Tab 13 or the
Legal Aid Association of California web site at www.pic.org.
1
Information of this type can help a program determine whether its services are reaching a particular
client group of interest, or whether they are being utilized by a reasonable cross-section of their client
base.
Page 2
2. What The Tool Measures
Any or all of the following:
• Background information about the customer
1
• Customer’s familiarity with the legal system
• How customer found out about the service
• Customer overall satisfaction
• Customer perceptions of specific aspects of service, such as helpfulness of the staff,
accessibility, wait time or responsiveness to specific concerns
• Customer suggestions for improving the service
3. Comment
Many legal services programs and projects use a tool of this type to measure client satisfaction. The
instrument is mailed or handed to the client at the conclusion of service. Response rates are typically low -- 10
to 30 percent -- for mail-back surveys. In court-based pro per projects, response rates can be higher when
clients are asked to complete the instrument before leaving the premises.
One method of increasing the response rate is to use a sampling approach rather than to give every customer
the survey. A few surveys filled out sporadically do not provide valid information. It is better to get responses
by all users of the service, for a limited period of time - perhaps one or two weeks. If this is done three or
four times a year, it should provide sufficient information for analysis while not overburdening the staff or
interfering with customer service.
Caution in using this tool is essential to address privacy and confidentiality concerns. Respondent
privacy is always a factor to be built into a survey. If the project has a lawyer-client relationship with its
customers, or deals with sensitive matters, then safeguards for client confidentiality must be provided. For
some types of projects -- for example, a domestic violence project -- adequate safeguards for the safety and
privacy of the respondent must be in place before going forward with attempts to contact them.
4. Project Types
• General or specialized civil legal aid projects
• Courthouse-based pro per projects
• Telephone legal hotlines
5. Guidelines for Using This Tool
Please see next page.
Page 1
Guidelines for Using the Customer Exit Survey
by John Tull
Edited by Ken Smith
General considerations. This tool is relatively easy to administer and can readily be tailored to obtain
data about different aspects of a project or its operation. The data obtained is relatively easy to tabulate
and analyze. At the same time, the tool has limitations associated with typically low return rates and the
fact that it generally solicits respondents' subjective perceptions rather than a professional’s perspective
about quality and effectiveness of the services provided. Because the questionnaire is self-administered,
it is limited in the number of questions that can be asked and the literacy level required of the respondent.
Caution in using this tool is essential to address privacy and confidentiality concerns. Respondent
privacy is always a factor to be built into a survey. If the project has a lawyer-client relationship with its
customers, or deals with sensitive matters, then safeguards for client confidentiality must be provided. For
some types of projects -- for example, a domestic violence project -- adequate safeguards for the safety
and privacy of the respondent must be in place before going forward with attempts to contact them. For
further information, see the ethical guidelines for human subject research at Tab 14 of the hard-copy
Toolkit or the Legal Aid Association of California web site at www.pic.org.
Potential uses. The tool is often used to obtain data about the degree to which clients were satisfied
with their interaction with personnel in a program ("Did the advocate representing you keep you
adequately informed about the progress of your case?) It can also seek data about client's reactions to
specific aspects of a project or a process within it. ("Were you treated courteously during the intake
process?" “Was the fact that you were being referred to a private attorney thoroughly explained to
you?")
It can also measure people's experience with processes external to the program that might affect the
outcome of their case. ("Did you feel that you were given an opportunity to present your case in the
hearing in which you represented yourself?").
The following table presents examples of purposes an exit survey might serve and gives examples of
questions that might be asked.
Research or Management Purpose Examples of Survey questions
• Measure overall satisfaction • How do you rate the service you were provided?
• Measure reactions to the intake process
and ease of access to the program
• Was it easy to apply for legal representation at the
program? Was the length of time you had to wait to
see someone about your problem acceptable?
• Measure clients’ awareness what is
happening at various steps of a process
within a project
• Were you informed that you needed to return to the
clinic for further help before actually going to court?
Was the fact that your case was being transferred to
a private attorney adequately explained to you?
• Assess how well clients are kept
informed of the progress in their case
• Were you kept well informed by the person
representing you about what was happening in your
case?
Research or Management Purpose Examples of Survey questions
1
Such an inquiry would probably need to be anonymous to assure the responses are not skewed by
concern on the respondents' part that the response might affect whether their case will be accepted.
Page 2
• Measure satisfaction with the outcome in
their case (not necessarily a measure of
quality or if the best outcome possible
was obtained)
• Were you satisfied with the results that you got in
your case?
• Correlate satisfaction among groups of
clients distinguished by culture,
language, geographic location or other
significant factors
• A number of inquiries are possible, but the survey
instrument needs to collect the demographic data
necessary for making the desired comparisons.
•
Guage clients’ perceptions of the roles
of key partners or agencies, particularly
where such information might affect how
the evaluating organization provides
assistance
•
Did you feel that the judge listened to you when you
were representing yourself in court?
It is important to be clear about the specific purpose of the survey, then to limit it to key questions that will
address that purpose. Experience teaches that client satisfaction surveys will not be filled out and
returned if they are long. There is a temptation to ask more questions than are necessary and
inadvertently discourage responses. The model instrument in this tool kit (see Exhibit 2) can easily be
altered to include only questions that are immediately germane to the program conducting the survey.
If the principal use of a survey is to develop data for reports that will be used to market a project to
others, including potential new partners and funders, then broader questions about satisfaction with the
results of the assistance and if the user would recommend the service to others would be appropriate. If,
on the other hand, the principal purpose is to provide information to management regarding appropriate
changes to improve performance of a project, then more focused questions about the specific aspects of
the program would be in order.
Administration of the survey. The greatest challenge with an end-of-service survey is getting an
adequate rate of return to support meaningful, valid conclusions. There are several way to distribute a
survey to potential respondents.
1. Administer while the respondents are still physically present at the service location. This
method has the highest likelihood of a high return rate. For example, if a program desires
feedback on its intake process, it can ask applicants to fill out a short survey at the end of the
process.
1
2. Mail to potential respondents with a close-out letter. Mailed surveys are appropriate when
on-site distribution and collection of survey instruments is not practical and large numbers of
people are being surveyed. Mailed surveys may be appropriate, for instance, at the close of a
case involving extended representation, or after a pro se client has gone to court.
Techniques to increase responses include:
2
Telephonic surveys have several advantages. They allow for follow-up questions to clarify
responses and explore the reasoning behind an answer. They also allow a survey of persons who are not
literate or who speak a language that is not common or is not written.
3
A full description of instruments that have been developed for evaluation of websites can be found
on the “LSTech” web site at www.lstech.org/TIG/eval/web_client_tools.html.
4
An evaluation consultant or a survey group at a local college should be able to provide useful advice
about how to select a statistically valid sample and how big the sample needs to be for achieving the
purpose of the survey.
Page 3
3. Attaching a stamped, self-addressed envelope.
4. Making follow-up phone calls to persons who have not responded.
5. Using a sampling approach rather than surveying all clients. For example, staff could
administer the survey to all clients served during a specific week, and follow up aggressively
with clients who do not respond.
6. Using a telephone interview. Such surveys are appropriate when the population surveyed is
relatively small; the evaluation involves a short, discrete time period or when the purpose of the
survey is to gather information about specific aspects of a project's operation.
2
7. Using technology. When the vehicle for providing service is technology, such as a website or
a kiosk, the survey instrument can be administered on-line or through the technology, itself. If
the identity and address of the user is known, as it may be with technology that supports the
development of self-help pleadings and the like, then direct contact may be possible for a
follow-up written or oral survey. Users of websites, however, will likely be anonymous and
difficult or impossible to track down.
3
Number of customers to be surveyed. A client satisfaction survey typically is administered to all
clients who are served by the project, or to a random sample of clients served. The size of sample that is
appropriate depends on a number of factors; for example, a survey that is designed to test the difference
in responses among various populations based on language, culture, age, geographic location or some
other significant factor would need to have a sufficient sample of each population group to provide a valid
measure of those differences.
4
Language. This is an important consideration when a project serves people whose primary or preferred
language is other than English. In this case, surveys should be administered in the predominant languages
of the population that is being surveyed. If there are multiple languages in a service area, providing
surveys in all the languages may not be practical. Some client populations will not be literate in any
language and some languages may not be written. In such cases, surveys can be administered by phone
on a sampling basis.
Limitations and benefits of the tool. An exit survey can be useful, but it is important to recognize its
limitations. For example, if the rate of response is low, there is no way to know whether respondents
represent a cross section of persons using the assistance. There is always a risk that a higher percentage
Page 4
of people with strong negative or positive feelings have responded than those whose reactions are
indifferent. It is also possible that responses are skewed in favor of persons who are better educated,
more comfortable in English, have more time (e.g., the elderly) or simply are more confident.
It is not unusual for client satisfaction surveys in legal services to show a high level of satisfaction and low
levels of variation among respondents. For example, most surveys show 95-99 percent of respondents
are “satisfied” or “highly satisfied.” While 98 percent satisfaction makes a good story for public relations
purposes, it doesn’t necessarily mean the clients are getting great service. Respondents are often so
grateful to be getting any help at all in the stressful situation they are in that they are not inclined to be
critical. Recognizing this, some managers follow up with every client who indicates less than high
satisfaction, to find out more about the reasons underlying this assessment. Because of the low numbers
of clients in this category, this can be a practical step and offers a very powerful method of learning about
quality-of-service issues from a client’s perspective.
Moreover, asking clients about their experience with the project provides benefits beyond simply getting
data for research or quality control purposes. It tells clients that their perspective matters. It tells funders
that the project is committed to providing good services. From this perspective, the client satisfaction
survey is more than a useful tool -- it is an essential tool for having running an effective legal services
project.
Exhibit 1:
Customer Exit Survey Questions
From Sonoma County Legal Aid
Self-Help Access Center Evaluation
Four questions were asked:
1. How do you rate the service you received today?
1=poor
2=satisfactory
3=good
4=very good
5=excellent
2. Were you able to get info or assistance that will help you solve your problem?
1=Not at all
2=Somewhat
3=Definitely
3. What type of problem did you need help with? (Check one)
___ Family
___ Housing
___ Other
4. What could the SHAC staff do to be more helpful?
Exhibit 2
Self-Help Center Evaluation
Model Customer Satisfaction Survey
Demographic or Background Questions
1. What is your ethnic or cultural background?
European Latino African-American Asian Native American
Other I’d rather not say
2. What language do you most often speak at home?
English Spanish Vietnamese Mandarin Cantonese
Other
3. What is the highest grade you completed in school?
0 – 5 6 – 8 9 – 11 High School Graduate/GED
Some College College Graduate Post Graduate Study
4. How old are you? __________
5. Have you ever spoken to an attorney before about a legal matter you had to deal with?
Yes No
6. Have you ever had to deal with a legal matter before without professional help?
Yes No
7. How did you hear about our Center?
Court Personnel Private Attorney Radio/Newspaper/Flyer
District Attorney Friend or Relative Used Center Before Saw a Sign
Another Government or Social Service Agency
A Church or Private Assistance Organization Other
General Satisfaction Questions
1. Overall, did you find the Center helpful?
Definitely Somewhat Not Sure Not really Not at all
2. Overall, did the staff seem to know what they were doing?
Definitely Somewhat Not Sure Not really Not at all
3. If you had another legal problem, would you come back to this Center, or recommend it to a
friend or relative?
Yes Probably Not Sure Probably Not No
4. How long did you have to wait for service?
Less than 5 minutes 5 to 10 minutes 10 to 20 minutes
20 to 30 minutes more than 30 minutes
5. How did you feel about how long you had to wait?
Very good Okay Not sure Unhappy Angry
6. Were the staff respectful and courteous to you?
Definitely Somewhat Not Sure Not really Not at all
7. In your opinion, were you able to explain your problem completely to a person you felt was
listening to you?
Definitely Somewhat Not Sure Not really Not at all
8. After explaining your situation, were you given information that helped you understand your
situation better?
Definitely Somewhat Not Sure Not really Not at all
9. Were you given advice about what to do next?
Yes No
10. Do you have a clear idea about what to do next?
Definitely Somewhat Not Sure Not really Not at all
11. Do you feel you will be able to take the next steps needed on your case (such as fill out and
file forms, or present your case to a judge)?
Definitely Somewhat Not Sure Not really Not at all
12. Do you have any suggestions for improving the service?
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
Page 11
Exhibit 3:
Sample Output
Excerpt from Self-Help Access Center evaluation report
Sonoma County Legal Aid,, Appendix A, page 11
SHAC clients are highly satisfied with the assistance they receive.
Clients of the Center were given
an “exit survey” to fill out
indicating their satisfaction with
the services they’d received.
Clients were uniformly
appreciative of the service they’d
received. When asked, “What
suggestions do you have for
improving the Center?” most said,
in various ways, “None. It’s
great.” Of the few who made
suggestions, most repeated the
theme heard from court clerks
and project partners: “Be open
for more hours, on more days.”
Client Satisfaction
Survey Results
Each client who received more than quick directions or a
brochure was asked to complete a short questionnaire indicating
their assessment of the service they’d received and suggestions
for improving service.
Clients gave SHAC high ratings.
Overall rating of service
..........................
4.86
(Average of ratings on 1 to 5 scale)
Clients felt they were able to get useful help.
Average of responses to question below
...............
2.31
“Were you able to get info or assistance that will
help you solve your problem?”
(1=”Not at all;” 2=”Somewhat”; 3=”Definitely.”)
The vast majority expressed appreciation for the assistance
they received. The following comments were typical:
! “There was nothing you could have done to be more
helpful.”
! “I think this is a great thing and will be an asset to
people.”
! “I got all the help I need and more.”
! “The Center is fine just the way it is. I don’t know
what I would have done without this help.”
Exhibit 4Exhibit 4::
Questions for I-CAN! Survey Module
January 24, 2003
1. What is your home zip code?
2. Was I-CAN! easy to use?
1. Very easy
2. Easy
3. Hard
4. Very Hard
3. Was I-CAN! helpful?
1. Very helpful
2. Somewhat helpful
3. Not very helpful
4. Not helpful at all
4. Where did you use I-CAN?
a. At home
b. At work
c. In a library
d. In the court house
5. Other
5. Was this location convenient for you?
a. Very convenient
b. Somewhat convenient
c. Not very convenient
d. Very inconvenient
6. Did you receive the Earned Income Tax Credit LAST year?
a. Yes
b. No
c. Don’t know or don’t remember
If no, why not?
(1) Applied, but did not qualify for it.
(2) Did not know about it.
(3) Too much paperwork.
(4) Couldn’t afford to have someone do my taxes for me.
(5) Don’t know or don’t remember
(6) Other
7. How did you complete your tax return last year?
a. Filled out the tax forms myself and mailed them in.
b. Had help from a friend
c. Had help from a tax preparation service [if selected, branch to question 8 after they
choose from the comparison below]
d Other
How did using I-CAN! compare to the way you did your tax return last year?
2. Much easier
3. Somewhat easier
4. About the same
5. Somewhat harder
6. Much harder
7. Did not file a tax return last year
8. Don’t know or don’t remember
8. If “Yes, did you receive an “instant refund” type of loan where the loan is repaid out of the tax
refund from the IRS?
a. No
b. Yes
c. Don’t remember or do not know
9. Would you use I-CAN! again?
a. No - check all that apply:
(1) Too hard to use
(2) Too far from home or too inconvenient
(3) Takes too long
(4) Not enough help available
(5) Want someone to help me in person
(6) Don’t like or don’t feel comfortable using a computer
(7) Other
b. Yes - check all that apply:
(1) Makes the application process easier
(2) Saves money
(3) Saves time
(4) Convenient locations
(5) The ability to file my tax return by computer rather than mailing it.
(6) Other
11. How did you hear about I-CAN?
9. Radio
10. TV
11. Newspaper
12. Legal aid office suggested I try it.
13. Library staff suggested I try it
14. Other
11. What information can you give us to help us make I-CAN! better?
12. Can we call you to ask a few questions to help us make I-CAN! better? (Any information you
give us will be confidential and for I-CAN!’s use only.) If so, what is the best number and time
to reach you?
Very Satisfied
Somewhat Satisfied
Somewhat Dissatisfied
Very Dissatisfied
Don’t Know
CLIENT SATISFACTION QUESTIONNAIRE
NAME:_______________________________________________________(Optional)
NAME OF YOUR LAW WORKER:_____________________________
We are interested in your experience with our program. In order to
help us improve client services, please take a few minutes to
complete the following survey. Please mark the appropriate box
which best describes your opinion.
Overall, would you say you were satisfied or dissatisfied with:
1. The way in which you applied for our services
2.
Time period in which you had to wait for an appointment with
the law worker
3. Preparedness and thoroughness of law worker
4.
The way your law worker explained your case and the legal
options available to you
5. Degree of respect and/or way in which you were treated
Were you satisfied with the outcome or result of your case? “Yes “No
Please Explain:
Did you feel comfortable with the representation which was provided? “Yes “No
Please Explain:
How quickly were your telephone calls usually returned by the law worker?
“ Same day “ Next day “ Couple days “ Almost never
If you need help again, would you prefer:
“ Same law worker “ Different law worker “ No preference
Are there any changes you would like to see made in the way services are provided?
Additional Comments:
Do you wish a follow-up call from the Executive Director? “Yes “ No If so, Phone #:
Please Return to: Attention: Executive Director
Northwestern Legal Services
1001 State Street, Suite 1200
Erie, PA 16501
Exhibit 5 (from Northwestern Legal Services, PA):
