1. Where can I find surveys that are
valid for our organization?
2. Is it more useful to use customized
surveys or standardized surveys?
3. What does an organization have to
do to develop its own customized 360-degree feedback survey?
4. What’s the best way to research
survey items?
5. How long should a survey be?
6. How do you validate a locally
researched survey?
7. What are the attributes of a
well-constructed survey item?
1. Where can I find surveys that
are valid for our organization?
When it comes to people in the workplace, one size doesn't
fit all. Most managers want to give their employees feedback,
and they want it to be appropriate to the realities of their
organization. Yet, it isn't easy to research, develop and validate
a survey from scratch. Most organizations don't have the internal
resources to do so, and hiring a consultant can be an expensive
project.
Off-the-shelf surveys
Off-the-shelf surveys may seem convenient, but not all are well-constructed,
even surveys associated with celebrity authors. And not all well-researched,
well-constructed surveys are likely to be valid for everyone in your organization.
Many of these focus on executive leadership, though few subjects are likely
to be executive leaders. Even if the scope of the survey matches a person’s
role, every organization is remarkably different—one size does not
fit all. Furthermore, most standardized surveys are lengthy; they take too
much time to complete. In almost every case, some customization will be needed
for the survey to be useful and valid for the specialized industry, culture
and needs of a given organization.
Customizing an off-the-shelf survey
Most organizations solve this problem by customizing a standard survey. They
obtain a well-researched, well-constructed survey. Then they add, eliminate
or revise items until the survey accurately describes behavior within the
organization. The procedures for researching and validating a customized
survey are well within the capabilities of most organizations. Basically,
knowledgeable managers and stakeholders review and revise the customized
survey until they can agree that the most important aspects of a person's
role are being assessed.
Access to customizable surveys
Where can you find excellent surveys that can be tailored to suit local conditions?
Unfortunately, not all 360 instruments permit customization. Many require
that they be administered as is; no customization is authorized. Others can
be customized, but only by the publisher and at significant expense.
Performance Support Systems publishes and uses the 20/20 Insight
GOLD feedback software, which was designed with survey customization
in mind. The program has built-in functions for easy customization
of survey items, scales, rater relationships, assessment media
and report formats. It also has an extensive library of more
than 1,000 well-researched well-constructed proprietary survey
items in over 140 survey categories:
• Permission to customize any survey at no charge
• Surveys for executives, manager, supervisors, team members, administrators,
instructors, salespeople and consultants
• Surveys for teams, groups or organizations
Copyright © 2004 Performance Support
Systems, Inc. All rights reserved.
2. Is it more useful to use customized
surveys or standardized surveys?
Like a tailored suit, there are advantages and disadvantages
to a customized survey. Your size and shape are unique, but
these dimensions can change over time. The suit that fits perfectly
now may not fit well later on.
Customized surveys
Customization is usually necessary to produce a locally valid survey. Customization
also increases ownership; a work unit can be involved in developing unit-specific
questions. Furthermore, today's top performance issues may change over time.
In subsequent survey administrations, the questions may not be as relevant,
and managers are likely to want to revise many of the questions. Continually
changing the questions allows you to reflect the realities of the workplace
as they change.
Standardized surveys
On the other hand, repeated customization limits
your ability to compare current scores with earlier scores.
Accordingly, an organization can standardize certain locally
customized and validated core surveys. Such a survey would
describe broadly applicable behaviors that don’t change
much over time, while allowing specificity in terms of the
type of customer (patient, distributor, taxpayer, buyer), titles,
and abbreviations. Standardization permits the compilation
of data from across the organization, which helps an organization
determine where to apply developmental resources. Also, when
a survey is kept standard over time, people can learn "how
they are doing" by seeing measures of progress.
Hybrid surveys
Many organizations prefer the hybrid approach, which are made possible by programs
like 20/20 Insight GOLD. The 20/20 Insight Survey Library stocks researched
surveys in over 100 mainstream categories. These can be customized and validated
for any segment within the organization, and other items tailored for the
organization can be added to the library. Using these modifications to the
Survey Library, core question sets can be kept constant, while other question
sets are allowed to evolve over time.
Copyright © 2004 Performance Support
Systems, Inc. All rights reserved.
3. What does an organization have
to do to develop its own customized 360-degree feedback
survey?
There are three ways to acquire a survey: use an off-the-shelf
survey, customize a standard survey, or develop your own from
scratch. Surveys that don't permit customization will probably
not reflect the specific work practices and competencies required
in your organization. If you don't find one that can be tailored
to your needs, your research can help you develop one yourself.
Whether you outsource the development of your survey or handle
it internally, the following is a classic approach.
Determine what you want to measure.
With the help of human resource professionals, managers can determine which
positions or job categories would benefit most from 360-degree feedback.
They can also help describe excellent performance: The competencies, practices,
and behaviors that exemplify the organization's values and drive the organization
toward its strategic goals. This information can be gathered from strategic
business plans, individual and group interviews, the analysis of organizational
processes, meetings with senior management and personnel documents. To simplify
review and discussion, cluster these competencies and practices into categories
like Problem Solving, Leadership, Operations Management, Customer Service,
etc.
In deciding what to measure, you can focus on behaviors and
competencies that are currently required for excellent performance,
or you can focus on the organization's future requirements.
Feedback based on current requirements helps people meet expectations;
feedback based on future requirements helps people make personal
career decisions and manage their own professional development.
You can also measure observers’ perceptions about the
relative importance of each behavior, which helps recipients
prioritize their development goals. However, some organizations
prefer that senior managers assign "importance" ratings.
Set up a steering committee.
This voluntary group includes people who will participate in the 360-degree
feedback process. They will devote up to 10 hours over several weeks reviewing
the purpose and policies of the feedback program, as well as the competencies
and practices to be measured. As a 20/20 Insight GOLD user, they can begin
by searching the Survey Library for stock survey sets, relevant portions
of which may be compiled as “straw man” draft surveys. They may
also test the draft survey and suggest revisions.
Prepare a pilot survey.
The review should consider choice of language, types of questions, and different
response scales. The language for instructions and questions must be clear
and direct. Consider the advantages of phrasing the questions in the first-person
(as a request for feedback from one person to another) rather than in the
third-person, which sounds like one person is asking another person to evaluate
a third person. Open-ended questions can increase the value of the feedback
with concrete examples and specific suggestions. However, they take more
time than numerical ratings and must be carefully worded to elicit helpful
responses. Scales that have a wide range of responses produce richer and
more useful data than abbreviated response scales. The committee's review
should result in relevant and clear survey questions and instructions that
provide valuable "actionable information" in the resulting feedback
reports.
Refine and revise the survey.
After the first 360 feedback trials, it will be helpful to ask the participant
groups to suggest improvements in the survey and the feedback process. Revised
surveys may be archived in the 20/20 Insight Survey Library for future use.
Going forward, the steering committee can oversee updating the survey based
on changes in organizational strategies, priorities and practices.
Copyright © 2004 Performance Support
Systems, Inc. All rights reserved.
4. What’s the best way to research
survey items?
Start by defining the goal of the survey this way: to give
individuals feedback about what stakeholders agree are the
most important elements of their job.
Feedback is information about behavior that tells a person
what to change to be more effective. Stakeholders are people
who care about how the individual does his or her job: incumbents,
supervisors, experts and customers. They have first-hand knowledge
of the position. If they agree that the assessment describes
the position accurately, the survey is valid. The survey doesn't
have to cover every aspect of workplace behavior. To make a
difference, the survey should focus on the most important elements
of the position.
360-degree feedback is best used when measuring aspects of
performance that are otherwise hard to quantify. A person’s
job may be divided into three main areas of performance:
• Business/technical
• Management/administrative
• Interpersonal/leadership
Whether a job has to do with construction, finance, manufacturing,
service or law enforcement, most business/technical aspects
are typically easy to quantify and measure and quantify. In
other words, it doesn't make sense to ask a lot of people for
their opinions about something that is easy to measure through
direct observation. Many resource management and administrative
tasks are also easy to quantify.
On the other hand, the interpersonal dimension of work is
extremely hard to measure. 360-degree feedback is the only
known means for getting fairly objective data about these important
performance issues. The following common areas of work are
mostly interpersonal in nature and therefore ideal for 360-degree
surveys:
• Executive leadership
• Supervisory leadership
• Team interaction
• Customer service
• Sales
• Instruction
• Administrative support
A proper research process is one that validates the survey.
By definition, the survey is valid if it measures what stakeholders
agree are the most important elements of the job. You determine
this by collecting their input to define and refine the survey.
The following procedure is used by most organizations to research
and validate locally developed surveys:
1. Query expert job incumbents, experienced supervisors
and customers
2. Consult local manuals and policies
3. Draft a list of behaviors, grouped by function
4. Confirm the list by direct on-the-job observation of subjects
5. Have stakeholders evaluate the survey using scales for importance and
relevance
6. Use their input to revise competencies
7. Conduct a pilot assessment to test the survey
8. Get feedback of constituents and revise again
The advantage of 20/20 Insight GOLD is that it can be used
to validate the survey before it’s administered to anyone.
Simply use the software to create a survey that collects input
from stakeholders using stock importance and relevance scales.
After refinement of the survey, the final version can be stored
in the 20/20 Insight Survey Library for future use.
Copyright © 2004 Performance Support
Systems, Inc. All rights reserved.
5. How long should a survey
be?
"What’s the ideal number of questions?" Because
a 360-degree assessment can collect so much important information,
you may be tempted to ask too many questions. Brevity is crucial
for four reasons:
• The quality of participant feedback diminishes after
too many questions
• Opportunity costs: time spent giving feedback is time away from other
work tasks
• The recipient can only focus effectively on one or two developmental goals
• A recipient can be overwhelmed by too much feedback
The ideal length of a survey depends on three considerations:
type and use, number of surveys to complete, and time allowed.
Type and use of survey
As a general rule, the number of questions may depend on what the feedback
is used for. If the assessment is used for individual development, the survey
should be brief—usually no more than 50 questions. Most organizations
focus on core competencies or key areas, thereby reducing the number of areas
for feedback.
Number of surveys to complete
In some "flattened" organizations, a manager may have as many as
50 direct reports, or teams may consist of more than ten members. If a respondent
has to complete several surveys, enthusiasm for the activity will decrease
with each successive attempt, reducing the value of the feedback. Limiting
the number of individuals a person rates is helpful. Feedback providers should
be limited to the people who are most familiar with the individual's performance.
Time to devote to this activity
To encourage respondent participation, the rating
process should be kept simple so that the assessment doesn’t
take more time than necessary. When assessments seek ratings
on more than one scale, it doubles the time required. With
two scales and comments after each question, 50 questions can
take more than an hour per assessment. Some organizations allow
time for ratings in the work schedule. 20/20 Insight GOLD supports
ratings via Internet, intranet or internal network, which save
time by eliminating the physical exchange of paper or diskettes.
Interactive voice response (IVR) has a reasonably high set-up
cost and is only suitable for pure rating responses (no narrative)
on short questionnaires.
The 20/20 Insight GOLD Survey Library features over 100 thoroughly
researched, comprehensive behavior sets. Organizations are
encouraged to use these resources as “start points” for
each new survey project, evaluating each item for inclusion
or deletion. In this way, the survey can focus only on top
priority behaviors and be kept brief.
Copyright © 2004 Performance Support
Systems, Inc. All rights reserved.
6. How do you validate a locally
researched survey?
No standard off-the-shelf survey can adequately describe the
realities of every organization or the various positions within
an organization. Organizations need to customize surveys so
they align with actual employee performance and the organization's
unique business practices. To be considered valid, what's being
measured should accurately describe what an individual does
at work. The survey should adequately sample the most important
observable competencies of a given job. The following guidelines
are an effective method for validating a local survey.
Assemble a Team of Subject Matter Experts (SMEs).
Identify five to seven people who understand the position. This group may include
current and former successful job incumbents and managers who are familiar
with the position. Include internal or external consultants, if involved.
Frequently, the internal HR person facilitates the process and the external
consultant contributes expertise.
Describe the responsibilities of the rated position.
Collect as much information as possible about the
job as it is presently being done. Have SMEs discuss how the
job is done on a day-to-day basis. Determine which aspects
of performance are crucial. Discuss examples of success and
failure, either real or simulated. Record all information.
Describe the knowledge, skills and abilities (KSAs)
needed to perform successfully.
Determine which are the most important KSAs for the job. Determine whether
others can observe these behaviors. The hard-to-measure behaviors become competencies
to assess with a 360-degree feedback survey. Compare these behaviors with those
found in the researched competency sets in the 20/20 Insight GOLD Survey Library.
Refine the survey to include only the most important competencies.
Revise the list of competencies.
Include other stakeholders in the final review of
a competency list (e.g., senior management, direct reports,
internal and external customers). Solicit input from qualified
professionals outside the organization—other HR professionals
or experts in technical aspects of the job. Revise the competencies
as appropriate.
Pilot test the survey in the organization.
Administer the survey to a sample group. Collect feedback about the survey
from the participants. Revise if necessary. After the survey has been reviewed
and confirmed by everyone who understands the position, it may be considered
valid for the position.
Periodically review the survey and repeat the process.
Over time, the requirements and expectations for a position can change. Review
the survey every one to two years to determine if competencies and survey
items are still valid for the position requirements.
Copyright © 2004 Performance Support
Systems, Inc. All rights reserved.
7. What are the attributes of a well-constructed
survey item?
The strength of an assessment derives from the behavioral
items being measured. These items must be carefully worded
to measure the right things. Carefully review the wording of
survey items. Ask for the opinions of the people who will be
assessed to avoid disagreements during the rating process.
General Guidelines
1. Does each item begin with an ACTION VERB?
2. Does the item describe an OBSERVABLE behavior?
3. Does the item describe ONLY ONE behavior?
4. Does the item describe an IMPORTANT behavior?
5. Is the item described as a POSITIVE, desired
behavior?
6. Is the item described in CLEAR LANGUAGE?
7. Is the item FREE OF BlAS—sexuaI,
racial or cultural?
8. Does this item, taken together with all
of the other items, SUFFICIENTLY DEFINE the category?
Evaluate every survey item for each of the eight guidelines.
This can be done before entering the items in the assessment
software.
1. Does each item begin with an ACTION
VERB?
Beginning an assessment item with an action verb is necessary
to identify the observable behavior, and doing so ensures that
items will have a uniform structure.
EXAMPLE:
Write it this way: Notifies other departments within an
hour of receiving a change of order.
NOT this way: Departments have notification of order change.
2. Does the item describe an OBSERVABLE
behavior?
Traits, characteristics, attitudes, motives, values, thoughts
and feelings are important aspects of professional people,
but these processes take place inside a person. If others cannot
directly observe a quality or characteristic, they cannot accurately
evaluate it.
EXAMPLE:
Observable item: Discusses possible costs and benefits with
team members before making money decisions.
Observable behaviors:
Creates Delivers Uses
Coordinates Asks Organizes
States Persuades Finds
Non-observable item: Analyzes possible costs and benefits
before making money decisions.
Non-observable behaviors:
Knows Appreciates Values
Tends Is Understands
Believes Reflects Thinks
3. Does the item describe ONLY ONE
behavior?
If an item describes more than one behavior, the person evaluating
may feel differently about each behavior. If two behaviors
are important enough to be included, use two assessment items
to represent them.
EXAMPLE:
Write it this way: Explains security procedures to students.
NOT this way: Explains security procedures to students and
sets the example for appropriate behavior.
4. Does the item describe an IMPORTANT
behavior?
Any skill may involve a number of actions, steps, tasks, procedures,
processes, performances and other behaviors. It would be impractical
to list them all in an assessment. The best approach is to
focus on the crucial activities.
EXAMPLE:
More important: Responds to customer requests before the
end of the shift.
Less important: Uses the unpublished phone line for outgoing
calls.
5. Is the item described as a POSITIVE,
desired behavior?
Items that are phrased negatively do not indicate desired
behavior so their measurement is inconsistent with items that
are worded positively. A high score for such items would indicate
a negative tendency, not a positive one.
EXAMPLE:
Positive: When discussing mistakes, talks about lessons
learned.
Negative: Blames others for mistakes.
6. Is the item described in CLEAR
LANGUAGE?
Much of business writing is excessively formal, complex and
passive. This makes the message hard to understand. To ensure
that everyone clearly understand the behavior represented by
an assessment item, say it with fewer, simpler words.
EXAMPLE:
Write it this way: Tells phone receptionists right away
when the stock level is low.
NOT this way: Makes timely notification to work station
technicians in instances of modifications to stock levels
resulting in inventory situations that are below established
parameters.
7. Is the item FREE OF BIAS - sexual,
racial or cultural?
An assessment must be understandable to everyone regardless
of sex, race, religion or heritage. It must measure each person
fairly.
EXAMPLE:
Free of bias: Delivers orders on time.
Culturally biased: Follows the Golden Rule when dealing
with people.
8. Does this item, taken together
with all of the other items, SUFFICIENTLY DEFINE the category?
A whole is the sum of its parts. Are all the parts there?
Do all the items relate to the category? Are any items more
closely related to another category of performance?
Copyright © 2004 Performance Support
Systems, Inc. All rights reserved
