394 Public Administration Review â€¢ May | June 2016
Bridging the Divide between Evidence and Policy in
Public Sector Decision Making:
A Practitioner â€™ s Perspective
Max K. Arinder recently retired after
34 years of service to the Mississippi Joint
Legislative Committee on Performance
Evaluation and Expenditure Review, having
served 15 years as chief analyst for planning
and support and 19 years as executive
director. He holds a PhD in experimental
psychology from the University of Southern
Mississippi and has served as staff chair
of the National Conference of State
Legislatures and the National Legislative
Program Evaluation Society.
E-mail: [emailÂ protected]
Abstract : While policy advocates can help bridge the divide between evidence and policy in decision making by
focusing on ambiguity and uncertainty, policy makers must also play a role by promoting and preserving deliberative
processes that value evidence as a core element in leveling raw constituent opinion, ultimately resulting in a better-
informed electorate. Building on existing research and analytic capability, state legislatures can increase the demand
for and delivery of relevant information, giving the institution the capacity to keep abreast of research in critical public
policy areas. By implementing data and time-conscious evaluative frameworks that emphasize evidence-based decision
making and longitudinal costâ€“benefit analytics at critical policy-making junctures, the institutional culture can
become less unpredictable and the â€œrules of the gameâ€ can be more transparent. In 2015, Mississippi â€™ s legislative leaders
created a system to review requests for new programs and funding using such an evidence screen.
Kimberley R. Isett, Brian W. Head, and Gary VanLandingham, Editors
Max K. Arinder
Evidence in Public
Bridging the Divide between Evidence and Policy in Public Sector Decision Making: A Practitionerâ€™s Perspective 395
As a longtime nonpartisan legislative staffer, I agree with Paul Cairney, Kathryn Oliver, and Adam Wellstead â€™ s assertion that there are significant
differences in academic and political cultures and in how academics
and policy makers view and use â€œevidenceâ€ in decision making.
Their article â€œTo Bridge the Divide between Evidence and Policy:
Reduce Ambiguity as Much as Uncertaintyâ€ provides a thought-
provoking perspective on how the scientific community could
use elements of public policy theory to better communicate in
a political culture, thus giving more weight to their empirical
observations in a deliberative process.
While I agree that there needs to be a culture shift in the way
scientists advocate for policy, experience tells me that there also
needs to be a culture shift in the policy-making institutions
themselves. Yes, policy makers often balance evidence, emotions,
and beliefs in making a decision. The question is whether we
can create an environment in which an optimum balance can be
achieved among these elements for a given policy question. The
answer can be found in an exploration of the â€œcultureâ€ of a given
The conclusions drawn by Cairney, Oliver, and Wellstead would
not be foreign in many of the legislative program evaluation shops
around the country. One of the ongoing challenges they face as
nonpartisan legislative staff is to get their work recognized and
used in policy debate. Most provide regular in-service training
on ways to achieve that goal through improved written and oral
communication. I see this as a parallel goal to those espoused for
scientists by Cairney, Oliver, and Wellstead .
A State Legislature as a Policy Culture
Professionals in the legislative arena see daily the swirl of facts,
opinions, feelings, and arguments surrounding policy makers. Amid
this swirl, some legislators routinely rely on empirical evidence in
key policy areas, while others use emotions and beliefs based largely
on pressing constituent concerns. â€œMy people tell meâ€ is often heard
around the capital, reflecting the importance of constituent opinion
in any decision-making process. This may well be the natural state
of any politically based public environment. Constituent opinion
is a strong voice and will certainly continue to be a part of any
legislative arena. But how well informed is that voice? True, we are
a society awash in information. But how good is that information?
A significant element of a sound policy culture should be the
promotion and preservation of a deliberative process that values
evidence as a core element in leveling raw constituent opinion.
Failure may well negate one of the strengths of representative
democracy itself: controlling tyranny by the majority.
Understanding the deliberative core of a particular political
institution is a critical first step in bridging any divide between
evidence and policy in decision making. The opportunities for
change can be found within the bounds of the long-standing
rules, values, and practices of legislative bodies. Generally, these
institutions will have both expectations for empirical rigor and
latitude for constituent opinion, whether that opinion be rigorously
developed or not. While legislatures can be, as Cairney, Oliver, and
Wellstead describe, â€œunpredictable policy-making environment[s]
in which attention can lurch from issue to issue,â€ they do possess
a basic culture that allows them to function fairly efficiently to
keep government service structures operational and to address
perceived needs. If we can understand this natural balance in a
given legislature, we can better see its strengths and understand
its weaknesses regarding the nature and utility of different types
of evidence in the decision-making process and perhaps find
opportunities for constructive change.
Changing the Policy Culture of a State Legislature
While the scientific community can provide an important impetus
to change through actions referenced in the work of Cairney,
Oliver, and Wellstead, a core change in a legislative culture must
ultimately be accomplished by the membership of the institution
itself, optimally with assistance from a sound technical staff
and supported by an informed constituency. Just as constituent
opinion is important in policy making, it is also an important
element in determining the policy culture of a legislature.
Paradoxically, achieving the needed constituent support for change
may depend on the will of policy makers to take the first steps
in developing a transparent, evidence-informed policy process
that increases public knowledge in critical ways. To do so, elected
officials must acknowledge that, while they are elected to represent
their constituentsâ€™ opinions on important policy issues, they are
also chosen to help determine whether those opinions, regardless
of how formed, can withstand empirical debateâ€”certainly a
political risk, but one worth taking in the interest of both the
common good and the health of the representative democratic
The wider processes of debate, coalition formation, and persuasion
to reduce ambiguity could be at home in such an environment
because the culture itself prescribes the method of framing policy
questions, demanding scientific evidence as a basic element when
needed. In such an environment, policy advocates would be
expected to meet established standards for rigor and completeness
if they expect their proposal or program to be seriously considered.
Such an environment would help with the demand for and delivery
of relevant information as well as give the institution the capacity to
keep abreast of research in critical public policy areas.
Can Such an Ambitious Culture Shift Be Achieved?
Since the 1970s, legislatures around the country have recognized
their lack of the research evaluation skills needed to identify
and appropriately use robust evidence. In response to that
need, legislatures have established internal audit and evaluation
capacities that enable them to carry out their role as a coequal
branch of government through legitimate research and analytic
capability. Over the years, this capacity has reached different levels
of development in different legislatures, but all are marked by
more scientifically trained teams that can, as recommended by
Cairney, Oliver, and Wellstead, help build research capacity in
government, reduce the loss of institutional memory, and generate
a clearer research question when policy makers commission
In addition, the bipartisan National Conference of State Legislatures
continually highlights the challenges legislatures face in developing
sound responses to pressing public needs. This work is underpinned
by an informal multistate agreement on the core values that should
396 Public Administration Review â€¢ May | June 2016
1.a. What public problem is this program seeking to address?
1.b. Briefly stated, how will this program address the public problem identified in Question 1.a?
1.c. Does this proposed program effort link to a statewide goal or benchmark identified in Building a Better Mississippi: The
Statewide Strategic Plan for Performance and Budgetary Success? (yes or no)
1.d. If the answer to Question 1.c was â€œyes,â€ specify the statewide goal or benchmark to which the proposed program links.
1.e. Explain where this program fits into your agencyâ€™s strategic plan; i.e., specify the agency goal, objective, and strategy
that the proposed program seeks to address.
What is the statewide extent of the problem identified in Question 1.a, stated in numerical and geographic terms?
3.a. What specific service efforts/activities will you be carrying out to achieve the outcomes identified in Question 5.a?
3.b. Describe all start-up activities needed to implement the program.
3.c. Provide a time line showing when each start-up activity will take place and the date when you expect the program to be fully operational.
3.d. Over the time period for which you are requesting funding:
i. How many of each of the service efforts and activities identified in Question 3.a do you intend to provide and in which
ii. How many individuals do you intend to serve?
4â€”Return on Investment
4.a. What are the estimated start-up costs for this program, by each start-up activity described in Question 3.b?
4.b. Once the program is fully operational, what is the estimated cost per unit of service?
4.c. List each expected benefit of this program per unit of service provided. If known, include each benefitâ€™s monetized value.
4.d. What is the expected benefit to cost ratio for this program, i.e., total monetized benefits divided by total costs?
5â€”Measurement and Evaluation
5.a. What specific outcomes do you expect to achieve with this program? Each outcome must be stated in measurable terms
that include each of the five elements specified in the following example:
Required Elements of a Measurable Outcome Example of a Measurable Outcome
1 Targeted Outcome
2 How the Outcome Is Calculated
Infant mortality rate (number of deaths of
children less than one year of age per 1000 live births)
Number of deaths of children less than one year of age during
a specified time period [generally one calendar year, unless
otherwise noted] divided by the number of live births during
the same period, multiplied by 1,000
Direction of Desired Change (increase, decrease or maintain)
In order to establish a performance baseline, for each outcome measure reported in answer to Question 5.a, report the
most recent data available at the time of your request and the reporting period for the data.
For each outcome measure reported in answer to Question 5.a, explain how you arrived at the expected rate of change
by the target date.
5.d. How often will you measure and evaluate this program?
5.e. What specific performance measures will you report to the Legislature for this program? At a minimum, you should
include measures of program outputs, outcomes, and efficiency.
6â€”Research and Evidence Filter
6.a. As defined in MISS. CODE ANN. Section 27-103-159 (1972), if there is an evidence base, research base, promising
practice or best practice on which your agency is basing its proposed new program, attach a copy of or online link to
the relevant research.
6.b. If there is no existing research supporting this program, describe in detail how you will evaluate your pilot program with
sufficient rigor to add to the research base as required by MISS. CODE ANN. Section 27-103-159 (1972).
7.a. For programs with an existing research base, explain the specific steps that you will take to ensure that the program is
implemented with fidelity to the evidence/researc h/best practice on which it is based.
7.b. If there is no existing research base for this program, explain the key components critical to the success of your pilot
program and how you will ensure that these components are implemented in accordance with program design.
4 Targeted % Change 18.5% decrease in the infant mortality rate per 1,000 live births
5 Date Targeted to Achieve Desired Change One year from full implementation of program
Figure 1 Seven Elements of Quality Program Design
Bridging the Divide between Evidence and Policy in Public Sector Decision Making: A Practitionerâ€™s Perspective 397
mark any truly representative political body and, if followed, should
establish a solid foundation for implementing a more evidence-
informed policy culture.
Finally, with the impetus of multistate participation in the
Pew-MacArthur Results First Initiative, a number of states are
implementing data and time-conscious evaluative frameworks
that emphasize evidence-based decision making and longitudinal
costâ€“benefit analytics as an important element in the policy-making
process. With Pew-MacArthur â€™ s technical assistance, participating
states are able to better use their own technical staffs in employing
methods that will help meet an evolving demand for clearer evidence
at critical points in the policy-making process, thus providing policy
makers with the information needed to justify decisions based on
evidence that can be balanced with constituent opinion.
Developing an Evidence-Sensitive Policy Culture
Through the years, there have been many efforts at introducing
â€œscientificâ€ management principles into governmental
administrative and budgetary practices. Sound in principle, these
efforts were embraced with great expectations but often fell short for
various reasons, not the least of which has been that they have been
tied more to political cycles and personalities than to foundational
changes in the way we think about program and budgetary
This was certainly true in Mississippi 20-plus years ago when
the legislature passed the Performance Budgeting and Strategic
Planning Act of 1994. The act itself is sound, but a retrospective
look at implementation reveals critical failures in follow-through
that compromised its utility as a budgetary tool that could be used
to build a priority and data-driven budget. For example, elements
of the act that would have provided the resources to analyze
and perfect strategic planning and performance data were not
implemented. As a result, legislators were presented with raw data
that often was not helpful in legislative deliberation.
However, Mississippi â€™ s current legislative leadership has
acknowledged this long standing flaw, has embraced the importance
of data analytics to sound policy processes, and has adopted
a strategic view of budgeting that can fundamentally alter the
budgetary culture of the state. Backed by a professional staff
dedicated to establishing and maintaining a framework of evidence,
performance, and cost-based data to support sound policy debate,
the budget and appropriations committees will now be able to
develop clearer budgetary recommendations to fund those programs
and services that help the state reach its overall policy objectives and
eliminate those that do not.
The key elements of this system can be summarized as follows: a
legislatively developed and maintained statewide strategic planning
effort; a comprehensive statewide program inventory; and mastery
of the Pew-MacArthur Results First Initiative as a tool for bringing
data-driven decision making and costâ€“benefit analytics to bear on
the state â€™ s budgetary process.
These three deceptively simple key elements contain other supporting
elements that also need to be developed or implemented. Examples
include a system for keeping the state-level strategic planning process
relevant across election cycles and responsive to executive branch
initiatives without losing its strategic value; a transparent, longitudinal
tracking system to monitor progress in achieving state-level outcomes;
increased capacity to identify program-level costs and monetize
relevant benefits; a needs assessment process that allows costâ€“benefit
analytics to be better utilized in selecting programs; a strategy for
assessing the efficiency and effectiveness of administrative support and
other nonintervention programs; an evidence-based research focus for
newly proposed intervention programs; routine use of costâ€“benefit
analytics and performance-based outcomes to identify programs for
possible elimination and resource redirection; and expanded capacity
for fidelity studies and performance evaluations, to name a few.
Creating such a framing process allows the institutional culture to
become less unpredictable and the â€œrules of the gameâ€ to be more
While the thoughts by Cairney, Oliver, and Wellstead on the
multilevel, unpredictable nature of policy environments are certainly
noteworthy, critical junctures exist in the policy process where a
disciplined approach to screening policy proposals can become
pivotal in a budget culture, determining in large part the future of
the initiative or program being advocated. It is at these junctures,
most critically in the appropriations committees, that political
leaders have an opportunity to insist that the rules of the game
require vetting of every proposal (regardless of politics) against a
core of critical questions designed to assess the potential and cost of
the program against anticipated benefits relative to the need being
For example, in 2015, Mississippi â€™ s legislative leaders created a
system to review requests for new programs and funding using an
evidence screen. This process, labeled the Seven Elements of Quality
Program Design (see figure 1 ), requires agencies to meet certain
criteria to qualify for funding. For instance, agencies must report
whether a requested program has â€œan evidence base, research base,
promising practice or best practiceâ€ model, describe the monitoring
system that will be used to ensure that evidence-based programs
are implemented with fidelity, and explain how they will measure
the results the program is achieving. While this does not guarantee
removal of politics from the process, it does create a point where the
quality of evidence and sound costâ€“benefit analytics can take center
stage for all to see.
This analytic tool reflects the spirit of the times in Mississippi and is
a clear example of how the evidence/policy gap can be reduced in a
real-world decision-making process. With tools like thisâ€”and this
is only one exampleâ€”policy makers now have the option of seeking
advice from an independent broker that they can trust, using the
information called for by the Seven Elements. Such an approach
allows legislative policy makers to ask key questions that are directly
in line with their political and public policy interests.
While agreeing that empirical facts cannot be completely separated
from human values, ongoing changes in the policy culture of state
legislatures over the last 40-plus years lead the author to believe
that we can make research-sensitive strategies a more important
part of our core value system in the selection and funding of policy
initiatives and programs. To do so, weâ€”scientists, independent
398 Public Administration Review â€¢ May | June 2016
reviewers, and policy makers alikeâ€”must all work
to support any efforts made in policy-making
institutions to shape the policy environment in
such a way that the demand for empirical evidence
becomes a required, appropriately targeted part of
the policy-making culture. While decisions may
still be made in light of other considerations, the
actual supporting data in such an environment will
be available and transparent for all to see, with one
desired outcome being a better informed electorate.
Public Administration Review,
Vol. 76, Iss. 3, pp. 394â€“398. Â© 2016 by
The American Society for Public Administration.
If there is an area of Public Administration process or practice where you feel there is a mismatch
between the accumulated evidence and its use, and the substantive content is salient to todayâ€™s
political or institutional environment, please contact one of the editors of Evidence in
Public Administration with a proposal for consideration: Kim Isett ([emailÂ protected]),
Copyright of Public Administration Review is the property of Wiley-Blackwell and its
content may not be copied or emailed to multiple sites or posted to a listserv without the
copyright holder’s express written permission. However, users may print, download, or email
articles for individual use.