Friday, February 24, 2017

Investment Choices

R&D activities are investments in the future, and so we must assess tradeoffs among competing investment options in terms of focus, benefits, costs, and risks. Managing NOAA’s R&D enterprise requires that the Agency take a portfolio perspective. A portfolio is a set of investments that yield benefits and have costs and associated risks. Portfolio Management is the setting of policy on the distribution of investments across categories, based on expected results.

There are several obvious categorization schemes that NOAA can use to understand its portfolio: by strategic goal, by NOAA Line Office, intramural versus extramural, and R&D as a proportion of all funding. Over the past few years, for example, non-federal partners have conducted between a quarter and a half of the R&D that NOAA funds. NOAA R&D represented 14% of the total appropriation request budget in 2009, it rose to 17% in 2011, and declined to 13% in 2013. In comparison, other environmental agencies allocate a somewhat smaller proportion (for 2012, the Department of Interior: 8%, Department of Agriculture: 10%, Environmental Protection Agency: 8%), other technology-heavy agencies allocate a much larger proportion (Department of Energy: 26%, National Aeronautics and Space Administration: 67%), and the average for all non-defense agencies has stayed consistent at around 11% over the past few decades.

Managing NOAA’s portfolio of R&D needs to take into account how the R&D activities address the breadth of NOAA’s responsibilities and fit together as a system of innovation; consequently, the set of activities must be balanced across a number of dimensions beyond those discussed above. The table below provides some principal dimensions that are important to NOAA, and the types of choices enabled by each dimension, as developed by NOAA for the NOAA Science Advisory Board. There is no one option within these dimensions that is inherently better or worse; rather, NOAA aims for a balance along the continuum for each dimension, given the returns on investments that the Agency seeks.

Portfolio balancing is done with respect to NOAA’s strategy for R&D in pursuit of the objectives in NOAA’s strategic plan. Evaluating and confirming or setting dimensional balance targets for NOAA’s R&D portfolio are components of the strategic planning phase for NOAA’s R&D enterprise. Should NOAA be aiming for more radical or incremental innovation, near-term or longer-term results, or extramurally or internally conducted R&D? These questions require identifying what the current balances are and the expected costs and benefits of changing them. The answers depend upon which goals and objectives NOAA is trying to accomplish, and which take priority. Recent input from NOAA’s Science Advisory Board suggests that NOAA may need to reexamine the balance of its R&D portfolio in a few of these dimensions.  

One potential dimension for assessment is disciplinary specialization, specifically with regard to the proportion of effort the Agency puts into social sciences compared with natural sciences and engineering. Because people both affect, and are affected by the natural environment, NOAA must understand these interactions. Similarly, a move toward studying ecosystems (as distinct from the species and habitats that compose them) might require a research portfolio that is more inter- and trans-disciplinary than uni- or multi-disciplinary. Another dimension in need of portfolio evaluation is output type, with regard to the proportion of attention and effort into activities of transition as compared with creation. “Transition” is the transfer of knowledge or technology from a research or development setting to a real-world setting. Surmounting the “valley of death” between research and applications is a challenge for many Federal agencies and NOAA is no exception.

Dimension  Choices
  Strategy   Goals and objectives from the NOAA strategic plan
  Time Horizon   Short-term, mid-term, or long-term results
  Risk Level   High, medium, or low chance of not achieving results
  Degree of change   Incremental or radical results
  Driver of change   “Push” from research or “pull” from stakeholders
  Comparative Advantage   Activities unique to NOAA, or those that others can also conduct
  Who Conducts   Internal or external, centralized or distributed
  Specialized Talent   Natural science, social science, multi-, inter-, and trans-disciplinary
  Output Type   Knowledge, technology, or transfer of knowledge/ technology

Planning R&D

NOAA 5Year R&D Plan

NOAA must continually strengthen the quality, relevance, and performance of its R&D products, balancing its portfolio of associated R&D activities to optimally achieve NOAA’s strategic objectives. The purpose of R&D planning is to establish objectives, priorities, performance expectations, resource requirements, and the desired balance for R&D activities, thereby enabling consistent and coordinated management of these activities, both within and across organizational units.

Planning activities build a shared understanding of the purpose and direction for NOAA’s R&D enterprise. NOAA’s Science Advisory Board has found that “the major challenge for NOAA is connecting the pieces of its research program and ensuring research is linked to the broader science needs of the Agency.” And further, that “the overall research enterprise should be viewed as a corporate program. Explicit linkages between research efforts across organizational lines must be forged and maintained for the Agency and the nation to obtain the full benefit from research”. The planning process forges these necessary linkages.

Effective plans capture expected cause-and-effect relationships between desired outcomes and the investments that are required to achieve them, thus providing a structure for implementation, monitoring, and evaluation. NOAA’s R&D Plan can also serve as an important tool for communicating the importance and intended value of NOAA R&D to the Administration, the Department of Commerce, Congress, academia, regulated and user communities, and the public at large. In this capacity, this R&D plan serves to foster strategic partnerships with the external research community, whose valuable contributions are critical to meeting NOAA’s mission. This plan highlights NOAA’s R&D foci so that the external research community knows which research aligns with NOAA’s gaps and priorities. In so doing, it also establishes a framework of objectives and targets with which stakeholders can expect to have the results of monitoring and evaluation reported.

Planning for R&D should be appropriate for the kind of R&D being planned (see portfolio dimensions in the previous section). Lower-risk, incremental advances may require a very sequential progress through a series of stage gates or technical readiness levels. More transformative advances might benefit less from a predefined set of hurdles than from multiple opportunities to iterate objectives with leadership and stakeholders as capabilities emerge from the work. All NOAA R&D, however, is directed, which means that it is guided by some objective that describes a vision of success.

Setting Priorities

NOAA plans for R&D within the Strategy Execution and Evaluation (SEE) cycle, within which NOAA manages performance. During the annual planning season, potential priorities permeate up from programs as options for Line Office and Agency leadership to consider. Leadership sets priorities at the corporate level, which then are translated to work plans at the program level. The NOAA Administrator states the Agency’s priorities in the Annual Guidance Memorandum (AGM) to focus the Agency’s attention on particular areas. Implementation Plans (IPs) are then updated to detail how capabilities across the Agency are being used to satisfy those priorities, and how progress is expected to occur over the next few years.

Priorities are choices among options. Prioritizing something means performance in the priority area takes precedence over performance in other areas, resulting in difficult, but necessary decisions. Priorities are best framed as ends rather than means (i.e., requirements rather than solutions), so that programs have flexibility to pursue the best routes to achieve them. Priorities are established periodically by analyzing the strategic context for NOAA R&D, and how it may have changed. If the context has changed, if NOAA is positioned to take action, and if this change warrants a change in strategic direction (including, but not limited to shifting investments), then priorities should change accordingly.

External changes often alter the context within which R&D is being conducted, for example: changes in science, technology, politics, budgets, economic outlook, environmental conditions, and evolving stakeholder needs. Changes can also be internal, for example: programmatic performance with respect to objectives. Context changes can be identified in several ways. Internal changes can be identified through program evaluation (see next section), as well as less formal findings and recommendations of program staff. External changes can be identified by systematically scanning the media environment for emerging trends and issues, as well as simply engaging stakeholders and partners in active dialogue.

Evaluating R&D

NOAA strategic planning process graphic

Evaluations of NOAA R&D inform NOAA on how well its R&D is progressing with respect to the R&D plan, and whether planning assumptions were valid. Evaluation begins with a logic model of how a program’s work is intended to result in strategic objectives. Based upon this model, NOAA can establish appropriate means of gauging performance as empirical means of assessing progress. NOAA policy on R&D assessment is consistent with the National Academies’ Best Practices, in which R&D is judged on three criteria: quality (scientific and technical merit of research as determined by peers), management (engagement with staff and stakeholders, as well as resource allocation and portfolio analysis), and impact (value of research results to people beyond the research community). With empirical data on these criteria (from internal and external sources), assessments can be made of process effectiveness and efficiency, of intended outcomes, of unintended impacts, and of benefits relative to costs. Through evaluation, NOAA can learn if a program works the way it is intended; identify unknown causes of program outcomes and unanticipated consequences; and make better decisions about whether to continue, halt, or change a program.

Evaluation is the end and the beginning of NOAA’s performance management system. The findings and recommendations of R&D evaluation provide raw material with which to develop objectives and targets and set priorities, which, once established, are the basis of future evaluations. Learning how to improve R&D involves asking questions such as: What R&D should be conducted to achieve desired outcomes? Is there sound logic connecting the R&D effort to the expected outcomes? Is the design of the program or project optimal? What execution needs are there in terms of time and resources? Did the research conducted achieve the desired outcomes? Did the research conducted have any unexpected results or impacts?

NOAA values peer reviews of its Laboratories/Centers, Programs, and Cooperative Institutes to ensure their quality, relevance and performance. National Sea Grant follows a rigorous review of all its state Sea Grant programs. Formal policy establishes that peer review panels evaluate each lab every five years and prepare recommendations that labs must then address through implementation plans.

NOAA’s program evaluation efforts are consistent with the performance management requirements of the Government Performance and Results Act (GPRA) and the 2010 GPRA Modernization Act, complying with, but not limited to, the performance management requirements of Congress and the Office of Management and Budget (OMB). NOAA meets or exceeds OMB rules for agencies to conduct peer review for Federal science according to established standards of quality, relevance, and scope set by the Information Quality Act and Peer Review Bulletin.

Engaging Stakeholders

NOAA’s capacity to achieve the objectives outlined in this plan depends on stakeholder engagement. Stakeholder engagement serves to identify user needs, from which NOAA’s R&D objectives and targets are ascertained. NOAA can effectively engage stakeholders by: strategically working with partners and the public; having two-way conversations to better identify society’s needs; and refining NOAA’s R&D to provide capabilities to meet those needs. NOAA’s next breakthrough in R&D may depend upon the unique knowledge or needs of a partner or customer. NOAA’s long-term success will be determined by its capacity to effectively engage individuals and other organizations. The most effective stakeholder engagement approach will depend on the situation, specific goals, objectives and desired outcome. In general, engaging stakeholders early and often leads to more successful partnerships and more valuable R&D. As a leader in oceanic and atmospheric R&D, but not the complete and sole source for these subjects, NOAA must work with others to meet the needs of society. Stakeholder engagement implies shared goals, objectives, and resources. Implicit to engagement is listening, dialogue, understanding, and mutual support.