Applying expert elicitation of viability and persistence to a lynx species status assessment

In 2015, the United States Fish and Wildlife Service initiated a review of the status of Canada lynx (Lynx canadensis) in the contiguous United States. Available research and monitoring, while substantial, lacked information on the demographic rates, abundance, and trends necessary to complete a full viability assessment. Therefore, alternative sources of information were needed to inform the species status assessment. We designed and conducted an expert elicitation to capture the knowledge, professional judgments, and opinions of lynx experts to assess the status of, and the drivers influencing, these lynx populations. We elicited the likelihood and level of uncertainty regarding future persistence over several time frames (at years 2025, 2050, and 2100). The elicitation revealed experts' concerns that expected climate‐driven losses in habitat quality, quantity, and related factors will likely result in declines. Experts expect resident populations of lynx will persist in all five currently occupied geographic units in 2025; in 4 or 5 of the units at 2050; and in 2 or 3 units at 2100. Experts expressed a high level of uncertainty regarding the rate and extent of decline due to projected climate warming and corresponding effects to these lynx populations. In the absence of adequate monitoring data, this type of expert elicitation is a useful method to aid classification decisions, such as providing the scientific information the Service relied upon to complete the November 5, 2017 5‐year review which recommended that the lynx distinct population segment be removed from the list of threatened and endangered species.


| INTRODUCTION
In 2000, the U.S. Fish and Wildlife Service (Service) designated Canada lynx (Lynx canadensis) in the contiguous (lower 48) United States as a distinct population segment (DPS) and listed it as threatened (65 FR 16052, 2000) under the U.S. Endangered Species Act (ESA, 1973, as amended) due to inadequate regulatory mechanisms. Subsequent research and monitoring of lynx status in the DPS-in addition to a 2014 court order regarding development of a recovery plan as required by the ESA and an upcoming 5-year review of the listing status-prompted the Service to call for an updated assessment of the status of the lynx DPS. This assessment provided the scientific information the Service relied upon to complete the November 5, 2017 5-year review (U.S. Fish and Wildlife Service (USFWS), 2017a), which recommended that the lynx DPS be removed from the list of threatened and endangered species.
Recent Service practice is to conduct a scientific review of a species' condition to support multiple decisions related to species management and ESA classification in a process termed a Species Status Assessment (SSA; Smith et al., 2018, U.S. Fish andWildlife Service, 2016). SSAs evaluate the status of a species with regard to the conservation biology principles of representation, redundancy, and resiliency, through an analysis of the best available scientific information regarding the species' individual and population-level ecological requirements, its current and possible future conditions, and the factors thought most likely to influence its viability. Representation is typically measured by the genetic diversity and adaptive capacity of a species, redundancy by the number of independent populations, and resiliency by the size and productivity of populations (see Supplement A for the complete definitions used for this SSA). The SSA should include a "prediction of the species response to a range of plausible future scenarios of environmental conditions and conservation efforts." (Smith et al., 2018).
In the case of the lynx DPS, despite continued research and monitoring, reliable estimates of abundance and vital rates such as those necessary for development of an empirical quantitative model predicting future lynx population status were lacking (U.S. Fish and Wildlife Service (USFWS), 2017b). While there have been studies of lynx demography in Canada and of the reintroduced population in Colorado (Devineau et al., 2010), there was concern these rates would not apply to the diverse ecological and demographic circumstances of the six geographic units spread across the DPS. In addition, there has been little monitoring of the current status of these geographic units to inform the starting conditions for a population projection model. In place of observationally derived empirical models that predict future species condition, expert opinion and professional judgment are appropriate and scientifically supported approaches (Morgan, 2014) often used to fill in the gaps in SSAs (e.g., Voorhies, Szymanski, Nail, & Fidino, 2019). Expert judgment can be obtained through formal elicitation procedures (e.g., U.S. Environmental Protection Agency (USEPA), 2011; Drescher et al., 2013;Morgan, 2014, Fleishman, Burgman, Runge, Schick, & Kraus, 2016 in which experts are selected based on a screening process, are prepped for the elicitation, and are facilitated through the elicitation process. Elicitation processes are designed to enable empirical analysis supported by standardized, transparent, and well-documented methodology that reduces cognitive biases .
In October 2015, to assess the future condition of lynx in the limited time available and in the absence of an empirical population model and data to support one, the Service, led by the Lynx SSA Team and Lynx SSA consultants (Supplement B) convened an expert panel to elicit the current and future conditions of lynx populations in the DPS. While published scientific literature and other available information on many aspects of lynx population dynamics in the DPS range was available, expert input was needed to complement this information and fill in the gaps. The expert elicitation was convened to obtain information on the factors likely to influence the future condition of lynx populations in the DPS, such as the condition and management of lynx habitats, and snowshoe hare (Lepus americanus, lynx primary prey) abundance and population dynamics. We also obtained information on the condition of lynx populations themselves, including potential future trends in lynx abundance and distribution. We therefore designed a process to elicit and capture the knowledge, professional judgments, and opinions of lynx experts to help assess the current and potential future status of, and the nature and magnitude of potential drivers influencing, lynx populations, and habitats within the DPS. We also sought expert knowledge to help evaluate the viability of the DPS in terms of representation, redundancy, and resiliency given uncertainty about future climate and habitat conditions, and to identify and make explicit uncertainty within and among experts.

| Study species
Lynx have large feet and long legs that make them highly adapted for hunting snowshoe hares in the deep or powdery snow that persists across much of its boreal forest distribution, most of which occurs in Canada and Alaska. The southern periphery of the boreal forest extends into parts of the northern contiguous United States, where it transitions to the Acadian forest in the Northeast (Seymour & Hunter Jr, 1992), deciduous temperate forest in the Great Lakes regions, and subalpine forest in the Rocky Mountains and Cascade Mountains in the west (Agee, 2000).
Currently, there are five geographic areas known to support resident lynx populations in the DPS: northern Maine (with occasional/sporadic breeding by small numbers of lynx in northernmost New Hampshire and Vermont); northeastern Minnesota; Northwestern Montana and Northeastern Idaho; North-Central Washington; and Western Colorado (Figure 1). Lynx captured in Canada and Alaska were released between 1999 and 2006 into southwest Colorado to establish the current resident population there. The Greater Yellowstone Area (GYA) of southwestern Montana and northwestern Wyoming is believed historically (and as recently as 2003-2004) to have supported a small number of resident lynx, at least intermittently. However, it is uncertain whether it ever supported any persistent resident populations, and no verified lynx occurrence has been documented in the GYA since 2010. While several other areas peripheral or proximal to the six geographic units described above are known or suspected to have supported small numbers of resident lynx historically, either intermittently or as small but persistent resident populations, due to their small numbers and higher uncertainty in current status we did not elicit persistence specifically for these peripheral lynx.

| Identifying experts
There were five different roles individuals filled at the lynx expert elicitation workshop (Supplement B). Lynx SSA Team members led the SSA process and produced the SSA report. Lynx SSA Consultants and Elicitation Facilitators conducted the elicitation process and aided the SSA Team as they completed the SSA process. Subject Matter Experts provided presentations on a specific topic and were not panelists whose input was elicited during the elicitation process. Candidate Panelists were evaluated for invitation to the elicitation workshop but did not attend. Lynx Expert Panelists were present at the elicitation and were the only individuals to provide input to the results presented below.
Lynx SSA Team members (Supplement B-Lynx SSA Team) reviewed the relevant literature and used their first-hand knowledge to identify experts involved in lynx and hare research or management, boreal forest ecology, and climate modeling. We then developed a priori selection criteria based on professional credentials, positions, areas of expertise, and pertinent experience to develop a list of candidate lynx experts. Selection criteria (Supplement C) helped ensure that invitations to participate were made only to scientists with expertise highly relevant to projecting future conditions for lynx and, further, that the selections were transparent, unbiased, and adequately captured the diversity of expertise and professional judgments related to the topics. Selection was not based on participants' affiliations; however, States and other partners in the SSA process were asked to review the draft list of workshop invitees and suggest alternate or additional qualified experts that meet the selection F I G U R E 1 Six geographic units within the range of the contiguous U.S. distinct population segment of Canada lynx (Lynx canadensis) criteria. Using the criteria, candidates for the lynx expert panel were contacted to determine their interest and ability to attend the workshop. While not an explicit item in the selection criteria experts were also drawn from each geographic unit and the expert panel included experts conducting studies across multiple units and Canada, the more intact and abundant portion of lynx' North American range (Supplement B-Lynx Expert Panelist) which can help to ensure location specific knowledge was represented on the panel and help to avoid having a single perspective of lynx ecology drive the results. Among the experts both initially interested in and able to attend the workshop, the team extended invitations to 13 candidates, 10 of whom ultimately were able to attend and participated as panelists (Supplement B-Lynx Expert Panelist). Only these panelists' inputs were used as data in the SSA process.

| Elicitation format
In accordance with the expert elicitation literature (e.g., Burgman, 2005, U.S. Environmental Protection Agency (USEPA), 2011, Gregory et al., 2012, Drescher et al., 2013, Morgan, 2014, the facilitators (Supplement B-Elicitation Facilitator) developed expert qualification standards for expert selection and a used a Delphibased approach to elicit opinions from the expert panel for each question, as described below. All estimates were provided in a 3-day workshop, one and a half days of presentations followed by one and a half days of elicitation, held in Bloomington, Minnesota from October 13-15, 2015. When estimates were combined, they were mathematically aggregated by taking the median.

| Questions
We asked the expert panelists to supply input on three categories of lynx condition: representation, redundancy, and resiliency (definitions in Supplement A). These categories are used in all SSA processes. At the time of this SSA, these definitions had not been standardized across SSA's and were therefore defined for panelists. Before questions in each category were asked, we visually displayed and read the definition of each category to the panelists and allowed them to ask any questions they had about the definition. After definitions were provided and panelists were ready to proceed, we worked through the questions for that category (Table 1), using the methodology described in the elicitation process section (below).

| Elicitation process
To ensure that all lynx experts had a common baseline of information prior to the future condition elicitation process, the lynx expert panelists first observed and discussed presentations by subject matter experts (Supplement B-Subject Matter Expert) summarizing the current state of available information on topics relevant to the future condition of lynx populations in the DPS (Canada Lynx Species Status Assessment Team, 2016). These presentations covered the historic distribution of lynx in the DPS, genetics, lynx status in southern Canada, climate change impacts, and snowshoe hare status. In addition, members of the lynx expert panel presented updates on lynx populations, research, and monitoring results in the geographic areas they were most familiar with. These subject matter and population update presentations were intended to inform the SSA team for their report and to provide experts with a summary of the current knowledge and provide a common point of reference for responses. While it may have reduced the diversity of elicitation responses to precede the elicitation process with these presentations, they served two purposes. One was to provide experts with a more in-depth format to transmit their knowledge to the Lynx SSA team. The other was to reduce sources of uncertainty the SSA team felt was less germane to capturing experts' future population status estimates. For example, initial population status and ecological conditions of these units is not entirely unknown and therefore could be reduced through sharing across experts such that expert responses varied mainly due the ecology of each unit.
Although invited experts were expected to contribute openly and effectively to group discussions, we did not seek consensus among experts; rather, we probed differences of opinion or interpretation of scientific and technical information. We also asked experts and others present at the workshop to focus on scientific questions and to refrain from discussing or recommending management or policy decisions related to the Service's authorities and responsibilities in implementing the ESA unless directly requested. Only responses from members of the expert panel were recorded. However, panelists were provided the opportunity to confer with the subject matter specialists and SSA Team members in attendance as needed prior to providing an official response for the record.
In questioning, we used a modified Delphi method (e.g., MacMillan & Marshall, 2006). We first elicited individual responses/scores to a question with experts submitting their scores independently via submission sheets with anonymous ID numbers. Following the initial response, the anonymous responses were displayed, and facilitators noted where there was high congruence among responses, as well as low congruence or outlying responses and then asked for voluntary expression of potential rationale supporting the median, endpoint, and other divergent responses. Anonymity was encouraged during this process, that is, panelists need not have stated which response was T A B L E 1 The questions asked of the expert panelists, by category, in their revised form as applicable, and the response type requested theirs or provide rationale related to their response, however panelists chose at times to voluntarily state which response was theirs. Following discussion panelists were given time to reconsider their response and provide their final response which was recorded for the record. Where aggregation of responses occurred, we report the median response.
In addition to elicited responses to each question (Table 1), we often received feedback from the experts on definitional issues and the wording of the questions themselves; based on which the elicitation facilitators and Lynx SSA Team revised the questions as needed. In the case of a revised question, the process was repeated starting with the revised question. That is, scores were elicited and displayed for the revised question, an additional opportunity for rationale, comment, and discussion was provided, which was followed by a third and final response submission.
The Lynx SSA Team and Elicitation Facilitators scripted questions for experts prior to the meeting. Because of the uncertainty related to the status of lynx populations in each of the geographic units and factors influencing lynx biology, questions were phrased in a manner to obtain quantifiable or discrete responses. The types of questions and the format of responses differed based on the information needed to inform the status assessment, as well as the best way to capture information relevant to the question being asked. For example, responses were requested in the form of lists when a set of influences was desired, and in the form of a 4-point elicitation (Speirs-Bridge et al., 2010)-for example, the high and low end values of a range, confidence that the range contains the true value, and the most likely value-when an uncertain quantitative value was desired, or in the form of graphed trajectories when probabilities of persistence over time were desired, and other forms as necessary (see Table 1). Based on prior discussions within the Lynx SSA Team that determined the range of uncertainty from Questions 2.5 and 3.1 regarding reestablishment time and probability of persistence would be most consequential to the SSA and listing process, as well as the limited time available for elicitation, a single point elicitation was used for Questions 2.3 and 2.4 regarding catastrophic events rather than the 4-point and 3-point elicitation graphic responses used in Questions 2.5 and 3.1, respectively.
All panel members were encouraged to respond to each question but also given the option of abstaining if they felt it was beyond the bounds of their expertise. With few exceptions, all 10 expert panelists responded, and as requested. In a few cases, as noted in the results, due to an expert's desire to provide more detail, or to limit follow-up with an expert due to the time available for elicitation, responses differed from the requested response type.
In the case of Questions 3.1 and 3.2 regarding probability of persistence and the factors driving persistence probability, we used a custom elicitation technique. We asked these questions concurrently as part of a probability-of-persistence exercise conducted for each geographic unit. We asked experts to graphically provide the probability of persistence-that is, the probability that resident lynx would not be functionally extirpated-of resident lynx through time for each geographic unit, as well as the major factors influencing persistence in those geographic units, one geographic unit at a time. Experts were asked to provide persistence probabilities and influencing factors for the nearterm (2025), mid-term (2050), and longer-term (2100). Unlike Question 2.5, and as noted can be preferable by Hemming, Burgman, Hanea, McBride, and Wintle (2018), experts chose to present the full range of outcomes through a 3-point elicitation; that is, providing their "most likely" persistence probabilities as well as "low" (worse case) and "high" (best case) persistence probability values to express the full range (100%) of their uncertainty at each future time frame (e.g., Hanea et al., 2018).
Prior to expert resiliency (probability of persistence) responses for Questions 3.1 and 3.2, the expert(s) most familiar with the geographic unit in question restated their presentations from earlier in the workshop as a 5 to 10-min summary of covering what they viewed as the most relevant information about the current and likely future status of lynx populations and habitats in that unit. They also presented any other conditions or issues they thought could affect (either positively or negatively) the probability of persistence of resident lynx in that unit. All experts then completed their graphs and lists of the factors that influenced the probabilities of persistence they selected for each time frame for the geographic unit in question. After all experts completed their responses, the graphs and influence lists from each expert for a given geographic unit were visually displayed and workshop participants were invited to gather around to view and participate in a facilitated discussion regarding what drove the responses for that unit. These questions were a mix of directed questions about unique responses, the role of factors noted in the responses, and open-ended questions to allow experts to describe their thinking. Experts and team members were also encouraged to ask clarifying questions about the responses. For each geographic unit, experts were given an opportunity to post revised responses following these discussions.

| Response processing
For most questions, we report the responses for each expert as provided, as well as the median response for the requested values. For Question 2.5 regarding the time to reestablishment of a lost geographic unit we calculated a 95% confidence interval for the shortest and longest plausible time periods for each expert using the confidence ranges provided from the 4-point elicitation using linear extrapolation (e.g., McBride et al., 2012). In cases where the linear extrapolation resulted in a negative number of years for the shortest time periods, we adjusted the lower bound of the interval to zero (as in Hemming, Walshe, Hanea, Fidler, & Burgman, 2018). We used the probabilities of persistence provided for each geographic unit from Question 3.1 to calculate a cumulative impact across the DPS given the responses at the individual geographic unit level. This calculation provides the probability that a given number of geographic units will persist into the future based on the probabilities provided for each individual geographic unit. The probabilities of persistence for each geographic unit were treated as independent, that is, not influenced by the probability of any other unit persisting and were used to determine the joint probability of persistence for a given number of geographic units in total. Thus, if there were only 2 units, the probability both units persist, or fail to, would be the product of the probabilities that each unit persists, or fails to, producing the joint probability that both persist or fail to. The probability of 1 unit persisting is the sum of the joint probabilities that unit one persists while unit two does not, and vice versa. The full set of probabilities for 6 units can be determined via a convolution. Computationally, these n unit persistence probabilities were calculated using a convolution of the Bernoulli probability distribution of persistence for each geographic unit via a custom convolve function executed in the statistical software R (see Supplement D for the R code and data used to produce these and the other summaries and figures presented). We present this in two forms: (a) the probability that a particular number of geographic units persist, and (b) the cumulative probability that at least a certain number of geographic units persist.

| RESULTS
Results are presented by question number for each of the three categories.
3.1 | Representation 1.1: Regarding the potential for genetic drift to impact lynx populations in each geographic unit, three experts responded that the Western Colorado geographic unit is susceptible to genetic drift, and two responded that the GYA geographic unit is susceptible to genetic drift. Five experts responded that none of the geographic units are susceptible to meaningful genetic drift, and two experts abstained from responding.
1.2: Regarding the unique ecological conditions of the geographic units, all experts abstained from responding with a discrete geographic unit-based response. In lieu of this, experts generally stated that there were no known adaptively unique habitats; however, maintaining genetic variability was deemed important for adaptive capacity and maintaining all populations was considered important to conserving adaptive capacity (full response contained in Canada Lynx Species Status Assessment Team, 2016).

| Redundancy
2.1: Regarding the possibility of extirpation resulting from catastrophic events, six of the ten experts did not list any catastrophic event that could result in the functional extirpation of resident lynx from any entire geographic unit. Among responses from the other four experts who did list an event potentially capable of catastrophic extirpation, disease, and fire were each listed three times, forest insect outbreak was listed twice, and a failure of winter conditions (i.e., warm or dry winters with insufficient snowfall) was listed once.
2.2: Regarding the likelihood that the entire DPS could be catastrophically extirpated, experts unanimously agreed that no single catastrophic event could eliminate resident lynx populations from all geographic units simultaneously.
2.3: Regarding the likelihood of any geographic unit being eliminated by a single catastrophic event, all respondents gave a relatively low probability (≤10%, median of 1%) that any single geographic unit could be eliminated by a single catastrophic event in the next 10 years (Table 2, Figure 2, Q3).
2.4: Regarding the probability of series of events causing extirpation in any geographic unit, panelist responses ranged from a 0.5% to a 60% chance of a series of events during a 10-year period causing extirpation, with a median response of 7.5% (Table 2, Figure 2, Q4).
2.5: Regarding the 4-point elicitation addressing reestablishment following a catastrophic event, panelists indicated that the most plausible time frame for reestablishment of a resident lynx population is 10-100 years, with a median response of 37.5 years (Table 2, Figure 3). The linear extrapolation produced a 95% interval with a median shortest and longest plausible reestablishment time of 0.5 years to 158.5 years.

| Resiliency
3.1: Regarding the probability of persistence at present and into the future, experts expressed certainty that all geographic units except the GYA unit currently support resident lynx populations (Table 3). Experts retained confidence in the persistence of lynx populations in those units through 2050, with median most likely probability of persistence values >=70% for all units other than the GYA (Figure 4). By 2100, confidence that populations will persist was lower with five of the 6 units having a median probability of persistence of 50% or less, the exception being the Northwestern Montana/Northeastern Idaho unit (Table 3). In each time period, experts had the greatest confidence in the persistence of the Northwestern Montana/Northeastern Idaho unit, and least confidence in the persistence of the GYA unit. Confidence in the persistence of the other 4 units was relatively similar in each time period.
Uncertainty in the probability of persistence increased as projections extended into the future (Table 3, Figure 4, gray area). Apart from the GYA, the median lowest probability of persistence was 70% in 2025 (North-Central Washington), 50% in 2050 (North-Central Washington and Western Colorado), and 5% in 2100 (Northeastern Minnesota). By 2100, the uncertainty expressed as the range of median lowest to median highest probabilities of persistence became substantial. In the case of the Northern Maine unit, the breadth of experts' median lowest to median highest persistence probabilities covered a large portion of the possible range (15-90%). By 2100, at least one expert expressed that lowest plausible probability was 0 and the highest was 100% in three of the 6 units, and nearly so in the remaining three (Figure 4, dotted lines).
The probability that populations will persist in all 6 units declined slightly by 2025, with a corresponding increase in the probability that populations will persist in 3 or 4 units. Expert responses suggested that populations will persist in 4-5 units by 2050 and that F I G U R E 3 Individual responses (in color) and summary boxplot of the number of years for a geographic unit to become reestablished following extirpation due to catastrophic events 2-3 units will most likely continue to support resident populations by 2100. (Figure 5, Median Likely column). Summarizing these probabilities another way, experts indicated they were 100% certain that 5 of the 6 units (all but the GYA) currently support resident lynx populations, that those 5 units are very likely (81%) to continue to support resident populations at year 2025, about as likely as not (47%) to do so at 2050, but less likely (5%) to do so at 2100, with extirpation of resident populations from several units possible by T A B L E 3 Probability of persistence (%)-aggregated median most likely value (top line) and range from median lowest to median highest (bottom line)-by geographic unit and time period   (2015), and in 2025, 2050, and 2100. The dashed line shows the aggregated median most likely probability of persistence responses across the 10 experts, with colored points for each expert's most likely response. The gray region shows the range from the aggregated median highest to median lowest probability of persistence, and the dotted lines show the extreme, that is, unaggregated, high-end, and lowend responses then, but with substantial uncertainty in end-ofcentury projections Figure 6.
There was some uncertainty about the current status based on the high-and low-end plausible persistence responses from experts, which range from an expert expressing certainty in all 6 units currently persisting to an expert expressing there is a 6% chance only 4 units are persisting and 96% chance that 5 units currently persist. The range in uncertainty expressed from the median range ( Figure 5, median low to median high), increased by time period. In the current time period, the most probable number of units persisting ranged from 5 units for the median low response to 6 units for the median high response. In 2025, this range was 5-6 units, in 2050, 3-5 units, and in 2100, 1-5 units ( Figure 5). The expert with the least confidence in lynx persistence by 2100 indicated only a 10% chance of 1 or more unit persisting by 2100, while the most confident expert indicated an 81% chance of all 6 units persisting in 2100.
3.2: Regarding factors influencing lynx status in the DPS, the near-and mid-term (up to 2050) drivers of persistence probability cited by experts, and common to all units, were climate-mediated increases in wildfire activity, reduced snow pack quality and a related increase in bobcat competition, forest insect-induced habitat changes, and the potential for disease. Longer-term drivers include the above set, but with a greater emphasis on the climate related drivers of changing snow conditions that influence lynx competitive relationship with bobcats and lost spruce-fir forests that support lynx prey species. Experts also listed potential reductions in the frequency and number of lynx immigrating to DPS populations from Canadian populations as a driver of future persistence of DPS populations.
In addition to the factors listed for all geographic units, some of the drivers listed were unique to one of more geographic units. For Northern Maine, experts listed forestry practice changes as a driver of lynx persistence. Landscape-level clear-cutting in the 1970s and 1980s in response to a spruce budworm (Choristoneura fumiferana) outbreak has resulted in the current unnaturally large amount and broad distribution of high-quality snowshoe hare habitat in Northern Maine. However, these regenerating clear cuts, which produced thick conifer understories that support high-density snowshoe hare populations, are developing into more mature stands that support fewer hares, and changes to state forestry regulations have resulted in a shift away from large clear-cuts to various partial-harvesting techniques that may support lower hare densities. In northeastern Minnesota, experts expressed some concern that the spruce budworm outbreak in eastern Canada could spread and impact that geographic unit. In the remaining western units (GYA, Western Colorado, Northwestern Montana/Northeastern Idaho, and North-Central Washington), experts expressed concern about climate-mediated increases in the frequency, size, and intensity of wildfires and forest insect outbreaks. They noted that the greater topographic relief in the western units may provide elevational refugia from climate impacts (upslope areas to which lynx and hare habitats and populations might relocate in response to continued climate warming), but expressed uncertainty about how snowfall, forest habitats, and lynx and hare populations might respond to climate impacts, and the timeline for those responses. 3.3: Regarding conservation efforts, experts provided a list of conservation actions to potentially increase persistence probability (Supplement D, Question 3.3). These responses included reducing CO 2 emissions, continued protections associated with listing, habitat management such as lynx focused forestry practices, promoting habitat connectivity and hare populations, population augmentation or reintroduction, as well as increased funding, research, and monitoring. We note that items in this list were offered not as established practices but as strategies for future consideration.

| DISCUSSION
The expert elicitation responses at this workshop helped inform and refine the Service's understanding of key aspects of the status of, and potential factors influencing the viability of the lynx DPS. The expert elicitation workshop helped to provide a summary of the current knowledge about the current and possible future representation, redundancy, and resiliency of the lynx DPS for use in the Service's SSA. The information obtained from this expert elicitation informed the conclusions in the Service's final SSA report (U.S. Fish and Wildlife Service (USFWS), 2017b) that: "We expect lynx populations in each geographic unit to become smaller and more patchily distributed due largely to projected climate-driven losses in habitat quality and quantity and related factors. However, the timing, rate, and extent of habitat decline due to projected climate warming and corresponding effects to lynx populations is highly uncertain. Despite some reduced top to bottom). The probability in a bar reaches 1 when at least that many geographic units are estimated to persist and drops toward zero as the probability of at least that many units persisting declines. Moving from left to right the grids show the range of uncertainty in expert responses with the left most and right most columns displaying the median of the low-end responses across experts (Median_Low), and median high-end response (Median_High). The middle column displays the median most likely response (Median_Likely). Therefore, looking down a column of grids provides a view of the trend in persistence through time and looking across a row of grids provides a view of the range of uncertainty in a single time period resiliency, we conclude that resident lynx populations are very likely to persist in all 5 units that currently support them (Units 1-4 and 6 [excluding the Greater Yellowstone Unit]) in the near-term (2025) and in all or most of those units at 2050, with corresponding maintenance of redundancy and representation in the DPS over that time span. We and the experts we consulted have low confidence in predicting the likely conditions of DPS populations beyond 2050. That said, smaller, more isolated populations would be less resilient and more vulnerable to demographic and environmental stochasticity and genetic drift and, therefore, at higher risk of extirpation. Although predictions out to 2100 are highly uncertain, it is possible that resident lynx populations could be functionally extirpated from some units by the end of the century. Should extirpations occur, this would indicate a loss of resiliency, reduced redundancy and representation, and an increased risk of extirpation of the DPS." The facilitated questioning of experts obtained during the elicitation workshop provided information in a useful and consistent form across experts, including explicit expressions of uncertainty that provided the bulk of the information reported in the Services conclusions regarding population viability and future probability of persistence. While the expert elicitation provided information for the SSA, a remaining challenge for listing decision makers in this case was dealing with the high amount of uncertainty that results from making long-range projections. In the long term, the effects of climate change on lynx populations are relatively certain, but the highly uncertain timing of when those effects will manifest for these geographic units makes the selection of the foreseeable future more consequential to listing in situations like this.
There are limitations to the strength of conclusions that can be drawn from the expert responses; however, available alternatives often have their own limitations such as the need model development or data collection that are difficult to complete within available timelines. In typical practice, from the date a proposal for listing is received the Service has 12 months to determine if there is substantial evidence to proceed with a listing determination, conduct an SSA, and make a listing determination-with a possible 6 month extension, if there is substantial disagreement within the scientific community concerning the biological appropriateness of the listing. In practice, this timeline may be extended, but often on an ad-hoc basis that requires planning SSA processes as if the 12-month timeline for the full listing process will be adhered to. This leaves roughly 6-8 months for an SSA report to be produced, a timeline which often restricts new data collection and requires use of current data and expert knowledge.
The compressed timeline for completion of the Service's SSA report, and the limited availability of the experts required that the bulk of the expert preparation and follow-up occurred at the elicitation workshop itself. This resulted in some on-the-fly crafting and reworking of questions, which may have limited the consistency with which questions were understood by experts and reduced the amount of time experts had for careful thought about the questions and their responses. The compressed schedule for elicitation exercises also did not allow the experts to become familiar with the process (e.g., through practice of the response types using training questions) ahead of the elicitation.
The compressed timeline may also have resulted in misrepresenting the true level of uncertainty the experts intended to express, due to semantic uncertainty resulting from misunderstanding the questions and response type influencing the experts rather than their uncertainty in the underlying value the question intended to address. For example, the results of the 4-point elicitation to Question 2.5 regarding reestablishment time, in which confidence range adjusted responses often fell below zero, indicate that in some cases there was likely some degree of misunderstanding of the implications of 4-point elicitation. Struggles with 4-point elicitation have been reported elsewhere (Hemming, Hanea, Walshe, & Burgman, 2020), and the confidence range adjusted results here may indicate under confidence in the responses provided by the experts and that linear extrapolation is not fitting the underlying probability distributions of these experts well. Expert requested summary presentations from the local experts to compensate for the compressed timeline and their incomplete knowledge of differences in units. Discussing the local expert's rationale during the feedback time between elicitation rounds in rather than prior to the first round of estimates or expanding the timeline, enabling experts to develop their own expertise, could have further minimized the potential for group think. In future processes, expanding the SSA timeline or using remote elicitation (e.g., McBride et al., 2012) could likely enable further follow-up on elicitation responses or enable additional rounds of elicitation separated by time to accumulate knowledge. Perhaps, an online training application or other techniques could be adopted by agencies that frequently utilize expert elicitation to better support the efficiency of these processes.
An additional consideration for future elicitations is the trade-off between directed questions with discrete response types, and more open-ended response types and questions. Expert panelists in the workshop struggled with this trade-off, often desiring to expound upon the topic a question addressed beyond what they provided specific to the requested response type. One means of addressing this concern, practiced at the workshop, was to take notes on the dialogue and include those notes in the workshop report (Canada Lynx Species Status Assessment Team, 2016). This enabled the collection of discrete responses, as well as for the Service to obtain and use additional information provided by the experts through reference to the workshop notes when completing the SSA.
The collection of discrete responses from this expert elicitation can provide a basis for future lynx research. Researchers could conduct studies to help validate the responses from experts, or future workshops could be held to update these results with new responses as future studies allow experts to consider additional information and update their knowledge. Researchers can also use the uncertainty expressed by the experts to conduct value-of-information analyses (e.g., Runge, Converse, & Lyons, 2011) to determine the value of improved information on the relationship between the factors experts listed and lynx persistence. Research can then be directed towards those areas where additional research could most cost effectively improve the understanding of lynx ecology, status, management, and conservation needs. With the status assessment completed based upon information obtain from the elicitation managers can also begin implementing management actions, and better target those actions based upon the estimated status in these geographic units.