Report on the proposal for a National Park in south-west Scotland - Accompanying report - Methodology for analysing consultation responses (Report number 2a)

May 2025

Introduction 

This is one of 21 reports that accompany NatureScot’s reporter Advice on the Scottish Government’s proposal for a new National Park in southwest Scotland. The advice was submitted on 5 May 2025. This report outlines how NatureScot undertook the analysis of the responses received to the consultation.

Survey Approach 

The public consultation was launched on 7 November 2024 and ran for 14 weeks, to 14 February 2025. Two weeks were added to the original 12-week period to allow for the Christmas holidays. 

A consultation paper with information on the options that had been developed during the engagement phase was provided on the NatureScot website. Alongside this paper, two consultation surveys were produced: a technical consultation (of around 30 questions, see annex A) aimed primarily at organisations and businesses, and a summary consultation (of around 10 questions, see annex A) aimed primarily at individuals and households. 

The survey questions were developed by a team within the core project staff to cover the issues we had been asked to provide advice on. It included staff with experience of survey design, data analysis and communications. The wording and sequencing of questions was refined during testing with a wider group of staff which led to changes to the questions, especially for the summary version of the survey. 

In developing the surveys, we recognised that some of the issues being covered were quite technical despite the efforts made to make them more accessible and without making either version of the survey too long. We also considered the best ordering of the questions – logically, the questions to gauge opinion on the proposal may have been best placed towards the end, after participants had considered the detailed proposals. However, in response to calls for a simple question on the principle of the proposed National Park we decided to put this upfront in both surveys. Rather than a straight yes or no question, we concluded that a more graduated set of responses was more useful at this stage of the process when a lot of uncertainty about the detail of the proposed National Park inevitably remained.

The surveys were available online, via the Smart Survey platform accessed from NatureScot’s website. They were also signposted from the online community information hub that had been active during the engagement or pre-consultation phase. Different formats of the survey were also available on request.

In the instructions for the consultation on our webpage, it was stated that respondents were only expected to complete one survey. To be as inclusive as possible, we did not place any restrictions on anyone if they wished to respond to more than one survey. All participants were asked to provide some basic contact information. Participants could also select whether they wished their responses to be published with their name, published anonymously, or not published.

A paper version of the summary consultation survey (using the same questions) was provided in a leaflet with additional information including pointers to the consultation paper and web site for further details. The consultation leaflet was posted to all addresses within and adjacent to the largest proposed park area (approximately 52,000 addresses). 

The survey closed at midnight on Friday 14 February 2025. Over 5,000 responses were received across all three formats as shown in the following table.

Analysis planning 

Given the large number of responses expected, NatureScot commissioned Progressive Partnership – an analytics consultancy – to provide support and expertise during the analysis phase. The division of roles was as follows.

Consultants:

• Provide guidance on qualitative analysis approaches
• Digitise the free-text paper responses
• Format and integrate data tables for analysis
• Analyse and present results from the quantitative questions in both surveys.
• Assist with data validation
• Produce summary data tables with agreed cross breaks.

NatureScot:

• Read all text responses
• Agree protocols for dealing with any multiple responses (see below)
• Carry out qualitative analysis (coding and QA)
• Undertake GIS analysis of postcode data
• Undertake quantitative analysis as part of the iterative process of reviewing responses, developing advice, and compiling reports
• Interpret the results and produce a final report for each survey.

NatureScot assembled an internal team of 22 staff to code the responses and undertake QA checks. The team was briefed, and instructions provided, in late January 2025. 

Review and validation of responses

To help understand the extent to which the survey responses were influenced by organised campaigns, Progressive Partnership identified responses where all or part of the message was considered consistent with information published by either side of the National Park debate. These were flagged as ‘campaign responses’ and were included in the analysis. Nine such responses were identified in the technical survey, 26 in the summary survey, and ten possible campaign responses in the summary (paper) survey. 

Each response was also assessed in terms of its validity. 

• Based on contact information provided, some participants were determined to have submitted more than one response to the same survey or responded to more than one survey. There are several legitimate scenarios in which a single email address, name, or other item of contact information could be used in submission of multiple responses. For instance, a couple might share an email address but choose to submit separate individual responses, or a business address might be used to submit a response on behalf of that business as well as by the owner of that business in an individual or household capacity. While there were no measures implemented to prevent people from completing more than one of the three consultation surveys, duplicate responses to the same survey, in the same capacity, and by the same respondent were judged invalid.
• All three surveys asked participants to provide contact information, including name, postcode, and email address. Participants who provided any contact information (including only partial information) were deemed to have complied and were thus included in the analysis. No further steps were taken to verify this information. In cases where no such information was provided, the response was judged invalid and excluded from the analysis.
• A small number of responses were submitted by NatureScot and other participants to test the functionality of the survey platform (in the case of other participants these responses included text such as “Test response, please ignore”). These responses were flagged accordingly and excluded from the analysis. 

Following analysis of supplied contact information, a total of 157 responses were ultimately considered invalid as follows:

• 101 invalid duplicates (i.e. multiple submissions to the same survey, using the same contact details, or in the same capacity (e.g. resident or business))
• 51 responses where no contact information was provided (no name, email address or physical address/postcode)
• 5 'test' responses. 

Removing these responses leaves a working total of 5,230 valid responses across the three surveys. 

We are confident that we have identified and removed invalid responses. However, more sophisticated attempts to influence the survey would be very difficult for us to identify with the time and resources available.

Review of validated duplicates

Given the inclusive approach to the consultation together with the complexity of verifying contact information, the lack of barriers to multiple submissions, and the challenge of interpreting duplicate responses, we accept that the total number of validated responses will include some instances of more than one response to the consultation surveys by the same respondent. 

In reviewing the scale of this, we identified 394 responses as either possible duplicates (i.e. part of the contact information is the same, but it remains inconclusive whether this amounts to an actual duplication) or acceptable duplicates (i.e. where people have responded to multiple surveys, or where someone submitted an individual response as well as a business response). Accepted duplicates represent 7.5% of the total of 5230 validated responses.

Further analysis of these accepted duplicates was undertaken to consider their overall impact.

• The balance of opinion among these 394 responses leaned more heavily towards 'no': with 250 (63%) strongly or tending to oppose versus 134 (34%) strongly or tending to support. There were 8 undecided and 2 blank.
• Based on the email address field (or, where the email field was left blank, the name field), there were 5,087 unique participants across the three surveys (excluding test responses and those without contact information), with 2,706 (53.2%) strongly or tending to oppose versus 2,167 (42.6%) strongly or tending to support. There were 162 undecided (3.3%), 30 blank and 22 conflicting/inconsistent. (Conflicting/inconsistent indicates instances where a single participant submitted more than one response and provided different answers in more than one of these (e.g. selecting ‘Strongly oppose’ in one response and ‘Undecided’ in another).
• Of these 5,087 participants, 4,879 submitted a single response and 208 participants (4%) submitted more than one response. Of these 4,879, 2585 (53%) strongly or tended to oppose versus 2,104 (43.1%) strongly or tended to support. There were 161 (3.3%) undecided and 29 blank.
• Looking just at individual responses to the online Summary survey (which comprises the bulk of responses), we find that 51.1% were strongly or tended to oppose compared with 45.4% strongly or tended to support (3.2% were undecided and 0.3% were blank). 

Given the investigation we have done, our assessment is that the presence of valid duplicates within and across the three surveys has not significantly altered the main conclusions of the quantitative analysis given the overall number of responses received. Nor was the survey designed to be a simple poll. A stricter approach to this issue, which excluded all possible validated duplicates, may have changed the numbers very slightly. However, on balance we consider that the approach taken has significant advantages both in terms of its inclusivity and for not requiring intrusive policing of personal information.

Analysis 

Survey responses (technical, summary online, summary paper, and ‘freehand’ – where the SmartSurvey format was not followed) were analysed and reported on separately. 

Responses as individual, business, households and organisations were analysed within the relevant survey but could also be reported on across the survey formats as appropriate. This was also possible with other data such as ‘live in the area’ or ‘in full time education’. 

In order to increase participation in the consultation, we chose to limit the request for more detailed respondent information. While we can infer broad age ranges and geographic data from the questions on capacity of response and postcodes, the survey data should not be considered a complete representation of the attitudes of any particular region, sector, or population group. 

Coding

As well as closed questions which lend themselves to quantitative analysis, each survey contained a number of free-text questions (see Annex A – list of consultation questions). As Reporter, NatureScot has a duty to read all responses submitted in connection the consultation and interpret these in compiling its advice. Word counts for each survey were as follows: 

In order to read and analyse the large volumes of text provided in response to these a qualitative coding approach was developed to systematically categorise and then quantify themes in the data. This approach utilised etic coding methods, in which the researcher judges what the text relates to according to criteria agreed in discussion with others and checked as part of the QA process – as structured and reflected by the questions in the consultation survey.

In preparation for this coding work, NatureScot staff involved in the project participated in training workshops  to familiarise themselves with the principles of qualitative coding and the specific mechanics of the task in the context of the consultation process. These workshops were supported by Progressive Partnership, who have many years of experience conducting qualitative coding of public consultations. 

To help overall consistency, probable common codes and issues were identified. This included ‘no’, ‘no National Park’ and ‘no ‘no’ option’ where this was said in response to questions other than Q1 (which asks about strength of support or opposition to the proposal). 

Once the surveys had closed, raw data was exported from SmartSurvey by NatureScot and shared withProgressive Partnership using a password-protected Sharefile. Progressive Partnership developed an Excel qualitative analysis template based on their in-house coding approach. A separate template was produced for each free-text question – a simplified example is shown in Figure 1.

Survey questions were assigned to groups of NatureScot staff on the basis of expertise and familiarity with the subject matter. The number of staff assigned to each question varied according to the number and length of responses received but was never less than two. This allocation followed good practice guidance for coding as well as being practical to allow for staff absences or time constraints. A core team of three led by the research and evidence officer provided an overview and technical help as required. Staff members read through each response to their assigned questions and identified codes and themes present in these responses, entering these into the templates provided by Progressive Partnership. 

A QA system was put in place to review the coding progress at key stages. A QA-lead was assigned to each question, drawing on more experienced staff members. The QA-leads served as important focal points for coordinating the coding work and resolving or escalating queries. At particular milestones throughout the coding process, and again once each question had been completed, QA-leads reviewed a sample of the coded responses for accuracy and consistency.

Quantitative analysis

Once coding of all questions was complete, the analysis templates were returned to Progressive Partnership who compiled summary data tables, including cross-breaks with other quantitative questions using the statistical program Snap. NatureScot staff were also able to conduct their own analysis of the qualitative and quantitative data using Excel and produce tabular outputs for use in reports and discussion of advice. 

At this stage, validation of the data took also place using contact information provided (see above). 

Although the surveys varied length and wording, a combined dataset was produced using those questions they held in common to facilitate comparison across key questions (such as area preference and support for the proposal). 

Location data 

Postcode data was used to provide additional insights, including the distribution of support and opposition, the influence of respondent location on option area preferences, and understanding the SIMD status of survey participants. This data required some cleaning prior to analysis, such as by changing ‘O’ to ‘0’ where necessary, and standardising length and spacing. 

In total 4,945 responses were successfully located using the supplied postcode. Of those that could not be located, some provided a non-locatable postcode (because of a typo in the postcode or the use of an incorrect code), some supplied a non-UK or non-standard postcode, and a proportion gave an incomplete or no postcode at all (although they may have provided other contact information).

Post-analysis

To inform the development of the advice, individual reports were prepared of the results of the analysis of each survey. As well as quantitative data, these reports include a short assessment of the main points made and a selection of quotes that illustrate the range of views expressed. Each of these individual reports have been published separately.

Annex A – list of consultation questions