From The Ground Up
The Publication of Archaeological Projects
a user needs survey
Appendix 3.2: Report on the pilot survey for the questionnaire
The aim of the pilot test was to establish:
The pilot sample
The pre-test questionnaire was sent out to representatives of all core constituencies The total number individuals sampled in the pilot survey varied, but a minimum of three individuals for each were selected (in some cases up to 14). The following numbers were returned (based on primary constituency):
Curators (national 8; regional 6): 14
Museum archaeologists: 2
Independents (active amateurs): 2
Members of local/regional societies: 2
University/college staff: 7
Cognate disciplines: 1
Other (heritage managers/facilitators): 3
These figures relate to primary constituency, but almost all the respondents identified with between two and four constituencies, most were members of national or regional societies and many had two employment/research-related constituencies.
Time taken to complete the questionnaire: This varied from 30 minutes to 2 hours, but most people took around 45 minutes to 1 hour. Comments on the amount of time people had to devote to the exercise ranged from those who said that they had no problem devoting time to it (mainly those who took around 30-60 minutes), and those who argued that it is much too long. Those who felt it to be too long outweighed those who were happy with its length, so steps have been taken to cut out questions which are considered superfluous to the main aim of the questionnaire in order to ensure a better response rate (see question by question below).
Clarity of questions and appropriateness: Most questions appeared to be pretty well understood. Problems arose with specific terms that are used regularly to characterise modes of behaviour or types/elements of project archaeological publication:
The only other problems arose when respondents made there own judgement about the broader agenda behind a particular question. So, for instance, a number of people answered negatively to 4.1 which asked whether they would like to see rapid summary publication, because they assumed, or were worried that, this would be used as a way to avoid further publication. Likewise with 5.6 where respondents were asked whether they would like to see widespread use of WWW and CD-ROM for publishing in various spheres, concern was expressed about whether this would be 'as well as', or instead of conventional publication and some people refused to answer because this was ambiguous. If consortium members had the same attitudes towards these issues and the same agendas it would be possible to ask people to respond to specific policy proposals. However, as this is not the case, I have tried to make the questions more general and more 'neutral'. Doubtless some respondents will still be influenced by former proposals which have been aired, but they will always answer with relation to the existing situation and their expectations about future developments.
Closed response categories: Again, these appeared to be generally appropriate with a few additional categories emerging through respondents' use of the 'other' category where they were asked to specify an alternative. Problems were more common in Section 4 where interviewees are asked to address complicated questions relating the principles of archaeological publication. As with the questions themselves, the main problems lie with certain terms that are regularly used as response categories which respondents obviously perceive in different ways and are too vague at present and cause confusion/differential responses:
These relative response categories aside the only other problem which emerged was that in certain cases where a single response out of many had been requested, respondents needed to use several to characterise their behaviour/needs/attitudes. These questions have been changed to multiple response questions where appropriate (see question by question below).
Open questions: These generally elicited a poor response rate, although many people stated that this was due to the length of time required by the overall questionnaire, so we may get an improved response after the questionnaire has been cut down. An open question asking for further comments will probably be more useful at the end of the two key sections (3 & 4) as well as at the very end of the questionnaire. The poor use of open sections following specific questions asking respondents to give reasons for their answer is particularly problematic in some instances, as the data collected will be of little value without further explanatory information, for instance as in the case of 4.6 - 4.8.
Success of the questionnaire: Generally the questionnaire appears successful in obtaining responses (c 60% response rate), although it was tested on groups of individuals who would be better disposed to responding than a general cross-section of the discipline (eg individuals working for our funding bodies, colleagues and friends).
In terms of acquiring the information which we need, it has been noted by pilot test respondents that their patterns of use and need are more complex than the questionnaire accommodates. This has been addressed in places by requesting more detailed information on patterns of use, but respondents already find the detail they are being asked for taxing so there has to be a compromise between achieving a good picture of the complexity of behaviour and needs, and the tendency towards irritation/non-response, which requests for such detailed information initiate.
Another complicating factor lies in the assessment of these complex patterns of use and need even if the information can be obtained. It is clear from the pilot survey that there are many factors which may lead people to use archaeological publications in different ways, and to have different needs and expectations of them. The most important of these are constituency; philosophical/theoretical position; age. Many individuals have two or three constituencies and even though they are asked to define a primary constituency (relating to employment if employed in archaeology) it is not clear which constituencies are the most important in determining responses in particular questions.
Finally, there is a general conflict between personal needs and behaviour and people's stance with relation to the overall principles of publication, which was noted by a number of respondents. Sections 2 & 3 should relate to respondent behaviour and needs. Section 4 to respondent attitudes towards the principles of project publication in general. This has been clarified using an introductory sentence at the beginning of each section.
General response to the questionnaire:
This ranged from very positive, in the case of some respondents who stated that they 'enjoyed the experience' and/or that they thought it would be very useful, to very negative, in the case of others who 'could not see the point'. Other's inevitably expressed the view that specific policy agendas underlie the questionnaire, for instance:
These views are more a result of individual concerns and prejudices, and the impact of pre-existing publication proposals than the subjects addressed and the phrasing used in the questionnaire. This is clearly evident in that some pilot test respondents expressed concern that there was 'a fix' on for electronic publication, whilst others suggested that we could get a negative response to electronic forms of publication because many people do not yet have full access, but that this should not be taken as a reason for abandoning that option. Likewise more generally, some respondents felt that there was a desire for change (for a minority, coupled with a dangerous radicalism) underlying the survey, whereas others commented that it was rather conservative (for a minority, it can only reinforce the existing status quo). This range of perceptions is only to be expected in a discipline characterised by diverse standpoints and interests, and it would be best to treat them as a source of information for analysis and recognition in the report. They may raise problems for the survey's role as a positive PR exercise, but I do not think they can be avoided, and vehement responses do appear to represent minority groups.
PUNS report index page