How Prosper did evaluation

Each of Prosper’s two pilot cohorts were 12 months in duration. Part of our cohort participant agreement was to commit to provide feedback. Entry, exit and pulse survey data were all attributable, rather than anonymous.

For full details on the participant agreement, see our how to recruit a cohort page.

The main surveys issued to the cohort were;

Entry survey

The entry survey provided a baseline from which to measure the impact of Prosper on the cohort members compared to the exit (or early leavers) survey. Completing the entry survey and the participants’ agreement was compulsory to participants entering the cohort. The cohort members had two weeks to complete the survey and we sent a maximum of four follow-up prompt emails.

The entry survey recorded the EDI details of the participants such as gender, sexuality, caring responsibilities, ethnicity, disciplinary backgrounds, and years of experience as a postdoc. In addition, this survey traced the attitudes towards career development, opportunities beyond academia and the postdocs’ confidence to self-direct their own career development. The entry survey also included open ended questions which included space for postdocs to set their own goals for the first phase (approximately first three months) of their Prosper pilot. Including open-ended questions in the surveys provided a wealth of data which added depth to the quantitative findings of the surveys and helped us to cohesively and comprehensively present the impact of Prosper.

Pulse surveys

The Prosper pulse surveys were shorter than the comprehensive entry and exit surveys, and were primarily geared towards tracking engagement and feedback on the content and sessions. Feedback was collected on the time the participants spent on Prosper (including the time spent on different resources) and the experience of live delivered sessions. A descriptive analysis (of averages and percentages) of the pulse survey data was used by the Prosper team to define topics for cohort focus groups and modify or develop additional resources. In each 12-month pilot cohort, we issued two pulse surveys. The first pulse survey broadly marked the end of the Reflect phase (after around three months) and the second pulse survey broadly marked the conclusion of the Explore phase (after around six months). The Act phase was covered in the exit survey.

The first Prosper pulse survey enquired if the postdocs felt that they have progressed in the career development goals they set in the entry survey. Open ended questions were included in the pulse surveys to ask the postdocs to set goals for the next phase (~ three-month period). This pattern of asking postdocs to consider their progress towards their goals and setting new goals was followed in the second pulse survey.

As these pulse surveys were issued in between the entry and exit surveys we aimed for them to be completed by the cohort within two weeks of being issued. We sent up to a maximum of four reminder emails to ensure we achieved a minimum of 80% of the cohort completing the survey within a month of issue. Reminders were also posted in the cohort instant messaging (Slack) chat. However, we found we typically had to send several reminders to the postdocs to ensure that a minimum of 80% of the cohort completed the survey within a month.

The exit survey recorded the impact of Prosper career development on postdocs as they concluded their engagement with the cohort. During both pilot cohorts the Prosper team developed two surveys; an early leavers survey completed by postdocs who left the cohort before the end due to varied reasons (such as end of their postdoc contract, and new jobs within and beyond academia) and an exit survey which was completed by postdocs who engaged with Prosper through to the end of the cohort (full 12 months).

Both the exit and early leavers surveys were very similar to each other and largely asked the same questions as the entry survey so the data could be directly compared. The questions and the order of the questions in the surveys was kept the same as the entry survey as far as possible. The exit and early leavers surveys included additional questions on the postdocs experience of applying for jobs within and/or beyond academia, their future plans and overall feedback on their experience of Prosper career development.

We found getting good response rates to the exit and early leavers surveys could be problematic as the postdocs may have left the institution, changed email address, or started a new role limiting their time and availability. Most early leavers completed the survey within one week. We ran the end of cohort surveys for two months with a maximum of four reminders being sent during this period.

The way in which the quantitative data collected through surveys is analysed is determined by the evaluation design, the nature of the questions and data collected through the surveys.

Since we collected a combination of quantitative data (through surveys) and qualitative data (from surveys and focus group discussions), we used different strategies to analyse the different kinds of data.

Quantitative analysis

For the quantitative data we did a combination of descriptive analysis (proportions, percentages and averages) for the majority of responses. We also used a more complex combination of factor analysis and analysis of variance (ANOVA) to measure changes in confidence.

Factor analysis: is a statistical analysis technique used to reduce large sets of correlated variables (data units or responses to questions) into a smaller number of groups or factors. This statistical tool can be executed on SPSS (Statistical Package for Social Sciences; a data management and analysis software developed by IBM) to group variables together into a single variable or factor. Grouping variables together in this way makes it easier to understand observed shifts. In our study we were able to condense 18 variables into 5 factors, which made it easier for us to understand and present our data.

ANOVA: is a statistical test which can be executed on SPSS and on Microsoft excel to examine the relationship between two or more groups of data and to test if our findings are statistically significant. In essence ANOVA allows you to test if there is a meaningful difference between your groups of data or not.

Qualitative analysis

We thematically analysed the qualitative data collected across journal entries, focus group discussions and postdoc responses to open ended survey questions. This involved identifying how these qualitative data or texts responded to different themes. We also analysed this data to identify emergent themes. For instance, when we examined postdoc responses to impediments to engagement, we identified struggles with journaling, time management and a lack of a sense of belonging. These are themes which emerged from a particular open-ended question in a survey.

Additionally, we found that some survey questions generated responses which lent themselves to attractive presentation in the form of word-clouds. Questions we found particularly useful in this way were; asking postdocs to identify the most useful resource, what the primary limitation to their engagement with their Prosper career development was and what three words would they use to describe their experience of career development.

A sample report based on this evaluation for cohort 1 can be found here.

We would recommend that as a minimum a descriptive analysis of the quantitative evaluation data should be performed. This can be easily done using software such as Microsoft excel to calculate proportions, averages or percentages. Additionally, themes can be extracted from any qualitative data collected. We have prioritised the analysis and presentation of quantitative analysis in our reports. We have used our qualitative analysis to complement the quantitative analysis. You can combine the results and analysis of the results in your report or however you wish to disseminate your evaluation results.