The How and the Why of Collecting Identity Data: Part 4

Collecting and Evaluating Identity Data

By Kristen Overstreet and Sherrie Hill 

November 18, 2021 

How to collect identity data

In this post, of a multipart series on identity data, we will discuss considerations for when you begin collecting and interpreting the data.  Some organizations choose to collect identity data through independent software applications (e.g., SurveyMonkey), while others use their journals’ submission systems.  There are pros and cons to either.

At the 2021 ISMTE Global Event in October, Dr. Teo Pulvirenti, from the American Chemical Society, discussed these pros and cons. Collecting data through an independent software application ensures that there will be limited access to the raw data collected, since you can easily control who within your organization will have access.  Because the data are not stored within the submission system, reviewers and authors might feel more comfortable revealing sensitive identity data. However, because it is a separate application, it does represent an additional process that must be completed by people who are typically very busy.  Therefore, your journal might find that the response rate is lower than it would be for questions asked within the submission system.

Some submission system vendors are working on better ways to collect identity information, including customizable fields and set lists of predetermined questions and response choices that the journal can use to build a survey. Journals can choose to collect identity information during the manuscript submission process or through user account creation, according to Anna Jester with eJournalPress, who also spoke at the ISMTE Global Event.  If the information is collected during the submission process, the submitting authors can only be asked to answer questions about themselves.  The submitting authors should never be asked to answer any questions relating to gender, sexuality, race, ethnicity, disability etc. on behalf of their co-authors.  Access to identity data stored within the submission system can be limited to a small group of individuals through role management permissions within the submission system.  It is critical that the number of journal personnel who have access to this identity information be kept as small as is reasonable.

Though there is an assumption that there will be a higher response rate to identity surveys conducted within a submission system, some people may have more significant privacy concerns when asked to enter this personal information in their user accounts or during manuscript submission.  According to Einhorn, Pollock, and Paolini, in a recent Scholarly Kitchen post, people may be reluctant to reveal their private identity information for fear of backlash, harm, bias, or misuse of the information.  They may also be concerned with being identified as belonging to a group that the journal wishes to include more fully and therefore become overburdened with requests from the journal to submit or review manuscripts, or they may simply want to protect their personal information.

Anna Jester also discussed in her presentation the need to provide clear information to the survey participants regarding what will and won’t be done with the data, including what roles within the organization will or won’t have access to these data.  For example, survey results will only be reported as the aggregate, and editors and reviewers will not have access to an individual’s identity information. It is also important to have documented data storage policies and procedures that include how the data will be kept confidential and any plans for updating the information, as well as the security measures used to keep the data secured.

 Initiating your survey and evaluating the data

After completing the survey design and approval process, your team can finally initiate the survey.

Try to have realistic expectations for your survey results.  When the American Chemical Society sent out their first identity survey to corresponding authors and reviewers (not using their submission system), they kept the survey open for 7 months and only had a 5% response rate.  However, when they surveyed their editors and editorial board members, about 29% of their editors responded and approximately 36% of the editorial board members responded during a 3-month period.  After all the hard work you put into creating a well-designed survey, you are likely hoping for a high response rate.  So, it can be very disappointing to get a low rate of return.  Just remember that this is not uncommon and may improve over time. People associated with your journal will become more familiar with your process of collecting identity data, as well as gain a deeper understanding of what your journal is trying to achieve by collecting it.

When reporting the aggregate identity data results, include the “did not respond” data so viewers can more fully understand how the data collected relate to the whole of the population who were queried.  Your journal should expect some inherent biases within the collected data.  For instance, do older individuals interpret their race, ethnicity, gender, or sexuality in the same way as younger individuals?  Are members of underrepresented groups more likely or less likely to fill out the survey? Are people further along in their career more or less likely to complete the survey than those who are just beginning their career?

Think of your first identity survey as the first step in a process of continuous improvement regarding your journal’s strategies.  The initial data that you collect will ideally allow your journal to develop a baseline that can be used to develop more specific goals for your journal. Your survey and your journal will evolve over time. Putting the needed time and effort into setting your goals, designing your survey, collecting your data, and properly reporting and storing your data will ensure you are on the right path toward meeting your initial goals of collecting accurate information, respecting all people, and data privacy in your efforts to ensure diversity, equity, and inclusion at your journals.