Citation analysis and bibliometrics

Responsible metrics - Choose your indicators with care. Don't make inappropriate comparisons.

Bibliometrics can be defined as the statistical analysis of publications. Traditionally, bibliometrics has focused on the quantitative analysis of citations and citation counts.

Such citation analysis can be useful to researchers in:

  • Helping to identify and prioritise publications to read
  • Informing the choice of targets for planned publications
  • Contributing to the demonstration of academic impact
  • Locating potential collaborators

Institutions can also benefit from the use of bibliometrics. Bibliometric analysis can be used to help identify an institution's research strengths, to benchmark its performance, and to inform research strategy development. The use of bibliometrics in the assessment of research performance is, however, not without its controversies. See below for more on the Limitations of bibliometric analysis.

Citation-based metrics

Article level metrics

Citation counts are sometimes used as an indicator of academic impact in the sense that citations from other publications suggest that the cited work has influenced the citing work in some way. Information on how to perform citation searches using Web of Science and other databases is available on the Searching the literature citation searches page.

Citation rates vary widely across disciplines. If you wish to compare citation counts from different fields you should use "normalised" or "field-weighted" citation metrics. Scopus provides field-weighted citation impact as one of its article metrics. It also provides Citation benchmarking data - this shows how citations received by an article compare with the average for similar articles. See the Scopus Article Metrics help page for more information.

Journal level metrics

It is also possible to use bibliometrics to calculate the impact factors of journal titles. This can help you target highly cited journals for your own publications.

The Journal impact factor is the most well known indicator but others are now available which attempt to take account of variations between subject areas and time periods.

  • You can use Journal Citation Reports (JCR) to get a list of the top ranked journals in your field or check an individual journal to see its impact and rank.
  • Scopus does not provide ranked lists of journals in a particular discipline, unlike JCR, but it has a good tool Compare Journals for selecting up to 10 journals and analysing a variety of citation parameters, including Impact per Publication (IPP) and Source Normalized Impact per Paper (SNIP).
  • SCImago, a free website, uses Scopus data to provide ranked listings of journals comparable to JCR.
  • Journal Metrics, also freely available and based on Scopus data, provides Impact per Publication (IPP), Source Normalized Impact per Paper (SNIP) and SCImago Journal Rank (SJR).
  • Google Scholar Metrics can be browsed to provide lists of journals by subject area ranked by their h5-index. Alternatively, it is possible to search for the h5-index of a specific journal.

Unfortunately, finding top ranked journals in multidisciplinary areas can be problematic. It can also be difficult to establish an impact factor for new journals.

Author (or group) level metrics

The h-index developed in 2005 by Professor Hirsch was designed to be a simple metric with which to quantify the output of an individual researcher. A researcher with an index of n has published n papers, each of which has been cited n times. For example if you have published 10 papers that have received at least 10 citations each then your h-index is 10. Web of Science, Scopus and Google Scholar can all be used to calculate your h-index.

Take care if you are using the h-index to make comparisons. The h-index is only meaningful when compared to others in the same discipline. As with all indicators, the h-index should only ever be used alongside other forms of evaluation in performance assessment.

You, or your research group, may also wish to compare your citation performance against that of other researchers or the normal for your field. It is also possible to undertake an analysis of the proportion of papers published which are amongst the most highly cited (e.g. the top 10%) in that field. See SciVal for more information.

SciVal and other tools

SciVal

All member of the University have access to SciVal, a subscription-based research performance assessment tool which uses data from Scopus. SciVal provides more advanced bibliometric analysis than that available from sources such as Scopus and Web of Science. SciVal also allows you to benchmark individual researchers, groups of researchers and institutions based on a variety of different metrics, including outputs in the top percentiles.

For more information, see the SciVal guidebook.

Access notes: SciVal is available both on and off campus:

At the SciVal Login screen you will be asked for an Elsevier username and password. If you have created a username and password for ScienceDirect or Scopus you can use those credentials to access SciVal. If you do not have an Elsevier username and password then choose Register Now.

Other tools

A range of other sources and tools are available including:

  • Essential Science Indicators - can be used to identify the influential individuals, institutions, papers, publications, and countries in a field of study.
  • Google Scholar Citations - allows authors to create public or private profiles, list publications and tracks citations via Google Scholar. Indexes recent materials not offered in ISI including both books and journal articles.
  • Publish or Perish software - has been developed in recent years by Professor Anne-Wil Harzing of Melbourne University, and is based on Google Scholar data. It has been particularly popular with social science researchers. You have to download the free software and install it on your desktop.
  • Highly Cited Researchers - a freely available list of authors based on data from Essential Science Indicators.

Limitations of bibliometrics

Bibliometrics and performance assessment

The use of bibliometric analysis is widespread within the academic community. The benefits of using bibliometric data to help assess research performance include the following:

  • It can be seen as a fair and 'objective' method (rather than relying solely on qualitative measures such as peer-review) in some disciplines
  • It may be considered cost-effective (as some data is readily available)
  • It is relatively transparent (as compared to some forms of peer review).

It is increasingly recognised, however, that bibliometrics should be used in combination with other qualitative measures when being used to assess quality. The Report of the Independent Review if the Role of Metrics in Research Assessment and Management has called for the research community to “develop a more sophisticated and nuanced approach to the contribution and limitations of quantitative indicators”. The Report also “proposes the notion of responsible metrics”.

The San Francisco Declaration on Research Assessment (DORA) also calls for improvements in the evaluation of research. It is particularly critical of the use of journal metrics, such as Journal Impact Factor, as a surrogate measure of the quality of research articles in assessing individuals contributions.

Discipline Variation

  • Citations patterns differ greatly between disciplines so direct comparisons cannot be made.
  • Bibliometrics predominantly focuses on journal article citations, but some disciplines such as the arts, humanities and social sciences publish research in different types of publication.
  • Different fields of research publish at different rates. For example, in biomedicine, there is generally a much stronger culture of publishing in journals and citing the work of peers than in engineering which makes more use of conference papers.

Bias and Discrepancies

  • Quality vs. quantity: The number of times a work has been cited should not be used to gauge the quality of the work, it really only measures the interest of other researchers in the work as recorded in the cited references.
  • Controversial papers, such as those based on fraudulent data, may be highly cited. A work may be heavily cited because other authors are refuting its research.
  • Citation bias. People may inappropriately cite their own work, their colleagues, or work from the journals in which they publish. A number of bibliometric tools allow you to exclude self-citations.
  • Experienced researchers have an advantage over early career researchers as they will have produced more outputs over a period of time and so will have more citations.
  • There is a bias towards English language material - this reflects the content of the main citation tools.
  • Time is needed before a meaningful citation analysis can be made, so new journals tend to fare badly.
  • Bibliographic tools cannot always reliably differentiate between researchers who share the same surname and initials, meaning that citation counts may not be accurate. Author identifiers such as ORCID iDs are important (for more on ORCID iDs see www.york.ac.uk/orcid).
  • Time-span: on average a publication may reach its citation count peak within the first two years following publication. The timespan chosen for a citation report may skew the results.
  • Only a small percentage of articles are highly cited and they are found in a small subset of journals. This small proportion accounts for a large percentage of citations.
  • Publication exclusion: only research articles, technical notes and reviews are "citable" items. Editorials, letters, news items and meeting abstracts are "non-citable items".
  • Review articles: authors and journals that frequently publish review articles tend to have their citation counts exaggerated because these types of articles are usually highly cited
  • Non-cited articles: citation counting does not take into account articles that were used but did not get cited.

Altmetrics

Alternative metrics

Altmetrics (or Alternative metrics) have been developed to complement traditional metrics based on citations within academic publications. Altmetrics have been devised to collect evidence of the societal impact of research in terms of its mentions in social media (blogs and Twitter), news reports, and in policy documents.

You will see that many publishers now incorporate altmetrics within their own websites and databases. Altmetric badges (or “doughnuts”), for example, are now embedded within many sources and you can also download their Altmetric Bookmarklet your own use. Scopus uses Altmetric data within its Article level metrics to display information on Mendeley and Twitter counts, as well as other indicators of what it terms “Engagement highlights”.

Help

Sources of help

The Library’s Research Support Team provide general guidance on the use of bibliometrics for individual researchers and postgraduate research students. Email: lib-research-support@york.ac.uk.

Contact the Research Strategy and Policy Office (RSPO) if you are interested in the strategic use of bibliometrics at an institutional or departmental level.