Indicators and methods

The databases of the Science and Technology Observatory (OST) are used to calculate indicators for specific analyses, such as the Report on the Scientific Position of France, or annual synthesisreports for the “Higher Education and Research in France, Facts and Figures” published by the Ministry for Higher Education, Research and Innovation (MESRI).
The indicators produced by the OST serve a variety of purposes. They are used, for example, to describe the scientific position of France in the world. They may also be used by institutions to elaborate their strategy or feed their self-evaluation process. Finally, they inform a variety of documents produced by the Ministry for Higher Education, Research and Innovation.
OST also works on an ongoing basis to adapt its methods and the databases it uses in line with international best practice.

Share of world publications

A country’s share of world publications is the ratio between the number of publications produced by that country and the number of publications produced worldwide. The publications of a country are those where at least one of the research laboratories that authored the publication is located in that country.

For various bibliometric indicators, either full or fractional countingmay be used. The full counting method gives a full weight of one to each publication of a country. The fractional counting method gives less weight to collaborative publications:For instance, if a publication has been co-authored by five researchers and two of these researchers are affiliated in country A, the publication has a weight of 2 / 5 = 0.4 in the calculation of the indicators for country A.

Impact index

Citations provide a means of inferring the relevance of a given document to the scientific community as documents are subsequently published.

The normalised impact measure at the country level is calculated as the ratio between the average number of citations received by the documents published by authors in a given country and the world’s citation average, over the same time period and for the same document type and subject area. The values show the relationship of the country's average impact to the  world average, which is 1, i.e. an index of 0.8 means that the document cited is 20% less than the world average and 1.3 means that the document is cited is 30% more than the world average.

A country’s expected impact index is the impact index that the country would have if its publications were cited as much as the average for publications in the journals in which it publishes (thus taking into account the reputation of those journals).

Specialisation index 

The specialisation Index of a country in a discipline is calculated by dividing that discipline share of publications within a given country by the world share of that particular discipline . For a country with  very  similar  distribution  by  discplines  to  the  world,  specialisation values  should  be  very  close  to  1.  A  value of 2 for a given discipline and country indicates that the weight of that discipline in the country is two times as large as for the entire world.

World share of all patent applications observed for a given patent office

A country’s share of all patent applications for a given office (for example, the European Patent Office - EPO) is the ratio between the number of patent applications from that country and the total number of patent applications recorded for that Office.

Most of the indicators are based on the date of publication of the patent applications (18 months after initial filing for the EPO, but up to 3 to 5 years for the United States Office) and the address of the inventor.

Technological specialisation index

As with publications, a country’s technological specialisation index is the ratio between the country's world share in a field of technology and the country's world share in all fields combined

  • More about the “patents database”  

Other indicators

Where applicable, other indicators may be calculated (citation classes, co-publication indicators, etc.).

For further information, see:

  • Methodology appendix to the report on “The Scientific Position of France in the World 2000-2015”, Hcéres-OST (March 2018) 
  • Higher Education and Research in France, Facts and Figures reports  

IPERU Programme

“University Research Institutions Output Indicators” (IPERU) are provided to the institutions in the “Higher Education and University Research” programme of the French Organic Law on Budget Acts (LOLF). The IPERU programme contributes to:

  • the management of these institutions (universities, engineering schools, Ecoles Normales Supérieures, etc.),
  • the production of indicators for the budget documents prepared by the Ministry for Higher Education, Research and Innovation (MESRI).

The IPERU indicators are used to describe the scientific and technological output of the institutions, monitor their development and assess their positioning in reference geographical areas. They are based on three data sources :

  • publications,
  • patents,
  • participation in EU Framework Programme for Research and Innovation projects.

Individual indicator reports are provided to the institutions on an annual basis. A report on the relative position of of higher education and research institutions in worldwide publications is also provided.

Indicators for the French national budget process

Pursuant to the French Organic Law on Budget Acts (LOLF), the State budget is organised into missions, programmes and actions to monitor implementation of the major public policies. Each programme has its strategy, objectives and performance indicators. The latter provide a measurement of target achievement in figures.

Since 2005, the OST has produced part of the budget document indicators (Annual Performance Projects – “PAP” - and Annual Performance Reports – “RAP”) for several programmes of the Interministerial Mission for Research and Higher Education (“MIRES”) and for the General Commission for Sustainable Development (“CGDD”) of the Ministry for Ecological Transition and Solidarity.

These indicators concern:

  • The scientific publications of the operators in the various programmes. They are listed in the OST database developed from Web of Science® by Clarivate Analytics (reference database for international publications);
  • Participation in the construction of the European Research Area by measuring participation in the projects of the Framework Programmes for Research and Innovation (FP) of the European Union.

Operators with varying sizes and missions

The indicators concern the following LOLF programmes:

  • Programme P150: Higher Education and University Research: higher education institutions under the supervision of the Ministry for Higher Education, Research and Innovation (MESRI)
  • Programme P172: Multidisciplinary Scientific and Technological Research: CEA (civil), CNRS, Inria, Inserm, Institut Curie, Instituts Pasteur Lille and Paris, BRGM, Irstea, Cirad, Ifremer, Inra, IRD
  • Programme P190: Research in the Areas of Sustainable Energy, Development and Mobility: CSTB, Ineris, IRSN, IFSTTAR, IFP Energies nouvelles
  • Programme P159: Geographic and Meteorological Expertise and Information: IGN and Météo France
  • Programme P193: Space research (space missions and CNES)

Methodology

For publications, the operators identify the addresses of their publications directly. The OST then performs the consolidation of the “meta-institution” formed by the concerned LOLF programme.

In the specific case of P193, the scope is based on the constitution of a body of publications using a list of keywords and cleaning that body by topic modelling, with the themes being validated by experts in space research.

For the FP for Research and Innovation, the OST identifies the participants in the Framework Programmes by their name and then also performs the consolidation of the “meta-institution” formed by the LOLF programme in question.

Indicators

The approach to performance is translated into the following indicators:

  • production measured by scientific publications, 
  • impact measured by citations, 
  • collaboration measured by co-publications with the 28 EU countries or the countries of the southern hemisphere, 
  • participation in the construction of the European Research Area measured by participations and coordinations under the FP for Research and Innovation. 

Depending on the programme, the indicators that are calculated may vary slightly.

The OST therefore provides: 

  • for the Interministerial Mission for Research and Higher Education (MIRES): basic indicators on scientific production and participation in the FP for Research and Innovation. for the Report on National Research and Higher Education Policies (“Jaune ESR”):  indicators on the scientific and technological production of France and on participations in the FP for Research and Innovation of France and its main partners (Germany, United Kingdom, Spain and Italy). 
  • View the public finance reference information website 

Territorial Strategy Reports on Higher Education and Research (STRATER)  

The STRATER project (regional diagnoses) headed by the Ministry for Higher Education, Research and Innovation (MESRI) offers a regional view of higher education, research and innovation. The Ministry disseminates regional booklets for the attention of all the different stakeholders and the OST provides the indicators on research and innovation activity for them.

The indicators are produced on the basis of three bodies of data:

  • Scientific publications listed in the OST database developed from Web of Science® by Clarivate Analytics (reference database for international publications) ;
  • European patent applications identified in the patents database of the OST, developed on the basis of the PATSTAT database ;
  • Participations in the FP for Research and Innovation. 

A geographical and institutional application that can be adapted to MESRI requirements

The geographical and institutional classification is updated in accordance with the specifications of the Ministry and, depending on the year, may correspond to regions, regional education authorities or, in a certain number of cases, “Communautés d’universités et établissements” in the Paris region.

The OST compiles tables and graphs of indicators that are then used in the regional leaflets, basing itself on: