Experts' evaluation
a) Questionnaire - Weight elicitation
The FEEM SI 2011 derives all the weights (measures) starting from a survey of experts' evaluations, implemented using the QUALTRIX software. The questionnaire elicits individual preferences and their valuation on the specific performance of the three main components of sustainability.

To find an example of the questionnaire click here

Each indicator can assume two states of performance: “BAD” and “GOOD”, which have to be interpreted as extreme conditions. With reference to each nest of the indicators’ tree, all possible combinations of these two states are presented to the questionnaire respondent, which must provide an evaluation respecting a monotonicity criterion.

b) Experts' profile
The preference elicitation obtained with the questionnaire allows capturing a broader view on sustainability throughout the world. The questionnaire respondents are experts from Belgium, France, Germany, Israel, Italy, Japan, Spain, Switzerland, UK and USA. The chart offers the percentage of country representation. Moreover, 40% of the experts are affiliated in academia, another 40% in international organisations and the remaining 20% of the experts are involved with think tank organisations. The goal for future FEEM SI issues is to increase the variety of respondents involving experts from other countries, namely developing ones.

c) Aggregation of experts’ evaluations
A consensus measure among decision makers is considered to derive a ‘representative’ weight assigned to each sustainability indicator. For this purpose, the “metric distance” measure is used to assign weights to valuations of each expert at each node in the decision tree. If the valuation of an expert is in agreement with others (namely, having lower distance measure), then it receives higher weight. Conversely, if a expert’s valuation is extremely different from other valuations (namely, having a higher distance measure), then the weight is lower. Therefore, a ‘consensus’ weight for each sustainability indicator is obtained where the majority of experts are in agreement.

Choquet methodology for aggregation
To assess the degree of interaction across indicators’ performance (or criterion), two extreme levels have been defined for each indicator (the best and worst performance) in the questionnaire. Next, a weight (or measure) is assigned not only to a single criterion, but also to any coalitions of criteria for each node in the decision tree. In doing so, the weight of two indicators having “best” performances is not necessarily their weighted sum, but can be greater (in the case of positive interaction) or lower (in the case of negative interaction) (please see detailed explanation in questionnaire-weight elicitation section). A suitable algorithm based on the Choquet integral aggregates all criteria into a single outcome taking into account all the coalition weights.

Interaction among criteria can be measured by the tendency of the experts’ preference toward “conservative” or “optimistic” behaviour. A “conservative” decision maker prefers that all (or many of) the criteria are fulfilled in order to give a positive evaluation, while an “optimistic” one is satisfied if an excellent performance is observed in at least one criterion independently on the level of the other criteria. This kind of behaviour is a characteristic of the set of weights and can be summarised into a numerical index (level of orness).

Shapley values
The ‘representative’ expert assigns a weight (or measure) not only to a single criterion, but also to any coalitions of criteria for each node in the decision tree. One can assess the relative importance of any indicator by summing its marginal contributions to the node (Shapley value): namely, the difference of coalition weights with and without the indicator. Moreover, each indicator’s contribution to the overall index can be calculated by multiplying the Shapley values at all nodes located above the indicator.

Node	Criterion	Shapley Value
FFEM SI	Economic	0.326
	Social	0.316
	Environment	0.358
Economic	Growth Drivers	0.379
	GDP Per Capita	0.346
	Exposure	0.275
Social	Population Density	0.250
	Well Being	0.406
	Vulnerability	0.344
Environment	Air Pollution	0.350
	Energy	0.330
	Natural Endowment	0.320
Growth Drivers	R&D	0.514
	Investment	0.486
Exposure	Relative Trade Balance	0.543
	National Debt	0.457
Well Being	Education	0.502
	Health	0.498
Vulnerability	Food Relevance	0.383
	Energy Security	0.283
	Private Health	0.333
Energy Security	Energy Imported	0.500
	Energy Access	0.500
Air Pollution	GHG per capita	0.508
	CO2 intensity	0.492
Energy	Energy intensity	0.477
	Renewables	0.523
Natural Endowment	Biodiversity	0.446
	Water	0.554
Biodiversity	Animals	0.504
	Plants	0.496

Sensitivity analysis
The subjective weights provided by the experts are crucial in the aggregation methodology of the FEEM SI; therefore, a sensitivity analysis has been carried out to assess the effects of variations among different degrees of “optimism” of experts. Since each country’s sustainability outcome differs depending on expert’s valuations, we produce the interval of sustainability outcomes - mean, maximum and minimum - for all countries and for each year

Click here to know more about the FEEM SI aggregation procedure and sensitivity analysis.

Detailed sensitivity analysis results are available in the Results section