EAGER: Metrics to Evaluate Customer Preference Models for use in Engineering Design Optimization

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

Engineers have begun to use customer preference models developed in marketing, economics, and psychology to design products that better meet customer desires. While such models are adequate for marketing purposes, they often introduce significant errors when used as-is in an engineering design context. Despite the prevalent use of customer preference models by many engineering researchers, the field does not have adequate methods for evaluating the accuracy or appropriateness of demand models for use in this context. This EArly-concept Grant for Exploratory Research (EAGER) award provides support for fundamental research to develop metrics and a test procedure to evaluate various errors associated with customer preference models that can mislead engineering designers. Results will allow engineering designers to construct demand models using estimation methods that minimize errors in design selection and optimization. The metrics developed through this work will also allow practitioners to evaluate demand models and select the most appropriate model for their design problem. In addition, several education and dissemination activities will be conducted to improve student learning of model evaluation techniques and facilitate use of the evaluation methods by federal agencies that employ demand models to inform their funding and regulation of technology development in the transportation sector. The research objectives are to produce (1) engineering-design specific metrics that will evaluate the estimation biases associated with demand models, (2) a demonstration of the significance of demand-model estimation biases on optimal design variable selection, and (3) identification of one or more demand estimation methods that reduce biases affecting design selection and optimization. The research will draw upon discrete choice analysis and econometric estimation to identify metrics and estimation methods that are appropriate for engineering design. Estimation biases of two types of parameters that affect demand gradients with respect to engineering design variables -- customer preference coefficients and aggregate demand estimates -- will be examined. Multiple metrics will be tested to compare demand-model predictions with synthetic customer purchase data in which biases between the estimates and true parameters are known. Several different demand estimation methods proposed in econometrics will be evaluated using the identified metric(s). Finally, an optimization case study will be used to illustrate the influence of demand model biases on optimal design variables by comparing results using the identified estimation method that reduces parameter biases with one that does not.