Enhancing Discrete Choice Models with Representation Learning

TL;DR

This paper introduces a novel approach to discrete choice models that combines knowledge-driven and data-driven representations, using neural networks to improve prediction accuracy while maintaining interpretability.

Contribution

It proposes a hybrid modeling framework that integrates neural network representations into traditional discrete choice models, enhancing their predictive power and parameter estimation.

Findings

L-MNL and L-NL models outperform traditional models in predictive accuracy.

The new models improve parameter estimation accuracy.

Source code is publicly available for reproducibility.

Abstract

In discrete choice modeling (DCM), model misspecifications may lead to limited predictability and biased parameter estimates. In this paper, we propose a new approach for estimating choice models in which we divide the systematic part of the utility specification into (i) a knowledge-driven part, and (ii) a data-driven one, which learns a new representation from available explanatory variables. Our formulation increases the predictive power of standard DCM without sacrificing their interpretability. We show the effectiveness of our formulation by augmenting the utility specification of the Multinomial Logit (MNL) and the Nested Logit (NL) models with a new non-linear representation arising from a Neural Network (NN), leading to new choice models referred to as the Learning Multinomial Logit (L-MNL) and Learning Nested Logit (L-NL) models. Using multiple publicly available datasets based on revealed and stated preferences, we show that our models outperform the traditional ones, both in terms of predictive performance and accuracy in parameter estimation. All source code of the models are shared to promote open science.

Tables

Table 1. Table 1 : Monte Carlo average log-likelihood ( L ​ L ¯ ¯ 𝐿 𝐿 \bar{LL} ) and standard deviation ( s . d . ( L ​ L ) formulae-sequence 𝑠 𝑑 𝐿 𝐿 s.d.(LL) ) for the different models. Based on the test set value, we conclude that our L-MNL learns the best general representation. An MNL model with true utility specification is given as a reference. Average of accuracies

Model	Train set		Test set		Accuracies [%]		${\rho }_{test}^2$
	$\overline{LL}$	$s.d.\left(LL\right)$	$\overline{LL}$	$s.d.\left(LL\right)$	${\overline{Acc}}_{train}$	${\overline{Acc}}_{test}$
*Logit(${𝒳}_{true}$)	-459	15	-94	8	78	77	0.32
Logit(${𝒳}_1$)	-604	11	-123	6	67	66	0.11
DNN($25,𝒬$)	-363	21	-112	13	84	74	0.19
DNN_L($25,𝒳=𝒬$)	-367	18	-108	12	83	75	0.22
L-MNL($25,𝒳,𝒬$)	-429	16	$-\mathrm{𝟗𝟕}$	$\mathrm{𝟖}$	80	$\mathrm{𝟕𝟔}$	$0.30$

Table 2. Table 2 : Monte Carlo relative errors for the different models in [ % percent \% ] with e ¯ ¯ 𝑒 \overline{e} the average relative error, s . d . formulae-sequence 𝑠 𝑑 s.d. its standard deviation and β p subscript 𝛽 𝑝 \beta_{p} , β a subscript 𝛽 𝑎 \beta_{a} are from Equation ( 21 ).

Model	${\overline{e}}_{{\beta }_p}$	$s.d.\left(e_{{\beta }_p}\right)$	${\overline{e}}_{{\beta }_a}$	$s.d.\left(e_{{\beta }_a}\right)$	${\overline{e}}_{{\beta }_p/{\beta }_a}$	$s.d.\left(e_{{\beta }_p/{\beta }_a}\right)$
*Logit(${𝒳}_{true}$)	6.4	$\pm$   4.9	14.4	$\pm$   10.7	10.8	$\pm$   9.7
Logit(${𝒳}_1$)	26.7	$\pm$   6.2	26.7	$\pm$   14.7	15.5	$\pm$   12.4
DNN_L($25,𝒳=𝒬$)	60	$\pm$   32.4	74	$\pm$   54	460	$\pm$   166
L-MNL($25,𝒳,𝒬$)	$7.1$	$\pm$   $5.1$	$15.2$	$\pm$   $11.7$	$11.3$	$\pm$   $10.3$

Table 3. Table 3 : Monte Carlo hypothesis testing for the different models, for β p subscript 𝛽 𝑝 \beta_{p} and β a subscript 𝛽 𝑎 \beta_{a} taken separately and for their ratio. Parameters β p subscript 𝛽 𝑝 \beta_{p} , β a subscript 𝛽 𝑎 \beta_{a} are from Equation ( 21 ).

Model	% of experiments not rejecting $H_0$
	${\beta }_p$ and ${\beta }_a$	${\beta }_p/{\beta }_a$
*Logit(${𝒳}_{true}$)	97.5	94
Logit(${𝒳}_1$)	34	$\mathrm{𝟗𝟕}$
DNN_L($25,𝒳=𝒬$)	25.5	37
L-MNL($25,𝒳,𝒬$)	$\mathrm{𝟗𝟓}$	$\mathrm{𝟗𝟒}$

Table 4. Table 4 : Swissmetro benchmark utility function, from Bierlaire et al. ( 2001 )

		Alternative
Variables		Car	Train	Swissmetro
ASC	Constant	Car-Const		SM-Const
TT	Travel Time	B-Time	B-Time	B-Time
Cost	Travel Cost	B-Cost	B-Cost	B-Cost
Freq	Frequency		B-Freq	B-Freq
GA	Annual Pass		B-GA	B-GA
Age	Age in classes		B-Age
Luggage	Pieces of luggage	B-Luggage
Seats	Airline seating			B-Seats

Table 5. Table 5: Variables in the Swissmetro dataset used for modeled component of the utility specification

Variable	Description
TT	Door-to-door travel time in [minutes], scaled by 1/100.
Cost	Travel cost in [CHF], scaled by 1/100.
Freq	Transportation headway in [minutes]
GA	Binary variable indicating annual pass holders (=1).
Age	IntegeEs variable scaled with the traveler’s age.
Luggage	Integer variable scaled with amount of luggage during travel.
Seats	Binary variable for special seats configuration in Swissmetro (=1).

Table 6. Table 6: Unused variables in the Swissmetro dataset

Variable	Description
Purpose:	Integer variable indicating the trip purpose (business, leisure, etc. )
First :	Binary variable indicating if first class (=1) or not (=0)
Ticket:	Integer variable indicating the ticket type (one-way, half-day, etc.)
Who:	Integer variable indicating who is paying the ticket (self, employer, etc.)
Male:	Binary variable indicating the traveler’s gender (0 = female, 1 = male)
Income:	Integer variable indicating the traveler’s income per year.
Origin:	Integer variable indicating the canton in which the travel begins.
Dest:	Integer variable indicating the canton in which the travel ends.

Table 7. Table 7 : Comparison of log-likelihood and parameters estimates for different models with utility specification of Bierlaire et al. ( 2001 ) . Number of observations = 7234.

Model	Parameters	Estimates	Std errors	t-stat	p-value
MNL ${\rho }_{test}^2=0.28$ $ℒ\left(\hat{\beta }\right)=-5764$ ${ℒ}_{test}\left(\hat{\beta }\right)=-1433$	$AS{C}_{Car}$	1.08	0.162	6.67	0.00
	$AS{C}_{SM}$	1.05	0.153	6.84	0.00
	${\beta }_{age}$	0.146	0.436	3.35	0.00
	${\beta }_{cost}$	-0.695	0.0423	-16.42	0.00
	${\beta }_{freq}$	-0.733	0.1132	-6.47	0.00
	${\beta }_{GA}$	1.54	0.167	9.24	0.00
	${\beta }_{luggage}$	-0.114	0.0488	-2.338	0.02
	${\beta }_{seats}$	0.432	0.115	3.76	0.00
	${\beta }_{time}$	-1.34	0.051	-26.18	0.00
L-MNL($100,{𝒳}_1,{𝒬}_1$) ${\rho }_{test}^2=0.41$ $ℒ\left(\hat{\beta }\right)=-\mathrm{𝟒𝟓𝟏𝟏}$ ${ℒ}_{test}\left(\hat{\beta }\right)=-\mathrm{𝟏𝟏𝟖𝟏}$	$AS{C}_{Car}$	0.106	0.174	0.61	0.54
	$AS{C}_{SM}$	0.454	0.163	2.80	0.01
	${\beta }_{age}$	0.390	0.045	8.63	0.00
	${\beta }_{cost}$	-1.378	0.048	-28.45	0.00
	${\beta }_{freq}$	-0.860	0.127	-6.77	0.00
	${\beta }_{GA}$	0.214	0.194	1.10	0.27
	${\beta }_{luggage}$	0.116	0.0529	2.19	0.03
	${\beta }_{seats}$	0.104	0.109	0.95	0.34
	${\beta }_{time}$	-1.563	0.056	-27.97	0.00
DNN_L(100,${𝒳}_1=𝒬$) ${\rho }_{test}^2=0.37$ $ℒ\left(\hat{\beta }\right)=-4964$ ${ℒ}_{test}\left(\hat{\beta }\right)=-1257$	$AS{C}_{Car}$	0.365	0.165	3.61	0.00
	$AS{C}_{SM}$	0.549	0.162	2.22	0.03
	${\beta }_{age}$	0.087	0.0423	2.07	0.04
	${\beta }_{cost}$	-0.897	0.046	-19.46	0.00
	${\beta }_{freq}$	-0.639	0.123	-5.20	0.00
	${\beta }_{GA}$	1.40	0.172	8.15	0.10
	${\beta }_{luggage}$	0.186	0.0523	3.52	0.00
	${\beta }_{seats}$	0.233	0.102	2.29	0.02
	${\beta }_{time}$	-1.146	0.049	-23.32	0.00
Logit(${𝒳}_{dum}$) (all 41 inputs) ${\rho }_{test}^2=0.33$ $ℒ\left(\hat{\beta }\right)=-5451$ ${ℒ}_{test}\left(\hat{\beta }\right)=-1322$	${\beta }_{cost}$	-1.062	0.059	18	0.00
	${\beta }_{freq}$	-0.79	0.118	6.69	0.00
	${\beta }_{time}$	-1.326	0.053	25.02	0.00
	…	…	…	…	…
L-MNL($100,{𝒳}_2,{𝒬}_2$) ${\rho }_{test}^2=0.44$ $ℒ\left(\hat{\beta }\right)=-\mathrm{𝟑𝟖𝟗𝟓}$ ${ℒ}_{test}\left(\hat{\beta }\right)=-\mathrm{𝟏𝟏𝟎𝟖}$	${\beta }_{cost}$	-1.671	0.0523	-31.94	0.00
	${\beta }_{freq}$	-0.865	0.0765	-11.30	0.00
	${\beta }_{time}$	-1.769	0.0389	-45.4	0.00

Table 8. Table 8 : Parameter ratio comparison

Model	Value of Time	Value of Frequency	Train Log-Likelihood	Test Log-Likelihood
Logit(${𝒳}_1$)	0.52	0.95	-5764	-1433
Logit(${𝒳}_{dum}$)	0.80	1.34	-5451	-1322
DNN_L(${𝒳}_1$)	0.78	1.40	-4964	-1257
L-MNL(${𝒳}_1$)	0.88	1.60	-4511	-1181
L-MNL(${𝒳}_2$)	0.94	1.93	$-\mathrm{𝟑𝟖𝟗𝟓}$	$-\mathrm{𝟏𝟏𝟎𝟖}$
CNL(${𝒳}_1$)	0.59	1.52	-5711	-1415
TPM(${𝒳}_1$)	0.72	1.59	-4752	-1350

Table 9. Table 9 : Performance of Swissmetro models with added nest structure. Increase in fit lowers the need of nests.

	Standard Models		Nest Factor	Nested Models
	Train LL	Test LL		Train LL	Test LL
Logit(${𝒳}_1$)	-5764	-1433	1.43	-5734	-1413
L-MNL(${𝒳}_1,{𝒬}_0$)	-5324	-1348	1.37	-5305	-1340
L-MNL(${𝒳}_1,{𝒬}_1$)	-4268	-1173	1.01	-4260	-1173
L-MNL(${𝒳}_2,{𝒬}_2$)	-3895	$-\mathrm{𝟏𝟏𝟎𝟖}$	1	-3894	$-\mathrm{𝟏𝟏𝟎𝟖}$

Table 10. Table 10 : Base model specification from Fernández-Antolín et al. ( 2016 )

		Alternative
Variables		Public Transportation	Car	Slow modes
ASC	Constant	PT-Const	CAR-Const
TT	Travel Time [min]	B-Time-PT	B-Time-CAR
MCost	$\frac{\text{Marginal Cost}}{\text{Income}}$	B-MCost-PT	B-MCost-CAR
Distance	Trip distance [km]			B-Dist
Work	Work related Trip		B-Work
French	French Speaking area		B-French
Student	Occupation is student	B-Student
Urban	Urban area	B-Urban
NbChild	Number of Children		B-NbChild
NbCar	Number of Cars		B-NbCar
NbBicy	Number of Bicycles			B-NbBicy

Table 11. Table 11: Added variables for representation learning term in Optima dataset. All variables gave -1 for missing values.

Variable	Description
Age:	Age of the respondent (in years)
HouseType :	1 is individual house (or terraced house), 2 is apartment, 3 is independent room
Gender:	1 is man, 2 is woman. -1 for missing value.
Education:	Highest education achieved101010More details at https://biogeme.epfl.ch/data.html . Categories from 1 to 8.
FamilSitu:	Family situation10. Categories from 1 to 7.
ScaledIncome:	Integer variable indicating the traveler’s income per year.
OwnHouse:	Do you own the place where you are living? 1 is yes, 2 is no
MotherTongue:	1 for german or swiss german, 2 for french, 3 for other,
SocioProfCat:	Socio-professional10 categories from 1 to 8.

Table 12. Table 12 : Models accuracy on training and testing sets of Optima

Model	Logit(${𝒳}_1$)	L-MNL(${𝒳}_1$ ,${𝒬}_1$)	NN40($𝒳\cup 𝒬$)	ICLV(${𝒳}_1$)
Accuracy Train [$\%$]	76.8	80.4	86.1	80.0
Accuracy Test [$\%$]	76.7	79.2	81.3	77.7

Table 13. Table 13 : Impact of unseen variables on parameter estimates and log-likelihood. e ¯ ¯ 𝑒 \overline{e} is the average relative error, s . d . formulae-sequence 𝑠 𝑑 s.d. its standard deviation and β p subscript 𝛽 𝑝 \beta_{p} , β a subscript 𝛽 𝑎 \beta_{a} are from Equation ( 21 ).

Model	${\overline{e}}_{{\beta }_p}$	$s.d.\left(e_{{\beta }_p}\right)$	${\overline{e}}_{{\beta }_a}$	$s.d.\left(e_{{\beta }_a}\right)$	${\overline{e}}_{{\beta }_p/{\beta }_a}$	$s.d.\left(e_{{\beta }_p/{\beta }_a}\right)$
L-MNL($25,𝒳,𝒬$)	$11.2$	$\pm$   $7.4$	$21.1$	$\pm$   $13.6$	$16.2$	$\pm$   $12.5$
Logit(${𝒳}_1$)	34.2	$\pm$   6.3	38.7	$\pm$   16.2	20.9	$\pm$   15.5

Model	${\overline{e}}_{{\beta }_p}$	$s.d.\left(e_{{\beta }_p}\right)$	${\overline{e}}_{{\beta }_a}$	$s.d.\left(e_{{\beta }_a}\right)$	${\overline{e}}_{{\beta }_p/{\beta }_a}$	$s.d.\left(e_{{\beta }_p/{\beta }_a}\right)$
L-MNL($25,𝒳,𝒬$)	$11.2$	$\pm$   $7.4$	$21.1$	$\pm$   $13.6$	$16.2$	$\pm$   $12.5$
Logit(${𝒳}_1$)	34.2	$\pm$   6.3	38.7	$\pm$   16.2	20.9	$\pm$   15.5

	${\overline{LL}}_{train}$	$s.d.\left(LL\right)$	${\overline{LL}}_{test}$	$s.d.\left(LL\right)$
L-MNL($25,𝒳,𝒬$)	$-\mathrm{𝟒𝟔𝟎}$	$\pm$   $\mathrm{𝟏𝟔}$	$-\mathrm{𝟏𝟎𝟏}$	$\pm$   $\mathrm{𝟕}$
Logit(${𝒳}_1$)	-620	$\pm$   11	-124	$\pm$   5

Table 16. Table 14 : Values of parameter estimates and likelihoods based on optimization strategy for β p = − 2 subscript 𝛽 𝑝 2 \beta_{p}=-2 and β a = 1 subscript 𝛽 𝑎 1 \beta_{a}=1 . Best results are obtained with joint optimization.

Strategy	${\hat{\beta }}_p$	${\hat{\beta }}_a$	$ℒ\left(\hat{\beta }\right)$	${ℒ}_{test}\left(\hat{\beta }\right)$
(1) $\widehat{𝜷}$ then $𝒘$	-$1.41$	$0.75$	-4927	-942
(2) $𝒘$ then $\widehat{𝜷}$	-$1.59$	$0.82$	-3924	-758
(3) $\widehat{𝜷}$ and $𝒘$	-$1.95$	$1.0$	-$\mathrm{𝟑𝟔𝟕𝟖}$	-$\mathrm{𝟕𝟐𝟏}$

Table 17. Table 15 : Values of parameter estimates and likelihoods for different models based on Equation ( 40 ). Ground truth is β T ​ T = − 1 subscript 𝛽 𝑇 𝑇 1 \beta_{TT}=-1 and β T ​ C = − 2 subscript 𝛽 𝑇 𝐶 2 \beta_{TC}=-2 . Only L-MNL is able to estimate correctly the parameters.

Models	${\hat{\beta }}_{TT}$	${\hat{\beta }}_{TC}$	${\hat{\beta }}_{TC}/{\hat{\beta }}_{TT}$	$L{L}_{train}$	$L{L}_{test}$
Logit(${𝒳}_a$)	-0.65	-1.63	2.50	-5412	-1354
Logit(${𝒳}_b$)	-0.25	-0.70	2.81	-7722	-1925
L-MNL($𝒳,𝒬$)	$-1.01$	$-1.99$	$1.96$	$-\mathrm{𝟐𝟓𝟏𝟔}$	$-\mathrm{𝟖𝟎𝟗}$

Equations

$$U_{in}=V_{in}+{\epsilon }_{in},$$

$$V_{in}=\sum _d{\beta }_d\cdot x_{din},$$

$$P_n(i)=\frac{e^{V_{in}}}{{\sum }_{j\in {\mathcal{C}}_n}{e}^{V_{jn}}}.$$

$$\mathcal{L}=\sum _{n=1}^N\sum _{i\in {\mathcal{C}}_n}{y}_{in}\log \left[P_n(i)\right],$$

$$\mathbf{U}$$

$$with$$

$$h_i^{(j+1)}=g(\sum _{k=0}^d{h}_{(s\cdot i+k)}^{(j)}{\beta }_k^{(j)}+{\alpha }_i^{(j)}),$$

$$(\mathbf{\sigma }({\text{V}}_n){)}_i=\frac{e^{V_{in}}}{{\sum }_{j\in {\mathcal{C}}_n}{e}^{V_{jn}}},$$

$$H_n(\mathbf{\sigma },{\text{y}}_n)=-\sum _{i\in {\mathcal{C}}_n}{y}_{in}\log \left[{\sigma }_i({\text{V}}_n)\right].$$

$$U_{in}$$

$$V_{in}=f_i({\mathcal{X}}_n;\mathbf{\beta })+r_i({\mathcal{Q}}_n,\mathbf{w}),$$

$${\mathbf{U}}_n=\mathbf{f}({\mathcal{X}}_n;\mathbf{\beta })+\mathbf{r}({\mathcal{Q}}_n;\mathbf{w})+{\mathbf{\epsilon }}_n.$$

$$\frac{\partial U_{in}}{\partial t_{in}}=\frac{\partial f_{in}}{\partial t_{in}}+\frac{\partial r_{in}}{\partial t_{in}}$$

$$\frac{\partial r_{in}}{\partial t_{in}}=0$$

$${\bar{\epsilon }}_{in}=r_i({\mathcal{Q}}_n;\mathbf{w})+{\epsilon }_{in}.$$

$$P_n(i)=\frac{e^{f_i({\mathcal{X}}_n;\mathbf{\beta })+r_i({\mathcal{Q}}_n;\mathbf{w})}}{{\sum }_{j\in {\mathcal{C}}_n}{e}^{f_j({\mathcal{X}}_n;\mathbf{\beta })+r_j({\mathcal{Q}}_n;\mathbf{w})}}.$$

$$r_{in}=\sum _{k=1}^H{w}_{ik}^{(L)}g({\mathbf{q}}_n^{(L-1)}{\mathbf{w}}_k^{(L-1)}+{\alpha }_k^{(L-1)})+{\alpha }_i^{(L)},$$

$${\left[{\mathbf{q}}_n^{(j+1)}\right]}_i=\sum _{k=1}^H{w}_{ik}^{(j+1)}g({\mathbf{q}}_n^{(j)}{\mathbf{w}}_k^{(j)}+{\alpha }_k^{(j)})+{\alpha }_i^{(j+1)},$$

$$P_n(i|\mathcal{C})$$

$$=\frac{e^{{\mu }_m{V}_{in}}}{{\sum }_{j\in {\mathcal{C}}_m}{e}^{{\mu }_m{V}_{jn}}}\cdot \frac{\exp \left(\frac{\mu }{{\mu }_m}\ln {\sum }_{p\in {\mathcal{C}}_m}{e}^{V_{pn}}\right)}{{\sum }_{k=1}^M\exp \left(\frac{\mu }{{\mu }_k}\ln {\sum }_{p\in {\mathcal{C}}_k}{e}^{{\mu }_k{V}_{pn}}\right)}$$

$$U_{in}=V_{in}+{\epsilon }_{in},$$

$$V_{in}=\underset{\text{known relation}}{\underbrace{{\beta }_p\cdot p_{in}+{\beta }_a\cdot a_{in}+{\beta }_b\cdot b_{in}}}+\underset{\text{unknown interactions}}{\underbrace{{\beta }_{qc}\cdot q_{in}{c}_{in}}}$$

$$p_{in}$$

$$q_{in}$$

$$k_{in}$$

$$x_{in}\sim \mathcal{U}([-1,1])$$

$$y_{1n}$$

$$y_{2n}$$

$$e_{\beta }$$

$$e_{{\beta }_i/{\beta }_j}$$

$$q_{in}^{\prime }=s\cdot p_{in}+\sqrt{1-s^2}\cdot q_{in},$$

$$t_{car}\cdot I_i={\alpha }_i+{\lambda }_i\cdot CarLoving\cdot t_{car}+{\omega }_i$$

$$CarLoving={\eta }_{car}+\omega$$

Full text

1 Introduction

†† ©2020. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/

Discrete Choice Models (DCM) have emerged as a powerful theoretical framework for studying choices made by humans. The goal of these models is to predict the choice outcome among a given set of discrete alternatives (e.g., choice of walking as a transportation mode, rather than taking the car or the bus), while understanding the behavioral process that led to the specific choice. For many years, the Multinomial Logit model (MNL), based on a simple parametric utility specification, has provided the foundation for the analysis of discrete choice. Despite its oversimplified assumptions regarding the actual decision-making process, MNL is still commonly used in practice since it enables a high level of interpretability.

Interpretability is critical for researchers and practitioners. For instance, parametric specifications based on linear or logarithmic functions allow for straightforward derivation of the value-of-time (VOT), i.e., the marginal rate of substitution between time and cost, that constitutes a highly relevant measure in a wide range of public transport policy. However, interpretability gain from using an MNL model with such simple parametric specifications often leads to a sacrifice in the predictive power of the model. Indeed, MNL does not allow to incorporate individual heterogeneity explicitly in the empirical investigation, and the assumed simple parametric model cannot adequately capture the full underlying structure of the data. Several prior studies, within the discrete choice literature, have shown that incorrectly assuming a parametric utility specification can cause severe bias in parameter estimates and post-estimation indicators (e.g., Torres et al. (2011), Van Der Pol et al. (2014)).

More advanced utility specifications, embedded into more complex models, would allow for a better fit of the data and, ultimately, a better prediction. In this regard, utility specifications based on Box-Cox transformation, piecewise-linear, exponential, or non-parametric functions have been proposed in the literature (e.g., Schindler et al. (2007), Kneib et al. (2007), Kim et al. (2016)), as well as advanced DCM models, such as the Mixed Logit Model (MLM) or the Latent Class Model (LCM) (e.g., Shen (2009), Xiong and Mannering (2013), Vij et al. (2013), Kim et al. (2016)). However, the main limitation of these choice models is that the standard procedure for estimating the parameters’ values is to assume that the model specification is known a priori, while in practice determining the utility specification for a particular application remains a difficult task.

Within the machine learning community, Neural Networks (NN) masters the field of representation learning. They have become the go-to solutions, as their superior prediction performance has been repeatedly demonstrated, including in transportation applications (e.g., Omrani (2015), Hagenauer and Helbich (2017), Pekel and Soner Kara (2017), Nam et al. (2017), Pirra and Diana (2018), Chang et al. (2019), Zhou et al. (2019)). Unlike discrete choice models, NN requires essentially no a priori beliefs about the nature of the true underlying relationships among variables. However, gaining in prediction accuracy often comes at the cost of losing interpretability, explaining why the discrete choice community often considers them as a black-box.

In this work, we propose a new approach to discrete choice modeling by integrating representation learning in the formulation. We propose to divide the systematic part of the utility specification into (i) a knowledge-driven (interpretable) part and (ii) a data-driven (representation learning) part, which aims at automatically discovering a good utility specification from available data. Our formulation partially replaces manual utility specification and allows for a better prediction performance without sacrificing interpretability. We demonstrate the effectiveness of our framework by augmenting the utility specification of the MNL and Nested logit models with a new non-linear representation, arising from a neural network. It leads to new choice models, referred to as the Learning Multinomial Logit (L-MNL) and Learning Nested Logit (L-NL) models, respectively. The source code is made publicly available to ease reproducibility and promote open science111https://github.com/BSifringer/EnhancedDCM.

The remainder of the paper is organized as follows: Section 2 is a non-exhaustive overview of the recent attempts to conduct multi-disciplinary research, combining machine learning (ML) and discrete choice modeling. Section 3 gives a brief background and presents details on how we implement MNL with modern neural network libraries. Section 4 describes our proposed L-MNL and L-NL models. Section 5 demonstrates evidence that our learning choice models outperform both existing hybrid neural network and MNL models based on simple parametric utility specifications. Our results on real and synthetic data show that the research community can benefit from a better integration of representation learning techniques and discrete choice theory. In that sense, our research also contributes to bridging the gap between knowledge-driven and data-driven methods.

2 Related work

Discrete choice and machine learning researchers have typically different perspectives and research priorities. The related work in each field is too broad to be covered here exhaustively, and we, therefore, limit ourselves to the most relevant works at the intersection of the two communities.

More than 20 years ago, researchers were already interested in applying data-driven methods such as neural networks in different transportation applications (e.g., Faghri and Hua (1992), Dougherty (1995), Shmueli et al. (1996)). Over the years, different types of NN, as well as decision trees, have been compared to MNL models (Agrawal and Schorling (1996), Lee et al. (2018), Zhao et al. (2020)), Nested Logit (NL) models (Mohammadian and Miller (2002), Hensher and Ton (2000)) and more generally to random utility models (Sayed and Razavi (2000), Cantarella and de Luca (2005), Paredes et al. (2017)) and statistical methods (West et al. (1997), Karlaftis and Vlahogianni (2011), Iranitalab and Khattak (2017), Golshani et al. (2018), Brathwaite et al. (2017)). However, the literature has mainly focused on comparing the models in terms of prediction power, without discussing the issue of interpretability.

The machine learning community has also generated a tremendous amount of research aiming at predicting, with high accuracy, a variety of choices. Among the most recent works, Pekel and Soner Kara (2017) or Jin et al. (2020) present comprehensive reviews of publications related to the application of NN or ML to predict travelers’ choice in public transportation. Hagenauer and Helbich (2017) compare seven machine learning methods to classify travel mode choice. The methods investigated are MNL and NN, but also Naive Bayes (NB) (Rish et al. (2001)), Gradient Boosting Machines (GBM) (Friedman (2001)), Bagging (BAG) (Breiman (1996)), Random Forests (RF) (Breiman (2001)), and Support Vector Machine (SVM) (Cortes and Vapnik (1995)). Pirra and Diana (2018) also propose to use SVM for travel mode choice while Lhéritier et al. (2018) rely on RF and GBM to predict airline itinerary choice. Not surprisingly, the results exhibited in these papers show outstanding results in the ability of these models to fit the data but do not tackle the issue of their interpretability.

Some recent publications have gone beyond the comparison of the two fields and proposed innovative behavioral studies based on data-driven methods. Two notable examples are the work of Wong et al. (2018), who use a restricted Boltzmann Machine (BM) (Ackley et al. (1985)) to represent latent behavior attributes, and the study of van Cranenburgh and Alwosheel (2019), who develop a novel NN based approach to investigate decision rule heterogeneity amongst travelers. Moreover, Wang et al. (2018c) have investigated the numerical extraction of behavioral information such as willingness to pay in a Dense Neural Network (DNN) using elasticity while Wang et al. (2018b) have investigated the theoretical transition from DCM to DNN and subsequent trade between predictability and interpretability. Recently, the main authors have also developed a new structuring of neural networks, based on alternative specific utility architecture (Wang et al. (2020)). Finally, Wong and Farooq (2020) have made use of a NN generative model to capture the joint distribution of multiple discrete-continous travel data. However, these papers do not have the objective of finding a utility specification that allows high predictability while maintaining straightforward interpretability.

The closest studies relative to our work are in the brand choice literature. Bentz and Merunka (2000) introduce a hybrid model, afterward called the NN-MNL model. Their model involves a two-stage approach that starts with the estimation of a NN model that aims at discovering non-linearity effects in the utility function. Then, if the NN identifies some non-linearities, the specification of the MNL model is modified to include new variables, specially created to account for the discovered non-linear effects. On their dataset, the re-specified MNL model, which includes the discovered non-linear effects, slightly outperforms the basis MNL model. Their work, similar to ours in spirit, aims at achieving better predictive power together with a greater understanding of the influencing factors. However, their approach is sequential, and their neural network is only used beforehand as a diagnostic and specification tool. Again, in the context of brand choice, Hruschka et al. (2002, 2004) and Hruschka (2007) investigate further the potential of neural network based choice models. In Hruschka et al. (2002), the NN-MNL model is compared against homogeneous and heterogeneous versions of linear utility MNL models. Meanwhile, in Hruschka et al. (2004), the comparison is made against two non-linear models, the generalized additive (GAM-MNL) model of Abe (1999) and a flexible functional form based on Taylor series approximations. The authors show that, by being capable of finding non-linear relationships from available data, the neural network based choice models with non-linear specifications outperform the other models in terms of prediction performance. However, the authors are unable to draw any conclusion regarding the significance of the factors with this method. Also, the post estimation analysis is limited to likelihood and choice elasticities. Finally, Hruschka (2007) proposes a Multinomial Probit (MNP) model with a neural network extension to model brand choice. Their model combines the heterogeneity across households with a multilayer perceptron. Linear and non-linear deterministic utility specifications are considered by specifying the same variables in linear and non-linear terms, hence loosing interpretability. The probit assumption also prevents the closed form of the choice probabilities and the model can only be estimated using a Markov Chain Monte Carlo simulation technique.

Our work significantly departs from these previous studies by proposing a flexible and general framework for the specification of the deterministic utility in any DCM model. More specifically, our approach combines, in a single joint optimization, the estimation of a standard DCM model, based on a carefully thought-out utility specification, with a representation learning model that aims at increasing the overall predictive performance. To our knowledge, this is the first formulation that benefits from the predictive power of representation learning techniques while keeping some key parameters interpretable, which allows us to derive insightful post-estimation indicators. In Section 5, we show the effectiveness of our learning choice method by comparing its performance with benchmarking models, including the NN-MNL model of Hruschka et al. (2002, 2004).

3 Background and Notations

Multinomial Logit

Discrete choice modeling complies with the Random Utility Maximization (RUM) theory (McFadden (1974)), which postulates that an individual is a rational decision-maker, who aims at maximizing the utility relative to their choice. Utility is a latent construct assumed to be partitioned into two components: a systematic (or deterministic) utility, $V_{in}$, and a random component, $\varepsilon_{in}$, that captures the uncertainty coming from the impossibility for the modeler to fully capture the choice context. Formally, the utility that individual $n$ associates with alternative $i$ from their choice set $\mathcal{C}_{n}$ is given as222Notation convention can be found in A.:

[TABLE]

For convenience, we consider the systematic part of the utility to be linear-in-parameter, as it is generally assumed:

[TABLE]

where $\bm{\beta}$ are the preference parameters (or estimators) associated with the explanatory variables (or the input features) $\bm{x}\in\mathcal{X}$ that describe the observed attributes of the choice alternative (e.g., the price or travel time associated with the mode), and the individual’s socio-demographic characteristics (e.g., the individual’s level of income or age).

Under the standard MNL assumption that error terms are independently and identically distributed (i.i.d.) and follow an Extreme Value Type I distribution with location parameter zero and the scale parameter 1 (i.e. $\varepsilon_{in}\overset{\text{i.i.d.}}{\sim}EV(0,1)$), the probability for individual $n$ to select choice alternative $i$ is given by

[TABLE]

The preference parameters $\bm{\beta}$ are typically estimated by maximizing the log-likelihood function given by:

[TABLE]

where $y_{in}$ is the observed choice variable (or true label) and is equal to $1$ if the choice of individual $n$ is the alternative $i$, and to [math] otherwise.

Implementing MNL as a Neural Network

A neural network consists of a function mapping the input space $x$ to an output of interest $U$ through several intermediate representations commonly referred to as hidden layers $\bm{h}^{(j)}$:

[TABLE]

where $\bm{q}^{(0)}=\bm{x}$ and $L$ is the last representation layer.

We make use of a Convolutional Neural Network (CNN) to retrieve the MNL formulation333Previous studies, such as Bentz and Merunka (2000), have successfully written MNL with other Neural Network architectures. We have used CNN for its weight-sharing architecture.. A CNN has weights in the shape of a filter that connects a layer $\bm{h}^{(j)}$ to the next by applying a convolution444 This method is often seen in image processing where a filter with a fix number of weights is applied to an equal amount of inputs by multiplying the terms together and then summing them over to obtain a single new value. A new image is obtained by sliding the filter over all inputs with a set step-size named stride. In our case, the operation is done without padding, i.e., without applying the filter out of the boundaries, effectively reducing the size from one layer (filtering) to the next. In image processing, the correlation formula is the same as a convolution up to a flip in the order of its entries. Without loss of generality, we write the correlation formula in Equation 7 for readability.. Therefore, the value of a neuron $i$ in the next layer $(j+1)$ can be written as:

[TABLE]

where $\{\beta_{1},...\beta_{d}\}=\bm{\beta}$ is the filter of size $(1\times d)$, $s$ the stride of the convolution, $\alpha_{i}$ a bias term and $g(\cdot)$ an activation function.

The MNL formulation is retrieved by using a single layer ($L=1$), setting the activation function to identity ($g(x)=x$) and the stride $s$ to $d$. Doing so, we get the utility functions $\textbf{V}_{n}=\{V_{1n},...V_{In}\}$ as defined in Equation (2).

Then, the probabilities can be obtained by using a softmax activation layer (Bishop (1995)) defined as:

[TABLE]

which can be identified as Equation (3) for all probabilities. The output of the network goes through a loss function, in our case categorical cross-entropy (CE)(Shannon (1948)):

[TABLE]

Minimizing (9) is equivalent to maximizing Equation (4) when summed over all individuals $n$.

An illustrative example is given in Figure 1 with :

[TABLE]

where $x_{1}=cost$ stands for travel cost, and $x_{2}=time$ for travel time. The choice set $\mathcal{{C}}$ is assumed to be the same for all individuals and contains the following mode alternatives: $\mathcal{{C}}=\{\text{Car, Train, {Swissmetro (SM)}}\}$. The filter is made of $\bm{\beta}=(\beta_{t},\beta_{c}$), and a stride of $d=2$ allows to recover each utility specification in the next layer from $cost$ and $time$ of every alternative.

4 Representation Learning in Discrete Choice Modeling

General Formulation

As previously mentioned, the standard statistical procedure for estimating the parameter values of choice models is to assume that the true utility specification is known a priori. Typically, the most common representation is inspired from linear regression and involves the observed attributes of the choice alternative and the individual’s socio-demographic characteristics. However, the influence of these explanatory variables on the utility of a choice alternative is unlikely to be known and to be precisely linear. The danger of an incorrect utility specification remains, therefore, highly present.

In this work, we propose a more flexible and data-driven approach that consists in expressing the systematic part of the utility function into two sub-parts, as follows:

[TABLE]

where

$\bm{f}(\mathcal{X}_{n};\bm{\beta})$ is the knowledge-driven part, assumed interpretable. The function $f$ is defined such that its unknown model parameters $\bm{\beta}$ are an interpretable combination of the explanatory variables (or input features) $\mathcal{X}_{n}$.

• -

$\bm{r}(\mathcal{Q}_{n},\bm{w})$ is the data-driven part, a representation that is learned from a set of explanatory variables (or input features) $\mathcal{Q}_{n}$, for which no a priori relationship is assumed.

Conditions between $\mathcal{X}$ and $\mathcal{Q}$ to keep $\bm{f}$ interpretable are described in section 4.2.

Replacing the systematic utility component in Equation (1) by its new expression given by Equation (11), we obtain the following utility expression:

[TABLE]

We may interpret this equation as a data-driven term finding the residual of a hand-modeled function. He et al. (2016) have shown the impact of residual network (ResNet) in the machine learning community, and its particular architecture has been widely used for increased interpretability by finding the residual of knowledge and equation based modeling (Liao et al. (2018), Wang et al. (2018a), Li et al. (2019)), or choice modeling (Wong and Farooq (2019)). Our work differs with the use of two input sets $\mathcal{X}$ and $\mathcal{Q}$ necessary for keeping discrete choice interpretability as we define it in section 4.2 and further explain in section 5.

Finally, the above specification may be applied to any discrete choice modeling kernel. However, the following applications will limit themselves to models with Gumbel distributed random terms.

Modeling

When modeling a utility specification with the formulation in Equation (12), one must understand the attributes of a parameter related to either $\mathcal{X}$ in $\bm{f}$ or $\mathcal{Q}$ in $\bm{r}$. The first assumes to have all the benefits of expert modeling in discrete choice, which includes the availability of the $\bm{\beta}$ parameters’ Hessian and thus their standard deviation approximation. The second produces a new representation of its associated input, which can span from discrete to continuous inputs such as signals and images, while the hessian is generally not available due to computational complexity. Therefore, when one studies the impact, or elasticity, of a feature $t_{in}$ on alternative $i$, this readily translates as:

[TABLE]

Bentz and Merunka (2000) and Wang et al. (2018c) have shown that behavioral insight can be captured and studied through the elasticities of a neural network. However, to ensure the same straightforward interpretability of behavioral choice models for a parameter $t$, we explicitly stipulate that the standard deviation must be available, and the elasticity must not depend on the representation term. In other words, we impose that:

[TABLE]

Equation (14) constitutes a critical assumption which ensures that insightful post-estimation indicators can be retrieved. This definition, as well as the general formulation is what sets us apart from old and new literature alike. Indeed, to only cite a few, Hruschka (2007) or Wong et al. (2018) are specific cases of our formulation when both sets are the same, i.e., $\mathcal{X}=\mathcal{Q}$, while Bentz and Merunka (2000), Otsuka and Osogami (2016) and Wang et al. (2018c) are cases where $\mathcal{X}=\emptyset$. All of which do not follow our goal of fixing a biased expert modeling component by leveraging the strength of data-driven methods in an added and non-overlapping representation term.

Moreover, the global theoretical effect of this new term may be derived. When one considers Equation (12) without the new term, then this value would be found in the random component such that:

[TABLE]

To have unbiased parameter estimates in $\bm{f}(\mathcal{X}_{n};\bm{\beta})$, one must avoid correlation between the specification and the random terms known as endogeneity, avoid correlation of the random terms between each alternative and overall avoid utility misspecification. Many methods have been developed to specifically fix for endogeneity, such as Guevara (2015), while others account for correlation between alternatives (see section 4.4). Our model aims at correcting for underfit due to misspecification and omitted variable bias thanks to data-driven methods. In this sense, the new error term $\varepsilon_{in}$ is more likely to fit the aforementioned criteria than $\bar{\varepsilon}_{in}$. The new representation term increases the probability of having unbiased parameters while increasing the prediction rate as it may easily find undiscovered specifications during the expert modeling process. Moreover, our general framework does not exclude the coupling with other methods that aim at fixing the random term, as we will observe in the following sections.

L-MNL Model Formulation

In the specific case where we choose to add a representation term to a Multinomial Logit for Equation (12), the standard MNL assumptions regarding the distribution of the error term $\varepsilon_{in}$ (described in Section 3.1) have to be made.

The likelihood of selecting the choice alternative $i$ for individual $n$, given the values of the model parameters ($\bm{\beta}$ and $\bm{w}$) and the influencing factors ($\mathcal{X}_{n}$ and $\mathcal{Q}_{n}$), is therefore naturally expressed as:

[TABLE]

For our L-MNL model, we choose to use a Dense Neural Network (DNN) as the learning method. More specifically, its representation term $r_{in}$ is the resulting function of a DNN with $L$ layers of $H$ neurons and a single output per utility function:

[TABLE]

where $g(\cdot)$ is the rectifier linear units (ReLU) activation function and $\bm{q}_{n}^{(j)}$ is recurrently defined by:

[TABLE]

with $\bm{q}_{n}^{(0)}$ being the vector of input features $\mathcal{Q}_{n}$.

The architecture of our L-MNL hybrid model is shown in Figure 2, where we see both the linear component, written as a CNN (see Section 3), and the learned representation term, obtained through a DNN. Our hybrid approach is both general and flexible. Other functions of $\bm{f}$ and other architectures for $\bm{r}$ may be chosen. For more complex DCM frameworks, new assumptions must be made on the error terms, probabilities, and likelihood function. For instance, an extension of our work to the nested logit model is shown in the next section.

L-MNL nested generalization

The Nested Logit formulation has been first introduced by Williams (1977). It relaxes the i.i.d. assumption of MNL random terms in the case where alternatives may have correlation among each other in the decision making process. They are gathered together in nests which then affects their final probability such that:

[TABLE]

where $M$ is the total number of nests, $\mu_{l}$ is the factor for nest $l$ and $\mathcal{C}_{l}$ is the set of alternatives belonging to that nest. This is the probability of choosing an alternative within the nest $m$ multiplied by the probability of choosing $m$ among all nests.

To add a representation learning term to the Nested Logit, one must only change the final function of L-MNL seen in Equation (8) and adapt it to Equation (4.4). This is done by removing the commonly used softmax layer seen in Figure 2 for a custom-built network with added trainable weights for the nest factors. The new architecture is depicted in Figure 10 of Appendix E. Further changes to the network would allow us to easily model Cross-Nested Logit (Vovsha (1997)) and Generalized Extreme Value models (McFadden (1978)) with an added representation term.

5 Experiments

Our experiments demonstrate how our L-MNL and L-NL models increase the predictability of the MNL and NL models respectively while keeping their interpretability. The former, predictability, is directly quantified in terms of likelihood using two real-world datasets. The latter interpretability, however, is more challenging to assess. To do so, we define interpretability as the ability of the model to have both a quantifiable uncertainty of its parameters as well as an understandable or meaningful association with its variables. This latter point usually translates as the ability to mathematically derive and interpret the impact, or elasticity, of a variable. We will use synthetic and semi-synthetic data for which the true parameters are known to assess the quality of the estimates and the interpretability of every model.

Benchmarking models

We want to compare our models (L-MNL and L-NL) against key previous works. As our approach is both general and flexible, we retrieve previous works with appropriate modeling decisions on the $\mathcal{X}$, $\mathcal{Q}$ input sets and $\bm{f}$, $\bm{r}$ functions of Equation (12).

Essentially, we compare the following models:

Logit($\mathcal{X}$): The standard MNL model with linear-in-parameter specification, and no learning component. The variables $x\in\mathcal{X}$ enter the linear utility specification and $\mathcal{Q}=\emptyset$. The model has $|\mathcal{X}|$ parameters.

• 2.

DNN($n,\mathcal{Q}$): A Dense Neural Network for every alternative with softmax loss. This is the NN-MNL model proposed by Hruschka et al. (2004) and based on Bentz and Merunka (2000) with $n$ neurons in the hidden layer ($H=n$) and a single layer deep. The variables $x\in\mathcal{Q}$ are the input features of the neural network. There is no variable entering the linear utility specification ($\mathcal{X}=\emptyset$). The model has around $n(|\mathcal{Q}|+|\mathcal{C}|+1)$ weights.

• 3.

DNN_L($n,\mathcal{X}=\mathcal{Q}$): A modified version of the NN-MNL model proposed by Hruschka et al. (2004). The variables $x\in\mathcal{X}(=\mathcal{Q}$) enter both the linear utility specification and the neural network. The latter can be seen as finding the residual from the linear input of $\mathcal{X}$ and the overall architecture is close to the concept of the more well known residual building block from ResNet (He et al. (2016)). The model has in the order of $(n+1)(|\mathcal{X}|+|\mathcal{C}|)$ weights.

• 4.

L-MNL($n,\mathcal{X},\mathcal{Q}$): Our learning logit model with $n$ neurons in the hidden layer ($H=n$). Part of the variables $x\in\mathcal{X}$ enter the linear utility specification and the remaining variables $q\in\mathcal{Q}$ are given as input features to the neural network. This separation satisfies the interpretability condition (Equation 14) for all $x$. The function $\bm{r}(\mathcal{Q})$ is defined by Equation (17) where we limit the complexity to a single layer ($L=1$). Therefore, the model has a total of $|\mathcal{X}|$ interpretable parameters and $n(|\mathcal{Q}|+|\mathcal{C}|)$ weights. We remind the reader that this architecture for the representation term is for the sake of example and more complex networks or methods depend on the datasets at hand and are open for future research.

Unless specified otherwise, all models have run for 200 epochs with an Adam optimizer (Kingma and Ba (2014)) running on default parameters from Keras python deep learning library (Chollet et al. (2015)). Every model with a Neural Network has a $20\%$ dropout regularizer following the DNN layer. The parameters in every model are initialized by the default random initialization of the used library, and no seed is set. Moreover, in our current framework, no notable difference in learning time between models are observed.

Synthetic data

We now use synthetic data to better analyze the performance of our L-MNL model in terms of both prediction performance and estimates accuracy. We start by describing how we generated synthetic data. Then, we perform Monte Carlo experiments to compare all benchmarking models on parameters estimation. Then, a scan on increasing number of neurons is introduced to analyze the impact of the NN architecture better. Finally, we investigate the case of strongly correlated variables between input sets $\mathcal{X}$ and $\mathcal{Q}$. Additional experiments to study the particular effects of complexly correlated variables, missing variables, or sequential versus joint optimization strategies can be found in B. To study the limits of the hybrid model, a special case study of very noisy and highly non-linear utility specifications using semi-synthetic data can be found in C. For all experiments, all benchmarking models remain the same as in the previous Section (5.1).

5.2.1 Data generation

Synthetic data provides a controlled environment for analyzing our L-MNL model. To generate our synthetic data, we consider a simple binary choice model where an individual $n$ has to choose between two options (e.g., choosing to use public transportation to go to work as opposed to taking an alternative mode). Following the generation process of Guevara (2015), we obtain the following utility specifications for $i=1,2$:

[TABLE]

with

[TABLE]

[TABLE]

where $\beta_{x}$ stand for parameters, and all variables $a_{in}$, $b_{in}$, $c_{in}$, $z_{in}$, $wz_{in}$, $h_{in}$ and error terms $\varepsilon_{pin}$, $\varepsilon_{qin}$, $\varepsilon_{pin}$ follow a uniform distribution.

For the following experiments we consider the non-linear interactions among variables to represent unknown and undiscovered causalities during the modeling phase. Our data generation process slightly differs from Guevara (2015) on two points: first of all, $p_{in}$ is not correlated with $q_{in}$ as this is kept for an experiment by itself in Section 5.2.4 and B.1. Secondly, we have added an interaction term, $c_{in}$ to allow a simulated situation where the modeler would misspecify the utility due to an undiscovered interaction. To avoid too large difference in magnitude order among our variables as well as the interaction term, we generated all variables as:

[TABLE]

Under the classical logit assumption that $\varepsilon_{in}\overset{\text{i.i.d.}}{\sim}EV(0,1)$ and given fixed values of parameter estimates $\bm{\beta}$ in the utility specification, the choice probabilities $P_{in}(\bm{x_{n}})$ are known, and the synthetic choice $y_{in}$ can then be seen as a Bernoulli random variable that takes the value 1 with probability $P_{in}(\bm{x_{n}})$ and 0 with probability $1-P_{in}(\bm{x_{n}})$. Formally, we assume that

[TABLE]

Using synthetic data allows us to compare our L-MNL model with standard choice models, not only in terms of prediction performance, but also in accuracy in parameter estimation since we know the true value of parameter estimates $\bm{\beta}$, i.e., the true linear and non-linear dependencies between the variables.

5.2.2 Monte Carlo experiment

We rely on Monte Carlo simulations to investigate how well each model performs in both prediction and parameter estimation of Equation (21). To do so, we select a true utility specification with $\beta_{p}=-1$, $\beta_{a}=0.5$, $\beta_{b}$ = 0.5, $\beta_{qc}=1$. For each experiment, we generate $1,000$ synthetic individual observations for the training set, and $200$ more for the testing set. The following results are obtained by performing $100$ of such experiments.

Our new choice model is defined as

L-MNL({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}25},\mathcal{X},\mathcal{Q}), with $\mathcal{X}=\{\bm{p},\bm{a},\bm{b}\}$, and $\mathcal{Q}=\{\bm{q},\bm{c}\}$,

and is compared with the following three benchmarking models and one reference model*555The reference model is considered differently than the benchmarks as it breaks the assumption that the modeler does not know the ground truth.:

Logit($\mathcal{X}_{1}$), with $\mathcal{X}_{1}=\{\bm{p},\bm{a},\bm{b},\bm{q},\bm{c}\}$,

• 2.

DNN({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}25},\mathcal{Q}), with $\mathcal{Q}=\{\bm{p},\bm{a},\bm{b},\bm{q},\bm{c}\}$,

• 3.

DNN_L($25,\mathcal{X}=\mathcal{Q}$), with $\mathcal{X}=\mathcal{Q}=\{\bm{p},\bm{a},\bm{b},\bm{q},\bm{c}\}$,

• 4.

*Logit($\mathcal{X}_{true}$), with $\mathcal{X}_{true}=\{\bm{p},\bm{a},\bm{b},\bm{qc}\}$.

The log-likelihood results from Monte Carlo experiments can be seen in Table 1. As expected, the best models in terms of predictive performance are the neural network-based choice models. Among them, despite overfitting on the train set, our L-MNL gives the best general representation of the data, achieving the best predictive performance in the test set (with a LL of -97 compared to -123 for the standard MNL model).

In order to evaluate the models in terms of accuracy in interpretable parameter estimation, we define the relative errors $e_{\beta}$ and $e_{\beta_{i}/\beta_{j}}$ as

[TABLE]

The relative error results666Note that the DNN($25,\mathcal{Q}$) benchmarking model does not appear in Table 2 since no variable enters the linear utility specification. from Monte Carlo experiments can be seen in Table 2. As the beta parameters of the model have different values, we normalize the performance by reporting the average relative error of the estimates for every model, as well as the standard deviation of all relative errors. We see that our L-MNL greatly outperforms every model in the ability to recover the true parameter values with a relative error smaller than $1\%$ away from the true model. It is worth noting that for the ratios of parameters, the MNL models perform second best, although they have high relative errors at parameters level. This phenomenon has been investigated in a few papers (e.g., Lee (1982), Cramer (2007)) where it has been shown that even when utility is misspecified, the MNL model is still able to retrieve good ratios between estimates. The hybrid models with $\mathcal{X}=\mathcal{Q}$, on the other hand, generate high errors in parameter estimates. It indicates that the NN component is also partially learning linear dependencies of $\bm{p}$ or $\bm{a}$, and prevents the linear function $f$, or the $\beta$ parameters, to reach a minimum as good as our L-MNL. This is a direct consequence from Equation (13) as the impact of the variable is shared between both data-driven and knowledge-driven terms.

Finally, we conduct hypothesis testing to determine whether the estimated parameters are statistically different from the true ones. Formally, we consider the following null and alternative hypotheses: $H_{0}:\widehat{\beta}=\beta$, and $H_{1}:\widehat{\beta}\neq\beta$, where the null hypothesis is rejected for a t-test above 1.96, which corresponds to a 5$\%$ confidence interval. The standard deviation of the estimates in $\mathcal{X}$ for the t-tests are obtained via Hessian approximation777Note that profile likelihood approaches such as Hessian approximation create confidence intervals that fail to have the desired frequentist coverage properties. See Berger et al. (1999) for further discussion on that topic..

The results are shown in Table 3. We see that the coefficients alone are almost always statistically different from the true ones for every model except L-MNL. The NN component of our model fixes the estimators by learning only on the data that are included in the linear component. The ratio of parameters is well retrieved with both MNL and L-MNL models. Concerning the residual block model, we see that its NN component compromises the ratios between parameters. Indeed, as seen in Equation (14), part of the Neural Network has overlapped over the estimates of $\beta_{p}$ and $\beta_{a}$, thus breaking both the estimates and their ratio.

To better illustrate the origin of ratio discrepancy of variables $x_{in}\in\mathcal{X}$ when $\frac{\partial r_{in}}{\partial x_{in}}\neq 0$ in hybrid models, we have shown in Section 5.2.4 that we retrieve the same biasing effect on our L-MNL when we make use of highly correlated data between both sets.

5.2.3 Choice of neural network architecture

Choosing the structure of an NN requires one to have enough capacity to capture the underlying pattern of the data without overfitting. In the following, we gradually increase the size of the NN component of our L-MNL, spanning from an underfit of the non-linear truth of Equation (21) to an overfit of the generated data. To illustrate these effects, we study the values of $\beta_{p}$ and $\beta_{a}$, as well as likelihoods of both train and test sets.

The results are depicted in Figure 3, where we take L-MNL($n,\mathcal{X},\mathcal{Q}$) with a single layer L=1, and scan from $n=0$ neurons in the hidden layer to $n=5000$. We see that from $n=0$ neurons ($\equiv$MNL) to about $n\sim 10$, the NN has not yet captured all the non-linearities of the original utility specification. This can be seen by the higher values in likelihood and how the parameter estimates have not yet reached their true minimum. From $n=10$ to $n=100$, the model reaches stable values where the linear terms are equivalent to their ground truth. The model performs almost as well as the true model, depicted by the boundary lines, leading us to believe that the NN component has successfully learned the non-linearities of the data. When we continue to increase the number of neurons, we see the effects of overfitting, namely a drop in train likelihood and an increase in test likelihood. We finally observe that the values of $\beta_{p}$ and $\beta_{a}$ also start to suffer from the overfitting, as the L-MNL is no longer a general representation of the data, but has become specific to the training set. This experiment also shows that for equal optimal likelihood values, the change in architecture and non-convex property of the neural network does not significantly affect the $\beta$ estimates as it converges to the ground truth. We can see that the standard deviation spread of the estimates over 100 experiments match with those of the true model also reported on the figure. Although, is has been shown in Draxler et al. (2018) and Garipov et al. (2018) that a neural network may have many different parameter estimates for similar valued optimum, it does not seem to be the case for our convex part of the model.

5.2.4 Impact of strongly correlated variables

Experimental results from Section 5.2.2 suggest that having identical variables (or features) in both sets lead to poor parameter estimates because of the neural network’s better ability to discover correlation with the observations for all its variables. In this sense, Equation (13) shows how a variable’s impact will be shared between both knowledge-driven and data-driven terms if it belongs to both input sets.

Although our L-MNL requests the variables in the two sets to be different, strong correlation between the variables can remain an issue. To investigate the impact of correlated variables, we replace the original explanatory variable $q_{in}$ with a new explanatory variable $q^{\prime}_{in}$ that is defined to be correlated to $p_{in}$. Formally, we define $q^{\prime}_{in}$ as

[TABLE]

where $s\in[0,1]$ is the correlation coefficient.

Our new choice model is defined as

L-MNL($100,\mathcal{X},\mathcal{Q}$), with $\mathcal{X}=\{\bm{p},\bm{a},\bm{b}\}$, and $\mathcal{Q}=\{\bm{q^{\prime}},\bm{c}\}$,

and is compared with the following two benchmarking models:

Logit($\mathcal{X}_{1}$), with $\mathcal{X}_{1}=\{\bm{p},\bm{a},\bm{b},\bm{q^{\prime}},\bm{c}\}$,

• 2.

Logit($\mathcal{X}_{2}$), with $\mathcal{X}_{2}=\{\bm{p},\bm{a},\bm{b}\}$.

The Monte Carlo mean relative errors for different levels of correlation are shown in Figure 4.

Overall we can see two effects: bias with high variance due to the correlated variables (Logit($\mathcal{X}_{2}$), L-MNL), and bias due to misspecification (Logit($\mathcal{X}_{1}$), Logit($\mathcal{X}_{2}$)). For a correlation coefficient below $s\leq 0.8$ we can see the L-MNL has much better estimates with respect to the true model than the Logit models. For $s\leq 0.4$ the L-MNL is practically as efficient as the true model. This is important as it suggests that in order to get insightful results, the modeler has to carefully check that the variables that enter as input features in the neural network are not the same or not too strongly correlated with the variables in the linear utility specification.

Finally, we also show in B.1, that our new choice model does not suffer from complexly correlated variables between both sets $\mathcal{X}$ and $\mathcal{Q}$.

A real case study: the Swissmetro dataset

For the following experiments, we use the openly available dataset Swissmetro (Bierlaire et al. (2001)). The Swissmetro dataset consists of survey data collected in Switzerland during March 1998. The respondents provided information to analyze the impact of a new innovative transportation mode, represented by the Swissmetro, a revolutionary mag-lev underground system. Each individual reported the chosen transportation mode choice for various trips, including the car, the train, or the Swissmetro. The original dataset contains 10,728 observations. We removed observations for which the information regarding the chosen alternative was missing (9 observations). For convenience, we also decided to discard the observations for which the three alternatives car, train, and Swissmetro were not all available (1,683 observations). Our dataset contains 9,036 observations. We split this initial dataset into a training set of 7,234 observations and a test set of 1,802 observations.

5.3.1 Models Comparison

In Table 5.3.1, we present a detailed comparison of all the benchmarking models with respect to log-likelihood in the training and testing sets as well as the parameters estimates. Since the true values of parameter estimates are unknown, we compare the values obtained with our L-MNL to the ones obtained using the MNL model described in Bierlaire et al. (2001), whose linear-in-parameter utility specification is described in Table 4 and variables described in Table 5. The results of parameter optimization for this MNL model are shown in Table 5.3.1888It is worth noting that the values of estimates that we obtain slightly differ from the ones depicted in the original paper. This is normal since the original Swissmetro dataset is different from the training dataset that we used..

We next estimate our new choice model L-MNL($100,\mathcal{X}_{1},\mathcal{Q}_{1}$), where $\mathcal{X}_{1}$ contains the same variables as the ones included in the utility specification of Bierlaire et al. (2001), and $\mathcal{Q}_{1}$ contains all the unused variables in the dataset. These variables are described in Table 6. The results of this model are depicted in Table 5.3.1. We see that adding the representation learning component in the utility specification significantly increases the log-likelihood, suggesting that these variables contain information that helps to explain travelers’ choice. However, we also observe that several estimates are not statistically different from zero (p-value $>$ 0.05). We have concluded in Section 5.2.2 and further developed in Section 5.2.4, that only strong correlation between variables can lead to biased parameter estimates. However, the correlation among the variables has been investigated and does not seem to be the issue here (see Table LABEL:tab:coeffcorrelation in Appendix for the correlation among variables). Moreover, the DNN_L(100,$\mathcal{X}_{1}=\mathcal{Q}$) model (see Table 5.3.1) has one-to-one correlation among variables in both sets and does not loose significance in its parameters. One explanation could be that the coefficients in L-MNL have lost their significance due to the neural network’s ability to learn better which variables and interactions are most correlated to the data. Therefore, the significance of the same parameters in the initial MNL model can originate from a bias due to the model’s underfit. Also, with many explanatory variables being omitted in the initial MNL model, it is worth noting that the model is more likely to be subject to endogeneity, i.e., correlation among the dependent variables and the error term. Endogeneity is a well-known cause of bias in parameter estimates and can cause bias in the parameter estimates of the initial MNL model.

Finally, in Table 5.3.1, we estimate the new choice model L-MNL($100,\mathcal{X}_{2},\mathcal{Q}_{2}$), where $\mathcal{X}_{2}$ contains only the variables needed to compute the Value of Time (VOT) and Value of Frequency (VOF), i.e., the variables $time$, $cost$, and $frequency$. All other variables constitute the set $\mathcal{Q}_{2}$ and are therefore given to the neural network. We observe that all parameter estimates are significant, while the log-likelihood has greatly increased compared to the standard MNL model. In other words, the representation term was able to get information from the previously non-significant variables in the linear specification.

Furthermore, as L-MNL has a bigger feature space than the aforementioned models, we have designed a dummy coded Multinomial Logit as another benchmark. The parameter space contains homogeneous $\beta_{x}$ for the variables in $\mathcal{X}_{2}$ and alternative specific $\beta_{iq}$ for all variables contained in $\mathcal{Q}$. The full feature space is written as $\mathcal{X}_{dum}=\mathcal{X}_{2}\cup\mathcal{Q}_{2}$ and the results can be seen in Table 5.3.1. The big difference in final likelihoods with our L-MNL models further demonstrate that this dataset indeed contains complex functions and interactions among its variables which may be difficult to capture by the modeler.

We show a comparison of VOT and VOF for the different models in Table 8. We have also added a Cross-Nested Logit (CNL), and a Triangular Panel Mixture (TPM) model to the results for comparison purposes, whose detailed parameter values can be found in F and for which the implementation was taken from Biogeme examples999https://biogeme.epfl.ch/examples.html. The difference in log-likelihood values between the L-MNL and the Logit($\mathcal{X}_{1}$) suggests that the MNL model suffers from underfitting that leads to significant ratio discrepancy. A similar conclusion was derived only under strong non-linearities in the dataset, suggested in C. The linear specification of the MNL model does not allow to capture the Swissmetro’s non-linear dependencies among variables. Moreover, we may observe for both added DCM methods, that their ratio values move towards those of L-MNL as their likelihood increases. This trend implies a better specification in their utility would allow them to reach even more similar values in ratios and likelihood with our new choice model.

In support of these conclusions, we incrementally increase the size of the NN component. We observe in Figure 5 the parameter values with respect to the number of neurons in our neural network. As for the synthetic data experiments in Section 5.2.3, we see that stable ratios for VOT and VOF are obtained for a neural network having from 10 to 200 neurons. We see the MNL ratios for n=0 neuron highlights the underfit of the MNL model. An overfit effect is observed starting from 500 neurons, where the test likelihood no longer improves.